With more organizations moving their servers to the cloud, managing SSH keys that are used to access these servers is fast becoming a critical identity and access management issue. Continue reading to gain a better understanding of the risks underlying poor SSH key management, and what to look for in a key management solution.
Where did the server go?
Not all that long ago, it was not uncommon to find a server sitting under the company’s reception desk. The server in question would typically be a repurposed desktop, its case rendered a grungy beige by years of cigarette smoke from its previous owner, who had been successful in convincing management that the pentium processor just didn’t cut the mustard anymore.
Frugal admins came by and installed the Linux or BSD flavor of the day, giving the abandoned machine a new life as a web, mail or other server. The web server would go down for a couple of hours twice a week, as the cleaning lady unplugged it in favor of the vacuum cleaner, but that was not really an issue. People could still find the company’s phone number in the yellow pages after all.
Grumbling admins, tired of lugging around CRT monitors and typewriter-sized keyboards, insisted that the company should have a server room, and that they should be entrusted with the keys. So the servers came out of hiding, from under desks and inside closets, and made their way to the server room. Closing the door to the server room was often not an option, as the excessive heat would randomly cause servers to shut down, again redirecting people to the yellow pages.
Groaning admins, weary of spending hot summer days in even hotter server rooms, argued that the servers should be moved to an off-site data center. The data center would keep the servers cool, it would provide redundant power supplies and network connections, and have the necessary security measures in place to prevent unauthorized access. All this was required to ensure that the servers would continue to happily hum along with the growing importance of the internet.
Still, all was not good. Frequent trips to the data center had to be made for a variety of reasons: the cabling in the data center had —frankly— become a total mess and would have to be completely redone, new servers had to be added, and obsolete and faulty hardware needed to be replaced. What’s more, the computing resources in the data center didn’t scale well, and certainly not on demand. After some successful (and a lot of failed) attempts at virtualization, admins started to make the case for moving an increasing number of their servers to the cloud.
As server resources moved further and further away from their locus of operation, the security implications of this shift may have not been given sufficient attention. Once, a server’s root password was a tightly guarded secret shared among a small number of admins, only to be keyed in from a physically connected keyboard. Now, that server might be running on shared hardware in a data center in Singapore, only to be accessed remotely from three different continents over an inherently insecure connection.
When it comes to UNIX-based (Linux, *BSD, etc.) operating systems, that remote connection is usually established using the Secure Shell (SSH) protocol, which transports the client-server communication over an encrypted channel. With the actual communication being secured, the question now becomes how the communication gets established to begin with, or, what kind of authentication is required to start the SSH session.
Depending on the server software in use, an SSH server will typically support a number of different authentication protocols. If configured correctly, authentication protocols that rely on supporting infrastructure (LDAP, Kerberos, NTLM, …) can indeed provide secure means of authentication, but are out of scope for this discussion. Instead, I will focus on the more commonly used password authentication and public key authentication.
Password authentication has been around since ancient times, and despite high-level players predicting the death of the password for well over a decade now, password-based authentication schemes remain by far the most ubiquitous method to authenticate to local and remote systems alike. In most cases, however, passwords are either too short to be secure, or too long to be remembered. The risks associated with password authentication have been widely discussed (disclosure, brute-force attacks, shoulder surfing, …), so I will limit myself to saying that, if you’re still running password-based authentication on your SSH server (especially if it’s publicly accessible), it’s probably time to start looking into alternatives.
Public key authentication
Public key cryptography relies on an asymmetrical key pair that participants in an exchange use to encrypt and decrypt messages. When used for purposes of authentication, public key cryptography works on the premise that a message can only be decrypted by a public key if it has previously been encrypted with its corresponding private key, and vice versa. In SSH, this mechanism allows the client to prove that it’s in possession of the private key by demonstrating that it is able to decrypt a message that was encrypted by the server with the corresponding public key.
Before addressing the risks associated with public key authentication, I’ll first list up the benefits:
- Keys are typically longer than the longest password you care to remember.
- Keys can optionally be protected using a passphrase. This effectively turns public key authentication into a two-factor authentication mechanism that uses something you have (the key) and something you know (the key’s passphrase).
- Unlike passwords, keys are not sent to the server upon authentication.
- Unlike passwords, keys can be used for non-interactive authentication.
Common issues with public key authentication
While the public key authentication scheme in itself is secure, shifting the authentication factor from something you know (a password) to something you have (a key file), does present additional challenges.
First, users and admins alike can be lulled into a false sense of security by the fact that public key authentication is in play. Caution is thrown to the wind, just because using an asymmetrical cryptographic key algorithm to authenticate sure sounds a whole lot more secure than typing the name of your long-lost childhood pet.
Secondly, a number of challenges derive from the fact that, for the mechanism to work, the server has to be in possession of the public keys of all the key pairs that will be used for authentication. In OpenSSH, the most widely deployed SSH server (with some resources quoting up to 97% market share), this is typically achieved by storing a user’s public keys in the user account’s ~/.ssh/authorized_keys file.
A common scenario is for a user to generate a key pair using a tool such as ssh-keygen, and then send the public key to an administrator. The admin will in turn install it on all the server instances that the user needs to be able to access. While this approach is easily managed when only a small number of users and instances are in play, it quickly becomes unwieldy as the operation increases in scale. To avoid having to deal with the cumbersome aspects of key management, a number of unsavory practices have evolved.
Without a key management system in place, key rotation puts a heavy burden on administrators. To change a user’s keys, the admin will first have to determine which servers and user accounts the user has access to, and then identify the public keys that need to be replaced—decidedly a non-trivial procedure in cases where multiple users access the same account, and the user no longer possesses the corresponding private key. Then, the admin can proceed to remove the obsolete key on all instances, and replace it with the newly generated one.
If this sounds like a laborious process, that’s because it is. So, in a lot of cases, a user generates a key pair once, and keeps on reusing it. Over the years, the private key—supposedly a closely guarded secret—might find itself being copied to other hard drives as the user switches computers, backed up on various USB drives “just in case”, or sent by email because “who knows when and where I might need to be able to access the server”. All of these events increase the risk of keys falling into the hands of a party with nefarious intentions.
“But, we protect our keys with a passphrase, so it doesn’t matter”, is the retort one often gets when pointing out these risks. Actually, it does matter:
- Who’s to say that the user, tired of having to enter the passphrase over and over (and unaware of how to correctly configure ssh-agent), has not just removed the passphrase from the key?
- Unlike the password used in client-server authentication, the passphrase protecting the private key is local to the key itself. There’s no server in the picture that can prevent or slow down any brute-force attempts by an attacker that somehow got hold of the key. Instead, said attacker is only limited by the complexity of the passphrase and the amount of raw computing resources they can bring to bear on the cracking attempt.
At best, if your key is compromised, the passphrase can buy you some extra time. This brings us back to the original problem with the use of long-lived keys: it gives potential attackers plenty of time.
A variation of the pattern described above, similarly caused by the administrative burden of key provisioning, is the sharing of keys. This typically happens in situations where multiple (or even all) users access the same user account on a server instance.
It often starts off innocuously, as Jack needs to run a quick query on the database server, and Jill is kind enough to oblige by briefly “lending” him her private key. Before long, the key is shared by dozens or even hundreds of users. In terms of security, this approach is akin to a country line dance: take one step forward by removing the dependency on a shared secret (password), and then take two steps back by sharing the private key among multiple users. It only takes one less responsible user for the key to be compromised.
Besides authentication, the usage of shared keys also has an impact on other security aspects:
Sharing keys means sharing user accounts, so the user/group permission system that is available on UNIX-based systems can no longer be used effectively. When it comes to authorization (who can do what), this has a greatest common divisor effect: the permissions granted to all users will be a superset of the permissions required by each individual user.
- Auditing and accountability
Shared keys and shared accounts also present extra issues in the realm of accountability. In case the key gets compromised, it will not be possible to identify the responsible party. Moreover, any form of server auditing (e.g. through the use of auditd) will lose meaning, as it will no longer be possible to reliably identify the user that executed a given command.
A common approach to reduce the hassle of having to manage user keys on a number of instances, is to introduce a so-called bastion host, and subsequently abuse it. In essence, a bastion host is a publicly accessible server that is used to route traffic to a number of servers that are otherwise shielded from the evil outside world. The concept of the SSH bastion host itself is sound, especially if used with SSH proxying as described e.g. here.
The problem with this setup is that it does absolutely nothing to alleviate the pains associated with SSH key management. In fact, it makes things even worse as not only the keys to the various servers, but also to the bastion host itself now need to be managed.
That’s why clever admins came up with the following approach: instead of creating individual user accounts on the servers that are fronted by the bastion host, let’s just create one account and have the bastion host access that. In terms of key management, this has the immediate benefit that the individual user keys now only need to be managed on the bastion server, while the actual servers can remain largely untouched.
Unfortunately, this approach comes with a number of very significant drawbacks:
- The bastion host now contains the keys to the kingdom. If the castle falls, the kingdom falls.
- The key used to authenticate the bastion host’s outgoing connections is essentially a shared key. As such, it suffers from the limitations discussed in the previous section: fine-grained authorization and auditing are no longer possible, accountability goes out the window.
The need for SSH key management
By now, I’ve hopefully established that there is a need for SSH key management, and that as the operation grows in scale, this becomes less and less of a trivial matter. So, what then should a successful SSH key management solution look like? In my opinion, it requires at least the following features:
- Individual keys
Keys need to be issued to individual users and should not be shared. It should be possible to revoke individual keys, and deleting or suspending a user should forcefully expire all of the user’s keys.
- Short-lived keys
Keys need to be short-lived. Rotating keys on a weekly or even daily basis greatly reduces the risks that are incurred in case a key falls into the wrong hands.
- Central management
Issuing a new key or revoking an existing key should not require manual intervention on the server instances that the key provides access to. The process is too laborious and error-prone, even for small and medium-sized infrastructures.
- Central auditing
Public key authentication to instances needs to be centrally logged. Combined with the use of individual keys, this allows administrators to identify the accountable party should irregularities occur.
- Strong authentication
It goes without saying that the SSH key management solution itself should be adequately protected, preferably requiring two-factor authentication.