The SSH Key Problem With Cloned Linux VM’s

First let me start off this by saying – the way this is effects you will differ entirely on your organizational procedures and security requirements.

We all love templates – don’t we? I mean they are the best! You configure your VM to your liking, OS patches, company policy settings etc.. etc.. and every new VM that you deploy – will have the exact same baseline.

Standardization… conformity… in the enterprise – all great.

Except.. a short while a go I found out something which is not exactly the best security practice (to put it mildly)

In vSphere you can create a Guest Customization specification and deploy your VM.

For Linux you can enter in some information, but not much

(Today I found out – that you also can run a custom script to configure the VM Configure a Script to Generate Computer Names and IP Addresses During Guest Operating System Customization in the vSphere Client )

But let’s get back on track.

So you have deployed a VM from a template – it now has an IP – and what is the first thing you would most probably do? SSH into the VM – because you now want to start doing the real work (amazing how we take the deployment of an OS for granted these days).

So if this is the first time you are connecting you will most probably get something like this:

Which to explain in simple terms is saying, “Hey – I don’t know this server – here are its details and RSA key. Do you still want to connect?”

And you would usually say – yes and enter your password – and all is fine and dandy.

What this does is add an entry to the .ssh/known_hosts file

But not only did I deploy one VM from this template – I deployed 2. So let’s repeat the process again.

So where is the problem? If you look carefully – you will see that key fingerprint on vm1 is the same as on vm2

30:d6:df:54:ca:26:b2:a4:df:65:e5:33:9f:21:df:55  
30:d6:df:54:ca:26:b2:a4:df:65:e5:33:9f:21:df:55

Identical.

And if we would now look into the known_hosts file then you would see this:

All exactly the same (each host created two lines for some reason on Ubuntu – usually it is one).

But why is that? Shouldn’t every VM be unique? I mean they have different MAC addresses, different UUID’s, different IP’s – VMware usually takes care of that.

Well it is pretty simple really – when SSH is installed, the OS package usually creates these files for you. But remember we are cloning from a template – after SSH was installed (that will usually be the case).

That now means – that every single VM that was deployed from the template now has the same exactly identical key.

That could be acceptable in your environment – maybe. Maybe not.

But take this example. You are providing vCloud services and your VM’s are spawned from the same templates. All … of… them!!!

Here you could have the same public keys in different organizations – different companies, I am sure you can see how bad this might become from a security perspective.

This can also cause havoc on certain monitoring systems and also will create a number of problems with SSH key authorization.

So how do you solve this? Unfortunately – there is no built-in way to do this with the current functionality in vSphere today. PowerCLI scripts – or other orchestration tools will need to be used to get around this.

What I would personally like is an option to run a guest OS script as part of the deployment process. Yes I know this exists for Windows VM’s today – but there is no such functionality for Linux.

I did a quick check on some of the VM’s in one of the environments I have access to - 50 VM’s

There are duplicates – actually I was surprised to see that there some were actually unique.

Some food for thought… (which reminds me – time for me to go out for lunch).

Thanks to @brian_smi for the background information from his blog post!

It would be interesting to hear your thoughts and comments on how or if this might pose a problem in your environment. Please feel free to leave them in the comments below.

At the moment my recommended solution would be to remove the ssh_host* files on the VM
before you power it down. The files will be recreated once the VM starts up (or a new VM is deployed from this template). Just make sure.. When you power on the template for maintenance – you must remove the files before your power it down again.

Thanks Erik!