Cloning Ubuntu 10.04 Server KVM guests efficiently

0.00 avg. rating (0% score) - 0 votes

If you need to create lots of similar virtual machine guests running on QEMU/KVM, it is a very good idea to prepare a template guest image from which to clone the other guests. You should do whatever customizations you like before cloning. For instance I like to configure LVM and file systems to my liking, install openssh-server, install nfs-common and configure NFS mounts, install all available updates, add users or set up authentication, copy ssh keys, and do many other things so that they will be working out-of-the-box after cloning a number of guests from the template.

After you have installed and set-up your template virtual server to your liking, and would want to start cloning multiple instances of it, some tricks are needed to make things work more automatically after cloning and starting up the final copy.

If you simply clone a vanilla Ubuntu server installation multiple times, you will face some problems:

  1. The network interfaces will be renamed at first boot because the MAC address will be different.
  2. The ssh keys will be the same on each server (it works, but not the best idea)
  3. The server hostname will be the same on each server
  4. Statically configured IP address will be the same on each server, causing potential IP address conflicts

For IP addresses, I suggest using DHCP for the template image, so that each cloned guest will receive a unique IP during the first boot-up. You can then configure static IP addresses for the clones. I will show you how to do it almost automatically.

There are also a couple of optional tips which you should do for a virtual guest template before anything else.

This long post is divided into three parts:

  • Part I. Configuring the template guest image
  • Part II. Cloning and configuring multiple guests from your template
  • Part III. Conclusions and recommendations

Part I is the longest, but it will show you how to set up your template so that only one script must be run after guest boot-up to change network settings and hostname for each new guest. Also, I will show you how to make sure network interface naming is consistent from eth0 onwards, and that the host ssh keys will be unique on each host.


Part I. Configuring the template guest image

Let’s start with the tips:

Tip 1: Make shutdown work

The shutdown command does not work by default from virt-manager nor virsh. To fix that, install acpid on the template guest image:

After that, shutdown should work from both virt-manager and virsh. For some strange reason, reboot works only from virt-manager, not virsh.

Tip 2: Make serial console work

This is very handy for console over ssh (no need for virt-manager, nor any kind of VNC connection for console). See earlier post:

http://koo.fi/tech/2010/11/27/serial-console-for-ubuntu-server-1004-kvm-guests

Tip 3: Copy your ssh public key

It makes sense to copy your public ssh key at this point, so that you don’t need to do it again for each guest. Simply go to your home directory on the template guest and run:

Paste your public key there, and press Ctrl-D. Pasting does not work in a VNC console, but does work using the serial console trick mentioned in tip 2, or ssh connection.

Next, let’s take a look at the problems mentioned in the start of the post:

Problem 1: Network interface naming

When Ubuntu boots up and detects a network interface with a MAC address which it has not seen before, it will take the next available eth number and add that to the udev rules file. The MAC addresses of network interfaces for cloned guests will be different from the template guest, so if you don’t delete the corresponding lines from the file before cloning, the cloned guest’s interface numbering will not start from eth0, but eth1 or something else depending on the number of interfaces you have configured.

In order to get your cloned guests’ network interfaces start being numbered from eth0 onwards, simply delete the udev file which contains the rules for interface naming:

That file will be regenerated at the first boot, so all clones will start their interface naming from eth0. But you must also remember that the next time you start up your template to make changes to it, it too will re-create the file. So remember to delete it every time you start up your template guest to make changes to it.

Problem 2: Individual SSH host keys

The ssh host keys are a bit more difficult thing. If you simply delete the keys from /etc/ssh/, they will not be regenerated automatically on Ubuntu. To make that happen, you need to run manually:

In order to make that happen automatically during the first boot of a cloned guest, one line has to be added to the /etc/init/ssh.conf file (marked with bold):

/etc/init/ssh.conf:

That will test the existence of host keys, and regenerate them if they are not present.

Before stopping your template guest, delete the ssh host keys:

Script to prepare template for cloning

I added a simple script, /usr/local/sbin/cloneprep, to ease the preparation of the template. It will simply delete your host ssh keys and udev persistent network rules (asking your permission first), and shut down the template guest afterwards so you can start cloning.

/usr/local/sbin/cloneprep:

Give execute permission with “sudo chmod u+x /usr/local/sbin/cloneprep”.

Remember to run cloneprep EVERY TIME you start up a template guest machine to change something. Don’t use halt or shutdown, but run cloneprep instead.

It is not yet time to run it though, as we will do some further preparations to the template in the following section.

Problems 3 and 4: Hostname and IP address

To address these problems efficiently, we must create three of configuration files and a configuration script that will be run on each guest after the first boot.

To create a dozen guests, I created the following files under the template guest’s /usr/local/etc.

/usr/local/etc/cloneaddresses:
This file will be used to look up the IP address for a given name.

(Use spaces instead of tabs in the cloneaddresses file. You can add as many spaces in between the names and addresses as you like, though.)

/usr/local/etc/clonehosts:
This file will be copied to /etc/hosts, with the GUESTNAME replaced with the guest’s actual name. You can copy your actual /etc/hosts file to clonehosts and edit that.

/usr/local/etc/cloneinterfaces:
This file will be copied to /etc/network/interfaces, with the GUESTIP replaced with the guest’s actual IP address. You can copy your actual /etc/network/interfaces file to cloneinterfaces and edit that.

I created the configuration script as /usr/local/sbin/cloneconf, and it looks like this:

/usr/local/sbin/cloneconf:

Give execute permission with “sudo chmod u+x /usr/local/sbin/cloneconf”.

That script must be run individually on each cloned guest machine. Either give the hostname as the single argument to the script, or the script will ask you for the hostname if you don’t do that. The hostname must be found in the cloneaddresses file. The rest of the settings will be determined from the configuration files.


Part II. Cloning and configuring multiple guests from your template

Cloning guests from a template is as easy as:

  1. Run the “cloneprep” script on the template guest and shut it down
  2. Clone as many new guests as you wish with the virt-clone command
  3. Start up each new guest
  4. Run “cloneconf” on each new guest

Clone

One clone:

Or a dozen clones:

The virt-clone command will create a new xml configuration file under /etc/libvirt/qemu with a new UUID and randomly selected MAC address. Also, it will copy the image file to the location specified in the command line.

If you configured your template for serial console operation (link to the instructions in Tip 1 above), you can start up the new guest and get the console instantly using this command:

To start your dozen guests simultaneously:

Then connect to the console of each individual guest with:

You can get a list of running guests with “virsh list”.

To end a serial console session, press Ctrl-]

Configure

The cloneconf script created in the first part of this post makes things easy.

After you have logged into a running guest machine (either VNC, serial console or ssh), run:

The hostname must be listed in the /usr/local/etc/cloneaddresses file created earlier. Otherwise the script will abort. It will show you changes it is about to make, and let you choose whether to commit them or not.

Reboot the guest after the script has run. Repeat for all clones.

The script will make changes to these files:

  • /etc/hostname
  • /etc/hosts
  • /etc/network/interfaces

It will replace the hostname and IP address with a suitable value.


Part III. Conclusions and recommendations

Customizing each cloned guest really just consists of changing or regenerating the following files:

  • /etc/hostname
  • /etc/hosts
  • /etc/network/interfaces
  • /etc/ssh/ssh_host_dsa_key
  • /etc/ssh/ssh_host_dsa_key.pub
  • /etc/ssh/ssh_host_rsa_key
  • /etc/ssh/ssh_host_rsa_key.pub
  • /etc/udev/rules.d/70-persistent-net.rules

After that, the rest of the template customization is up to you.

I recommend creating one master template with the instructions above, with very little installed software – the bare minimum you will install on ALL of your virtual machine guests. Configure all the authentication and NFS stuff, and whatever your environment requires.

Then clone this master template to create new templates for your web servers, mail servers etc. You can run cloneprep on them again even if you already ran cloneconf, to reverse a cloned guest back to a template (although change /etc/network/interfaces back to DHCP in order to avoid IP address conflicts).

After all this messing around with templates, deploying a new server is really quick and easy. You will have the right software installed, and nearly configured. And you can even customize the cloneconf script to configure whatever software you like.

I hope this helps!

Here you can download the scripts and configuration files in one package: clonetools.tar.gz

2 thoughts on “Cloning Ubuntu 10.04 Server KVM guests efficiently”

  1. Just the collection of details I needed, thanks!

    Regarding having/wanting to delete /etc/udev/rules.d/70-persistent-net.rules on every shutdown. Myself I ended up with adding the following lines to /lib/udev/rules.d/75-persistent-net-generator.rules instead.

    # ignore KVM virtual interfaces
    ENV{MATCHADDR}==”52:54:00:*”, GOTO=”persistent_net_generator_end”
    # ignore VMWare virtual interfaces
    ENV{MATCHADDR}==”00:0c:29:*|00:50:56:*”, GOTO=”persistent_net_generator_end”

    They were introduced in Ubuntu 10.10 in in response to https://bugs.launchpad.net/bugs/341006

Leave a Reply to librarian Cancel reply