If you need to create lots of similar virtual machine guests running on QEMU/KVM, it is a very good idea to prepare a template guest image from which to clone the other guests. You should do whatever customizations you like before cloning. For instance I like to configure LVM and file systems to my liking, install openssh-server, install nfs-common and configure NFS mounts, install all available updates, add users or set up authentication, copy ssh keys, and do many other things so that they will be working out-of-the-box after cloning a number of guests from the template.
After you have installed and set-up your template virtual server to your liking, and would want to start cloning multiple instances of it, some tricks are needed to make things work more automatically after cloning and starting up the final copy.
If you simply clone a vanilla Ubuntu server installation multiple times, you will face some problems:
- The network interfaces will be renamed at first boot because the MAC address will be different.
- The ssh keys will be the same on each server (it works, but not the best idea)
- The server hostname will be the same on each server
- Statically configured IP address will be the same on each server, causing potential IP address conflicts
For IP addresses, I suggest using DHCP for the template image, so that each cloned guest will receive a unique IP during the first boot-up. You can then configure static IP addresses for the clones. I will show you how to do it almost automatically.
There are also a couple of optional tips which you should do for a virtual guest template before anything else.
This long post is divided into three parts:
- Part I. Configuring the template guest image
- Part II. Cloning and configuring multiple guests from your template
- Part III. Conclusions and recommendations
Part I is the longest, but it will show you how to set up your template so that only one script must be run after guest boot-up to change network settings and hostname for each new guest. Also, I will show you how to make sure network interface naming is consistent from eth0 onwards, and that the host ssh keys will be unique on each host.
Table of Contents
Part I. Configuring the template guest image
Let’s start with the tips:
Tip 1: Make shutdown work
The shutdown command does not work by default from virt-manager nor virsh. To fix that, install acpid on the template guest image:
1 |
sudo apt-get install acpid |
After that, shutdown should work from both virt-manager and virsh. For some strange reason, reboot works only from virt-manager, not virsh.
Tip 2: Make serial console work
This is very handy for console over ssh (no need for virt-manager, nor any kind of VNC connection for console). See earlier post:
http://koo.fi/tech/2010/11/27/serial-console-for-ubuntu-server-1004-kvm-guests
Tip 3: Copy your ssh public key
It makes sense to copy your public ssh key at this point, so that you don’t need to do it again for each guest. Simply go to your home directory on the template guest and run:
1 2 3 4 |
mkdir -p .ssh chmod 700 .ssh cd .ssh cat >authorized_keys |
Paste your public key there, and press Ctrl-D. Pasting does not work in a VNC console, but does work using the serial console trick mentioned in tip 2, or ssh connection.
Next, let’s take a look at the problems mentioned in the start of the post:
Problem 1: Network interface naming
When Ubuntu boots up and detects a network interface with a MAC address which it has not seen before, it will take the next available eth number and add that to the udev rules file. The MAC addresses of network interfaces for cloned guests will be different from the template guest, so if you don’t delete the corresponding lines from the file before cloning, the cloned guest’s interface numbering will not start from eth0, but eth1 or something else depending on the number of interfaces you have configured.
In order to get your cloned guests’ network interfaces start being numbered from eth0 onwards, simply delete the udev file which contains the rules for interface naming:
1 |
rm -f /etc/udev/rules.d/70-persistent-net.rules |
That file will be regenerated at the first boot, so all clones will start their interface naming from eth0. But you must also remember that the next time you start up your template to make changes to it, it too will re-create the file. So remember to delete it every time you start up your template guest to make changes to it.
Problem 2: Individual SSH host keys
The ssh host keys are a bit more difficult thing. If you simply delete the keys from /etc/ssh/, they will not be regenerated automatically on Ubuntu. To make that happen, you need to run manually:
1 |
sudo dpkg-reconfigure openssh-server |
In order to make that happen automatically during the first boot of a cloned guest, one line has to be added to the /etc/init/ssh.conf file (marked with bold):
/etc/init/ssh.conf:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
respawn respawn limit 10 5 umask 022 # replaces SSHD_OOM_ADJUST in /etc/default/ssh oom never pre-start script test -x /usr/sbin/sshd || { stop; exit 0; } test -e /etc/ssh/sshd_not_to_be_run && { stop; exit 0; } test -c /dev/null || { stop; exit 0; } <strong> # Regenerate ssh host keys if they are missing: test -f /etc/ssh/ssh_host_dsa_key || dpkg-reconfigure openssh-server</strong> mkdir -p -m0755 /var/run/sshd end script # if you used to set SSHD_OPTS in /etc/default/ssh, you can change the # 'exec' line here instead exec /usr/sbin/sshd |
That will test the existence of host keys, and regenerate them if they are not present.
Before stopping your template guest, delete the ssh host keys:
1 |
rm -f /etc/ssh/*host*key* |
Script to prepare template for cloning
I added a simple script, /usr/local/sbin/cloneprep, to ease the preparation of the template. It will simply delete your host ssh keys and udev persistent network rules (asking your permission first), and shut down the template guest afterwards so you can start cloning.
/usr/local/sbin/cloneprep:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
#!/bin/sh echo "TEMPLATE PREPARATION BEFORE CLONING" echo echo "This script will prepare this host for cloning. The network interface" echo "name rules and ssh host keys will be deleted, but they will be" echo "regenerated at the next boot (run sudo dpkg-reconfigure openssh-server" echo "to regenerate ssh keys if needed)." echo echo "Are you sure you want to remove the ssh keys and network interface" echo -n "rules on this host [y/N]? " read YN if [ "$YN" != "y" ]; then echo abort. exit 1 fi echo echo Removing persistent network interface rules... rm -f /etc/udev/rules.d/70-persistent-net.rules echo Removing host ssh keys... rm -f /etc/ssh/*host*key* echo echo Your image is now ready for cloning. The machine will shut down. echo for i in 12 11 10 9 8 7 6 5 4 3 2 1; do echo "Halting machine in $i seconds (Ctrl-C to abort)"; sleep 1; done halt |
Give execute permission with “sudo chmod u+x /usr/local/sbin/cloneprep”.
Remember to run cloneprep EVERY TIME you start up a template guest machine to change something. Don’t use halt or shutdown, but run cloneprep instead.
It is not yet time to run it though, as we will do some further preparations to the template in the following section.
Problems 3 and 4: Hostname and IP address
To address these problems efficiently, we must create three of configuration files and a configuration script that will be run on each guest after the first boot.
To create a dozen guests, I created the following files under the template guest’s /usr/local/etc.
/usr/local/etc/cloneaddresses:
This file will be used to look up the IP address for a given name.
1 2 3 4 5 6 7 8 9 10 11 12 |
guest01 192.168.1.101 guest02 192.168.1.102 guest03 192.168.1.103 guest04 192.168.1.104 guest05 192.168.1.105 guest06 192.168.1.106 guest07 192.168.1.107 guest08 192.168.1.108 guest09 192.168.1.109 guest10 192.168.1.110 guest11 192.168.1.111 guest12 192.168.1.112 |
(Use spaces instead of tabs in the cloneaddresses file. You can add as many spaces in between the names and addresses as you like, though.)
/usr/local/etc/clonehosts:
This file will be copied to /etc/hosts, with the GUESTNAME replaced with the guest’s actual name. You can copy your actual /etc/hosts file to clonehosts and edit that.
1 2 3 4 5 6 7 8 9 |
127.0.0.1 localhost 127.0.1.1 GUESTNAME # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters |
/usr/local/etc/cloneinterfaces:
This file will be copied to /etc/network/interfaces, with the GUESTIP replaced with the guest’s actual IP address. You can copy your actual /etc/network/interfaces file to cloneinterfaces and edit that.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address GUESTIP netmask 255.255.255.0 gateway 192.168.1.1 |
I created the configuration script as /usr/local/sbin/cloneconf, and it looks like this:
/usr/local/sbin/cloneconf:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
#!/bin/sh echo "GUEST CONFIGURATION AFTER CLONING" echo GUESTNAME="" if [ "$1" = "" ]; then echo -n "What is the hostname of this guest? " read GUESTNAME else GUESTNAME="$1" fi GUESTIP="$(cat /usr/local/etc/cloneaddresses | egrep ^$GUESTNAME | awk '{print $2}')" if [ "$GUESTIP" = "" ]; then echo "Hostname not found. Available hostnames:" echo cat /usr/local/etc/cloneaddresses exit 1 fi echo "This script will configure the host with the following settings:" echo Hostname: $GUESTNAME echo IP address: $GUESTIP echo echo /etc/hostname: echo $GUESTNAME echo echo /etc/hosts: cat /usr/local/etc/clonehosts | sed -e "s/GUESTNAME/$GUESTNAME/g" echo echo /etc/network/interfaces: cat /usr/local/etc/cloneinterfaces | sed -e "s/GUESTIP/$GUESTIP/g" echo echo -n "Are you sure you want to continue [y/N]? " read YN if [ "$YN" != "y" ]; then echo abort. exit 1 fi echo $GUESTNAME > /etc/hostname cat /usr/local/etc/clonehosts | sed -e "s/GUESTNAME/$GUESTNAME/g" > /etc/hosts cat /usr/local/etc/cloneinterfaces | sed -e "s/GUESTIP/$GUESTIP/g" > /etc/network/interfaces test -f /etc/ssh/ssh_host_dsa_key || dpkg-reconfigure openssh-server echo echo Done. The machine will now be rebooted to make changes effective. echo for i in 12 11 10 9 8 7 6 5 4 3 2 1; do echo "Rebooting in $i seconds (Ctrl-C to abort)"; sleep 1; done reboot |
Give execute permission with “sudo chmod u+x /usr/local/sbin/cloneconf”.
That script must be run individually on each cloned guest machine. Either give the hostname as the single argument to the script, or the script will ask you for the hostname if you don’t do that. The hostname must be found in the cloneaddresses file. The rest of the settings will be determined from the configuration files.
Part II. Cloning and configuring multiple guests from your template
Cloning guests from a template is as easy as:
- Run the “cloneprep” script on the template guest and shut it down
- Clone as many new guests as you wish with the virt-clone command
- Start up each new guest
- Run “cloneconf” on each new guest
Clone
One clone:
1 |
sudo virt-clone --original=template1 --name=guest01 --file=/var/lib/libvirt/images/guest01.img |
Or a dozen clones:
1 |
for i in {01..12}; do sudo virt-clone -o template1 -n guest$i -f /var/lib/libvirt/images/guest$i.img; done |
The virt-clone command will create a new xml configuration file under /etc/libvirt/qemu with a new UUID and randomly selected MAC address. Also, it will copy the image file to the location specified in the command line.
If you configured your template for serial console operation (link to the instructions in Tip 1 above), you can start up the new guest and get the console instantly using this command:
1 |
sudo virsh create /etc/libvirt/qemu/guest01.xml --console |
To start your dozen guests simultaneously:
1 |
for i in {01..12}; do sudo virsh create /etc/libvirt/qemu/guest$i.xml; done |
Then connect to the console of each individual guest with:
1 |
sudo virsh console guest01 |
You can get a list of running guests with “virsh list”.
To end a serial console session, press Ctrl-]
Configure
The cloneconf script created in the first part of this post makes things easy.
After you have logged into a running guest machine (either VNC, serial console or ssh), run:
1 |
sudo cloneconf <hostname> |
The hostname must be listed in the /usr/local/etc/cloneaddresses file created earlier. Otherwise the script will abort. It will show you changes it is about to make, and let you choose whether to commit them or not.
Reboot the guest after the script has run. Repeat for all clones.
The script will make changes to these files:
- /etc/hostname
- /etc/hosts
- /etc/network/interfaces
It will replace the hostname and IP address with a suitable value.
Part III. Conclusions and recommendations
Customizing each cloned guest really just consists of changing or regenerating the following files:
- /etc/hostname
- /etc/hosts
- /etc/network/interfaces
- /etc/ssh/ssh_host_dsa_key
- /etc/ssh/ssh_host_dsa_key.pub
- /etc/ssh/ssh_host_rsa_key
- /etc/ssh/ssh_host_rsa_key.pub
- /etc/udev/rules.d/70-persistent-net.rules
After that, the rest of the template customization is up to you.
I recommend creating one master template with the instructions above, with very little installed software – the bare minimum you will install on ALL of your virtual machine guests. Configure all the authentication and NFS stuff, and whatever your environment requires.
Then clone this master template to create new templates for your web servers, mail servers etc. You can run cloneprep on them again even if you already ran cloneconf, to reverse a cloned guest back to a template (although change /etc/network/interfaces back to DHCP in order to avoid IP address conflicts).
After all this messing around with templates, deploying a new server is really quick and easy. You will have the right software installed, and nearly configured. And you can even customize the cloneconf script to configure whatever software you like.
I hope this helps!
Here you can download the scripts and configuration files in one package: clonetools.tar.gz
Just the collection of details I needed, thanks!
Regarding having/wanting to delete /etc/udev/rules.d/70-persistent-net.rules on every shutdown. Myself I ended up with adding the following lines to /lib/udev/rules.d/75-persistent-net-generator.rules instead.
# ignore KVM virtual interfaces
ENV{MATCHADDR}==”52:54:00:*”, GOTO=”persistent_net_generator_end”
# ignore VMWare virtual interfaces
ENV{MATCHADDR}==”00:0c:29:*|00:50:56:*”, GOTO=”persistent_net_generator_end”
They were introduced in Ubuntu 10.10 in in response to https://bugs.launchpad.net/bugs/341006
How do you change root password and regenerate ssh keys not on Ubuntu guests?