Installing LinuxMCE on Virtual Machine via KVM
KVM: To badly go where (almost) no man has gone before
In this overview I will try to follow the installation path to get LCME 8.10 into a virtual machine which uses PCI(-E) devices from the host. Normally the option to install on a dedicated computer is the best, but as I want to host several systems, including LCME, websites etc and the ease of testing out new stuff combined with the fact that I want to keep it as green as possible lead me to the conclusion that virtualization was the way to go. Over the last 2 years I've been building systems that have failed in doing what I wanted (XEN and KVM based), but with the latest state of soft- and hardware I'll hope I'll have a better change. The below is my queste for the wanted setup up, step by step.
Virtualization the way I want to do it can only be done with specific hardware/firmware which uses VT-d, a virtualization PCI Passthrough technique embedded in the bios.
So I will state my machine details first:
- Motherboard: ASUS P5E VM DO with bios 10.1
- CPU: Q6600
- 6 Gb DDR-2 Memory
- 1 onboard GbE Nic
- 1 PCI-E GbE Nic
- 1 Hauppauge PVR-150 (for the moment)
han@obelix:~$ lspci 00:00.0 Host bridge: Intel Corporation 82Q35 Express DRAM Controller (rev 02) 00:02.0 VGA compatible controller: Intel Corporation 82Q35 Express Integrated Graphics Controller (rev 02) 00:03.0 Communication controller: Intel Corporation 82Q35 Express MEI Controller (rev 02) 00:03.2 IDE interface: Intel Corporation 82Q35 Express PT IDER Controller (rev 02) 00:03.3 Serial controller: Intel Corporation 82Q35 Express Serial KT Controller (rev 02) 00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connection (rev 02) 00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02) 00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02) 00:1a.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02) 00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02) 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 02) 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02) 00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02) 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation 82801IO (ICH9DO) LPC Interface Controller (rev 02) 00:1f.2 IDE interface: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 4 port SATA IDE Controller (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA IDE Controller (rev 02) 00:1f.6 Signal processing controller: Intel Corporation 82801I (ICH9 Family) Thermal Subsystem (rev 02) 01:00.0 IDE interface: JMicron Technologies, Inc. JMB368 IDE controller 02:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06) 03:01.0 Multimedia video controller: Internext Compression Inc iTVC16 (CX23416) MPEG-2 Encoder (rev 01) 03:02.0 FireWire (IEEE 1394): Agere Systems FW323 (rev 70)
Step 1) Install Ubuntu 9.04 server AMD 64 (should be easy enough if you are able to get LMCE up and running :-) )
Step 2) Make the system current:
sudo apt-get update sudo apt-get upgrade
Step 3) Add some management tools:
sudo wget http://prdownloads.sourceforge.net/webadmin/webmin_1.480_all.deb sudo apt-get install libnet-ssleay-perl libauthen-pam-perl libio-pty-perl libmd5-perl sudo dpkg -i webmin_1.480_all.deb sudo apt-get install mc
I use webmin to give my server a fixed ip address and perform other administrative tasks. MC (Midnight Commander) is a nice tool to move files around the server.
Step 4) To be able to do remote monitoring of virtual machines, install virt-manager on your remote machine. If your are using Jaunty on your remote system, install ssh-askpass as per [[1]]
sudo apt-get install ssh-askpass
or use ssh keys
Step 5) Get the up to date kernel sources & unpack them
sudo apt-get install linux-source sudo tar -jxvf linux-2.6.31.tar.bz2 -C /usr/src
Step 6) Install dependencies for make menuconfig.
sudo apt-get install gcc ncurses* kernel-package fakeroot
Step 7) Configure the new kernel. (In this case for Karmic 2.6.31)
cd /usr/src/linux-2.6.31 # cd into directory cp /boot/config-2.6.31-14-server .config # this will make sure you start of with the right settings from the previous kernel make oldconfig # here you are being asked for all of the changes and if you want to implement them, my advise: make a note and say no, in the next step it can all be adjusted sudo make menuconfig # here you are able to make the wanted changes, for VT-D to work you will need to enable at least:
* set "Bus options (PCI etc.)" -> "Support for DMA Remapping Devices" to "*" * set "Bus options (PCI etc.)" -> "Enable DMA Remapping Devices" to "*" * set "Bus options (PCI etc.)" -> "PCI Stub driver" to "*"
(This line broke my setup so I took it out * optional setting: set "Bus options (PCI etc.)" -> "Support for Interrupt Remapping" to "*")
Step 8) Build the new kernel.
sudo make-kpkg clean sudo fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers
You can grab a coffee, make a hot meal, have a sleep ;-)
Step 9) Install The New Kernel
After the successful kernel build, you can find two .deb packages in the /usr/bin directory.
cd .. ls -l linux*
Install them:
sudo dpkg -i linux-image-2.6.30.4-custom_2.6.30.4-custom-10.00.Custom_amd64.deb sudo sudo dpkg -i linux-headers-2.6.30.4-custom_2.6.30.4-custom-10.00.Custom_amd64.deb
Step 10) Check that menu.list has been updated with your new kernel
cat /boot/grub/menu.lst sudo reboot
Step 11) Deinstall networkmanager
aptitude remove network-manager
Step 12) Setup your network, change /etc/network/interfaces to look like this
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The first bridge interface auto bridge_ext iface bridge_ext inet static address 0.0.0.0 #no external access without going through DCERouter first netmask 255.255.255.0 broadcast 10.0.0.255 gateway 10.0.0.138 bridge_ports eth0 bridge_stp off bridge_maxwait 5 # The second bridge interface auto bridge_int iface bridge_int inet static address 192.168.80.240 netmask 255.255.255.0 broadcast 192.168.80.255 gateway 192.168.80.1 bridge_ports eth1 bridge_stp off bridge_maxwait 5
With the above the host will use dcerouter as its way into the internet and thus the protection offered by the firewall. The host has no direct access to the 10.0.0.0 network.
Step 12) Install Linuxmce
Using virt-manager you can now install LinuxMCE. As we are all experts on that I will only give away my xml config file (should be enough ;-))
<name>DCERouter</name>
<uuid>f1a8bb7e-9e07-4f72-1337-d04429d0d2de</uuid>
<memory>2097152</memory>
<currentMemory>2097152</currentMemory>
<vcpu>2</vcpu>
<os>
<type arch='i686' machine='pc'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<source file='/storage2/100-virtualmachines/DCERouter.img'/>
<target dev='hda' bus='ide'/>
</disk>
<disk type='file' device='cdrom'>
<target dev='hdc' bus='ide'/>
<readonly/>
</disk>
<interface type='bridge'>
<mac address='54:52:00:0a:09:ee'/>
<source bridge='bridge_ext'/>
<target dev='vnet0'/>
<model type='virtio'/>
</interface>
<interface type='bridge'>
<mac address='54:52:00:14:87:a4'/>
<source bridge='bridge_int'/>
<target dev='vnet1'/>
<model type='virtio'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target port='0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='yes' keymap='en-us'/>
<sound model='es1370'/>
<hostdev mode='subsystem' type='pci' managed='no'>
<source>
<address domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
</source>
</hostdev>
</devices>
</domain>
</domstatus>
Step 13) Enjoying building a test enviroment, inline upgrade etc.
Step 14) I chose to put my media on the host via a Samba share which will be detected as a NAS. This will simplify things when performing upgrades and test. Because I am using LVM also expansion is easy ;-)
Hints/Known Issues/Working Hardware
AMD/ATI Hardware
- NVidia Chipset 3600 with Opteron Socket F has full KVM support and works fine with LMCE 8.10.
- ATI-driver 9.8 supports kernel 2.6.30
Keyboard Layout
- As you can't choose the keyboard-layout in virtual-manager, simply change the keymap in the xml-file e.g. from "en-us" to "de".
Networking
- Step 11) is not needed if the physical network devices are configured as 'auto ethx inet manual'
- The virtual LMCE runs fine with only one NIC provided to it. The setup automatically configures a sub-NIC which works well. I would suggest to have the host provide the internet-access, so vLMCE can't mess it up at the installation.
PCI-Passthrough at KVM
- PCI-Passthrough only works if the full PCI-device is transferred to the virtual machine. So if the PCI-Slot has sub-functions (e.g. Firewire) all of the 'functions' have to be adressed to the virtual machine with the <hostdev> section of Step 12). To look up your PCI-device-tree, use 'lspci -t'.
(Tyan S2915-E)
[0000:00]-+-00.0
+-01.0
+-01.1
+-02.0
+-02.1
+-04.0
+-05.0
+-05.1
+-06.0-[0000:01]----04.0 (<- PCI-Slot = DVB-card)
\-04.1 (<- Firewire, either disable in Bios or ->KVM)
+-06.1
+-08.0
+-09.0
+-0d.0-[0000:02-04]--+-00.0-[0000:03]----04.0
| \-00.1-[0000:04]--
+-0f.0-[0000:05]--+-00.0
| \-00.1
+-18.0
+-18.1
+-18.2
+-18.3
\-18.4
- PCIe-devices do not have this restriction.
- Please note that each change in the .xml-files is activated only after rebooting the libvirtd. You can do this manually by killing and restarting the process "libvirtd" or by a host reboot.
