/usr/local.org

User Tools

Site Tools

You are here: Home » Documents » Tips 'n tricks (IT) » Xen
docs:tips_n_tricks:xen.html

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
docs:tips_n_tricks:xen.html [16.04.2022 12:38 CEST] – created peterdocs:tips_n_tricks:xen.html [13.08.2022 23:36 CEST] (current) – [Create new DomU] peter
Line 1: Line 1:
-Removed as it contains internal information. +====== Xen ====== 
-Recreating ...+===== Start Instance ===== 
 +after you have written a config file ''/etc/xen/abcd.cfg'' you have to create the instance by 
 + 
 +  xm create abcd.cfg 
 + 
 +(I didn't expect the suffix ''.cfg'' to be neccessary) 
 + 
 +===== Get a console ===== 
 + 
 +  xm console <Domain> 
 + 
 +to disconnect type ''ctrl-]'' (which means to press Str-AltGr-9 on a german keyboard) 
 + 
 +===== Kill a running instance ===== 
 + 
 +  xm destroy <Domain> 
 + 
 +this will not(!) destroy your data or config, it will just "poweroff" the running virtual machine without a clean shutdown. (don't know what happens to snapshots) 
 + 
 +===== Create new DomU ===== 
 +//Assuming, you have a bridge //xenbr0// up and running on your Dom0 with 10.1.0.1 beeing the outbound gateway and a disk partitioning scheme defined in// ''/etc/xen-tools/partitions.d/my-disk-set'' 
 + 
 +  * create ''/etc/xen-tools/skel/root/.ssh/authorized_keys'' 
 +  * run ''create_guest.sh //<name>//'' 
 + 
 +<code bash create_guest.sh> 
 +#!/bin/bash 
 + 
 +set -o errexit 
 + 
 +name="${1:-xenguest}" 
 + 
 +xen-create-image --hostname="${name}" \ 
 + --randommac \ 
 + --ip=10.1.0.2 \ 
 + --gateway=10.1.0.1 \ 
 + --netmask=255.255.255.0 \ 
 + --mirror=http://http.debian.net/debian/ \ 
 + --dist=bullseye \ 
 + --lvm=vg_1 \ 
 + --bridge=xenbr0 \ 
 + --vifname=vif."${name}".0 \ 
 + --partitions=my-disk-set \ 
 + --vcpus=1 \ 
 + --memory=512Mb \ 
 + --arch=amd64 \ 
 + --nokeep \ 
 + --nohosts \ 
 + --boot \ 
 + --password "" \ 
 + --pygrub 
 + 
 +xl console "${name}" 
 +</code> 
 + 
 +<code authorized_keys /etc/xen-tools/skel/root/.ssh/authorized_keys> 
 +# replace 202204260000 by some day in the near future and add your own sshkey 
 +expiry-time="202204260000" ssh-rsa AAAAB3N... 
 +</code> 
 + 
 +===== Matching physical and virtual Xen block (and other) devices and eventually their ids ===== 
 + 
 +  xenstore-ls -f -s /local/domain/0/backend/vbd | egrep '(domain|frontend|dev|params) = '  
 + 
 + 
 +__References:__ 
 + 
 +  * [[https://serverfault.com/a/259112|ukautz' answer]] to question [[https://serverfault.com/questions/153196/xen-find-vbd-id-for-physical-disks|"Xen find VBD id for physical disks" on Server Fault]]. 
 +  * https://xenbits.xen.org/docs/4.9-testing/man/xenstore-ls.1.html 
 +===== Turn a domU into a new dom0 running with serial console ===== 
 + 
 +This is for domU booted by //pygrub// 
 + 
 +  * on domU as guest of dom0 
 +    * give all filesystem a label 
 +    * replace device by label in /etc/fstab on domU 
 +    * replace dev by label for root fs in /boot/grub/menu.lst of domU 
 +    * install lvm2 and mdadm 
 +      * run //update-initramfs// if not done automatically 
 +  * on dom0 run ''update-grub2'' 
 +  * boot into grub menu 
 +    * edit config for booting into the domU system on bare metal 
 +      * add ''console=tty0 console=ttyS0,57600'' to linux kernel line 
 +    * continue booting 
 +  * in domU system on bare metal, install grub-pc and 
 +    * edit ''/etc/default/grub'' 
 +      * ''GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,57600"'' 
 +      * ''GRUB_TERMINAL=serial'' 
 +    * run ''grub-install --no-bootsector /dev/mapper///<lvm device>//''((Why doesn't it run on deb install time???)) 
 +    * run ''update-grub2'' ((Why doesn't it run on deb install time???)) 
 +    * label filesystems if not already done (see above) 
 +      * don't forget to relabel swap space 
 +      * edit ''/etc/fstab'' if not already done (see above) 
 +    * update /boot/grub/custom.cfg on (previous) dom0, so it boots domU system on bare metal via config ... 
 +<note important>One important thing is to **not** load raid((that is modules //diskfilter// and //mdraid1x//)) and lvm modules in the same ''insmod'' line in ''grub/custom.cfg''!</note> 
 +  *  
 +    * install xen packages (xen-system, xen-tools, grub-xen-host, ...) 
 +    * edit ''/etc/network/interfaces'' to setup ip etc. correctly 
 +    * allow root login by ssh (if you want it) 
 +    * run ''mdadm --detail --brief --scan >> /etc/mdadm/mdadm.conf'' on domU system on bare metal 
 +    * run u''pdate-initrams -k all -c'' on domU system on bare metal 
 +  * reboot and ... 
 + 
 +==== Create a generic grub installation ==== 
 +<code bash grub-install.sh> 
 +lvcreate -L 1G -n lv_GRUB0 /dev/vg_1 
 +mke2fs -L "$HOSTNAME:GRUB0" -t ext4 /dev/vg_1/lv_GRUB0  
 +mkdir /GRUB0 
 +mount LABEL="$HOSTNAME:GRUB0" /GRUB0/ 
 +grub-install --boot-directory /GRUB0 --recheck /dev/sdb 
 +grub-install --boot-directory /GRUB0 --recheck /dev/sda 
 +</code> 
 +<code grub /GRUB0/grub/grub.cfg> 
 +serial --speed=57600 --unit=0 --word=8 --parity=no --stop=1 
 +terminal_input serial 
 +terminal_output serial 
 +set timeout=600 
 +set default="dom0: LVM vg_1/dom0-root boot/grub/grub.cfg" 
 + 
 +menuentry 'Old dom0: /dev/md/0(/dev/sdb1,/dev/sda1) grub/grub.cfg' { 
 +        insmod gzio 
 +        insmod part_msdos 
 +        insmod diskfilter 
 +        insmod mdraid1x 
 +        insmod lvm 
 +        insmod ext2 
 +        set root='mduuid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' 
 +        echo 'Loading (/dev/md/0)/grub/grub.cfg' 
 +        configfile /grub/grub.cfg 
 +} 
 +menuentry 'dom0: LVM vg_1/dom0-root boot/grub/grub.cfg' { 
 +        insmod gzio 
 +        insmod part_msdos 
 +        insmod diskfilter 
 +        insmod mdraid1x 
 +        insmod lvm 
 +        insmod ext2 
 +        echo "Setting root='lvm/vg_1-dom0--root'" 
 +        set root='lvm/vg_1-dom0--root' 
 +        echo 'Loading (/dev/vg_1/dom0-root)/boot/grub/grub.cfg' 
 +        configfile /boot/grub/grub.cfg 
 +} 
 +menuentry "hd0 (MBR)" { 
 +        insmod chain 
 +        set root=(hd0) 
 +        chainloader +1 
 +} 
 +menuentry "hd1 (MBR)" { 
 +        insmod chain 
 +        set root=(hd1) 
 +        chainloader +1 
 +} 
 +menuentry "Reboot" { 
 +        reboot 
 +} 
 +menuentry "Halt" { 
 +        halt 
 +} 
 +</code> 
 +==== Use old dom0 as domU on new dom0 ==== 
 +//Assuming //grub-xen-host// is installed and ''/dev/md/0'' is ''/boot'' of old dom0// 
 +<code xen /etc/xen/olddom0.cfg> 
 +kernel = '/usr/lib/grub-xen/grub-x86_64-xen.bin' 
 + 
 +vcpus       = '1' 
 +memory      = '2048' 
 + 
 +root        = '/dev/xvda1 ro' 
 + 
 +disk        = [ 
 +                  'phy:/dev/md/0,xvda1,w', 
 +                  'phy:/dev/md/1,xvda2,w', 
 +                  'phy:/dev/md/3,xvda3,w', 
 +                  'phy:/dev/vg_1/lv_home,xvda4,w', 
 +                  'phy:/dev/sda2,xvda5,w', 
 +                  'phy:/dev/sdb2,xvda6,w', 
 +              ] 
 + 
 +name        = 'olddom0' 
 + 
 +dhcp        = 'dhcp' 
 +vif         = [ 'bridge=xenbr0,vifname=vif.oddom0.0,mac=00:16:3e:XX:XX:XX' ] 
 +</code> 
 +==== Attach network to domU ==== 
 +//assuming lspci shows 0000:02:00.0 as your network adapter// 
 + 
 +<note important>This does not work with Xen-4.14 on debian bullseye, see https://patchew.org/Xen/20200619073315.8414-1-paul@xen.org/. 
 + 
 +Using Xen-4.11 from debian buster on debian bullseye works.</note> 
 +Add 
 + 
 +  xen-pciback hide=(0000:02:00.0) 
 + 
 +to ''/etc/initramfs-tools/modules'', run ''update-initramfs -u'' 
 +Add 
 + 
 +  pci = [ "0000:02:00.0" ] 
 + 
 +to ''/etc/xen/guest.cfg'' 
 + 
 + 
 +Add (assuming your network driver is //e1000e//) 
 + 
 +  GRUB_CMDLINE_LINUX_XEN_REPLACE="$GRUB_CMDLINE_LINUX blacklist=e1000e" 
 + 
 +to ''/etc/default/grub'' 
 +<note warning>If you install GRUB from the new Dom0, make sure no outdated ''device.map'' is lying around. Best use the ''--recheck'' option to ''grub-install''</note> 
 +Manual steps: 
 +<code bash> 
 +rmmod e1000e 
 +rmmod xen-pciback 
 +modprobe xen-pciback "hide=(0000:02:00.0)" 
 +modprobe e1000e 
 +# check result 
 +ip link 
 +xl pci-assignable-list 
 +</code> 
 +==== Internal network ==== 
 +On dom0 (new one if you came from above): 
 + 
 +  xl network-attach //olddom0// 
 +  brctl addbr xenbr0 
 +  ifconfig xenbr 10.1.0.2 netmask 255.255.255.0 
 +  route add default gw 10.1.0.2 
 + 
 +On domU (with internet access - former dom0 if you came from above) 
 + 
 +//assuming eth0 is the outer (physical) network interface and the xen virtual interface showed up as eth1// 
 + 
 +  ifconfig eth0 10.1.0.1 netmask 255.255.255.0 
 +  INET_IP="`ifconfig eth0 | sed -n -e 's/^[[:space:]]*inet \([.0-9]\+\) .*$/\1/gp'`" 
 +  iptables -t nat -A POSTROUTING -o eth0 -s 10.1.0.0/24  ! -d  10.1.0.0/24 -j SNAT --to "$INET_IP" 
 +  sysctl -w net.ipv4.conf.all.forwarding=1 
 + 
 +=== Make network card names persistent with your own (traditional) naming scheme === 
 +//In domU connected to external interface// 
 +== The ifupdown way == 
 +  * Give parameeter ''net.ifnames=0'' to the kernel at boot time  
 +  * Use ''mac/xx.xx.xx.xx.xx.xx.xx=eth0'' in ''/etc/network/interfaces'' 
 +<code interfaces /etc/network/interfaces.d/ifrename> 
 +rename mac/00:16:3e:XX:XX:XX=vif0 
 +# rename mac/aa:bb:cc:??:??:??=eth0 # use this if you expect the network card to be replaced by another one of same type 
 +rename mac/aa:bb:cc:dd:ee:ff=eth0 
 +</code> 
 +== The systemd way == 
 +<code properties /etc/systemd/network/10-persistent-eth0.link> 
 +# https://wiki.debian.org/NetworkInterfaceNames 
 +[Match] 
 +MACAddress=aa:bb:cc:dd:ee:ff 
 + 
 +[Link] 
 +Name=eth0 
 +</code> 
 + 
 +<code properties /etc/systemd/network/11-persistent-vif0.link> 
 +# https://wiki.debian.org/NetworkInterfaceNames 
 +[Match] 
 +MACAddress=00:16:3e:XX:XX:XX 
 + 
 +[Link] 
 +Name=vif0 
 +</code> 
 +=== Use persistent name in /etc/network/interfaces === 
 + 
 +<code interfaces /etc/network/interfaces.d/vif0> 
 +auto vif0 
 +iface vif0 inet static 
 + address 10.1.0.1 
 + netmask 255.255.255.0 
 +</code> 
 +==== Allow ssh root login ==== 
 +<code text /etc/ssh/sshd_config> 
 +: 
 +PermitRootLogin yes 
 +: 
 +</code> 
 +==== Further reading ==== 
 +  * man xl.cfg 
 +  * https://wiki.xenproject.org/wiki/Xen_PCI_Passthrough#How_can_I_tell_if_I_have_IOMMU_.2F_VT-D_support.3F 
 +  * https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-xen-vhost.html 
 +  * https://xenbits.xen.org/docs/4.13-testing/misc/xen-command-line.html 
 +  * https://xenbits.xen.org/docs/4.6-testing/misc/xl-network-configuration.html 
 +  * https://xenproject.org/2015/01/07/using-grub-2-as-a-bootloader-for-xen-pv-guests/ 
 +  * https://wiki.debian.org/Xen 
 +  * Bugs I encountered 
 +    * [[https://patchew.org/Xen/20200619073315.8414-1-paul@xen.org/|[PATCH for 4.14] libxl: allow passthrough to PV guests regardless of whether IOMMU is enabled]] 
 +      * Workaround: Fixed that by downgrading to Xen 4.11 from Debian Buster :-/ 
 +    * [[https://github.com/xen-tools/xen-tools/issues/60|xen-create-image --nodhcp option results in dhcp being configured #60]] 
 +      * Workaround: Do not use ''--nodhcp'', it is not needed if //ip// and //netmask// is given.8-) 
 +    * [[http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=dcc0bf5dec61b3dd1cc00683b5b9b5bfe0a318de|libxl: fix pci device re-assigning after domain reboot]] 
 +      * Workaround: Don't reboot domU with PCI passthrough - shutdown and boot instead.8-) 
docs/tips_n_tricks/xen.html.1650105485.txt.gz · Last modified: 16.04.2022 12:38 CEST by peter
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki