Differences

This shows you the differences between two versions of the page.

@@ Line 1: / Line 1: @@
+====== Xen ======
+===== Start Instance =====
+after you have written a config file ''/etc/xen/abcd.cfg'' you have to create the instance by
+  xm create abcd.cfg
+(I didn't expect the suffix ''.cfg'' to be neccessary)
+===== Get a console =====
+  xm console <Domain>
+to disconnect type ''ctrl-]'' (which means to press Str-AltGr-9 on a german keyboard)
+===== Kill a running instance =====
+  xm destroy <Domain>
+this will not(!) destroy your data or config, it will just "poweroff" the running virtual machine without a clean shutdown. (don't know what happens to snapshots)
+===== Create new DomU =====
+//Assuming, you have a bridge //xenbr0// up and running on your Dom0 with 10.1.0.1 beeing the outbound gateway and a disk partitioning scheme defined in// ''/etc/xen-tools/partitions.d/my-disk-set''
+  * create ''/etc/xen-tools/skel/root/.ssh/authorized_keys''
+  * run ''create_guest.sh //<name>//''
+<code bash create_guest.sh>
+#!/bin/bash
+set -o errexit
+name="${1:-xenguest}"
+xen-create-image --hostname="${name}" \
+ --randommac \
+ --ip=10.1.0.2 \
+ --gateway=10.1.0.1 \
+ --netmask=255.255.255.0 \
+ --mirror=http://http.debian.net/debian/ \
+ --dist=bullseye \
+ --lvm=vg_1 \
+ --bridge=xenbr0 \
+ --vifname=vif."${name}".0 \
+ --partitions=my-disk-set \
+ --vcpus=1 \
+ --memory=512Mb \
+ --arch=amd64 \
+ --nokeep \
+ --nohosts \
+ --boot \
+ --password "" \
+ --pygrub
+xl console "${name}"
+</code>
+<code authorized_keys /etc/xen-tools/skel/root/.ssh/authorized_keys>
+# replace 202204260000 by some day in the near future and add your own sshkey
+expiry-time="202204260000" ssh-rsa AAAAB3N...
+</code>
+===== Matching physical and virtual Xen block (and other) devices and eventually their ids =====
+  xenstore-ls -f -s /local/domain/0/backend/vbd | egrep '(domain|frontend|dev|params) = '
+__References:__
+  * [[https://serverfault.com/a/259112|ukautz' answer]] to question [[https://serverfault.com/questions/153196/xen-find-vbd-id-for-physical-disks|"Xen find VBD id for physical disks" on Server Fault]].
+  * https://xenbits.xen.org/docs/4.9-testing/man/xenstore-ls.1.html
+===== Turn a domU into a new dom0 running with serial console =====
+This is for domU booted by //pygrub//
+  * on domU as guest of dom0
+    * give all filesystem a label
+    * replace device by label in /etc/fstab on domU
+    * replace dev by label for root fs in /boot/grub/menu.lst of domU
+    * install lvm2 and mdadm
+      * run //update-initramfs// if not done automatically
+  * on dom0 run ''update-grub2''
+  * boot into grub menu
+    * edit config for booting into the domU system on bare metal
+      * add ''console=tty0 console=ttyS0,57600'' to linux kernel line
+    * continue booting
+  * in domU system on bare metal, install grub-pc and
+    * edit ''/etc/default/grub''
+      * ''GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,57600"''
+      * ''GRUB_TERMINAL=serial''
+    * run ''grub-install --no-bootsector /dev/mapper///<lvm device>//''((Why doesn't it run on deb install time???))
+    * run ''update-grub2'' ((Why doesn't it run on deb install time???))
+    * label filesystems if not already done (see above)
+      * don't forget to relabel swap space
+      * edit ''/etc/fstab'' if not already done (see above)
+    * update /boot/grub/custom.cfg on (previous) dom0, so it boots domU system on bare metal via config ...
+<note important>One important thing is to **not** load raid((that is modules //diskfilter// and //mdraid1x//)) and lvm modules in the same ''insmod'' line in ''grub/custom.cfg''!</note>
+  *
+    * install xen packages (xen-system, xen-tools, grub-xen-host, ...)
+    * edit ''/etc/network/interfaces'' to setup ip etc. correctly
+    * allow root login by ssh (if you want it)
+    * run ''mdadm --detail --brief --scan >> /etc/mdadm/mdadm.conf'' on domU system on bare metal
+    * run u''pdate-initrams -k all -c'' on domU system on bare metal
+  * reboot and ...
+==== Create a generic grub installation ====
+<code bash grub-install.sh>
+lvcreate -L 1G -n lv_GRUB0 /dev/vg_1
+mke2fs -L "$HOSTNAME:GRUB0" -t ext4 /dev/vg_1/lv_GRUB0
+mkdir /GRUB0
+mount LABEL="$HOSTNAME:GRUB0" /GRUB0/
+grub-install --boot-directory /GRUB0 --recheck /dev/sdb
+grub-install --boot-directory /GRUB0 --recheck /dev/sda
+</code>
+<code grub /GRUB0/grub/grub.cfg>
+serial --speed=57600 --unit=0 --word=8 --parity=no --stop=1
+terminal_input serial
+terminal_output serial
+set timeout=600
+set default="dom0: LVM vg_1/dom0-root boot/grub/grub.cfg"
+menuentry 'Old dom0: /dev/md/0(/dev/sdb1,/dev/sda1) grub/grub.cfg' {
+        insmod gzio
+        insmod part_msdos
+        insmod diskfilter
+        insmod mdraid1x
+        insmod lvm
+        insmod ext2
+        set root='mduuid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
+        echo 'Loading (/dev/md/0)/grub/grub.cfg'
+        configfile /grub/grub.cfg
+}
+menuentry 'dom0: LVM vg_1/dom0-root boot/grub/grub.cfg' {
+        insmod gzio
+        insmod part_msdos
+        insmod diskfilter
+        insmod mdraid1x
+        insmod lvm
+        insmod ext2
+        echo "Setting root='lvm/vg_1-dom0--root'"
+        set root='lvm/vg_1-dom0--root'
+        echo 'Loading (/dev/vg_1/dom0-root)/boot/grub/grub.cfg'
+        configfile /boot/grub/grub.cfg
+}
+menuentry "hd0 (MBR)" {
+        insmod chain
+        set root=(hd0)
+        chainloader +1
+}
+menuentry "hd1 (MBR)" {
+        insmod chain
+        set root=(hd1)
+        chainloader +1
+}
+menuentry "Reboot" {
+        reboot
+}
+menuentry "Halt" {
+        halt
+}
+</code>
+==== Use old dom0 as domU on new dom0 ====
+//Assuming //grub-xen-host// is installed and ''/dev/md/0'' is ''/boot'' of old dom0//
+<code xen /etc/xen/olddom0.cfg>
+kernel = '/usr/lib/grub-xen/grub-x86_64-xen.bin'
+vcpus       = '1'
+memory      = '2048'
+root        = '/dev/xvda1 ro'
+disk        = [
+                  'phy:/dev/md/0,xvda1,w',
+                  'phy:/dev/md/1,xvda2,w',
+                  'phy:/dev/md/3,xvda3,w',
+                  'phy:/dev/vg_1/lv_home,xvda4,w',
+                  'phy:/dev/sda2,xvda5,w',
+                  'phy:/dev/sdb2,xvda6,w',
+              ]
+name        = 'olddom0'
+dhcp        = 'dhcp'
+vif         = [ 'bridge=xenbr0,vifname=vif.oddom0.0,mac=00:16:3e:XX:XX:XX' ]
+</code>
+==== Attach network to domU ====
+//assuming lspci shows 0000:02:00.0 as your network adapter//
+<note important>This does not work with Xen-4.14 on debian bullseye, see https://patchew.org/Xen/20200619073315.8414-1-paul@xen.org/.
+Using Xen-4.11 from debian buster on debian bullseye works.</note>
+Add
+  xen-pciback hide=(0000:02:00.0)
+to ''/etc/initramfs-tools/modules'', run ''update-initramfs -u''
+Add
+  pci = [ "0000:02:00.0" ]
+to ''/etc/xen/guest.cfg''
+Add (assuming your network driver is //e1000e//)
+  GRUB_CMDLINE_LINUX_XEN_REPLACE="$GRUB_CMDLINE_LINUX blacklist=e1000e"
+to ''/etc/default/grub''
+<note warning>If you install GRUB from the new Dom0, make sure no outdated ''device.map'' is lying around. Best use the ''--recheck'' option to ''grub-install''</note>
+Manual steps:
+<code bash>
+rmmod e1000e
+rmmod xen-pciback
+modprobe xen-pciback "hide=(0000:02:00.0)"
+modprobe e1000e
+# check result
+ip link
+xl pci-assignable-list
+</code>
+==== Internal network ====
+On dom0 (new one if you came from above):
+  xl network-attach //olddom0//
+  brctl addbr xenbr0
+  ifconfig xenbr 10.1.0.2 netmask 255.255.255.0
+  route add default gw 10.1.0.2
+On domU (with internet access - former dom0 if you came from above)
+//assuming eth0 is the outer (physical) network interface and the xen virtual interface showed up as eth1//
+  ifconfig eth0 10.1.0.1 netmask 255.255.255.0
+  INET_IP="`ifconfig eth0 | sed -n -e 's/^[[:space:]]*inet \([.0-9]\+\) .*$/\1/gp'`"
+  iptables -t nat -A POSTROUTING -o eth0 -s 10.1.0.0/24  ! -d  10.1.0.0/24 -j SNAT --to "$INET_IP"
+  sysctl -w net.ipv4.conf.all.forwarding=1
+=== Make network card names persistent with your own (traditional) naming scheme ===
+//In domU connected to external interface//
+== The ifupdown way ==
+  * Give parameeter ''net.ifnames=0'' to the kernel at boot time
+  * Use ''mac/xx.xx.xx.xx.xx.xx.xx=eth0'' in ''/etc/network/interfaces''
+<code interfaces /etc/network/interfaces.d/ifrename>
+rename mac/00:16:3e:XX:XX:XX=vif0
+# rename mac/aa:bb:cc:??:??:??=eth0 # use this if you expect the network card to be replaced by another one of same type
+rename mac/aa:bb:cc:dd:ee:ff=eth0
+</code>
+== The systemd way ==
+<code properties /etc/systemd/network/10-persistent-eth0.link>
+# https://wiki.debian.org/NetworkInterfaceNames
+[Match]
+MACAddress=aa:bb:cc:dd:ee:ff
+[Link]
+Name=eth0
+</code>
+<code properties /etc/systemd/network/11-persistent-vif0.link>
+# https://wiki.debian.org/NetworkInterfaceNames
+[Match]
+MACAddress=00:16:3e:XX:XX:XX
+[Link]
+Name=vif0
+</code>
+=== Use persistent name in /etc/network/interfaces ===
+<code interfaces /etc/network/interfaces.d/vif0>
+auto vif0
+iface vif0 inet static
+ address 10.1.0.1
+ netmask 255.255.255.0
+</code>
+==== Allow ssh root login ====
+<code text /etc/ssh/sshd_config>
+:
+PermitRootLogin yes
+:
+</code>
+==== Further reading ====
+  * man xl.cfg
+  * https://wiki.xenproject.org/wiki/Xen_PCI_Passthrough#How_can_I_tell_if_I_have_IOMMU_.2F_VT-D_support.3F
+  * https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-xen-vhost.html
+  * https://xenbits.xen.org/docs/4.13-testing/misc/xen-command-line.html
+  * https://xenbits.xen.org/docs/4.6-testing/misc/xl-network-configuration.html
+  * https://xenproject.org/2015/01/07/using-grub-2-as-a-bootloader-for-xen-pv-guests/
+  * https://wiki.debian.org/Xen
+  * Bugs I encountered
+    * [[https://patchew.org/Xen/20200619073315.8414-1-paul@xen.org/|[PATCH for 4.14] libxl: allow passthrough to PV guests regardless of whether IOMMU is enabled]]
+      * Workaround: Fixed that by downgrading to Xen 4.11 from Debian Buster :-/
+    * [[https://github.com/xen-tools/xen-tools/issues/60|xen-create-image --nodhcp option results in dhcp being configured #60]]
+      * Workaround: Do not use ''--nodhcp'', it is not needed if //ip// and //netmask// is given.8-)
+    * [[http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=dcc0bf5dec61b3dd1cc00683b5b9b5bfe0a318de|libxl: fix pci device re-assigning after domain reboot]]
+      * Workaround: Don't reboot domU with PCI passthrough - shutdown and boot instead.8-)