manuals:vps:kvm
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| manuals:vps:kvm [2023/07/31 14:56] – Update for vpsAdminOS, remove old/incompatible distros Aither | manuals:vps:kvm [2026/08/17 11:09] (current) – Link source and virtualization tests and correct the shared translation ID aither | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== Using KVM on vpsFree.cz | + | < |
| - | VPS are [[information:vpsadminos|linux containers]], | + | |
| - | which is shared between all VPS. In case you need your own kernel or if you'd like to use a different | + | |
| - | operating system, you can create your own virtual machines inside the VPS using QEMU/KVM. | + | |
| - | ===== Configuration ===== | + | < |
| + | source=" | ||
| + | test=" | ||
| + | /> | ||
| - | Go to the details of your VPS in vpsAdmin | + | ====== KVM and libvirt inside a VPS ====== |
| - | * TUN/TAP – enables the creation of virtual interfaces, | + | ===== Installing libvirt ===== |
| - | * KVM – enables KVM (for hardware support of virtualization). | + | |
| - | ===== libvirt ===== | + | vpsAdminOS runs VPSes as containers, not as virtual machines. KVM inside a VPS |
| - | Virtual | + | is therefore not nested virtualization: |
| - | with [[https://virt-manager.org|virt-manager]]. | + | node's hardware virtualization directly. This guide uses the current **Debian |
| + | (latest)** template, | ||
| - | ===== KVM on Alpine Linux ===== | + | < |
| + | The **KVM** feature is required for virtualization. **TUN/TAP** is required for | ||
| + | TAP-based virtual-machine network interfaces. Both features are enabled by | ||
| + | default for new VPSes in the **Features** section of the VPS detail in | ||
| + | vpsAdmin. | ||
| + | </ | ||
| - | Install the required packages (ip6tables is optional): | + | {{:en: |
| - | < | + | Install QEMU, libvirt and '' |
| - | apk update | + | |
| - | apk add qemu-system-x86_64 qemu-openrc qemu-img bridge iptables ip6tables | + | < |
| - | </ | + | # |
| + | set -euo pipefail | ||
| - | Configure the bridge for Qemu/KVM virtual machines – create the /// | + | apt-get update |
| + | apt-get install --yes \ | ||
| + | iptables \ | ||
| + | libvirt-clients \ | ||
| + | libvirt-daemon-system \ | ||
| + | qemu-system-x86 \ | ||
| + | qemu-utils \ | ||
| + | virtinst | ||
| - | <code shell> | + | virsh --connect qemu:/// |
| - | auto br0 | + | |
| - | iface br0 inet static | + | |
| - | pre-up brctl addbr br0 | + | |
| - | | + | |
| - | | + | |
| - | post-down brctl delbr br0 | + | |
| </ | </ | ||
| - | Since OpenVZ rewrites | + | Use the '' |
| + | libvirt over SSH, for example '' | ||
| + | with a key. Do not expose an unencrypted libvirt TCP socket | ||
| - | Give the user in the //qemu// group permissions to manage the newly-created bridge: | + | ===== Disk images and ZFS ===== |
| - | <code shell> | + | Use a separate [[manuals: |
| - | echo "allow br0" > /etc/qemu/bridge.conf | + | you change ZFS properties for the workload later without affecting the VPS root |
| - | chown root:qemu /etc/qemu/bridge.conf | + | dataset. Prepare the storage as follows: |
| - | chmod 0640 /etc/qemu/bridge.conf | + | |
| + | * <vpsadmin-nav id=" | ||
| + | * < | ||
| + | * < | ||
| + | |||
| + | For example, from the standard 120 GiB allocation you can leave 20 GiB for the | ||
| + | root dataset and assign 100 GiB to '' | ||
| + | virtual-machine disk 80 GiB and leave 20 GiB free for growth and snapshots. | ||
| + | |||
| + | The subdataset is [[manuals:vps: | ||
| + | root dataset. | ||
| + | |||
| + | {{: | ||
| + | |||
| + | Add the mounted directory to libvirt as a persistent storage pool: | ||
| + | |||
| + | <code bash> | ||
| + | # | ||
| + | set -euo pipefail | ||
| + | |||
| + | virsh --connect | ||
| + | vm-images dir --target / | ||
| + | virsh --connect | ||
| + | virsh --connect qemu:///system pool-autostart vm-images | ||
| + | virsh --connect | ||
| </ | </ | ||
| - | Configure | + | Then create disks through libvirt, '' |
| + | '' | ||
| - | If you have configured iptables, all you need to add is this rule: | + | Leave the subdataset' |
| + | show a reason | ||
| + | '' | ||
| + | records when appropriate. A '' | ||
| + | files. | ||
| - | < | + | For images on ZFS, '' |
| - | iptables | + | copy-on-write layer. Choose '' |
| + | snapshots, or other image-format features. | ||
| + | |||
| + | See [[https:// | ||
| + | |||
| + | ===== Virtual-machine networking ===== | ||
| + | |||
| + | The VPS network interface is connected to the node through a routed layer-3 | ||
| + | link. Do not add the VPS interface to a bridge managed by libvirt. Two setups | ||
| + | are suitable for connecting virtual machines: | ||
| + | |||
| + | ^ Setup ^ Addresses in the domain ^ Advantages ^ Disadvantages ^ | ||
| + | | libvirt NAT | Private IPv4 and private ULA IPv6 | Works with the ordinary VPS addresses and easily serves several domains. | Inbound services need port forwarding and traffic passes through NAT44 and NAT66. | | ||
| + | | Routed public addresses | A public IPv4 ''/ | ||
| + | |||
| + | NAT is usually simpler for one or a few domains with a small number of exposed | ||
| + | services. Use routed addresses when a domain needs its own public addresses, | ||
| + | all ports, or protocols that are awkward to handle with port forwarding. For | ||
| + | backend-only communication between domains, you can attach another isolated | ||
| + | libvirt network; outbound access still uses one of the setups above. | ||
| + | |||
| + | ==== Public addresses on the VPS with dual-stack NAT ==== | ||
| + | |||
| + | The public IPv4 ''/ | ||
| + | '' | ||
| + | '' | ||
| + | with NAT66. The '' | ||
| + | normally IPv4-only and uses '' | ||
| + | explicit dual-stack network with a distinct IPv4 subnet. NAT66 is enabled with | ||
| + | ''< | ||
| + | [[https:// | ||
| + | details. | ||
| + | |||
| + | The ULA '' | ||
| + | If you later join several private networks through a VPN or routing, generate a | ||
| + | different random ULA ''/ | ||
| + | '' | ||
| + | |||
| + | The essential part of the network definition is: | ||
| + | |||
| + | <code xml> | ||
| + | < | ||
| + | < | ||
| + | <forward mode=' | ||
| + | <nat ipv6=' | ||
| + | </ | ||
| + | <bridge name=' | ||
| + | <ip address=' | ||
| + | <ip family=' | ||
| + | </ | ||
| </ | </ | ||
| - | If not, you can follow our paragraph on [[# | + | Attach the created domain to '' |
| + | '' | ||
| + | stable addresses and gateways in the domain: | ||
| + | <code conf> | ||
| + | auto ens3 | ||
| + | iface ens3 inet static | ||
| + | address 192.168.124.10/ | ||
| + | gateway 192.168.124.1 | ||
| - | ==== Creating and Running a Virtual Machine ==== | + | iface ens3 inet6 static |
| + | address fd5f: | ||
| + | gateway fd5f: | ||
| + | </ | ||
| - | This manual presupposes that you will be using [[https:// | + | A port forward consists of DNAT to the domain address |
| + | corresponding forwarded traffic. For an IPv4 web service, these are the | ||
| + | essential commands: | ||
| - | Creating a new virtual machine consists only of preparing an image disk, creating a symlink for the init script and modifying a simple configuration script. Let’s say that the new virtual machine is called “myvirt.” | + | <code bash> |
| + | iptables -t nat -I PREROUTING 1 -p tcp -d VPS_IPV4 --dport 80 \ | ||
| + | -j DNAT --to-destination 192.168.124.10:80 | ||
| + | iptables -t filter -I FORWARD 1 -p tcp -d 192.168.124.10 --dport 80 \ | ||
| + | -j ACCEPT | ||
| + | </ | ||
| - | Prepare a raw image for myvirt with the required size: | + | Equivalent '' |
| + | however, are not preserved across libvirt network changes. The complete script | ||
| + | below therefore creates the network, a forwarding configuration file, and a | ||
| + | hook that restores the rules when the network starts or reconnects. By default, | ||
| + | it exposes | ||
| + | IPv4 and IPv6. | ||
| - | <code shell> | + | Save it as, for example, '' |
| - | mkdir -p /var/lib/ | + | with one public IPv4 and one address from the VPS IPv6 '' |
| - | qemu-img create | + | VPS: |
| - | chown qemu: | + | |
| - | chmod 0600 /var/lib/ | + | <code bash> |
| + | PUBLIC_IPV4=VPS_IPV4 PUBLIC_IPV6=VPS_IPV6 \ | ||
| + | | ||
| </ | </ | ||
| - | Copy the default configuration file /etc/conf.d/qemu to / | + | Run the script when creating the network and again when changing its input |
| + | values, not after every VPS reboot. The network is set to autostart and its hook | ||
| + | loads the rules when it starts. | ||
| + | |||
| + | <code bash> | ||
| + | # | ||
| + | set -euo pipefail | ||
| + | |||
| + | : " | ||
| + | : " | ||
| + | |||
| + | guest_ipv4=${GUEST_IPV4: | ||
| + | guest_ipv6=${GUEST_IPV6: | ||
| + | host_ipv4=${HOST_IPV4: | ||
| + | host_ipv6=${HOST_IPV6: | ||
| + | connection=qemu:/// | ||
| + | network=dualstack-nat | ||
| + | hook=/etc/libvirt/ | ||
| + | config=/etc/libvirt/ | ||
| + | xml=$(mktemp) | ||
| + | trap 'rm -f " | ||
| + | |||
| + | network_uuid=$(virsh --connect " | ||
| + | uuid_element= | ||
| + | if [[ -n $network_uuid ]]; then | ||
| + | uuid_element=" | ||
| + | fi | ||
| + | cat >" | ||
| + | < | ||
| + | < | ||
| + | $uuid_element | ||
| + | <forward mode=' | ||
| + | <nat ipv6=' | ||
| + | </ | ||
| + | <bridge name=' | ||
| + | <ip address=' | ||
| + | < | ||
| + | <range start=' | ||
| + | </ | ||
| + | </ | ||
| + | <ip family=' | ||
| + | </ | ||
| + | EOF | ||
| + | |||
| + | if virsh --connect " | ||
| + | | grep -Fx " | ||
| + | printf '%s is active. Shut down its attached domains, run ' " | ||
| + | printf 'virsh net-destroy %s, then rerun this script.\n' | ||
| + | exit 1 | ||
| + | fi | ||
| + | |||
| + | install -d -m 0755 /etc/ | ||
| + | install -d -m 0755 / | ||
| + | cat >" | ||
| + | # | ||
| + | set -euo pipefail | ||
| + | |||
| + | PATH=/ | ||
| + | network=${1-} | ||
| + | action=${2-} | ||
| + | config=/ | ||
| + | nat_chain=VPSFREE_KVM_DNAT | ||
| + | filter_chain=VPSFREE_KVM_FWD | ||
| + | |||
| + | [[ $network == dualstack-nat ]] || exit 0 | ||
| + | exec 9>/ | ||
| + | flock 9 | ||
| + | |||
| + | delete_jumps() { | ||
| + | local tool | ||
| + | for tool in iptables ip6tables; do | ||
| + | while " | ||
| + | " | ||
| + | done | ||
| + | while " | ||
| + | " | ||
| + | done | ||
| + | done | ||
| + | } | ||
| + | |||
| + | delete_chains() { | ||
| + | local tool | ||
| + | for tool in iptables ip6tables; do | ||
| + | if " | ||
| + | " | ||
| + | " | ||
| + | fi | ||
| + | if " | ||
| + | " | ||
| + | " | ||
| + | fi | ||
| + | done | ||
| + | } | ||
| + | |||
| + | cleanup() { | ||
| + | delete_jumps | ||
| + | delete_chains | ||
| + | } | ||
| + | |||
| + | valid_ip_address() { | ||
| + | local family=$1 address=$2 | ||
| + | |||
| + | perl -MSocket=AF_INET, | ||
| + | my ($family, $address) = @ARGV; | ||
| + | my $af = $family eq " | ||
| + | exit(defined(inet_pton($af, | ||
| + | ' " | ||
| + | } | ||
| + | |||
| + | if [[ $action == stopped ]]; then | ||
| + | cleanup | ||
| + | exit 0 | ||
| + | fi | ||
| + | [[ $action == started || $action == reconnect ]] || exit 0 | ||
| + | |||
| + | entries=() | ||
| + | while read -r family protocol public_ip public_port guest_ip guest_port extra \ | ||
| + | || [[ -n ${family:-} ]]; do | ||
| + | [[ -n ${family:-} && $family != \#* ]] || continue | ||
| + | [[ $family == ipv4 || $family == ipv6 ]] || { | ||
| + | printf ' | ||
| + | exit 1 | ||
| + | } | ||
| + | [[ $protocol == tcp || $protocol == udp ]] || { | ||
| + | printf ' | ||
| + | exit 1 | ||
| + | } | ||
| + | for address in " | ||
| + | valid_ip_address " | ||
| + | printf ' | ||
| + | " | ||
| + | exit 1 | ||
| + | } | ||
| + | done | ||
| + | [[ $public_port =~ ^[0-9]{1, | ||
| + | printf ' | ||
| + | exit 1 | ||
| + | } | ||
| + | public_port=$((10# | ||
| + | guest_port=$((10# | ||
| + | ((public_port >= 1 && public_port <= 65535)) || exit 1 | ||
| + | ((guest_port >= 1 && guest_port <= 65535)) || exit 1 | ||
| + | [[ -z ${extra:-} ]] || { | ||
| + | printf 'Too many fields in %s\n' " | ||
| + | exit 1 | ||
| + | } | ||
| + | entries+=(" | ||
| + | done <" | ||
| + | |||
| + | cleanup | ||
| + | for tool in iptables ip6tables; do | ||
| + | " | ||
| + | " | ||
| + | done | ||
| + | for entry in " | ||
| + | IFS=' | ||
| + | <<<" | ||
| + | tool=iptables | ||
| + | destination=" | ||
| + | if [[ $family == ipv6 ]]; then | ||
| + | tool=ip6tables | ||
| + | destination=" | ||
| + | fi | ||
| + | " | ||
| + | --dport " | ||
| + | " | ||
| + | --dport " | ||
| + | done | ||
| + | for tool in iptables ip6tables; do | ||
| + | " | ||
| + | " | ||
| + | done | ||
| + | HOOK | ||
| + | chmod 0755 " | ||
| + | |||
| + | cat >" | ||
| + | # FAMILY PROTOCOL PUBLIC_IP PUBLIC_PORT GUEST_IP GUEST_PORT | ||
| + | ipv4 tcp $PUBLIC_IPV4 80 $guest_ipv4 80 | ||
| + | ipv4 tcp $PUBLIC_IPV4 2222 $guest_ipv4 22 | ||
| + | ipv6 tcp $PUBLIC_IPV6 80 $guest_ipv6 80 | ||
| + | ipv6 tcp $PUBLIC_IPV6 2222 $guest_ipv6 22 | ||
| + | EOF | ||
| - | <code shell> | + | virsh --connect " |
| - | cd /etc/conf.d | + | virsh --connect " |
| - | cp qemu qemu.jarvis | + | systemctl restart libvirtd.service |
| - | vi qemu.jarvis | + | if ! virsh --connect " |
| + | | ||
| + | virsh --connect " | ||
| + | fi | ||
| + | " | ||
| + | virsh --connect " | ||
| </ | </ | ||
| - | Most importantly, add the prepared image: | + | Add another port to ''/ |
| + | '' | ||
| + | expose HTTPS on port 8443 over both protocols with: | ||
| - | < | + | < |
| - | disk1_file="/ | + | ipv4 tcp VPS_IPV4 8443 192.168.124.10 443 |
| - | disk1_format=" | + | ipv6 tcp VPS_IPV6 8443 fd5f: |
| </ | </ | ||
| - | You will probably also need to add the installation CD of a distribution that you have already downloaded: | + | The supported protocols are '' |
| + | is rejected before | ||
| + | the file: | ||
| - | < | + | < |
| - | cdrom0_file=" | + | /etc/libvirt/hooks/network.d/ |
| </ | </ | ||
| - | Create a symlink for the init script and run myvirt. | + | Port 2222 keeps the domain' |
| + | Restrict the VPS firewall further according to who should reach each service. | ||
| - | <code shell> | + | ==== Private IPv4 on the VPS with public addresses in the domain ==== |
| - | cd / | + | |
| - | ln -s qemu qemu.myvirt | + | |
| - | rc-service qemu.myvirt start | + | In this setup, the VPS has a private IPv4 ''/ |
| + | ''/ | ||
| + | separate IPv6 ''/ | ||
| + | private VPS IPv4 works on the internal vpsFree.cz network and reaches the | ||
| + | Internet through NAT on our routers; libvirt does not translate the domain' | ||
| + | public addresses. | ||
| + | |||
| + | For IPv4, first ask support to assign a private IPv4 ''/ | ||
| + | |||
| + | < | ||
| + | Under **Networking → Routable addresses**, | ||
| + | **Address**, | ||
| + | The public address is not configured on any interface inside the VPS; vpsAdminOS | ||
| + | routes it to the VPS via the private address. | ||
| + | </ | ||
| + | |||
| + | < | ||
| + | For IPv6, use another ''/ | ||
| + | addresses** and route it through an address in the VPS's ordinary ''/ | ||
| + | you already have a larger routed network such as a ''/ | ||
| + | ''/ | ||
| + | the gateway on the libvirt bridge; for example, '' | ||
| + | '' | ||
| + | </ | ||
| + | |||
| + | IPv4 uses a small transit network only between the VPS and the domain: | ||
| + | |||
| + | < | ||
| + | Internet → public /32 → VPS → 192.168.123.1/ | ||
| + | ↓ | ||
| + | | ||
| </ | </ | ||
| + | In the libvirt XML, the public IPv4 is a separate route through the domain' | ||
| + | transit address. The routed IPv6 ''/ | ||
| + | bridge: | ||
| - | ==== Configuring iptables | + | <code xml> |
| + | < | ||
| + | < | ||
| + | <forward mode=' | ||
| + | <bridge name=' | ||
| + | <ip address=' | ||
| + | <ip family=' | ||
| + | <route family=' | ||
| + | | ||
| + | </ | ||
| + | </ | ||
| - | If you aren’t using any tool to generate iptables rules (like e.g. [[https:// | + | The VPS must forward both protocols: |
| + | |||
| + | <code bash> | ||
| + | sysctl -w net.ipv4.ip_forward=1 | ||
| + | sysctl -w net.ipv6.conf.all.forwarding=1 | ||
| + | </code> | ||
| - | Download the modified rule template with the added masquerade for our bridge to /etc/iptables: | + | The following script creates this network and persists forwarding in |
| + | '' | ||
| + | ''/ | ||
| + | first address of the routed IPv6 ''/ | ||
| - | < | + | < |
| - | rmdir / | + | PUBLIC_IPV4=PUBLIC_IPV4 \ |
| - | wget -O / | + | IPV6_GATEWAY=ROUTED_IPV6_PREFIX::1 \ |
| + | | ||
| </ | </ | ||
| - | Modify | + | As with NAT, run the script when creating the network or changing its |
| + | addresses. Network autostart and the settings in '' | ||
| + | reboots. | ||
| - | < | + | < |
| - | # /etc/conf.d/iptables | + | #!/usr/bin/env bash |
| + | set -euo pipefail | ||
| - | IPTABLES_SAVE="/etc/iptables" | + | : " |
| - | # | + | : "${IPV6_GATEWAY:? |
| - | SAVE_ON_STOP="no" | + | |
| - | IPFORWARD="yes" | + | host_transit_ipv4=${HOST_TRANSIT_IPV4: |
| + | guest_transit_ipv4=${GUEST_TRANSIT_IPV4: | ||
| + | connection=qemu:/// | ||
| + | network=public-routed | ||
| + | forwarding_config=/etc/sysctl.d/ | ||
| + | xml=$(mktemp) | ||
| + | trap 'rm -f "$xml"' EXIT | ||
| + | |||
| + | export LC_ALL=C | ||
| + | network_uuid=$(virsh --connect | ||
| + | uuid_element= | ||
| + | if [[ -n $network_uuid ]]; then | ||
| + | uuid_element=" | ||
| + | fi | ||
| + | cat >" | ||
| + | < | ||
| + | < | ||
| + | $uuid_element | ||
| + | <forward mode=' | ||
| + | <bridge name=' | ||
| + | <ip address=' | ||
| + | <ip family=' | ||
| + | <route family=' | ||
| + | | ||
| + | </ | ||
| + | EOF | ||
| + | |||
| + | if virsh --connect " | ||
| + | | grep -Fx " | ||
| + | printf '%s is active. Shut down its attached domains, run ' " | ||
| + | printf 'virsh net-destroy %s, then rerun this script.\n' | ||
| + | exit 1 | ||
| + | fi | ||
| + | |||
| + | install -d -m 0755 " | ||
| + | cat >" | ||
| + | net.ipv4.ip_forward = 1 | ||
| + | net.ipv6.conf.all.forwarding = 1 | ||
| + | EOF | ||
| + | sysctl --quiet --load " | ||
| + | |||
| + | virsh --connect " | ||
| + | virsh --connect " | ||
| + | virsh --connect " | ||
| + | virsh --connect " | ||
| </ | </ | ||
| - | …and the /etc/conf.d/ip6tables configuration file (IPv6): | + | If the network is already active, the script leaves it unchanged. Shut down its |
| + | attached domains, stop it with '' | ||
| + | script, | ||
| + | UUID. | ||
| - | <code shell> | + | Attach the domain to '' |
| - | # /etc/conf.d/ip6tables | + | following as '' |
| + | '' | ||
| - | IP6TABLES_SAVE=" | + | <code conf> |
| - | SAVE_RESTORE_OPTIONS=" | + | auto ens3 |
| - | SAVE_ON_STOP=" | + | iface ens3 inet static |
| - | IPFORWARD=" | + | address 192.168.123.2/30 |
| + | up ip address add PUBLIC_IPV4/32 dev ens3 | ||
| + | up ip route replace default via 192.168.123.1 src PUBLIC_IPV4 | ||
| + | down ip address del PUBLIC_IPV4/ | ||
| + | |||
| + | iface ens3 inet6 static | ||
| + | address GUEST_IPV6/ | ||
| + | | ||
| </ | </ | ||
| - | Run iptables | + | '' |
| + | firewall. Allow only traffic that the domain should receive, | ||
| + | for both IPv4 and IPv6. The public IPv4 is configured only in the domain. The | ||
| + | IPv6 gateway and domain addresses come from the second, routed ''/ | ||
| + | address from the VPS's ordinary ''/ | ||
| + | protocol uses NAT in this setup. | ||
| + | |||
| + | ===== Installer ISO on NFS ===== | ||
| + | |||
| + | It is safer to copy an installer ISO to a local dataset first. On an older NFSv3 | ||
| + | export without working network locking, QEMU can hang while waiting for a lock | ||
| + | or fail with an error such as '' | ||
| + | '' | ||
| + | |||
| + | For one client and a read-only installer ISO only, you can use '' | ||
| + | |||
| + | <code bash> | ||
| + | # | ||
| + | set -euo pipefail | ||
| + | |||
| + | : " | ||
| + | : " | ||
| + | |||
| + | mountpoint=${ISO_MOUNTPOINT: | ||
| + | install -d -m 0755 " | ||
| + | mount -t nfs -o ro, | ||
| + | " | ||
| - | <code shell> | + | findmnt |
| - | rc-service iptables start | + | |
| - | rc-service ip6tables start | + | |
| - | rc-update add iptables boot | + | |
| - | rc-update add ip6tables boot | + | |
| </ | </ | ||
| - | ==== Contacts ==== | + | '' |
| + | locks. Never use it for a writable or shared virtual-machine disk. Locking is | ||
| + | part of the NFSv4 protocol and this option is not a solution there. See | ||
| + | [[https:// | ||
| - | * [[jakub@jirutka.cz|Jakub Jirůtka]] (on [[irc:// | ||
manuals/vps/kvm.1690815366.txt.gz · Last modified: by Aither
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
