--- title: "proxmox in a container on fedora" description: "violently disregarding all documentation" author: "slonkazoid" created_at: 2024-06-29T02:54:00+03:00 tags: - containers - virtualization --- # proxmox in a container on fedora ![Proxmox-VE running a vm called "test". it's booted into Debian Bookworm with the GNOME desktop environment. There is a terminal with a neofetch output showing it's running Fedora 40, as well as `pidof pvedaemon`, which returned '7987'](/media/proxmox/debian.png) violently disregarding all documentation - [fediverse thread](https://donotsta.re/notice/AjOaiP9SGU5Nbdvfns) ## what i ran proxmox virtual environment on fedora, utilizing systemd-nspawn containers. ## why i wanted to see if i could. ## when today. ## how i took my dying nvme and slapped on a new partition table, created a new luks2 partition and formatted it as btrfs. created a few subvolumes (standard stuff) and installed debian on it from fedora using `debootstrap`. then i went to the proxmox documentation and ignored all of it, then found the [manual installation guide](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm#Install_a_standard_Debian_Bookworm_(amd64)) and read it. i set up the system all nice and well, installed daemons and stuff, and set up the repos. after breaking all the rules, i booted the system in systemd-nspawn using the appropriate tools, then installed proxmox-ve in it. proxmox-ve depends on the proxmox kernel, which i couldn't even use because this was a regular container, not a virtual machine. i rebooted the container. pve-cluster was now failing to start, because /dev/fuse didn't exist. there was a [stale issue on the systemd repo](https://github.com/systemd/systemd/issues/17607) about just this and it happened to contain a patch that 'fixes' it. i swiftly figured out how to patch and build a system package on fedora and compiled systemd several times before i got it to work. i also had to regenerate the patch file cuz it didn't apply with fedora's weird patch applying command. i also had to [DeviceAllow=](https://man.archlinux.org/man/systemd.resource-control.5#Device_Access) it in `systemctl edit systemd-nspawn@<container-name>.service`. i am still not sure if this was required or i could have just bound it like how i did with /dev/kvm below. at this point, i could launch and use proxmox-ve. i uploaded a debian netinst iso and. ``` TASK ERROR: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS. ``` this looks like a scary error but i just had to bind /dev/kvm and DeviceAllow= it as well. after doing this, kvm on the host broke, cuz it was now owned by GID 104 (input), which was group kvm in the container. this normally wouldn't be a problem, but i had to set PrivateUsers=off to work around other permissions. i fixed it by editing /etc/group and changing some GIDs around in the container. while doing all this, i was trying to get [{net,}working](#networking). it was horrible. some insignificants touch-ups here and there and everything was working. well except for dhcp cuz i didn't set that up. ## networking 70% of the time i spent was probably on doing, or trying to do, networking. thanks to my prior experience with systemd-nspawn, i knew the default networking setup was to just expose all interfaces to the container. i wanted all my (non-proxmox) vms and containers in a LAN so this was not the way to go for me. apparently systemd-nspawn@.service actually [uses a virtual ethernet interface and NATs it](https://wiki.archlinux.org/title/systemd-nspawn#Use_a_virtual_Ethernet_link), which inexplicably did not happen on my machine. i checked the systemd-networkd link definitions of nspawn, and they matched with the ArchWiki ones, yet still did not NAT anything. i then tried just using libvirt's default `virbr0` interface as a bridge. it worked perfectly[*](#proxmox-ruins-it-all) first try and resulted in the desired effect. but this depended on libvirtd running and i, at first, couldn't figure out how to get that running consistently. after a few hours of trying to configure my own bridge interface, i realized that it wasn't going anywhere so i gave up. not before breaking some stuff and needing to fix it, though. instead i configured it to share the net ns as usual. ``` interface activation failed kvm: -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown: network script /var/lib/qemu-server/pve-bridge failed with status 7680 TASK ERROR: start failed: QEMU exited with code 1 ``` i gave up rather quickly and went back to using libvirt's `virbr0`. to fix the [*](#proxmox-ruins-it-all) mentioned previously, i added masquerade rules to /etc/network/interfaces: ``` auto vmbr0 iface vmbr0 inet static address 192.168.55.1/24 gateway 192.168.55.1 bridge-ports none bridge-stp off bridge-fd 0 # epic rules post-up iptables -t nat -A POSTROUTING -s '192.168.55.1/24' -o host0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s '192.168.55.1/24' -o host0 -j MASQUERADE ``` this worked well, but did not have a dhcp server and i was too exhausted from networking to set that up. i also had to forward the port 8006 from the container network into enp6s0. of course i tried DNAT before everything. this worked for a few minutes and then started getting rejecting by libvirt's firewall chain????? i tried using the [Port= option](https://man.archlinux.org/man/systemd.nspawn.5#%5BNETWORK%5D_SECTION_OPTIONS) which was also initially getting caught by firewalld. i disabled firewalld, and it stopped matching the rules, everything was ACCEPT by default and everyone was happy. it didn't work, the exact same way as before. i have no idea why. i ended up going with the worst possible option and making a service that runs `socat` to forward the port, and removed a system service in the process. (which then made my system unbootable and required recovery from my other OS) ### proxmox ruins it all proxmox's web interface lets you create virtual bridge interfaces. these interfaces override the default route with themselves and break container networking (alongside proxmox netowrking). ## moral of story don't use proxmox. ## footnotes i'm working on another post which will hopefully be up at [/posts/png](/posts/png) soon. it's slightly more important than a single-day project so i don't want to publish post as shitty as this one.