---
title: "proxmox in a container on fedora"
description: "violently disregarding all documentation"
author: "slonkazoid"
icon: /media/proxmox/debian.png
icon_alt: "Proxmox-VE running a vm called \"test\". it's booted into Debian Bookworm with the GNOME desktop environment. There is a terminal with a neofetch output showing it's running Fedora 40, as well as `pidof pvedaemon`, which returned '7987'"
created_at: 2024-06-29T02:54:00+03:00
tags:
 - containers
 - virtualization
---

# proxmox in a container on fedora

![Proxmox-VE running a vm called "test". it's booted into Debian Bookworm with the GNOME desktop environment. There is a terminal with a neofetch output showing it's running Fedora 40, as well as `pidof pvedaemon`, which returned '7987'](/media/proxmox/debian.png)

violently disregarding all documentation

- [fediverse thread](https://donotsta.re/notice/AjOaiP9SGU5Nbdvfns)

## what

i ran proxmox virtual environment on fedora,
utilizing systemd-nspawn containers.

## why

i wanted to see if i could.

## when

today.

## how

i took my dying nvme and slapped on a new partition table, created a new luks2
partition and formatted it as btrfs. created a few subvolumes (standard stuff)
and installed debian on it from fedora using `debootstrap`. 

then i went to the proxmox documentation and ignored all of it, then found the
[manual installation guide](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm#Install_a_standard_Debian_Bookworm_(amd64))
and read it. i set up the system all nice and well, installed daemons and stuff, and set up the repos.

after breaking all the rules, i booted the system in systemd-nspawn using the
appropriate tools, then installed proxmox-ve in it. proxmox-ve depends on the
proxmox kernel, which i couldn't even use because this was a regular container,
not a virtual machine.

i rebooted the container. pve-cluster was now failing to start, because
/dev/fuse didn't exist. there was a [stale issue on the systemd repo](https://github.com/systemd/systemd/issues/17607)
about just this and it happened to contain a patch that 'fixes' it. i swiftly
figured out how to patch and build a system package on fedora and compiled
systemd several times before i got it to work. i also had to regenerate the
patch file cuz it didn't apply with fedora's weird patch applying command.  
i also had to [DeviceAllow=](https://man.archlinux.org/man/systemd.resource-control.5#Device_Access)
it in `systemctl edit systemd-nspawn@<container-name>.service`. i am still
not sure if this was required or i could have just bound it like how i did
with /dev/kvm below.

at this point, i could launch and use proxmox-ve. i uploaded a debian netinst
iso and.

```
TASK ERROR: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
```

this looks like a scary error but i just had to bind /dev/kvm and DeviceAllow=
it as well. after doing this, kvm on the host broke, cuz it was now owned by
GID 104 (input), which was group kvm in the container. this normally wouldn't
be a problem, but i had to set PrivateUsers=off to work around other permissions.
i fixed it by editing /etc/group and changing some GIDs around in the container.

while doing all this, i was trying to get [{net,}working](#networking).
it was horrible.

some insignificants touch-ups here and there and everything was working.
well except for dhcp cuz i didn't set that up.

## networking

70% of the time i spent was probably on doing, or trying to do, networking.

thanks to my prior experience with systemd-nspawn, i knew the default networking
setup was to just expose all interfaces to the container. 
i wanted all my (non-proxmox) vms and containers in a LAN so this was not the
way to go for me. apparently systemd-nspawn@.service actually
[uses a virtual ethernet interface and NATs it](https://wiki.archlinux.org/title/systemd-nspawn#Use_a_virtual_Ethernet_link),
which inexplicably did not happen on my machine. i checked the systemd-networkd
link definitions of nspawn, and they matched with the ArchWiki ones, yet still
did not NAT anything.

i then tried just using libvirt's default `virbr0` interface as a bridge. it
worked perfectly[*](#proxmox-ruins-it-all) first try and resulted in the desired
effect. but this depended on libvirtd running and i, at first, couldn't figure
out how to get that running consistently.

after a few hours of trying to configure my own bridge interface, i realized
that it wasn't going anywhere so i gave up. not before breaking some stuff
and needing to fix it, though. instead i configured it to share the net ns as
usual.

```
interface activation failed
kvm: -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown: network script /var/lib/qemu-server/pve-bridge failed with status 7680
TASK ERROR: start failed: QEMU exited with code 1
```

i gave up rather quickly and went back to using libvirt's `virbr0`. to fix the [*](#proxmox-ruins-it-all) mentioned previously,
i added masquerade rules to /etc/network/interfaces:

```
auto vmbr0
iface vmbr0 inet static
        address 192.168.55.1/24
        gateway 192.168.55.1
        bridge-ports none
        bridge-stp off
        bridge-fd 0
		# epic rules
        post-up   iptables -t nat -A POSTROUTING -s '192.168.55.1/24' -o host0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '192.168.55.1/24' -o host0 -j MASQUERADE
```

this worked well, but did not have a dhcp server and
i was too exhausted from networking to set that up.

i also had to forward the port 8006 from the container network into enp6s0. of
course i tried DNAT before everything. this worked for a few minutes and then
started getting rejecting by libvirt's firewall chain????? i tried using the
[Port= option](https://man.archlinux.org/man/systemd.nspawn.5#%5BNETWORK%5D_SECTION_OPTIONS)
which was also initially getting caught by firewalld. i disabled firewalld, and
it stopped matching the rules, everything was ACCEPT by default and everyone
was happy. it didn't work, the exact same way as before. i have no idea why.  
i ended up going with the worst possible option and making a service that
runs `socat` to forward the port, and removed a system service in the process.
(which then made my system unbootable and required recovery from my other OS)

#### UPDATE 2024-06-29T18:46:00+03:00

i fixed the networking. i set up my own bridge interface with masquerade and
dnat instead of socat. here is most of the configuration i had to do for that
to work. oh also, i disabled firewalld.

`/etc/sysctl.conf`:

```toml
net.ipv4.ip_forward=1
```

`/etc/sysconfig/nftables.conf`:

```
table filter {
	chain FORWARD {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname vbr123 oif enp6s0 accept
		oifname vbr123 iif enp6s0 accept
	}
}

table nat {
	chain PREROUTING {
		type nat hook prerouting priority -100;
		iif enp6s0 tcp dport 8006 dnat to 192.168.123.9
	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		iifname vbr123 oif enp6s0 counter masquerade
	}
}
```

`/etc/systemd/network/50-vbr123.netdev`:

```ini
[NetDev]
Name=vbr123
Kind=bridge
```

`/etc/systemd/network/50-vbr123.network`:

```ini
[Match]
Name=vbr123

[Network]
DHCPServer=yes
Address=192.168.123.1/24
ConfigureWithoutCarrier=yes

#[Link]
#MTUBytes=65536

[DHCPServerStaticLease]
MACAddress=f2:71:54:ec:c1:ce
Address=192.168.123.9

#[DHCPServer]
```

`/etc/systemd/system/systemd-nspawn@fucking-around.service.d/override.conf`:

```ini
[Unit]
Requires=sys-devices-virtual-net-vbr123.device
After=sys-devices-virtual-net-vbr123.device

[Service]
DeviceAllow=/dev/fuse rwm
DeviceAllow=/dev/kvm rwm
# + other stuff...
```

#### UPDATE 2024-06-30T02:19:00+03:00

i have created a bridge interface in the container itself, mostly based on
[ArchWiki's advice](https://wiki.archlinux.org/title/Systemd-networkd#Network_bridge_with_DHCP).
i have also registered it in /etc/network/interfaces (using proxmox gui) so
proxmox also recognizes it. it has DHCP now. only thing left to do is DNS.  
here are the related files:

`/etc/systemd/network/50-vmbr0.netdev`:

```ini
[NetDev]
Name=vmbr0
Kind=bridge
MACAddress=f2:71:54:ec:c1:ce
```

`/etc/systemd/network/50-vmbr0.netdev`:

```ini
[Match]
Name=vmbr0

[Network]
DHCP=yes
```

`/etc/network/interfaces`:

```
# ... block of comments

auto lo
iface lo inet loopback

auto vmbr0
iface vmbr0 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*
```

this was honestly way easier to set up than the host's masqueraded bridge
interface. took me a few minutes and it worked perfectly (well, minus DNS)
on the first try, or the first vm boot at least. i will make sure to update
this with the changes as i polish it further.

#### UPDATE 2024-06-30T02:38:00+03:00

it hasnt been half an hour and i got dns working. i just had to configure
resolved to listen on virbr123 and advertise it in the DHCP server.  
here's how i did that:

`/etc/systemd/resolved.conf.d/listen-on-vbr123.conf`:

```ini
[Resolve]
DNSStubListener=yes
DNSStubListenerExtra=192.168.123.1
```

`/etc/systemd/network/50-vbr123.network`:

```
[Match]
Name=vbr123

[Network]
DHCPServer=yes
Address=192.168.123.1/24
ConfigureWithoutCarrier=yes

#[Link]
#MTUBytes=65536

[DHCPServerStaticLease]
MACAddress=f2:71:54:ec:c1:ce
Address=192.168.123.9

[DHCPServer]
EmitDNS=yes
DNS=192.168.123.1
```

### proxmox ruins it all

proxmox's web interface lets you create virtual bridge interfaces.
these interfaces override the default route with themselves and break container
networking (alongside proxmox netowrking).

## moral of story

don't use proxmox.

## footnotes

i'm working on another post which will hopefully be up at [/posts/png](/posts/png)
soon. it's slightly more important than a single-day project so i don't want
to publish post as shitty as this one.