No-NAT IPv4 on LXC

In a previous post I added IPv6 to my Linux containers. This worked fine. Since not all software fully supports IPv6, I wanted my devices to communicate freely over IPv4 as well, which in my case meant having them on the same subnet and not behind NAT (some devices use broadcasts for service discovery).

Again IP-addresses are replaced with RFC5737 and RFC3849 documentation ranges.

Setup bridge on the host

The first thing to do is setup a software bridge, so we can bridge between the containers and the interface on the host. This is the scariest part, so make sure you have physical access (or remote console) to your host so you can fix configuration mistakes.

I’m using Debian so I had to update my /etc/network/interfaces file on the host. I have removed the eth0 entry and in it’s place I added the br0 interface in its stead:

auto br0
iface br0 inet static
	bridge_ports eth0
        bridge_stp on
        bridge_maxwait 0
        bridge_fd 0
	address 203.0.113.42
	netmask 255.255.255.0
	gateway 203.0.113.1

Basically I changed iface eth0 to iface br0 and added the bridge_* lines. This means that 203.0.113.42 is the IP address of my host (previously on eth0) and 203.0.113.1 is the gateway (first hop) of my host.

I’m sure it’s possible to change the network on the running host, but I screwed up somehow and ended up locking myself out from the system, so I rebooted the machine and it came up again. Now eth0 interface doesn’t have an IP address and br0 does. Running ‘brctl show’ yields:

# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.00ff12345678	yes		eth0

Update LXC guest config files

After configuring the bridge (and possibly the LXC), we have to update our guests’ config files. Since they do not use DHCP anymore, we need to configure an IP-address AND the gateway. The lxc.network.link is also changed from virbr0 to br0 since we do not want the NAT-ted interface, but the bridged interface.

The gateways point to the host itself, not to the actual gateway of the host. I did this
to be sure all (outgoing) traffic goes through the firewall as well.

Please stop the virtual host before editing the file.

The relevant network stuff from an LXC config file:

lxc.network.type                        = veth
lxc.network.flags                       = up
lxc.network.hwaddr                      = 00:FF:12:34:56:78
lxc.network.link                        = br0
lxc.network.name                        = eth0
lxc.network.ipv4			= 203.0.113.100/24
lxc.network.ipv4.gateway		= 203.0.113.42
lxc.network.ipv6			= 2001:db8::100/64
lxc.network.ipv6.gateway                = 2001:db8::42

You should now restart your container and test if everything works correctly. You might have to reconfigure services to a new IP-address (I assume it will be in a different network, since you don’t want it to be NAT-ted anymore).

If you have started your containers (in my example four) you should see their interfaces (vethXXXXX) with brctl show:

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0		8000.00ff12345678	yes		eth0
                                                        veth9E7B01
                                                        vethM3V9D0
                                                        vethS404A0
                                                        vethFO4S0G
virbr0          8000.00ff87654321       yes             virbr0-nic

Edit lxc network configuration (probably not needed)

To be honest, I’m not sure this step is actually necessary. I started out with this and it worked, so I left it because I didn’t feel like rebooting a couple of times to test it properly (I want to be sure it comes up properly after a reboot). Anyway, if you tried the rest and it’s not working, you should probably do this step as well.

To show the current virsh config, run virsh net-dumpxml default:

<network>
  <name>default</name>
  <uuid>(some unique ID)</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='(some MAC address)'/>
  <ip address='192.0.2.0' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.0.2.2' end='192.0.2.254'/>
    </dhcp>
  </ip>
  <ip family='ipv6' address='2001:db8::1' prefix='64'>
  </ip>
</network>

Then edit it with virsh net-edit default. I left the NAT stuff, but I removed the DHCP on it and added an ip block. The ones I filled in are the IP-addresses of the host (and thus the gateway for the containers).

<network>
  <name>default</name>
  <uuid>(some unique ID)</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='(some MAC address)'/>
  <ip address='192.0.2.1' netmask='255.255.255.0'>
  </ip>
  <ip family='ipv6' address='2001:db8::1' prefix='64'>
  </ip>
</network>

This will actually put the ipv4 and ipv6 addresses on the virbr0 interface, that’s why I think this step is probably not necessary.