VMware + LXD + External DHCP Server for Containers

For the last few years I have been running a few services on VMware’s Photon Docker operating system.

Photon OS: https://vmware.github.io/photon/

However, I was not satisfied with the docker container configuration that was developed for large scale consumable service environments. I decided to migrate all of the services that were running in docker containers and a few individual VMs to containers managed by LXD/LXC.

Ubuntu LXD: https://www.ubuntu.com/cloud/lxd

Initial Issues

One of the requirements for this environment is the containers showing up as dependent boxes on the local network and have the IP addresses allocated by the LAN DHCP server. By default, LXD creates a NAT with its own DHCP which is then used for networking the containers. These containers are not reachable from the external network without implementing routing on the LXD host. In more robust environments this would most likely be ideal, but for serving up a few basic services on a local network, I found it a bit overkill.

Network Layout

After replacing the default network on LXD with a bridge, the containers were not able to see LAN or recieve an IP address from the LAN DHCP server. The DHCP server was seeing the request from the containers for IP address and responding but the containers were never recieving. After many hours of frustration, I resolved the issue as followed.

Solution

The environments of the tutorials demonstrating the setup of the bridged configuration were bare-metal systems and did not account for issues arising from installing in a virtualized environment.

Promiscuous mode must be set to “Accept” on the vNIC/vSwitch that the LXD container is connected to in the VMware environment.

vSwitch Settings

From VMwares documentation:

Promiscuous mode is a security policy which can be defined at the virtual switch or portgroup level in vSphere ESX/ESXi. A virtual machine, Service Console or VMkernel network interface in a portgroup which allows use of promiscuous mode can see all network traffic traversing the virtual switch. By default, a guest operating system’s virtual network adapter only receives frames that are meant for it. Placing the guest’s network adapter in promiscuous mode causes it to receive all frames passed on the virtual switch that are allowed under the VLAN policy for the associated portgroup. This can be useful for intrusion detection monitoring or if a sniffer needs to analyze all traffic on the network segment.

After toggling this setting, the containers were able to receive their IP address assignments from the DHCP server and were able to be communicated with from the LAN.

This would not be an ideal set up for a production environment, however, It is a very easy fix for a development set up.