Container Networking: Overlay networks
Overlay networks are meant to network containers hosted on different hosts. To create such a network we use the ‘overlay’ driver. The overlay driver is a native driver that helps to create a single layer2 broadcast domain across containers hosted on multiple Docker hosts. The hosts can be connected in the same subnet or may be in different subnets with router(s) inbetween. In the example here, we will create a single layer2 subnet 10.0.0.0/24 across hosts Docker01 and Docker03.
The target network which we will want to build would look like this:
In a docker environment with 2 hosts, each host has 1 container inside:
(Host) Docker01 : 172.16.255.101
Container1 : eth0: 10.0.0.2(Host) Docker03 : 172.16.255.103
Container2 : eth0: 10.0.0.3
When you create an overlay network, Docker will create a namespace for the network on the host. It will then create a bridge device (say br0) and a vxlan interface. When you create a container attached to this network, it will be attached to the bridge. When you then send traffic between the containers on different hosts, the network device on the container sends it to the vxlan device and the bridge br0, down to the host. The host uses the vxlan header to route the packet to the destination node.
Earlier, creating an overlay network was a bit more complicated, requiring a separate Consul and a key-value store. From Docker 17.06, things have gotten much easier. The recommended way to create an overlay network these days is in just 2 steps now:
1. Create a swarm network between the nodes that you want to network together.
2. Create an overlay network on top, and the swarm nodes will automatically discover each other.
Before starting off with creating an overlay network using swarm below, make sure that the following ports are open and reachable on all Docker host nodes:
- TCP port 2377
- TCP and UDP port 7946
- UDP port 4789
So lets get started:
1. Create a swarm manager node. If you are just creating an overlay network, choose any of the Docker host nodes and run the following command to create a swarm and assign this node as the swarm’s Manager.
Docker03:~ $ docker swarm init
….
docker swarm join — token <xxxxxxx> 172.16.255.101:2377
…- Verify the node’s state in the swarm:
Docker01:~ $ docker node lsID HOSTNAME STATUS AVAILABILITY MANAGER STATUSpzwktqplzgee7ozof0krr5dt5 * Docker01 Ready Active Leader
2. Add in the other nodes into the swarm network, by running the following command. They will each join in as a Worker node.
Docker03:~ $ docker swarm join — token <xxxxxx> 172.16.255.101:2377- Before this step, make sure that the Docker host machine have IP reachability to each other. A simple test would be to first ping the host machines to check whether it works.
- Check the nodes in the swam (from Manager node only):
Docker01:~ $ docker node lsID HOSTNAME STATUS AVAILABILITY MANAGER STATUSpzwktqplzgee7ozof0krr5dt5 * Docker01 Ready Active Leaderzdnk7vyhnvlg4dwkc08drctt3 Docker03 Ready Active
- Any node’s status in a swarm can be checked from the Manager node only. From this output, we see that host ‘Docker01’ is the Manager in this swarm, while host ‘Docker03’ is a Worker node in the swarm.
3. Create the overlay network on top of the nodes in the swarm:
Docker01:~ $ docker network create — driver=overlay — attachable my-overlay-network
Docker01:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
41349f735332 bridge bridge local
c753f318e62f docker_gwbridge bridge local
99e6a78287cf host host local
cvh3xki82cci ingress overlay swarm
7obl3n1z5vzk my-overlay-network overlay swarm
8290b71ecedb none null localThat’s it. Your overlay network is up and ready for use.
When you create an overlay network on one node(in a swarm), the other do not automatically discover the presence of the network:
Docker03:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
1b6d5a047710 bridge bridge local
1b091486939e docker_gwbridge bridge local
6a3a75471f3d host host local
cvh3xki82cci ingress overlay swarm
c755b8f4cb4e none null localThe Docker03 host machine will recognize and connect into this network only when it hosts a container that connects into this overlay network.
Docker03:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
1b6d5a047710 bridge bridge local
1b091486939e docker_gwbridge bridge local
6a3a75471f3d host host local
cvh3xki82cci ingress overlay swarm
c755b8f4cb4e none null localDocker03:~ $ docker container run -dit — network my-overlay-network — name container3 nginx:alpine
c21e1b6445fb112f0180e66a226cc16193e85e36d3ed0b0e6657a44c56bbae0aDocker03:~ $ docker container ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c21e1b6445fb nginx:alpine “nginx -g ‘daemon …” 9 seconds ago Up 6 seconds 80/tcp container3Docker03:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
1b6d5a047710 bridge bridge local
1b091486939e docker_gwbridge bridge local
6a3a75471f3d host host local
cvh3xki82cci ingress overlay swarm
7obl3n1z5vzk my-overlay-network overlay swarm
c755b8f4cb4e none null local
Until this point, we have created an overlay network in Docker01 host, a container on Docker03 host that connects into the overlay network which automatically created the overlay network on Docker03. But we havent yet connected a container on Docker01 to the overlay network.
So let us see how the overlay network looks like in Docker01 at this point:
Docker01:~ $ docker network inspect 7obl3n1z5vzk
[
{
“Name”: “my-overlay-network”,
“Id”: “7obl3n1z5vzk9j6zx8xu3irsd”,
“Created”: “0001–01–01T00:00:00Z”,
“Scope”: “swarm”,
“Driver”: “overlay”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: null,
“Config”: []
},
“Internal”: false,
“Attachable”: true,
“Ingress”: false,
“Containers”: null,
“Options”: {
“com.docker.network.driver.overlay.vxlanid_list”: “4097”
},
“Labels”: null
}
]Note the “Containers: {}” in the output. It has not recognized the container on Docker03 connected to this network. This is by desgin of overlay networks. Containers hosted on other hosts will show up as “Peers” and not as “Containers”. Only those locally-hosted containers connected to this network will show up in the “Containers” key.
Checking on Docker03 host, we can verify this:
Docker03:~ $ docker network inspect 7obl3n1z5vzk
[
{
“Name”: “my-overlay-network”,
“Id”: “7obl3n1z5vzk9j6zx8xu3irsd”,
“Created”: “2018–08–05T15:12:11.082630937Z”,
“Scope”: “swarm”,
“Driver”: “overlay”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: null,
“Config”: [
{
“Subnet”: “10.0.0.0/24”,
“Gateway”: “10.0.0.1”
}
]
},
“Internal”: false,
“Attachable”: true,
“Ingress”: false,
“Containers”: {
“c21e1b6445fb112f0180e66a226cc16193e85e36d3ed0b0e6657a44c56bbae0a”:
{
“Name”: “container3”,
“EndpointID”:
“250b1877adfd1383b233f4fe8c6f909cc7f07125367239ff29f1094dcfa9b2f6”,
“MacAddress”: “02:42:0a:00:00:02”,
“IPv4Address”: “10.0.0.2/24”,
“IPv6Address”: “”
}
},
“Options”: {
“com.docker.network.driver.overlay.vxlanid_list”: “4097”
},
“Labels”: {},
“Peers”: [
{
“Name”: “Docker03–54941c69b81a”,
“IP”: “172.16.255.103”
}
]
}
]Let us now go ahead and create a container on Docker01 and see what happens:
Docker01:~ $ docker container run -dit — name container1 — network my-overlay-network nginx:alpine
Docker01:~ $ docker network inspect my-overlay-network
[
{
“Name”: “my-overlay-network”,
“Id”: “7obl3n1z5vzk9j6zx8xu3irsd”,
“Created”: “2018–08–05T15:31:45.957714062Z”,
“Scope”: “swarm”,
“Driver”: “overlay”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: null,
“Config”: [
{
“Subnet”: “10.0.0.0/24”,
“Gateway”: “10.0.0.1”
}
]
},
“Internal”: false,
“Attachable”: true,
“Ingress”: false,
“Containers”: {
“8e2f525943839a6f51882c89311a7c6c87e7bbae25aa4e0425ef7bb3b157cf4d”:
{
“Name”: “container1”,
“EndpointID”: “b89636f23b47784411db31fd5f134328fba0d339e749ce787be67f3795ee66fc”,
“MacAddress”: “02:42:0a:00:00:03”,
“IPv4Address”: “10.0.0.3/24”,
“IPv6Address”: “”
}
},
“Options”: {
“com.docker.network.driver.overlay.vxlanid_list”: “4097”
},
“Labels”: {},
“Peers”: [
{
“Name”: “Docker03–54941c69b81a”,
“IP”: “172.16.255.103”
},
{
“Name”: “Docker01–60cd697a6982”,
“IP”: “172.16.255.101”
}
]
}
]Now that we have a container on host Docker01, the overlay network shows the locally hosted container under the “Containers” key and the others under the “Peers” in the inspection output.
Note: Check (using ifconfig) on the host machine, that it is transparent to this user-defined overlay network. This means, it does not have any adapter created on it for the overlay network.
Now, inspect the “docker_gwbridge” network. You will see 2 containers attached to it (parameters will be similar if run on other hosts).
Docker03:~ $ docker network inspect docker_gwbridge
[
{
“Name”: “docker_gwbridge”,
“Id”:
“1b091486939e1ab652013e8416be32aa6d41d2f74bf8f145bcd24e31d3109c5d”,
“Created”: “2018–08–05T14:51:25.57401118Z”,
“Scope”: “local”,
“Driver”: “bridge”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: null,
“Config”: [
{
“Subnet”: “172.18.0.0/16”,
“Gateway”: “172.18.0.1”
}
]
},
“Internal”: false,
“Attachable”: false,
“Ingress”: false,
“Containers”: {
“c21e1b6445fb112f0180e66a226cc16193e85e36d3ed0b0e6657a44c56bbae0a”:
{
“Name”: “gateway_c21e1b6445fb”,
“EndpointID”:
“8da6ce5b6f510a4cc09b1d653eca6eec8b13094c31b42b2ccb4f80400797873d”,
“MacAddress”: “02:42:ac:12:00:03”,
“IPv4Address”: “172.18.0.3/16”,
“IPv6Address”: “”
},
“ingress-sbox”: {
“Name”: “gateway_ingress-sbox”,
“EndpointID”:
“0d3e2cf0f6d91ac4afc040cd7134e06cfa1a70bd764b2aaa3de0d48640d2e759”,
“MacAddress”: “02:42:ac:12:00:02”,
“IPv4Address”: “172.18.0.2/16”,
“IPv6Address”: “”
}
},
“Options”: {
“com.docker.network.bridge.enable_icc”: “false”,
“com.docker.network.bridge.enable_ip_masquerade”: “true”,
“com.docker.network.bridge.name”: “docker_gwbridge”
},
“Labels”: {}
}
]If you login into the container itself, you will see eth0 connected to the user-defined ‘my-overlay-network’. eth1 will be connected to this docker_gwbridge network with IP=172.18.0.3 (as is also seen in “Containers” key of the network inspection result). The docker_gwbridge has gateway set to 172.18.0.1 on each host.
Between the host nodes, traffic is transported using VxLAN using UDP port 4789. Using vxlan has some advantanges:
1. Since it is IP traffic, it is easy to encrypt using IPSec.
2. An exact overhead of 50bytes.
3. No need for L2 connectivity between hosts. An IP (layer3) connectivity between hosts works as vxlan does the tunneling on top.
4. It does not have the limit of 4096 IDs as in normal vlans.
A packet capture of the packets on the eth0 device on Docker03 shows how the packets look like on the physical network between the hosts:
Happy containerizing !!!
I regularly write about different topics in tech including career guidance, the latest news, upcoming technologies, and much more. This blog was originally posted in my blogs at anirban-mukherjee.com
