How Tos, OpenStack

Docker Networking – Weave

CONNECTING MULTI-HOST DOCKER CONTAINERS

In the Docker Networking post I mentioned few solution for multi-host Docker containers connectivity. In this blog post I’ll describe my hands-on experience with Weave.

Weave solves multi-host Docker containers connectivity by creating vRouter in each host in order to facilitate a connection between containers running on different hosts. The communicating vRouters using both TCP and UDP on port 6783. TCP to communicate topology and UDP, using a proprietary tunnelling, to forward data between the containers. On top of that, Weave installation create also a weave bridge. Each container is connected to that bridge via a veth pair and this bridge is also connected to the vRouter. It looks something like this (with an additional interface I’ll refer at the end of this post):

Weave

The vRouter captures traffic from its bridge-connected interface and forward the capture packets over UDP to the peer vRouter. No vRouter is used for intra-host container communication.

Installing Weave is pretty simple. For the first host

ubuntu@docker-instance1:~$ sudo wget -O /usr/local/bin/weave \
 https://github.com/weaveworks/weave/releases/download/latest_release/weave

followed by

ubuntu@docker-instance1:~$ sudo chmod a+x /usr/local/bin/weave

Here we start to have difference between the two host. On host 1 we’ll simply run weave launch to create the vRouter.

ubuntu@docker-instance1:~$ sudo weave launch

Running netstat provide information on port 6783

ubuntu@docker-instance1:~$ netstat -l | grep 6783
tcp6       0      0 [::]:6783               [::]:*                  LISTEN     
udp6       0      0 [::]:6783               [::]:* 

Our iptables has also been changed

ubuntu@docker-instance1:~$ sudo iptables -L -v | grep 6783
    0     0 ACCEPT     tcp  --  !docker0 docker0  anywhere             172.17.0.2           tcp dpt:6783
25676 1800K ACCEPT     udp  --  !docker0 docker0  anywhere             172.17.0.2           udp dpt:6783

We can see that docker0 has some involvement when traffic being communicated between two weave vRouters. More about that later on.

While on host 2 we’ll have to provide host 1 IP address or Hostname/FQDN if our DNS is running.

ubuntu@docker-instance2:~$ ubuntu@docker-instance2:~$ sudo weave launch 192.168.1.10
Unable to find image 'zettio/weavetools:0.9.0' locally
Pulling repository zettio/weavetools
......
Status: Downloaded newer image for zettio/weavetools:0.9.0
Unable to find image 'zettio/weave:0.9.0' locally
Pulling repository zettio/weave
......
Status: Downloaded newer image for zettio/weave:0.9.0
5e289b7f963031c067a7357b70c5a69edfbaa6c304f84a2f23a75918f259a3e1

Executing the docker ps command shows we have one new container. This the vRouter we have just created.

ubuntu@docker-instance2:~$ sudo docker ps
CONTAINER ID        IMAGE                COMMAND                CREATED             STATUS              PORTS                                            NAMES
5e289b7f9630        zettio/weave:0.9.0   "/home/weave/weaver    42 seconds ago      Up 41 seconds       0.0.0.0:6783->6783/tcp, 0.0.0.0:6783->6783/udp   weave

Executing ifconfig show some additions to our pre-weave interface list

ubuntu@docker-instance2:~$ ifconfig
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99  
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:37 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:64981 (64.9 KB)  TX bytes:3981 (3.9 KB)

eth0      Link encap:Ethernet  HWaddr fa:16:3e:d5:3a:ee  
          inet addr:192.168.1.11  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fed5:3aee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:4470 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2617 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5633075 (5.6 MB)  TX bytes:284682 (284.6 KB)
........

veth0ff1825 Link encap:Ethernet  HWaddr 0a:3c:14:b1:f0:c4  
          inet6 addr: fe80::83c:14ff:feb1:f0c4/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1450  Metric:1
          RX packets:79 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:66927 (66.9 KB)  TX bytes:4559 (4.5 KB)

vethwepl1574 Link encap:Ethernet  HWaddr 76:24:d5:2d:e4:a5  
          inet6 addr: fe80::7424:d5ff:fe2d:e4a5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65535  Metric:1
          RX packets:7 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:578 (578.0 B)  TX bytes:1066 (1.0 KB)

weave     Link encap:Ethernet  HWaddr 7a:77:6c:8d:ef:9a  
          inet6 addr: fe80::7877:6cff:fe8d:ef9a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65535  Metric:1
          RX packets:7 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:480 (480.0 B)  TX bytes:578 (578.0 B)

We now have weave Linux bridge and two new veth pair ends – vethwepl1574 and veth0ff1825. Note that the weave vRouter (container) is not seen while executing ifconfig.

Looking on the available Linux bridges shows vethwepl1574 is attached to weave Linux bridge and
veth0ff1825 was actually created to connect docker0 to the new container – the weave vRouter.

ubuntu@docker-instance2:~$ sudo brctl show
bridge name	bridge id		STP enabled	interfaces
docker0		8000.56847afe9799	no		veth0ff1825	
weave		8000.7a776c8def9a	no		vethwepl1574

Confused? Let’s understand what weave created by using weave status command:

ubuntu@docker-instance2:~$ sudo weave status
weave router 0.9.0
Encryption off
Our name is 7a:77:6c:8d:ef:9a
Sniffing traffic on &{10 65535 ethwe 92:e7:92:a0:2d:f5 up|broadcast|multicast}
MACs:
76:24:d5:2d:e4:a5 -> 7a:77:6c:8d:ef:9a (2015-04-11 05:19:03.674119069 +0000 UTC)
92:e7:92:a0:2d:f5 -> 7a:77:6c:8d:ef:9a (2015-04-11 05:19:03.147495753 +0000 UTC)
7a:77:6c:8d:ef:9a -> 7a:77:6c:8d:ef:9a (2015-04-11 05:19:03.49414463 +0000 UTC)
Peers:
Peer 7a:77:6c:8d:ef:9a (v2) (UID 12042608006521930194)
   -> 7a:9a:89:e2:9d:76 [192.168.1.10:6783]
Peer 7a:9a:89:e2:9d:76 (v2) (UID 6230488938819459894)
   -> 7a:77:6c:8d:ef:9a [192.168.1.11:43142]
Routes:
unicast:
7a:77:6c:8d:ef:9a -> 00:00:00:00:00:00
7a:9a:89:e2:9d:76 -> 7a:9a:89:e2:9d:76
broadcast:
7a:77:6c:8d:ef:9a -> [7a:9a:89:e2:9d:76]
7a:9a:89:e2:9d:76 -> []
Reconnects:

We can see that the local vRouter 7a:77:6c:8d:ef:9a – peered with the vRouter on host1 – 7a:9a:89:e2:9d:76.The ‘Sniffing traffic’ line shows details of the virtual ethernet interface that weave is using to receive packets on the local machine.
The ‘MACs’ section lists all MAC addresses known to this router. The ‘Peers’ section lists all peers known to this router, including itself.

Before we move on and create weave containers let’s summarize what we have seen so far. After executing weave launch we noticed the following:

  • a new docker container – weave vRouter running on UDP and TCP port 6783
  • a new Linux bridge, also name weave, has been created
  • the weave bridge can communicate with containers created on the same host with vethwepl1574
  • traffic from a container on the host would move to the weave linux bridge and then to the weave vRouter (a docker container) and from there to its peer on the other host using proprietary tunnelling (encapsulation) on UDP port 6783.
  •  docker0 and its related veth pair end veth0ff1825 are not removed by the weave installation. More about that later in the post.

Creating Containers

We’ll create now our first container on host1 (considering the vRouter container hasn’t been created by us) and immediately run ifconfig inside the container.

root@docker-instance1:~# C=$(weave run 10.2.1.1/24 -t -i ubuntu)
root@docker-instance1:~# docker attach $C
root@b58406b2d92f:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:05  
          inet addr:172.17.0.5  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:5/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:648 (648.0 B)  TX bytes:648 (648.0 B)

ethwe     Link encap:Ethernet  HWaddr 9e:a5:11:2a:46:b8  
          inet addr:10.2.1.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::9ca5:11ff:fe2a:46b8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65535  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1296 (1.2 KB)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

We’ll create now our first container on host2 (considering the vRouter container hasn’t been created by us) and immediately run ifconfig inside the container.

root@docker-instance2:~# C=$(weave run 10.2.1.2/24 -t -i ubuntu)
root@docker-instance2:~# docker attach $C
root@49a348728a38:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:03  
          inet addr:172.17.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:3/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:648 (648.0 B)  TX bytes:648 (648.0 B)

ethwe     Link encap:Ethernet  HWaddr c6:55:70:8c:97:e6  
          inet addr:10.2.1.2  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::c455:70ff:fe8c:97e6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65535  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:648 (648.0 B)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The ethwe interface is the one communicating with vethwepxxx in order to pass traffic outside of the container.

Running ping from container on host2 to the one on host1 we can understand better how traffic really flows. First I promised you to check whether docker0 is involved in inter-host traffic and you can see it yourself with the following tcpdump output.

 ubuntu@docker-instance2:~$ sudo tcpdump -i docker0
.......
listening on docker0, link-type EN10MB (Ethernet), capture size 65535 bytes
04:51:59.272819 IP 172.17.0.2.6783 > docker-instance1.6783: UDP, length 118
04:51:59.274112 IP docker-instance1.6783 > 172.17.0.2.6783: UDP, length 118
04:52:00.274697 IP 172.17.0.2.6783 > docker-instance1.6783: UDP, length 118
04:52:00.275708 IP docker-instance1.6783 > 172.17.0.2.6783: UDP, length 118

Now the weave Linux bridge is using the same IP addresses we have first configured to the containers.

ubuntu@docker-instance2:~$ sudo tcpdump -i weave
.....
listening on weave, link-type EN10MB (Ethernet), capture size 65535 bytes
04:57:44.852676 IP 10.2.1.2 > 10.2.1.1: ICMP echo request, id 23, seq 445, length 64
04:57:44.854199 IP 10.2.1.1 > 10.2.1.2: ICMP echo reply, id 23, seq 445, length 64
04:57:44.855210 ARP, Request who-has 10.2.1.2 tell 10.2.1.1, length 28
04:57:44.855226 ARP, Reply 10.2.1.2 is-at c6:55:70:8c:97:e6 (oui Unknown), length 28
04:57:45.854075 IP 10.2.1.2 > 10.2.1.1: ICMP echo request, id 23, seq 446, length 64

It is now interesting to see what the vRouter container does with the ping traffic sent from one container to another. We’ll need to enter inside the vRouter container to find out but it looks like the only command that works is the following:

ubuntu@docker-instance2:~$ docker inspect weave
[{
    "AppArmorProfile": "",
    "Args": [
        "-wait",
        "20",
        "-name",
        "7a:77:6c:8d:ef:9a",
        "-iface",
       "ethwe",
        "192.168.1.10"
    ],
...........
        "ExposedPorts": {
            "6783/tcp": {},
            "6783/udp": {}
        },
        "Hostname": "af2806e04854",
        "Image": "zettio/weave:0.9.0",
        "MacAddress": "",
        "Memory": 0,
        "MemorySwap": 0,
        "NetworkDisabled": false,
        "OnBuild": null,
        "OpenStdin": false,
        "PortSpecs": null,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": null,
        "WorkingDir": "/home/weave"
    },
...........
    "NetworkSettings": {
        "Bridge": "docker0",
        "Gateway": "172.17.42.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.5",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
 .....

Looking at the above we can see that the vRouter using 172.17.0.5 IP address to communicate with docker0. Thus routing traffic from 10.2.1.x/24 network to 172.17.x.x/16 network.

References and further reading:
Weave Features Page
Weave GitHub Page
Weave Troubleshooting
Weave How It Works

Advertisements

2 thoughts on “Docker Networking – Weave

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s