Dock in the clouds

10.06.2014

Tired of having virtualbox and a thousand docker containers dragging your macbook air down in the mud? Put your docker host in the clouds!

In this post we will create a docker host running in the cloud and hook that up to our local development environment using a SSH based "VPN".

Her is a diagram.

    +---------------+               SSH                +----------------+
    |  development  | tun0         tunnel         tun0 |  docker host   |
    |               | <------------------------------->|                |
    |   (client)    | 10.0.0.1                10.0.0.2 | (cloud_server) |
    +-------+-------+     point to point connection    +-------+--------+
       eth0 |                                                  | eth0
192.168.0.2 |                                                  | 10.0.0.10
            |                    _  _                          | docker0
            |                   ( `   )_                       | 172.17.42.1
            -----------------  (    )    `)  -------------------
                              (_   (_ .  _) _)
                                  INTERNET

The Cloud Server

Pick a cloud, any cloud.

We'll stick with an Ubuntu 12.04 LTS on AWS EC2, but feel free to choose digitalocean or any other supplier and/or linux distribution. Create your VPS (follow provider instructions) and log in.

$ ssh <cloud_server_ip>

Prepare Docker

Follow the appropriate instructions to install docker. By default the docker daemon binds to a unix socket only. To be able to communicate with docker over the network, we need to bind to a network interface.

$ sudo vi /etc/init/docker.conf
    "$DOCKER" -d $DOCKER_OPTS -H unix:///var/run/docker.sock -H tcp://0.0.0.0:4243

WARNING! Binding to 0.0.0.0 will expose the docker host on all network interfaces on the server!! That might not be what you want. If you only want to talk to the docker host over the tunnel, you can bind to 10.0.0.2. Let's keep it exposed while we are testing, but remember to go back and fix that later.

Prepare SSH

To make this work, we need to permit root login over ssh. Now, that might seem like a security hazard. But, we are only going to allow login using keys (not passwords), and later on we are going to limit the access of root over ssh to only run a specific command.

$ sudo vi /etc/ssh/sshd_config
    PermitRootLogin yes
    PermitTunnel yes
    PasswordAuthentication no # <- optional, but recommended!
$ sudo service ssh restart

Friendly tip: When modifying sshd_config and restarting the ssh service; test your new configuration on a different session/terminal, keeping the one you already have open, just in case you messed up something.

We also need to remove any initial scripting under root's autorized_keys added to prevent login as root.

$ sudo vi /root/.ssh/authorized_keys

Remove anything resembling the following:

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10"

...so that your file starts with rsa-ssh <long_key>.

Permit forwarding

$ sudo su -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

The Client

If your on OSX (like me); install tuntap.

Grab the docker client!

$ curl https://get.docker.io/builds/Darwin/x86_64/docker-latest -o docker
$ chmod +x docker
$ sudo cp docker /usr/local/bin/

Thats it!

Connecting

Connection to your docker host is as simple as:

(client) $ sudo ssh -w 0:0 -i key.pem root@<cloud_server_ip>

Passing the -w flag to ssh will have ssh create a tunnel device on each end of the connection. The 0:0 indicates which tunnel device at each end. Since we have specified 0 at each end, both the device on the server and the client will be tun0. Now we need to route some traffic so we can talk to docker on the server.

Tunnels & Routes & Docker

Setup the tunnel devices:

(server) $ sudo ifconfig tun0 10.0.0.2 pointopoint 10.0.0.1
(client) $ sudo ifconfig tun0 10.0.0.1 10.0.0.2
(client) $ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=58.643 ms
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=58.410 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=58.586 ms
^ woohoo!

Setup routing:

(client) $ sudo route -n add -net 10.0.0.0 10.0.0.2
(client) $ sudo route -n add -net 172.0.0.0 10.0.0.2

Set the docker host:

(client) $ export DOCKER_HOST=tcp://172.17.42.1:4243

Et voilà:

(client) $ docker ps
CONTAINER ID        IMAGE                           COMMAND                CREATED             STATUS              PORTS

OHHHH

Addtional setup & security

Now, all this is quite a bit to remember, so I definately recommend scripting parts of the setup. One neat thing you can do that will both simplify the process and strengthen security is to only allow root to run a specific command over ssh, and have that command be opening the tunnel.

First, add the tun0 interface to /etc/network/interfaces

$ sudo vi /etc/network/interfaces
    iface tun0 inet static
       address 10.0.0.2
       pointopoint 10.0.0.1
       netmask 255.255.255.0
       # Forward traffic into server side network
       iptables -t nat -A POSTROUTING --source 10.0.0.2 -j SNAT --to-source 10.0.0.10

Add a command to root's authorized_keys (first thing in the file) to bring up the interface on connection.

$ sudo vi /root/.ssh/authorized_keys
    tunnel="0",command="/sbin/ifdown tun0;/sbin/ifup tun0" ssh-rsa ....

Only allow commands for root over ssh.

$ sudo vi /etc/ssh/sshd_config
    PermitRootLogin forced-commands-only
$ sudo service ssh restart

Now, whenever you ssh in as root the server will try to bring up the tunnel interface.

DNS

If you want simple dns based service discovery for you containers over you new cloud bridge, apply the same tools and techniques discussed in my previous article Vagrant Skydocking to this cloud bridge. You will be pinging redis.staging.yourapp in no time.

Closing thoughts

I have been applying this technique for connecting to different VPCs I manage on amazon. Once scripted it's really nice being able to connect to a specific environment with one command and be hands on the docker host(s) and the containers in that environment.

Originally I had been hoping to also use a VPC for development purposes. Unfortunately the latency from my location to the datacenter gets annoying when trying to get efficient feedback loops. This is however not a fault of this approach, but the distance from me to the datacenter. Your milage may vary. Setting up a development host closer to home did the trick.

Time to dance!

DANCE

Credits

♥ goes out to ssh.
And all the amazing unix hackers out there!
Including the authors of this and that.
Gifs from here.
Thank you internet of folks!