ssh to docker container breaks after 45 seconds

debugcn Published at Dev

azzid

I have docker running on a synology nas. The nas has a private network (172.17.0.0/16) where docker places the containers. It also has a public facing interface on my lan, call it 10.11.12.10/24.

                   internet
                      ^
                      |
                      |
                 +----+------+
                 |  pfSense  |
                 +-----+-----+
                       |
                 +-----+-----+
                 |   switch  |
                 +-+-------+-+
                   |       |
                   |       |
               +---+--+  +-+---+
      +------->|  AP  |  | NAS |
      |        +------+  +--+--+
      |                     |
+-----+--+               +--+--------+
| laptop |               | container |
+--------+               +-----------+

It is totally possible to set up port forwarding from the lan interface to a docker container and access ssh on a container that way, but I was feeling lazy so I thought I could just add a route in my pfsense firewall that would redirect all traffic to 172.17.0.0/16 from my lan directly to the nas.

The route seem to work, it is possible to ssh to docker containers directly from my lan. But no such ssh session can live longer than 45 seconds.

➜  ~ date; ssh 172.17.0.2 -l root
Wed Oct 10 21:19:32 CEST 2018
Last login: Wed Oct 10 19:16:07 2018 from 10.11.12.182
root@ubuntu1:~# while true; do date; sleep 5; done
Wed Oct 10 19:19:42 UTC 2018
Wed Oct 10 19:19:47 UTC 2018
Wed Oct 10 19:19:52 UTC 2018
Wed Oct 10 19:19:57 UTC 2018
Wed Oct 10 19:20:02 UTC 2018
Wed Oct 10 19:20:07 UTC 2018
packet_write_wait: Connection to 172.17.0.2 port 22: Broken pipe
➜  ~
➜  ~

The last packets before it stops seen with tcpdump on the laptop looks like this:

16:38:59.072210 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633305, win 5348, options [nop,nop,TS val 561669488 ecr 134368964], length 0                               
16:38:59.103814 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [.], seq 9633305:9634753, ack 5966, win 199, options [nop,nop,TS val 134369004 ecr 561669467], length 1448           
16:38:59.225048 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561669637 ecr 134369004], length 0                               
16:38:59.324726 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134369225 ecr 561669467], length 1448          
16:38:59.324789 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561669736 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:00.035790 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134369666 ecr 561669467], length 1448          
16:39:00.035854 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561670445 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:00.751550 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134370548 ecr 561669467], length 1448          
16:39:00.751642 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561671159 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:02.493382 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134372312 ecr 561669467], length 1448          
16:39:02.493451 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561672900 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:06.179874 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134375840 ecr 561669467], length 1448          
16:39:06.179964 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561676574 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:13.249689 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134382896 ecr 561669467], length 1448          
16:39:13.249741 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561683642 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:27.376570 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134396992 ecr 561669467], length 1448          
16:39:27.376645 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561697743 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0

Corresponding tcpdump on pfsense only see traffic from the laptop as replies does not need to be routed

16:38:59.083374 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9632853, win 5358, options [nop,nop,TS val 561669485 ecr 134368962], length 0                                 
16:38:59.083422 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9632905, win 5361, options [nop,nop,TS val 561669488 ecr 134368962], length 0                                 
16:38:59.083471 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9632957, win 5359, options [nop,nop,TS val 561669488 ecr 134368962], length 0                                 
16:38:59.083510 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633001, win 5358, options [nop,nop,TS val 561669488 ecr 134368963], length 0                                 
16:38:59.083542 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633053, win 5356, options [nop,nop,TS val 561669488 ecr 134368963], length 0                                 
16:38:59.083581 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633105, win 5355, options [nop,nop,TS val 561669488 ecr 134368963], length 0                                 
16:38:59.083680 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633157, win 5353, options [nop,nop,TS val 561669488 ecr 134368963], length 0                                 
16:38:59.083731 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633209, win 5351, options [nop,nop,TS val 561669488 ecr 134368964], length 0                                 
16:38:59.083778 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633261, win 5350, options [nop,nop,TS val 561669488 ecr 134368964], length 0                                 
16:38:59.083824 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633305, win 5348, options [nop,nop,TS val 561669488 ecr 134368964], length 0                                 
16:38:59.228487 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561669637 ecr 134369004], length 0                                 
16:38:59.328679 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561669736 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:00.039683 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561670445 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:00.755280 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561671159 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:02.497019 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561672900 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:06.183600 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561676574 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:13.255414 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561683642 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:27.380228 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561697743 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0

The nas sees the following

16:38:59.061977 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632365:9632409, ack 5966, win 199, options [nop,nop,TS val 134368960 ecr 561669467], length 44            
16:38:59.062027 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9628757, win 5360, options [nop,nop,TS val 561669467 ecr 134368941], length 0                               
16:38:59.062216 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632409:9632453, ack 5966, win 199, options [nop,nop,TS val 134368960 ecr 561669467], length 44              
16:38:59.062437 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632453:9632505, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 52              
16:38:59.062620 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632505:9632549, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 44              
16:38:59.062837 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632549:9632601, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 52              
16:38:59.062984 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632601:9632653, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 52              
16:38:59.063200 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632653:9632697, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 44              
16:38:59.063401 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632697:9632749, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.063605 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632749:9632801, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.063800 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632801:9632853, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.064021 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632853:9632905, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.064220 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632905:9632957, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.064447 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632957:9633001, ack 5966, win 199, options [nop,nop,TS val 134368963 ecr 561669467], length 44              
16:38:59.064698 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633001:9633053, ack 5966, win 199, options [nop,nop,TS val 134368963 ecr 561669467], length 52              
16:38:59.064938 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633053:9633105, ack 5966, win 199, options [nop,nop,TS val 134368963 ecr 561669467], length 52              
16:38:59.065172 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633105:9633157, ack 5966, win 199, options [nop,nop,TS val 134368963 ecr 561669467], length 52              
16:38:59.065422 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633157:9633209, ack 5966, win 199, options [nop,nop,TS val 134368964 ecr 561669467], length 52              
16:38:59.065663 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633209:9633261, ack 5966, win 199, options [nop,nop,TS val 134368964 ecr 561669467], length 52              
16:38:59.065880 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633261:9633305, ack 5966, win 199, options [nop,nop,TS val 134368964 ecr 561669467], length 44              
16:38:59.105416 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [.], seq 9633305:9634753, ack 5966, win 199, options [nop,nop,TS val 134369004 ecr 561669467], length 1448             
16:38:59.326452 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134369225 ecr 561669467], length 1448            
16:38:59.767432 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134369666 ecr 561669467], length 1448            
16:39:00.649466 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134370548 ecr 561669467], length 1448            
16:39:02.413454 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134372312 ecr 561669467], length 1448            
16:39:05.941490 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134375840 ecr 561669467], length 1448            
16:39:12.997468 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134382896 ecr 561669467], length 1448            
16:39:27.093471 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134396992 ecr 561669467], length 1448

I'm not sure I read it right, but the last few packages have the same ack/seq values. Leading me to believe that both nodes see each others packets but are unable to realise that the packets are part of the same session.

Any idea?

azzid

Adding the route to the pfsense makes the traffic flow weird.

Traffic from the laptop bounces over the pfsense, back down to the nas and then to the container. Replies from the container however will go via the nas directly to the laptop on layer 2. No routing needed.

A consequence of that is that the firewall on the nas that only allow related packages get confused and invalidates the relation between the ssh packages and the first ssh connect packet.

My guess is that iptables conntrack on the nas is killing the session.

# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DEFAULT_FORWARD
-N DOCKER
-N DOCKER-ISOLATION
-A FORWARD -j DEFAULT_FORWARD
-A DEFAULT_FORWARD -j DOCKER-ISOLATION
-A DEFAULT_FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DEFAULT_FORWARD -o docker0 -j DOCKER
-A DEFAULT_FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A DEFAULT_FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN

Removing the route from the pfsense and configuring it directly on the laptop instead seem to work a lot better.

How to add route on macOS:

sudo route add 172.17.0.0/16 10.11.12.10

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at2021-07-17

Comments

0 comments

From Dev

Related Related

Article

ssh to docker container breaks after 45 seconds

ssh to docker container breaks after 45 seconds

SSH directly into a docker container

Vagrant ssh to Docker container

SSH to docker container is not working

Vagrant ssh to Docker container

SSH to a docker container

Docker - ssh by container name

docker: SSH access directly into container

Unable to ssh into GCE docker container

SSH reverse tunnel into Docker container

Unable to ssh into GCE docker container

Azure Container: SetPermissions works after few seconds

Docker and systemd - service stopping after 10 seconds

docker start a container after stopping

docker start a container after stopping

eventlet.monkey_patch() breaks ipdb in docker container

Does the GOGS Docker Container run an SSH Daemon?

How to Ssh into docker container that is running inside Vagrant?

Using SSH keys inside docker container

SSH in Docker container causes HTTP 404

How to SSH to docker container in kubernetes cluster?

SSH Agent forwarding inside docker compose container

How to Ssh into docker container that is running inside Vagrant?

How to setup SSH access to a docker container?

Unable to ssh from the Docker host to a container

Docker container doesn't start, showing as 'Exited n seconds ago'

SSH into a docker container from another container on a different host

ssh to Ubuntu Server hangs after 30 seconds of inactivity

Docker, how to remote ssh from command line host to a docker container?

Update Docker container after making changes to image