Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.1k views
in Technique[技术] by (71.8m points)

docker - Swarm mode routing mesh not working, instead is working like host mode by default

Description

Swarm mode routing mesh not working, instead, it is working like using host mode by default.

We were deploying a swarm of 3 masters nodes and 8 worker nodes, each of them in a different instance of a cloud service OpenStack using Terraform and Ansible. The swarm and routing mesh was working perfectly since it stopped working and started working like in a host mode. We didn't change anything nor done any update or deploy new services. We tried to restart the swarm and re-deploy the swarm and all services, but nothing worked, we couldn't make it work in routing mesh mode again. So, we decided to destroy all instances and start from scratch (the issue reported below). We did a clean installation of Ubuntu 18.04 LTS and docker as we did before. Then we set 1 master node and 2 workers (this time manually) and deploy one service, but the swarm is still working like in host mode.

The only way to access the services is by the IP address of the node where it is running, otherwise, there is no answer (time out). We tried to access using the IP of the manager or the other worker instances, but it is not possible to access to the service. That is why we supposed that the swarm is using host mode by default instead of the ingress network and routing mesh.

We also tried with different services like Mongo or Cassandra but the behaviour is the same, the swarm looks like working using host mode. You can only access the service by using the instance IP address where the service is running.

Any ideas to how to bypass the host most and go back to the routing mesh? We want to access to any service only by using the IP address of the manager nodes which are supossed to be in Drain mode.

Steps to reproduce the issue:

  1. [manager] sudo docker swarm init --advertise-addr 158.39.201.14
  2. [worker-0] sudo docker swarm join --token SWMTKN-1-3np0cy0msnfurecckl4863hkftykuqkgeq998s1hix6jsoiarq-758o52hyma iyzv74w3u1yzltt 158.39.201.14:2377
  3. [worker-1] sudo docker swarm join --token SWMTKN-1-3np0cy0msnfurecckl4863hkftykuqkgeq998s1hix6jsoiarq-758o52hyma iyzv74w3u1yzltt 158.39.201.14:2377
  4. [manger] sudo docker stack deploy -c docker-compose.yml nh

Describe the results you received:

curl http://[worker-0-ip]:8089/bigdata 200 OK

curl http://[worker-1-ip]:8089/bigdata FAIL TIME OUT

Describe the results you expected:

curl http://[worker-0-ip]:8089/bigdata 200 OK

curl http://[worker-1-ip]:8089/bigdata 200 OK

Additional information you deem important (e.g. issue happens only occasionally):

This issue was not happening 2 days ago and suddently it started happening. We didn't made any modification nor touch the servers.

docker-compose.yml

version: '3.7'

networks:
  news-hunter:
    name: &network news-hunter

x-network: &network-base
  networks:
    - *network

services:
   blazegraph:
    <<: *network-base
    image: lyrasis/blazegraph:2.1.5
    ports:
      - published: 8089
        target: 8080
    deploy:
      placement:
        constraints:
          - node.role == worker 

IP tables of manager, worker-1 and worker-2 (all are the same): sudo iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-INGRESS  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination

Chain DOCKER-INGRESS (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:8089
ACCEPT     tcp  --  anywhere             anywhere             state RELATED,ESTABLISHED tcp spt:8089
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Manager ports: sudo netstat -tuplen

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      101        46731      14980/systemd-resol
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          17752      865/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      0          17757      865/sshd
tcp6       0      0 :::8089                 :::*                    LISTEN      0          306971     24992/dockerd
tcp6       0      0 :::2377                 :::*                    LISTEN      0          301970     24992/dockerd
tcp6       0      0 :::7946                 :::*                    LISTEN      0          301986     24992/dockerd
udp        0      0 127.0.0.53:53           0.0.0.0:*                           101        46730      14980/systemd-resol
udp        0      0 158.39.201.14:68        0.0.0.0:*                           100        46591      14964/systemd-netwo
udp        0      0 0.0.0.0:4789            0.0.0.0:*                           0          302125     -
udp6       0      0 fe80::f816:3eff:fef:546 :::*                                100        46586      14964/systemd-netwo
udp6       0      0 :::7946                 :::*                                0          301987     24992/dockerd

Worker ports: sudo netstat -tuplen

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      101        44998      15283/systemd-resol
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          15724      1010/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      0          15726      1010/sshd
tcp6       0      0 :::8089                 :::*                    LISTEN      0          300227     25355/dockerd
tcp6       0      0 :::7946                 :::*                    LISTEN      0          283636     25355/dockerd
udp        0      0 0.0.0.0:4789            0.0.0.0:*                           0          285465     -
udp        0      0 127.0.0.53:53           0.0.0.0:*                           101        44997      15283/systemd-resol
udp        0      0 158.39.201.15:68        0.0.0.0:*                           100        233705     15247/systemd-netwo
udp6       0      0 :::7946                 :::*                                0          283637     25355/dockerd
udp6       0      0 fe80::f816:3eff:fee:546 :::*                                100        48229      15247/systemd-netwo

Services running: sudo docker service ls

ID                  NAME                MODE                REPLICAS            IMAGE                      PORTS
m7eha88ff4wm        nh_blazegraph       replicated          1/1                 lyrasis/blazegraph:2.1.5   *:8089->8080/tcp

Stack: sudo docker stack ps nh

ID                  NAME                IMAGE                      NODE                DESIRED STATE       CURRENT STATE         ERROR               PORTS
tqkd9t4i03yt        nh_blazegraph.1     lyrasis/blazegraph:2.1.5   nh-worker-0         Running             Running 3 hours ago

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.16
 Git commit:        369ce74a3c
 Built:             Thu Feb 13 01:27:49 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.16
  Git commit:       369ce74a3c
  Built:            Thu Feb 13 01:26:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: hpcm67vxrmkm1wvlhfqbjevox
  Is Manager: true
  ClusterID: gnl96swlf7o3a976oarvjrazt
  Managers: 1
  Nodes: 3
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 158.39.201.14
  Manager Addresses:
   158.39.201.14:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

This is an indication that the overlay ports for vxlan are being blocked between nodes in the cluster. The ports used by vxlan are:

  • TCP and UDP port 7946 for communication among nodes
  • UDP port 4789 for overlay network traffic

Source: https://docs.docker.com/network/overlay/

The iptables shown indicate this is not being done within the Linux hosts themselves (input and output policies configured to allow by default), so I'd look towards the network policies and system used to run the VMs. E.g. VMware NSX uses these ports and blocked the embedded VMs as a result.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...