Patroni, Docker Swarm, and the Silent Killer: A Deep Dive into MTU Mismatch

Background

I was experimenting with patroni for PostgreSQL HA in multi-node docker swarm environment. One of the issues I used to repeatedly observe was about the loadbalancer (postgres-service, which was HAProxy bundled with patroni) repeatedly crashing after I performed a switchover, promoting a secondary node to primary. This suggested the new primary (patroni leader) wasn't being detected as healthy.

Debugging

My initial line of thought was it is due to corrupted data directory for PostgreSQL, I went ahead and delete the docker volumes in all nodes in the multi-node cluster and restored the backup and everything came up fine. 

I got curious! What if I hadn't deleted the docker volumes? What could be the reason for postgres-service crashing?

I had a suspicion that PostgreSQL is unable to recover as WAL files sync is taking huge time between the nodes. But how could it be since all the nodes were in the same kvm and associated with same bridge interface. We migrated wal files directory to shared directory using GlusterFS as NFS, problem should get solved automatically. Even with WAL files on the shared GlusterFS volume, the database failed to recover consistently. This was my first indication that the corruption was likely a symptom of a deeper, systemic issue, not the root cause.

I monitored the setup for couple of days but postgres-service was still crashing repeatedly. I noticed a pattern: a switchover to the original primary always succeeded, but any switchover to the configured secondary always resulted in the load balancer crashing. PostgreSQL logs were still opaque, only indicating the node was not healthy. Generally while debugging setup's for crashes and oom etc, I have developed a habit to look at holy grail mother of all logs i.e. syslog and dmesg. I looked at syslog for all the nodes but no clue. It felt like looking at a deadend. I had no clue what to look for and where to look for. I had already spent couple of weeks debugging this issue but all roads lead to deadend. 

I started reading more about Patroni architecture, docker swarm architecture and raft protocol. Suddenly it hit me that I haven't touched even the surface of patroni. I started looking at what all containers patroni was dependent on. Zookeeper!. I shifted my focus to see if zookeeper. Health checks confirmed it was up, but was the communication reliable? My attempts to manually re-register postgres nodes in ZooKeeper were dead ends, showing the client-server handshake was likely not the problem. And post re-registering,  postgres-service came up fine in configured secondary i.e. operational primary node. Again tried switchover to configured primary and then again switchover to configured secondary. Still same issue postgres-service unable to come up.

Here is what the patroni and zooKeeper logs looked like on the node that was failing to become the primary:

Simulated Patroni Agent Log (High loop_wait and retry_timeout):

The agent was failing to renew its leader lock, leading to timeouts:

Code snippet
2025-06-21 16:45:01,875 ERROR: Failed to update leader lock
Traceback (most recent call last):
  File "patroni/dcs/zookeeper.py", line 400, in _update_leader_znode
  File "kazoo/client.py", line 1238, in set
kazoo.exceptions.ConnectionLoss: [Errno 110] Connection timed out while talking to ZooKeeper
2025-06-21 16:45:01,875 WARNING: Connection to DCS failed: ConnectionLoss
2025-06-21 16:45:05,875 DEBUG: Sleeping for 5 seconds before trying again (loop_wait)

Manual ZooKeeper Debugging:

I ran basic diagnostics (simulated output showing successful connection, but high latency/timeouts were the real problem):

Bash
# Basic ZK health check
$ echo ruok | nc ZK_IP 2181
imok

# Checking ZK node status (showing a client connection)
$ echo stat | nc ZK_IP 2181 | grep "Mode"
Mode: leader
# Client connections were present, but the UPDATE requests were failing!

The data showed the connection could be established, but sustained communication (like Patroni's large state updates) was failing under the pressure of the retry_timeout.

I was thinking maybe Patroni is having trouble registering the node to Zookeeper itself. If zookeeper ports are open, why was the connection timing out? I tried ping, nc and traceroute to figure out if there is any packet drop or any intermittent connection drop issue.

I tried running the commands both from from host and container. Below is from patroni container to zookeeper container.

Bash
# Test 1a: Netcat to ZooKeeper Port (TCP Connection)
# This confirmed the door was open and ZK was listening.
$ nc -zv ZK_IP 2181
Connection to ZK_IP 2181 port [tcp/*] succeeded!

# Test 1b: Standard Ping to ZooKeeper Host (Small, 64-byte Packet)
# No packet loss. Everything looked fine!
$ ping -c 3 ZK_IP

PING ZK_IP (192.168.1.10) 56(84) bytes of data.
64 bytes from ZK_IP (192.168.1.10): icmp_seq=1 ttl=64 time=0.25 ms
...
--- ZK_IP ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

The data showed the port was open and basic connectivity checks were passing.

I also ran traceroute to map the path, but since the nodes were on the same KVM/bridge, it showed a direct, low-latency connection, offering no clues either:

Bash
# Test 1c: Traceroute
$ traceroute ZK_IP
traceroute to ZK_IP (192.168.1.10), 30 hops max, 60 byte packets
 1  ZK_IP (192.168.1.10)  0.30 ms  0.22 ms  0.24 ms

Rootcause: MTU Discovery 

I finally remembered the key to diagnosing packet size issues: forcing the ping utility to use a large packet that cannot be fragmented the "Don't Fragment" flag (-M do). I tested for a size (1472 bytes, which plus headers equals the full 1500 MTU) that should succeed on a standard network:

Bash
# Test 2: Ping with Don't Fragment (-M do) and Large Packet (-s 1472)
# This exposed the hard limit on the local network stack.
$ ping -c 3 -M do -s 1472 ZK_IP

PING ZK_IP (192.168.1.10) 1472(1500) bytes of data.
ping: local error: Message too long, mtu=1280
ping: local error: Message too long, mtu=1280
ping: local error: Message too long, mtu=1280

--- ZK_IP ping statistics ---
3 packets transmitted, 0 received, 100% packet loss

The ZooKeeper communication failure was the final clue. If basic connection checks worked, but large updates failed, the issue had to be packet size.

I checked the Maximum Transmission Unit (MTU) on the host network interfaces:

Bash
# ip link show output on the other (healthy) node
$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1280 qdisc pfifo_fast state UP mode DORMANT group default qlen 1000
    link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff

# On the problematic node (simulated discovery):
$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DORMANT group default qlen 1000
    link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff

The realization was immediate: the problematic node had its primary interface MTU set to 1500 bytes, while all other nodes and the network assumed 1280 bytes!

The database nodes were in Docker Swarm using overlay networks. Overlay networks add about 50 bytes of encapsulation overhead (e.g., VXLAN) to every packet.

  • Healthy Node: 

  • Problematic Node: 


RCA:
When the Patroni agent on the problematic node (set to 1500 MTU) sent a large state packet, it assumed the path could handle 1500 bytes. However, the packet would inevitably travel to another node that was limited to 1280 MTU. This larger packet was either silently dropped or caused network instability along the path (a behavior common when the "Don't Fragment" flag is set by the network stack). The immediate ping failure with mtu=1280 confirmed that the network path's safe ceiling was 1280, and the 1500-MTU node was trying to send too much. This caused the timeouts seen in the Patroni logs, leading to the failover failure.

Solution:

The solution required a two-step fix across the entire cluster:

  1. Uniform Host MTU: We standardized the MTU of the underlying physical network interfaces on all host nodes to 1280 bytes (or confirmed the network path supported 1280).

  2. Explicit Docker MTU Configuration: We forced Docker Swarm to use the correct overlay MTU (1230) to account for the VXLAN overhead by updating /etc/docker/daemon.json on all nodes.

  3. Explicit Docker mtu declaration for docker_gwbridge and other internal overlay networks
    docker network create --driver=bridge --opt com.docker.network.driver.mtu=1280 --opt com.docker.network.bridge.name=docker_gwbridge docker_gwbridge
    docker network create -d overlay --subnet="169.254.5.0/24" --gateway="169.254.5.1" -o com.docker.network.driver.mtu=1230      --attachable core-net

After restarting the Docker daemon and recreating docker swarm networks and post redeploying the Patroni stack, the problem vanished. The load balancer came up instantly after a switchover, confirming the new primary was being detected as healthy.

Post-Fix Status (Healthy Cluster):

Bash
$ octotales database status
+ Cluster: octo (7515327116225277980) ------+----+-----------+------------------+
| Member    | Host         | Role    | State     | TL | Lag in MB | Tags             |
+-----------+--------------+---------+-----------+----+-----------+------------------+
| postgres1 | 169.254.2.47 | Leader  | running   | 24 |           |                  |
| postgres2 | 169.254.2.58 | Replica | streaming | 24 |         0 |                  |
| postgres3 | 169.254.2.41 | Replica | streaming | 24 |         0 | nofailover: true |
+-----------+--------------+---------+-----------+----+-----------+------------------+

Lesson Learned: When dealing with complex distributed systems like Patroni on Docker Swarm, always suspect the network layer first. A tiny, silent mismatch in the MTU can cause massive application-level failures that masquerade as corruption or mysterious timeouts.

Comments

Post a Comment

Popular posts from this blog

How to write a Git Commit Message?

I recently realised about the importance of git commit messages when one of my co-workers who was looking at my  Lisp Interpreter  told me that my commit messages were not in a format. Until then I had never bothered to look into what makes a great commit message. So what are the ingredients of a great commit message? A common standard format to save soul searching. 1. Style 2. Content 3. Metadata There are seven rules of a great Git commit message. 1. Separate the subject line from the body. 2. Limit the subject line to 50 characters. 3. Capitalize the subject line. 4. Do not end the subject line with a period. 5. Use imperative mood in the subject line. 6. Wrap the body in 72 characters. 7. Use the body to explain what , why and how [1]. I use a trello to remind me the same. https://trello.com/b/mJA6Ceyi/how-to-write-a-git-commit Links: https://git-scm.com/book/en/v2 https://chris.beams.io/posts/git-commit/ [1] Chris Beams. "...
Read more

Function Composition in JavaScript

Composing Functions is a great way to abstract two or more function under the hood. It is a great style to apply functions on your data. It is like the UNIX pipe command which takes the output of one command and applies it as input for the second command. Function Composition of two functions f and g can be defined as `f(g(x))` where it will be evaluated from right to left. x <- Your data g <- Your First function f <- Your Second Function The order of evaluation in this case would be g(x) whose output would act as input to the function f. const compose = f => g => x => f(g(x)) const add = x => x + 5 const multiply = y => y * 10 compose(add)(multiply)(2)  This so compose(add) would return a closure and then we pass another function which will return a closure and then we pass data to be acted on.     
Read more