Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Not a Bug
-
Affects Version/s: 22.0.1
-
Fix Version/s: None
-
Component/s: inets
-
Labels:None
Description
Packet counts reported by distribution link are either too low or too high.
For example, the network interface reports ~3kpps TX in total, but send_cnt via erlang:port_info/2 reports 0.4pps (7500x fewer pps).
On the same host, the network interface reports ~3kpps RX, but recv_cnt reports 16kpps (5x more).
Furthermore, if send_cnt is 0.4pps how can send_bytes be 100MB/s ? A TCP packet cannot exceed 64K in Linux, and according to recv_cnt / recv_bytes received packets do not exceed 1.5KB (the Ethernet MTU site), but sent packets exceed 150MB...
I've attached the relevant metrics screenshots. Will package the entire setup in Docker Compose and share so that you can reproduce locally.
Activity
This doesn't feel like a bug, marking it as such.
Would like to double-check that our understanding is accurate:
1. Sent inet packets are arguments passed to gen_tcp:send/2 or equivalent. inet packets are split into TCP packets by the system kernel. The number reported by send_cnt is the message payload (inet packet) that gen_tcp:send/2 is called with.
2. Received inet packets are data returned by gen_tcp:recv/2. The maximum size of these packets is determined by the inet dist buffer, which defaults to 1460 bytes. This can be configured by setting i.e.:
-kernel inet_dist_connect_options [{buffer,1048576}] -kernel inet_dist_listen_options [{buffer,1048576}]
|
Increasing this setting as per the above configuration yields the following result (attaching screenshot shortly).
Gerhard Lazu
added a comment - This doesn't feel like a bug, marking it as such.
Would like to double-check that our understanding is accurate:
1. Sent inet packets are arguments passed to gen_tcp:send/2 or equivalent. inet packets are split into TCP packets by the system kernel. The number reported by send_cnt is the message payload (inet packet) that gen_tcp:send/2 is called with.
2. Received inet packets are data returned by gen_tcp:recv/2. The maximum size of these packets is determined by the inet dist buffer, which defaults to 1460 bytes. This can be configured by setting i.e.:
-kernel inet_dist_connect_options [{buffer,1048576}] -kernel inet_dist_listen_options [{buffer,1048576}]
Increasing this setting as per the above configuration yields the following result (attaching screenshot shortly). The names of your distributed nodes kind of implies that you are running the Erlang distribution over TLS, is that correct? If you do, then that changes what is considered a packet as it will batch things and optimize in order to minimize the overhead of TLS.
1. Sent inet packets are arguments passed to gen_tcp:send/2 or equivalent. inet packets are split into TCP packets by the system kernel. The number reported by send_cnt is the message payload (inet packet) that gen_tcp:send/2 is called with.
Yes. Though I would like to amend that "inet packets are split *or joined* into TCP packets by the system kernel".
2. Received inet packets are data returned by gen_tcp:recv/2. The maximum size of these packets is determined by the inet dist buffer, which defaults to 1460 bytes.
Yes, if the packet mode is set to 0. If it is not set to 0, then it is the amount of inet packets sent by the other side.
Lukas Larsson
added a comment - The names of your distributed nodes kind of implies that you are running the Erlang distribution over TLS, is that correct? If you do, then that changes what is considered a packet as it will batch things and optimize in order to minimize the overhead of TLS.
1. Sent inet packets are arguments passed to gen_tcp:send/2 or equivalent. inet packets are split into TCP packets by the system kernel. The number reported by send_cnt is the message payload (inet packet) that gen_tcp:send/2 is called with.
Yes. Though I would like to amend that "inet packets are split * or joined * into TCP packets by the system kernel".
2. Received inet packets are data returned by gen_tcp:recv/2. The maximum size of these packets is determined by the inet dist buffer, which defaults to 1460 bytes.
Yes, if the packet mode is set to 0. If it is not set to 0, then it is the amount of inet packets sent by the other side. The reason why you didn't see any distribution traffic is because the nodes sometimes don't auto-cluster. It's a race condition that we were able to reproduce on a different host, the quick fix is to restart everything: docker-compose down && docker-compose-up
Yes, the distribution uses TLS. You can see the relevant TLS processes on the same dashboard, towards the end of the page. If you want to disable TLS, remove all lines in rabbitmq-env.conf and restart the rabbitmq-dist-tls nodes: docker-compose restart rabbitmq-dist-tls{1..3}
So inet packets larger than the TCP window are split into TCP packets by the system kernel, and packets smaller than the TCP window are joined into TCP packets by the system kernel, correct?
I can confirm that when TLS is disabled, the number of inet packets matches on both the send & receive sides. I can also see that the size of inet packets is in the TCP window size ballpark. See attached screenshots (tested on OTP 22.0.2)
Gerhard Lazu
added a comment - - edited The reason why you didn't see any distribution traffic is because the nodes sometimes don't auto-cluster. It's a race condition that we were able to reproduce on a different host, the quick fix is to restart everything: docker-compose down && docker-compose-up
Yes, the distribution uses TLS. You can see the relevant TLS processes on the same dashboard, towards the end of the page. If you want to disable TLS, remove all lines in rabbitmq-env.conf and restart the rabbitmq-dist-tls nodes: docker-compose restart rabbitmq-dist-tls{1..3 }
So inet packets larger than the TCP window are split into TCP packets by the system kernel, and packets smaller than the TCP window are joined into TCP packets by the system kernel, correct?
I can confirm that when TLS is disabled, the number of inet packets matches on both the send & receive sides. I can also see that the size of inet packets is in the TCP window size ballpark. See attached screenshots (tested on OTP 22.0.2) So inet packets larger than the TCP window are split into TCP packets by the system kernel, and packets smaller than the TCP window are joined into TCP packets by the system kernel, correct?
smaller packets *may be* joined.
I'm not really surprised that the packet size that you get from the inet driver varies quite a bit under TLS. The TLS dist uses all kinds of tricks that will vary the packet to bytes ratio in order to be secure and performant at the same time.
Lukas Larsson
added a comment - So inet packets larger than the TCP window are split into TCP packets by the system kernel, and packets smaller than the TCP window are joined into TCP packets by the system kernel, correct?
smaller packets * may be * joined.
I'm not really surprised that the packet size that you get from the inet driver varies quite a bit under TLS. The TLS dist uses all kinds of tricks that will vary the packet to bytes ratio in order to be secure and performant at the same time. Thank you, all captured.
Gerhard Lazu
added a comment - Thank you, all captured.
For extra context, this fits within prometheus.erl - Add a collector for distribution related metrics. Distribution uses TLS, Erlang 22.0.1 runs on Ubuntu 18.04 LTS within Docker with Kernel 4.9.125-linuxkit.