It appears that the TCP flow control is no longer working, when using the SSL module. This started with the 21.2 release and seems to be broken from there on.
In a nutshell, the SSL module will always receive data from the socket, even if it is in passive mode and not receiving and data will queue up inside the TLS connection process from the SSL module (I think it queues up in state#user_data_buffer from ssl_connection.hrl). There seems to be no way to protect against a fast sender right now.
I'm attaching a small test program, which reproduces this behavior. Essentially, it will send 500 MB of data via SSL to another process (on the same node), which has the socket in passive mode and is not calling ssl:recv() at all. With 21.1.4 the send will fail after ~2 MB have been sent with the send timeout. In 21.2 (and newer versions I have tested) the sender will be able to send all 500 MB and memory usage goes above 500 MB, because the data has to be queued somewhere. I can reproduce the problem in passive mode, by not calling ssl:recv(), but also when using active once and waiting between re-activations.
We are tunneling data between several SSL connections and we rely on this behaviour. Depending on network conditions, we may not be able to forward the data as fast as we can receive it, so we have to block the receiving socket, by not calling ssl:recv(). With the new versions we are experiencing server crashes, because the servers run out of memory.
I also have a dump of the state for an affected SSL connection process, which has about 800 MB of data queued. I can provide a download via mail, if it helps. Given that the problem is fairly easy to reproduce, I don't think it is necessary though.
Tested versions without the bug: 21.1.4
Tested versions with the bug: 21.2, 21.3, 21.3.7
Compile flags: -O3
SSL: LibreSSL 2.8.3