Resolution: Cannot Reproduce
Affects Version/s: OTP-22.0.2
Fix Version/s: None
I was recently doing some fiddling with scripts and applications and noticed a rather significant performance regression when comparing older Erlang versions (19.3.1 and to a lesser extent 20.3.1) with newer ones (22.214.171.124 and 22.0).
All the measurements in this issue have been measured by running the scripts at https://github.com/ferd/snit/tree/perf-regression (the branch is important) with various versions, on Linux. The script runs on port 8080 and expects localhost to map to 127.0.0.1.
1. Compile the snit application with rebar3 on OTP-19.3.1 (the last one I checked to be fast enough)
2. Run the application by calling rebar3 as demo shell and then booting the test server by calling snit_infinite_stream:run().
3. In another terminal window, run the client script by calling escript tls_script.erl on OTP-19.3.1, and note the delays per 10,000 packets sent after a few seconds
4. Stop the script
5. Change the version of the current terminal window (the one running the client) to OTP-22.0
6. run the client script by calling escript tls_script.erl on OTP-22.0, and note the delays per 10,000 packets sent after a few seconds
The end result is that, according to my experiments, the OTP-22 client is 2x-5x slower than the OTP-19 version. OTP-20.3.1 appeared to be a bit slower than 19, but not significantly so.
For example, in my case, OTP-19.3.1 gives runs ranging from 100ms to 175ms per 10,000 packets sent over the loopback interface, and OTP-22.0 gives me ranges of 380ms to 600ms per 10,000 packets. The effect is more significant when packets are smaller, which to me points towards a cost impacted by packet handling rather than any cryptographic functions, but I haven't dug in too much detail.
I've also attached snapshots I had taken of the msacc values over 1 second that appeared representative of each. The more detailed view (with the dirty scheduler) are over OTP-22. You can particularly see port usage eating up stuff from the emulator.
I would usually not worry too much about small regressions, but 200%-500% performance decreases are worrisome.
This might also be related to https://bugs.erlang.org/browse/ERL-934 ? It was marked as fixed before 126.96.36.199, but I replicated the performance drop with it as well.