This "Handshake Failure - malformed_handshake_data" error shows up for an Erlang server and a Java client, with Erlang 22.0 and Java 11 or more. Note the handshake does work for different combinations: Erlang 21.3 / Java 11, Erlang 21.3 / Java 8, Erlang 22.0 / Java 8, but NOT for Erlang 22.0 / Java 11.
The client and server should agree on a cipher suite because they have some in common, but it looks like the Erlang server does not understand what the Java client presents.
I provide a script to reproduce with the simplest Erlang server and Java client, and the Java SSL logs for different combinations of Erlang and Java. This requires several JDK installed, SDKMAN! is an easy tool to install JDKs and activate them (pick the "zulu" builds). I focus on Java 8 and Java 11 because they are the current LTS versions.