PR-838, which reimplemented various parts of the crypto application to use OpenSSL EVP APIs was merged manually. During the manual merge the macro define for HAVE_RSA_OAEP_MD was lost. The result being that setting rsa_mgf1_md and rsa_oaep_md for any other value than sha causes the nif to return not_sup.
Support for configurable MGF1 digests and OAEP labels is available in OpenSSL version 1.1.0, a cursory look at libressl does not seem to reveal support.
The defective functionality has been returned in a fork by defining HAVE_RSA_OAEP_MD alongside HAVE_CHACHA20_POLY1305, i.e. OpenSSL >= 1.1.0 and NOT libressl.