Having no upper limit memory limitation in an Erlang process is problematic for Erlang processes that consume memory based on messages received, in a way where the memory usage is not easily anticipated. Two main areas that people have asked about in the past are:
- A way to limit the heap of an Erlang process
- A way to limit the incoming message queue of an Erlang process
The #2 problem above can be avoided, by adding a queue data structure to the Erlang process, so it is avoidable without modifications to the Erlang VM, and it is difficult to make an incoming message queue limit correct when some messages would want to always be received, ignoring the limit (e.g., system messages). However, the #1 problem requires modification to the Erlang VM to be implemented in a dependable way. An ad-hoc solution without Erlang VM modification would likely require an Erlang process that runs at a high priority with a timer, to check Erlang processes that opt-in to checking, with the potential to be missed during the timer time period (potentially not preventing the death of the Erlang VM due to memory consumption).
In the past (around 2007 supposedly, but I don't have a link for reference) there was a discussion about a max_heap_size for Erlang processes and how it would be enforced. The simplest way that would satisfy the problem would be the best, which should be comparing the current Erlang process heap size to the max_heap_size when the GC attempts to provide more Erlang process heap memory, so the max_heap_size is enforced as an upper-limit on the amount of memory available (so no chance for extra garbage collections due to the higher amount of memory). I have not looked into the source code to see how this would be implemented, so my view may be naive.
The feature should be important for the fault-tolerance goal of Erlang, since the availability of an infinite amount of memory is not realistic. setrlimit is the normal POSIX function used on UNIX for OS processes. The value of max_heap_size can initially be set to infinity so there is no impact on source code that will not use max_heap_size. The source code that does use the max_heap_size would be able to avoid having the Erlang VM crash due to unanticipated memory consumption (from the absence of impossible validation or erroneous source code).