ERlang HEAP overflow-Collection of common programming errors

Even your edit is not tail-recursive:

loop() -> 
  receive
     {sys, Msg} ->
         handle_sys_msg(Msg),
         loop();
     {From, Msg} ->
          Reply = handle_msg(Msg),
          From ! Reply,
          loop();
      _ -> continue 
  end,
  loop().

The order of execution for one function is: receive ... end, loop(). Now, if you get a {sys, _} message, loop/0 will be called from within the receive, transforming the order of execution above into something equivalent to:

 loop() ->
      receive
          loop() ->
               receive
                  ...
               end,
               loop(),
      end,
      loop() ->
         ...

The problem is that if you call loop() from within the receive, the VM still has to store the return point in order to run the loop() in place after the receive.

To make your function tail-recursive, you would need to do either:

loop() -> 
  receive
     {sys, Msg} ->
         handle_sys_msg(Msg);
     {From, Msg} ->
          Reply = handle_msg(Msg),
          From ! Reply;
      _ -> continue 
  end,
  loop().

or

loop() -> 
  receive
     {sys, Msg} ->
         handle_sys_msg(Msg),
         loop();
     {From, Msg} ->
          Reply = handle_msg(Msg),
          From ! Reply,
          loop();
      _ -> loop()
  end.

Where calling loop() really is always the last thing to be done in the function.