How does a process handle the situation when it finds no memory to use?-Collection of common programming errors

On some demand-paged virtual memory systems, the operating system refuses to allocate anonymous pages (i.e. pages containing data without a filesystem source such as runtime data, program stack etc.) unless there is sufficient swap space to swap out the pages in order to free up physical memory. This strict accounting has the advantage that each process is guaranteed access to as much virtual memory they allocate, but is also means that the amount of virtual memory available is essentially limited by the size of the swap space.

Memory accounting in the Linux kernel attempts to compensate for programs which tend to allocate more memory than they use by tracking the amount of memory actually in use by processes, and overcommits the amount of virtual memory. In other words, the amount of virtual memory allocated by the kernel can exceed the amount of physical memory and swap space combined on the system. In practice this means that memory allocation through malloc() will never fail. While this leads to better utilization of physical memory and swap space, the downside is that when the amount of of memory in use exceeds the amount of physical memory and swap space available, the kernel must somehow free memory resources in order to meet the memory allocation commitment.

The kernel mechanism that is used to reclaim memory to fill the overcommitment is called the out-of-memory-killer (OOM-killer). Typically the mechanism will start killing off memory-hogging “rogue” processes to free up memory for other processes. They behaviour of the OOM-killer and memory accounting algorithm can be adjusted via sysctl settings or /proc/sys/vm. If the vm.panic_on_oom setting is non-zero, the kernel will panic instead when the system runs out of memory.

The heuristic used by the OOM-killer can be modified through the vm.oom_kill_allocating_task setting. By default the OOM-killer will scan through the task list and select a task rogue task utilizing a lot of memory to kill. The OOM-killer can also be configured to kill the task that triggered the out-of-memory condition.

The kernel memory accounting algorithm can be tuned with the vm.overcommit_memory setting. The default is to perform some weak heuristic checks before overcommiting, but the memory accounting algorithm can also be set to a strict mode, in which the virtual address space limit is determined by the value of vm.overcommit_ratio settings according to the following formula:

    virtual memory = (swap + physical memory * (overcommit_ratio / 100))

When strict memory accounting is in use, the kernel will no longer allocate anonymous pages unless it has enough free physical memory or swap space to store the pages. This means it is essential that the system is configured with enough swap space. Unless there is enough physical memory or swap space to satisfy the memory commitment, calls to malloc() may fail. In these cases it is up to the program itself to determine an appropriate course of action. Some might just give up and fail outright, but they could also fall back to a slower, but more memory efficient algorithm to perform the action they would have needed the memory allocation for.