Searching for Linux optimization some time ago I got two cool features: hugepages and ksm.
By default, Linux allocates memory in ‘blocks’ of 4 kilobytes. This ‘blocks’ are called pages. For each page allocated, some space is spent by control data. So, with thousands of pages, you could got a considerably eaten by control data. Nowadays almost programs use more than 4 kilobytes memory, so if the system allocates bigger pages, the overhead will decrease.
That way, Linux can allocate 4 megabytes page size for x86 systems and 2 megabytes page size for x86_64. You can enable it defining into your .config file of kernel source:
Once you have a kernel compiled with this two options enabled, you can allocate a number of hugepages using sysctl tool:
sysctl -w vm.nr_hugepages = 10
This will tell Linux to reserve space to 10 hugepages. In case of x86_64 system, it means 2 megabytes * 10 = 20 megabytes. That pages are allocated contiguously. So, it is recommended you allocate hugepages in the boot time. Hugepages can’t be moved to swap spaec too.
The Kernel Samepage Merging is a mechanism used mainly by hypervisiors to save memory. When enabled, a thread of kernel scans memory search for pages with the same content but with different owner. When it occurs, Linux merges them and map both to the same location. The new common page is marked as copy-on-write too. When a process need write that page, Linux breaks it again in two pages. To use KSM, you kernel have to be compiled with CONFIG_KSM=y. With kernel supporting it, you can enable scans with:
echo 1 > /sys/kernel/mm/ksm/run
Now, your kernel will scan 100 pages each 20 millisecs by default. You can modify this writing into the files pages_to_scan and sleep_millisecs that you can find in /sys/kernel/mm/ksm folder.
For more details, find hugetlbpage.tx and ksm.txt in Documentation folder of your Linux source.