In order to take full advantage of the performance of a shared memory machine that uses Non-Uniform Memory Access (NUMA), you need to minimize communication between NUMA nodes; Threads on one NUMA node should refrain from accessing memory that belongs to another NUMA node unless it is absolutely necessary. In order to do this on a Linux system, there are two primary options:

1.  If you do not have access to the source code of the software that you are running or you don’t have the time or the expertise necessary to modify the source code to improve its performance, the NUMA Control tool (numactrl) may be useful. With numactrl, you can specify which NUMA nodes a particular program should use and which specific CPU cores should be used. Example: Suppose a particular system has 6 CPUs per NUMA node, the command

 numactl --physcpubind=0-5 --membind=0 YourProgram options 

runs the program named YourProgram on CPUs 0-5 and instructs the operating system to allocate memory on NUMA node 0 (this assumes that CPUs 0-5 belong to node 0).

When is this useful?  If YourProgram was not optimized for NUMA machines, it may scale very poorly when more than one NUMA node is used. If, for example, YourProgram performs the same type of computations on individual files (analysis, for example), then you could simultaneously launch YourProgram on NUMA node 0 using CPUs 0-5, on node 1 using CPUs 6-11, and so on until you have used all of the nodes available in the system. Each instance of YourProgram would perform computations on a separate file, of course. The several separate instances of YourProgram running on separate NUMA nodes will be more efficient than one instance of the program that uses all of the nodes simultaneously in an inefficient manner.

2. If you are a developer of the code in question or you possess the source code and sufficient desire and programming skill, you can use the NUMA library (libnuma) to make the code NUMA-aware. Among other things, this allows you to perform memory allocations that are local to specific NUMA nodes. In addition to the man page linked above, there are several guides available online to get you started with libnuma; just do a Google search for “libnuma examples.”  

 

This entry was posted on Thursday, December 6th, 2012 at 1:30 am and is filed under regular update. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.