Understanding NUMA Nodes and NUMA Topology in Large IBM Power Servers

Introduction

As server architectures continue to grow in scale, modern systems can contain hundreds of processor cores and terabytes of memory. Managing memory efficiently becomes critical to achieving optimal performance. This is where NUMA (Non-Uniform Memory Access) architecture comes into play.

In a NUMA-based system, processors and memory are organised into groups called NUMA nodes. Each NUMA node consists of a set of CPU cores and memory that are physically closer to each other. Accessing local memory within the same NUMA node is significantly faster than accessing memory located in another node. This design helps improve scalability and reduces memory access latency in large enterprise servers such as IBM Power systems.

Understanding NUMA nodes and NUMA topology is essential for system administrators, performance engineers, virtualization architects, and Linux developers, especially when working with large configurations, virtualization technologies, and memory-intensive workloads.

NUMA Node: A group of CPUs and memory that are physically close to each other.

NUMA Topology: The layout/map showing how all NUMA nodes are connected inside a server.

For your PowerVM work, think of NUMA as "keeping the CPU and its memory in the same neighbourhood so they can talk faster." 

NUMA Node = CPU neighbourhood + its local memory.

Best performance occurs when a workload's CPUs and memory stay in the same neighbourhood. When CPUs must fetch data from another neighbourhood, performance drops due to remote memory access.

Step 1: Understanding the Diagram

The diagram shows:

• 256 CPU cores
• 1 TB Memory
• 16 NUMA Nodes
• Each NUMA node contains:
• 16 CPU cores
• 64 GB Memory

NUMA Node Layout

NUMA Node	CPU Cores	Memory
Node 0	0-15	64 GB
Node 1	16-31	64 GB
Node 2	32-47	64 GB
Node 3	48-63	64 GB
Node 4	64-79	64 GB
Node 5	80-95	64 GB
Node 6	96-111	64 GB
Node 7	112-127	64 GB
Node 8	128-143	64 GB
Node 9	144-159	64 GB
Node 10	160-175	64 GB
Node 11	176-191	64 GB
Node 12	192-207	64 GB
Node 13	208-223	64 GB
Node 14	224-239	64 GB
Node 15	240-255	64 GB

Step 2: Physical Interpretation

Imagine the server has 4 processor sockets.

Socket 0

 ├── NUMA 0

 ├── NUMA 1

 ├── NUMA 2

 └── NUMA 3

Socket 1

 ├── NUMA 4

 ├── NUMA 5

 ├── NUMA 6

 └── NUMA 7

Socket 2

 ├── NUMA 8

 ├── NUMA 9

 ├── NUMA 10

 └── NUMA 11

Socket 3

 ├── NUMA 12

 ├── NUMA 13

 ├── NUMA 14

 └── NUMA 15

Each NUMA node has its own local memory.

Step 3: How Workloads Get Mapped

Example 1: Database Workload

Suppose Oracle or PostgreSQL starts.

Linux scheduler may place:

Database Process

     |

     +--> CPUs 0-15

     |

     +--> Memory from Node 0

Result:

CPU ---> Local Memory

Fastest possible access ✅

Example 2: Java Application

Suppose JVM requires:

• 32 cores
• 128 GB memory

Linux may place it across:

Node 4

Node 5

CPUs 64-95

Memory 128 GB

This is still good because nodes 4 and 5 are neighbours.

Example 3: SAP HANA

Large memory workload:

64 Cores

256 GB Memory

Linux may distribute across:

Node 8

Node 9

Node 10

Node 11

Socket 2

Everything remains inside one socket.

Performance is excellent.

Step 4: Bad Mapping Example

Suppose workload threads run here:

CPU Node 0

but memory gets allocated here:

Memory Node 15

Diagrammatically:

CPU Core 2

     |

     |

     +---------------------------+

                                 |

                                 V

                           Memory Node 15

Now every memory access crosses multiple interconnects.

Effects:

• Higher latency
• Lower throughput
• More cache misses
• Poor benchmark numbers

Step 5:  LPM

Before migration:

LPAR

  CPUs -> Node 0

  Memory -> Node 0

Everything local.

CPU 0

  |

  +--> Memory 0

After migration (bad case):

CPU 0

  |

  +-------------------> Memory Node 12

Remote memory access.

Application slows down.

Step 6: Example MaxConfig Workload Placement

Workload A (Kernel Build)

16 Cores

32 GB Memory

Placement:

NUMA Node 0

Workload B (Stress-ng)

32 Cores

64 GB Memory

Placement:

NUMA Node 1

NUMA Node 2

Workload C (Database)

64 Cores

256 GB Memory

Placement:

NUMA Node 4

NUMA Node 5

NUMA Node 6

NUMA Node 7

Workload D (AI/ML)

128 Cores

512 GB Memory

Placement:

NUMA Node 8-15

How Linux Decides

You can view actual placement using:

numactl --hardware

Show NUMA layout.

numastat -p <pid>

Show memory used by each NUMA node.

numactl --show

Show current NUMA policy.

lscpu | grep NUMA

Show NUMA configuration.

In simple terms: NUMA is all about keeping processors and their data in the same neighbourhood so they can communicate faster and work more efficiently

Conclusion

NUMA architecture plays a crucial role in the performance and scalability of modern multi-socket servers. By grouping processors and memory into NUMA nodes, systems can provide faster local memory access and reduce communication overhead across the server. The NUMA topology serves as a roadmap that describes how these nodes are interconnected and how data flows between them.

For large Power systems and MaxConfig environments, understanding NUMA behaviour is essential for workload placement, performance tuning, virtualization, and Live Partition Mobility (LPM). Keeping CPU threads and memory within the same NUMA node whenever possible helps minimise latency and maximise application performance. As server configurations continue to grow in size and complexity, NUMA awareness becomes a key factor in achieving efficient and predictable system performance.