Introduction
When debugging DAX-related issues, engineers often encounter terms like:
PMEM
devdax
fsdax
folio
compound page
page order
MAX_FOLIO_ORDER
vmemmap_shift
This blog explains how these concepts fit together and why a seemingly simple operation such as creating a DAX namespace can trigger complex memory-management behavior inside the kernel.
What is Persistent Memory (PMEM)?
Persistent Memory (PMEM) is storage that retains data across power cycles while being directly addressable like memory.
Traditionally:
| Technology | Fast | Persistent |
|---|---|---|
| DRAM | ✅ Yes | ❌ No |
| SSD/HDD | ❌ No | ✅ Yes |
| PMEM | ✅ Yes | ✅ Yes |
PMEM combines characteristics of both memory and storage.
Examples include:
Intel Optane Persistent Memory
NVDIMM
Future CXL-attached persistent memory devices
Where is PMEM Located?
Think of PMEM as being physically attached to the memory subsystem rather than accessed through traditional storage controllers.
Traditional storage path:
Application
↓
Page Cache
↓
SSD / NVMe
----------------------PMEM path:
Application
↓
CPU Memory Bus
↓
Persistent Memory
On Power systems and x86 systems, PMEM is exposed to Linux as a memory region that can be mapped directly into applications.
You can view PMEM devices using:
ndctl list
or
daxctl list
What is DAX?
DAX stands for Direct Access.
DAX allows applications to bypass the page cache and directly access persistent memory.
Without DAX:
Application
↓
Page Cache
↓
Storage Device
With DAX:
Application
↓
Persistent Memory
This removes:
Extra memory copies
Page cache overhead
Additional latency
Benefits:
Faster databases
Lower latency
Better HPC performance
Reduced memory consumption
Types of DAX
1. Filesystem DAX (fsdax)
Uses a filesystem such as:
XFS
ext4
Example:
PMEM
└── XFS (DAX enabled)
└── File
Applications access files directly from persistent memory.
2. Device DAX (devdax)
Provides direct raw access to the PMEM device.
Example:
/dev/dax0.0
Applications mmap() the region directly.
No filesystem is involved.
Creation example:
ndctl create-namespace \
--mode=devdax
What is a Folio?
A folio is a Linux kernel memory object used to represent one or more physically contiguous pages.
Before folios:
struct page
represented every individual page.
Now:
struct folio
can represent:
1 page
2 pages
512 pages
16384 pages
as a single unit.
Benefits:
Less memory-management overhead
Better performance
Improved page-cache scalability
Understanding Page Order
Linux measures compound page size using order.
Formula:
Size = PAGE_SIZE × 2^order
For ppc64le:
PAGE_SIZE = 64 KB
Examples:
| Order | Size |
|---|---|
| 0 | 64 KB |
| 1 | 128 KB |
| 2 | 256 KB |
| 4 | 1 MB |
| 5 | 2 MB |
| 10 | 64 MB |
| 14 | 1 GB |
For example:
64 KB × 2^14
= 1 GB
Therefore:
Order 14 = 1 GB object
What is a Compound Page?
A compound page is a collection of many base pages treated as a single memory object.
Example:
Compound Page
│
├── Page 0
├── Page 1
├── Page 2
├── ...
└── Page N
Instead of handling thousands of pages individually, Linux manages them as a single object.
This improves:
TLB efficiency
Memory management
Mapping performance
What is a DAX Folio?
A DAX folio is simply a folio backing DAX memory.
Normal memory:
RAM
↓
Folio
DAX memory:
Persistent Memory
↓
ZONE_DEVICE
↓
DAX Folio
A DAX folio represents PMEM pages that can be directly mapped into applications.
How a 1 GB DAX Folio Gets Created
Suppose we create:
ndctl create-namespace \
--mode=devdax \
-a 1073741824
Notice:
1073741824 = 1 GB
The kernel interprets this alignment requirement and calculates:
PAGE_SHIFT = 16
because:
64 KB pages
and
1 GB = 2^30
Therefore:
Order = 30 - 16
= 14
Linux now attempts to create:
Order-14 folio
which represents:
16384 pages
or:
1 GB DAX folio
Why Does MAX_FOLIO_ORDER Matter?
The kernel limits how large a folio can be.
Inside memory management code:
folio_set_order(folio, order);
Linux verifies:
VM_WARN_ON_ONCE(order > MAX_FOLIO_ORDER);
This prevents creation of excessively large folios.
If:
Order = 14
but
MAX_FOLIO_ORDER < 14
then Linux generates:
VM_WARN_ON_ONCE()
indicating the folio exceeds supported limits.
Real Example
User creates:
ndctl create-namespace \
--mode=devdax \
-a 1073741824
Kernel receives:
1 GB aligned PMEM range
Kernel computes:
Order 14 folio
During initialization:
memremap_pages()
↓
memmap_init_zone_device()
↓
memmap_init_compound()
↓
prep_compound_head()
↓
folio_set_order()
Kernel checks:
order > MAX_FOLIO_ORDER
and generates a warning if the architecture does not support such a large folio.
Commands Useful for DAX Investigation
Commands Useful for DAX Investigation
List PMEM devices
ndctl list
Show namespaces
ndctl list -N
Show DAX devices
daxctl list
View PMEM regions
ndctl list -R
Check memory block size
getconf PAGE_SIZE
Check DAX device
ls -l /dev/dax*
View DAX mappings
dmesg | grep dax
View compound page warnings
dmesg | grep -i folio
Key Takeaways
PMEM is persistent memory that combines storage persistence with memory-like access.
DAX allows direct access to PMEM without page cache involvement.
devdax exposes PMEM as a raw memory device.
Folios are Linux memory objects representing one or more pages.
Order defines folio size as
PAGE_SIZE × 2^order.On ppc64le with 64 KB pages, a 1 GB DAX alignment translates to order 14.
The kernel creates a DAX folio to represent that PMEM region.
If the calculated order exceeds MAX_FOLIO_ORDER, Linux triggers a warning during DAX initialization.
Final One-Line Summary DAX device
ls -l /dev/dax*
View DAX mappings
dmesg | grep dax
View compound page warnings
dmesg | grep -i folio
Key Takeaways
PMEM is persistent memory that combines storage persistence with memory-like access.
DAX allows direct access to PMEM without page cache involvement.
devdax exposes PMEM as a raw memory device.
Folios are Linux memory objects representing one or more pages.
Order defines folio size as
PAGE_SIZE × 2^order.On ppc64le with 64 KB pages, a 1 GB DAX alignment translates to order 14.
The kernel creates a DAX folio to represent that PMEM region.
If the calculated order exceeds MAX_FOLIO_ORDER, Linux triggers a warning during DAX initialization.
DAX enables direct access to persistent memory, and a DAX folio is the kernel's representation of that PMEM region; large DAX alignments can create large compound folios whose order may exceed architecture-specific folio limits.
No comments:
Post a Comment