Sunday, November 22, 2026

Reported AI-Assisted Proof of the Riemann Hypothesis

A Mathematician at Anthropic claims that, working with Claude AI, he has proven the Riemann Hypothesis and potentially the Generalized Riemann Hypothesis. The claim is extraordinary and, if verified, could be one of the biggest breakthroughs in mathematics and AI. However,  the result remained unconfirmed and awaited rigorous review by the mathematical community.

The Riemann Hypothesis, proposed in 1859, is one of mathematics' most famous unsolved problems and is closely tied to the distribution of prime numbers. It is also one of the Clay Mathematics Institute's Millennium Prize Problems.

According to the claim:

  • A short 4-page note outlines the proof.

  • The proof is supported by a massive formalization effort in Lean.

  • Over 250,000 pages of AI-generated mathematical material were reportedly condensed into verifiable formal code.

  • The approach is claimed to extend to the Generalized Riemann Hypothesis (GRH) and potentially further variants.

It emphasizes that:

  • The proof has not yet undergone independent verification.

  • Mathematical experts have not validated the result.

  • The broader mathematical community is still reviewing the claim.

  • Similar announcements in mathematics can take months or years to verify.

Who is behind the claim?

The claim comes from Levent Alpoge, described as:

  • A respected number theorist

  • Princeton PhD

  • Harvard Society of Fellows member

  • Research mathematician at Anthropic

Because of his strong academic reputation, the claim is receiving serious attention, although reputation alone does not constitute proof.


Discussion on AI's Role in Mathematics

What happens when AI starts solving major mathematical problems?

The presenter references comments attributed to Terence Tao regarding AI-assisted mathematical discoveries, particularly around the idea that:

  • Mathematics is not only about proofs.

  • It is also about discovery, intuition, and understanding.

  • When AI generates complex proofs, humans may lose part of the intellectual journey that traditionally accompanies mathematical breakthroughs.

The presenter expresses mixed feelings:

  • Amazement at AI's capabilities.

  • Concern that the discovery process may become less human-centered.

  • Recognition that mathematicians are still needed to verify and understand the results.


Key Uncertainties

At the time of the video:

  • No complete proof was publicly reviewed.

  • Only a brief note or remark appeared available.

  • Independent verification had not begun in earnest.

  • It was unclear how long validation would take.

  • There was no official mathematical consensus.


Potential Impact if True

If the proof is ultimately verified:

  1. The Riemann Hypothesis would be resolved after more than 160 years.

  2. The Generalized Riemann Hypothesis might also be settled.

  3. It would represent one of the most important achievements in modern mathematics.

  4. It would likely become the strongest demonstration yet of AI contributing to frontier mathematical research.

  5. It could fundamentally change how mathematicians collaborate with AI systems.

--------------

The Riemann Zeta Function is a mathematical function that encodes information about prime numbers, and the Riemann Hypothesis states that all its non-trivial zeros lie on the line Re(s)=1/2. Proving this would reveal fundamental hidden structure in the distribution of prime numbers and solve one of mathematics' greatest unsolved problems.

What is the Riemann Zeta Function?

The Riemann Zeta Function is a mathematical function that captures hidden patterns in the distribution of prime numbers.

It is defined as:

which was first proved by Euler.


Why do mathematicians care?

At first glance, the zeta function looks unrelated to prime numbers.

But Euler discovered:


This means the zeta function is directly connected to every prime number.

In simple terms:

The zeta function acts like a compressed database containing information about the primes.

Just as a Linux kernel trace can reveal hidden system behavior, the zeta function reveals hidden structure in prime numbers.


What are "zeros" of the zeta function?

A zero is a value of (s) where:




Examples:

f(x)=x-5

has a zero at:

x=5

because

5-5=0

Similarly, mathematicians look for values of (s) where:



Two kinds of zeros

1. Trivial zeros

These are known:

[ -2,-4,-6,-8,.......

No mystery here.


2. Non-trivial zeros

These occur in the complex plane:

[ s = a + bi ]

where:

  • (a) = real part

  • (b) = imaginary part

These are the mysterious ones.

Examples:

[ 0.5 + 14.134725i ]

[ 0.5 + 21.022040i ]

[ 0.5 + 25.010857i ]


What is the Riemann Hypothesis?

In 1859, Bernhard Riemann made the following conjecture:

Every non-trivial zero of the zeta function lies on the vertical line where the real part equals 1/2.

Mathematically:


This line is called the critical line.


Visualization

Complex Plane

Imaginary
    ^
    |
    |
    |
----+----------------->
    |
    |
    |

      Real Axis

0     1/2      1
      |
      |
      |
      |
      |

Riemann claimed:

All non-trivial zeros lie exactly here
            |
            |
            |
            |
      Re(s)=1/2

Why is this important?

Prime numbers appear random:

2
3
5
7
11
13
17
19
23
29
...

Nobody can predict the next prime exactly.

But the Riemann Hypothesis says:

There is much more order in prime numbers than we currently know.

If true:

  • Prime-number estimates become extremely accurate.

  • Many results in number theory become provably correct.

  • Hundreds of mathematical theorems become stronger.


Linux Analogy

Imagine a server running:

top

and CPU usage seems random.

Then somebody discovers:

All CPU spikes follow one hidden scheduler rule.

Suddenly thousands of performance observations make sense.

The Riemann Hypothesis is similar:

Prime numbers look random
        ↓
Zeta function reveals hidden structure
        ↓
All zeros appear on one line
        ↓
Prime number behavior becomes highly predictable

Why is it a Millennium Prize Problem?

The Clay Mathematics Institute listed it among the seven greatest unsolved mathematical problems.

Prize:

$1,000,000

for a valid proof.

Other famous problems include:

  • P vs NP

  • Navier-Stokes Existence and Smoothness

  • Yang-Mills Mass Gap

  • Birch and Swinnerton-Dyer Conjecture


What happens if it is proven?

If the recent claim is verified:

  1. One of the most famous problems in mathematics is solved.

  2. Researchers gain a much deeper understanding of prime numbers.

  3. The proof would likely become one of the most important mathematical achievements of the century.

  4. If AI significantly contributed, it would mark a historic milestone for AI-assisted scientific discovery.

-------

A proof of the Riemann Hypothesis would not immediately break today's cryptography. Quantum computers are the much bigger threat, and PQC (Post-Quantum Cryptography) is designed specifically to address that threat.

How is the Riemann Hypothesis related to cryptography?

Modern public-key cryptography such as:

  • RSA

  • Diffie-Hellman (DH)

  • Elliptic Curve Cryptography (ECC)

relies heavily on prime numbers.

Since the Riemann Hypothesis is about understanding the distribution of primes, people naturally ask:

"If RH is solved, can attackers suddenly factor RSA keys?"

The answer is generally No.

Why not?

The Riemann Hypothesis helps us understand:

  • How many primes exist in a range

  • How primes are distributed

  • Error bounds in prime-counting functions

But it does not provide:

  • A fast factoring algorithm

  • A method to solve discrete logarithms

  • A way to derive private keys from public keys

Think of it this way:

Riemann Hypothesis
    ↓
Better map of the prime-number landscape

RSA attack
    ↓
Need a fast way to find one specific factor

Those are very different problems.


Could it have indirect effects?

Yes.

A proof of RH could:

  • Improve number-theory algorithms

  • Tighten security estimates

  • Improve prime-generation techniques

  • Strengthen mathematical proofs behind cryptographic protocols

But cryptographers generally do not believe that proving RH would suddenly render RSA insecure.

In fact, many cryptographic results are already stated as:

"Assuming RH is true..."

so a proof would make those results stronger, not weaker.


The Real Threat: Quantum Computing

The major threat to cryptography comes from Shor's Algorithm.

A large fault-tolerant quantum computer could:

RSA 2048
ECC P-256
Diffie-Hellman

all become breakable in polynomial time.

Classical Computer

Factoring RSA-2048:

Thousands to millions of years

(estimated)

Quantum Computer with Shor's Algorithm

Potentially:

Hours to days

on a sufficiently powerful quantum machine.

That is why governments and enterprises are preparing now.


Role of PQC (Post-Quantum Cryptography)

PQC is designed to remain secure even if quantum computers become practical.

Instead of relying on factoring primes, PQC relies on different mathematical problems.

Examples

CRYSTALS-Kyber (ML-KEM)

Used for:

Key Exchange

Based on:

Lattice mathematics

NIST standard:

ML-KEM

CRYSTALS-Dilithium (ML-DSA)

Used for:

Digital Signatures

Based on:

Lattice problems

NIST standard:

ML-DSA

SPHINCS+

Used for:

Digital signatures

Based on:

Hash functions

Why PQC Matters

Current security:

Internet
 → TLS
 → RSA/ECC
 → Secure

Future with quantum computers:

Internet
 → TLS
 → RSA/ECC
 → Vulnerable

Future with PQC:

Internet
 → TLS
 → ML-KEM + ML-DSA
 → Quantum resistant

Why organizations are migrating now

The concern is called:

"Harvest Now, Decrypt Later"

An attacker can:

  1. Record encrypted traffic today.

  2. Store it for years.

  3. Wait for quantum computers.

  4. Decrypt historical data.

So even if practical quantum computers arrive 10 years from now, data encrypted today could be at risk.

This is why:

  • NIST

  • IBM

  • Microsoft

  • Google

  • AWS

  • Cloudflare

are already deploying PQC.

Future  releases are increasingly integrating:

  • ML-KEM (Kyber)

  • ML-DSA (Dilithium)

  • Hybrid TLS handshakes

where classical and PQC algorithms run together.


Simple Comparison

TopicImpact on Cryptography
Riemann Hypothesis provenBetter understanding of primes; no known direct break of RSA/ECC
Faster classical factoring algorithm discoveredPotential threat to RSA
Large-scale quantum computer + Shor's algorithmDirect threat to RSA, ECC, DH
Post-Quantum Cryptography (PQC)Defense against quantum attacks

Conclusion

The discussion started with a reported claim that Anthropic researcher Levent Alpoge, working with Claude AI, may have found a proof of the Riemann Hypothesis, one of the most famous unsolved problems in mathematics. However, the claim remains unverified, and the mathematical community would need extensive review and formal validation before accepting it.

From a mathematical perspective:

  • The Riemann Zeta Function is a special function that encodes deep information about the distribution of prime numbers.
  • The Riemann Hypothesis states that all non-trivial zeros of the zeta function lie on the critical line .
  • A proof would be a historic breakthrough and could significantly advance number theory and our understanding of prime numbers.

From a practical engineering and security perspective:

  • A proof of the Riemann Hypothesis would not directly break RSA, ECC, or current cryptographic systems.
  • It would improve theoretical understanding of prime numbers and may strengthen some mathematical results used in cryptography.
  • The real long-term threat to modern cryptography comes from quantum computing, particularly Shor's algorithm, which can efficiently factor large numbers and solve discrete logarithm problems.

This is where Post-Quantum Cryptography (PQC) becomes important:

  • PQC algorithms such as ML-KEM (Kyber) and ML-DSA (Dilithium) are being standardized and deployed to protect against future quantum attacks.
  • Organizations are transitioning now because of the "harvest now, decrypt later" risk, where encrypted data captured today could be decrypted years later by quantum computers.

If the Riemann Hypothesis is eventually proven with AI assistance, it would be remembered primarily as a landmark achievement in mathematics and AI-driven scientific discovery, not as a threat to cryptographic security. The major security challenge remains quantum computing, and Post-Quantum Cryptography is the industry's planned defense against that future risk.

Saturday, November 7, 2026

From Chips to Cloud: The Invisible Pyramid Powering the AI Revolution

An AI query served from Azure, AWS, or Google Cloud starts its journey years earlier inside semiconductor factories across the Netherlands, United States, Japan, South Korea, Taiwan, and Singapore. The modern AI revolution depends on hundreds of companies working together across multiple layers of the technology stack, from lithography machines and wafer fabrication equipment to memory chips, advanced packaging technologies, AI accelerators, servers, and hyperscale cloud data centers.

The semiconductor industry resembles a pyramid where every layer depends on the layer below it. If one layer slows down, the entire AI ecosystem feels the impact.



The Semiconductor Supply Chain Pyramid

                    ┌─────────────────────┐
                    │   AI APPLICATIONS   │
                    │ ChatGPT, Copilot,   │
                    │ Claude, Gemini      │
                    └─────────▲───────────┘

                    ┌─────────────────────┐
                    │ CLOUD PROVIDERS     │
                    │ AWS Azure Google    │
                    │ Meta Oracle xAI     │
                    └─────────▲───────────┘

                    ┌─────────────────────┐
                    │ AI SERVERS          │
                    │ Dell HPE Lenovo     │
                    │ Supermicro IBM      │
                    └─────────▲───────────┘

                    ┌─────────────────────┐
                    │ GPU DESIGNERS       │
                    │ NVIDIA AMD          │
                    │ Broadcom Intel      │
                    └─────────▲───────────┘

                    ┌─────────────────────┐
                    │ PACKAGING & TEST    │
                    │ ASE Amkor TSMC      │
                    │ CoWoS BESI DISCO    │
                    └─────────▲───────────┘

                    ┌─────────────────────┐
                    │ MEMORY MAKERS       │
                    │ Samsung SK hynix    │
                    │ Micron Kioxia       │
                    └─────────▲───────────┘

                    ┌─────────────────────┐
                    │ FOUNDRIES           │
                    │ TSMC Samsung Intel  │
                    └─────────▲───────────┘

                    ┌─────────────────────┐
                    │ EQUIPMENT VENDORS   │
                    │ ASML AMAT LRCX KLA  │
                    └─────────────────────┘

Layer 1: Equipment Vendors - The Foundation of Everything

ASML (Netherlands)

ASML produces the world's most advanced lithography equipment.

Without ASML:

  • No 5nm chips

  • No 3nm chips

  • No 2nm chips

  • No advanced NVIDIA GPUs

The company's EUV (Extreme Ultraviolet) lithography machine is arguably the most complex machine ever built.

An EUV system contains:

  • 100,000+ components

  • Mirrors from Germany

  • Lasers from the United States

  • Precision mechanics from Europe

Cost per machine:

$200M to $400M+

Architect Insight:

Every advanced AI chip begins with ASML.


Applied Materials (USA)

Applied Materials helps build the transistor structures inside the chip.

Major technologies:

  • Deposition

  • Materials Engineering

  • CMP

  • Process integration

Think of ASML printing the chip while Applied Materials creates the materials that make the chip work.


Lam Research (USA)

Critical for:

  • Etching

  • 3D NAND fabrication

  • HBM manufacturing

Without Lam Research:

  • Modern NAND flash would not exist.

  • High-density HBM stacks would be difficult to manufacture.


KLA (USA)

KLA ensures quality.

Responsibilities:

  • Defect detection

  • Wafer inspection

  • Process control

  • Yield optimization

Architect Perspective:

A defect discovered late may waste millions of dollars of production.


Tokyo Electron (Japan)

Provides:

  • Coaters

  • Developers

  • Deposition tools

  • Etch systems

Another critical supplier for advanced fabs.


Layer 2: Foundries - Where Chips Are Born

These companies convert designs into physical silicon.

TSMC (Taiwan)

Produces chips for:

  • NVIDIA

  • AMD

  • Apple

  • Broadcom

  • Qualcomm

Critical technologies:

  • 5nm

  • 3nm

  • 2nm

When NVIDIA announces a new GPU, TSMC manufactures it.


Samsung Foundry (South Korea)

Produces:

  • Memory

  • Logic chips

  • Mobile SoCs

Strong presence in:

  • HBM

  • Advanced packaging

  • Foundry services


Intel Foundry (USA)

Transitioning into a major foundry provider.

Focus areas:

  • Intel 18A

  • Advanced packaging

  • US-based manufacturing


Layer 3: Memory Makers - Feeding the AI Beast

AI cannot run on compute power alone.

Memory has become the bottleneck.

Samsung

Produces:

  • DRAM

  • NAND

  • HBM

Used in:

  • AI accelerators

  • Enterprise storage

  • HPC systems


SK hynix

Current leader in:

HBM3
HBM3E

Supplying memory for:

  • NVIDIA H100

  • NVIDIA B100/B200

  • Future AI accelerators

Many analysts consider HBM the most critical component in AI hardware today.


Micron

Competes in:

  • HBM

  • DDR5

  • Enterprise memory

Strategic supplier to hyperscalers and HPC vendors.


Layer 4: Packaging & Advanced Integration

This is where modern AI systems are fundamentally different.

Twenty years ago:

One Chip = One Package

Today:

GPU + HBM + Interposer
+ Chiplets + Advanced Packaging

TSMC CoWoS

CoWoS enables:

  • GPU integration

  • HBM integration

  • Extremely high bandwidth

Without CoWoS:

  • Modern NVIDIA accelerators cannot exist.


ASE & Amkor

Provide:

  • Packaging

  • Assembly

  • Final testing

They convert bare silicon into production-ready products.


BESI

Provides advanced die bonding.

Critical for:

  • Chiplet architectures

  • High-bandwidth memory assembly


Layer 5: Chip Designers

These companies create the architecture.


NVIDIA

Known for:

  • H100

  • H200

  • B100

  • B200

  • NVLink

AI training leadership.


AMD

Known for:

  • MI300

  • MI350

  • Infinity Fabric

Strong AI and HPC strategy.


Broadcom

Essential for:

  • AI networking

  • Custom AI silicon

Increasingly important in hyperscale infrastructure.


Intel

Provides:

  • CPUs

  • Gaudi accelerators

  • Networking technology


Layer 6: AI Infrastructure Builders

These companies assemble complete systems.

Dell

Builds:

  • AI clusters

  • Enterprise AI systems

  • Storage infrastructure


HPE

Major supplier of:

  • HPC supercomputers

  • AI factories

  • Exascale systems


IBM

Known for:

  • Power Systems

  • Storage

  • AI integration

  • Enterprise AI deployments


Supermicro

Rapidly growing provider of:

  • GPU servers

  • AI racks

  • Liquid-cooled infrastructures


Layer 7: Cloud Providers and AI Factories

The final consumers of the semiconductor pyramid.

Microsoft Azure

Deploys:

  • NVIDIA clusters

  • AI supercomputers

  • Copilot infrastructure


AWS

Largest cloud consumer of GPUs.

Developing:

  • Trainium

  • Inferentia


Google

Develops TPUs while also deploying NVIDIA infrastructure.


Meta

Consumes enormous amounts of:

  • GPUs

  • HBM memory

  • Networking equipment

for foundation models and AI training.


How Dependencies Flow

A simplified dependency chain looks like:

ASML
  ↓
TSMC / Samsung
  ↓
HBM (SK hynix, Samsung, Micron)
  ↓
CoWoS Packaging
  ↓
NVIDIA / AMD GPUs
  ↓
Dell / HPE / IBM Servers
  ↓
Azure / AWS / Google
  ↓
ChatGPT / Copilot / AI Services

Notice how every layer depends on the previous one.

If ASML shipments slow:

Foundry production slows
          ↓
GPU supply drops
          ↓
Cloud expansion slows
          ↓
AI deployment slows

This is why architects look beyond individual products and study the entire ecosystem.


====

NOTE:

Without ASML lithography systems:

  • TSMC cannot build advanced nodes.
  • Samsung cannot fabricate HBM and advanced logic.
  • NVIDIA cannot obtain cutting-edge GPUs.
  • Cloud providers cannot deploy AI clusters.

This dependency chain is what makes the semiconductor ecosystem one of the most interconnected industries in the world.


What Data Center Architects Care About

Modern AI architects evaluate:

Silicon Layer

  • Process node (3nm, 2nm)

  • Yield rates

  • Supply availability

Memory Layer

  • HBM capacity

  • Memory bandwidth

  • HBM supply constraints

Packaging Layer

  • CoWoS availability

  • Chiplet integration

AI Accelerator Layer

  • GPU performance

  • NVLink architecture

  • Power consumption

Infrastructure Layer

  • InfiniBand

  • Ethernet fabrics

  • Storage throughput

Cloud Layer

  • Scalability

  • TCO

  • Energy efficiency


Conclusion

The AI revolution is not powered by a single company. It is powered by an interconnected semiconductor supply chain spanning continents and thousands of specialized technologies. ASML enables advanced manufacturing, Applied Materials and Lam Research enable fabrication, TSMC and Samsung produce silicon, SK hynix and Micron provide memory, ASE and Amkor package devices, NVIDIA and AMD design accelerators, and cloud providers transform all of that innovation into services consumed by billions of users.



Designing the AI Factory: How NVLink, InfiniBand, and PCIe Shape Modern HPC and AI Infrastructure

Artificial Intelligence (AI) and High-Performance Computing (HPC) have entered an era where data movement is just as critical as compute performance. Modern AI models often span multiple GPUs, servers, and even entire data center clusters, making interconnect technology a key architectural decision. Three technologies play a central role in this ecosystem: PCIe, NVLink, and InfiniBand.

While all three are responsible for moving data, they operate at different layers of the infrastructure. PCIe provides the foundational connectivity inside a server, NVLink enables high-speed communication between GPUs within a node, and InfiniBand extends that communication across multiple servers in a cluster. Understanding the strengths and architectural roles of each interconnect is essential for designing scalable, efficient, and cost-effective AI and HPC systems. 

Most people focus on:

  • GPU FLOPS

  • GPU Memory

  • GPU Count

Architects focus on:

  • Communication topology

  • Data movement efficiency

  • Memory access hierarchy

  • Scale-out vs Scale-up designs

  • Network congestion

  • Power efficiency

  • Cost per trained model

The diagram should explain that AI infrastructure has evolved into a hierarchy.


Application Layer
      |
AI Frameworks
(PyTorch, TensorFlow)
      |
GPU Compute Layer
(H100, B100, MI300)
      |
GPU Fabric Layer
(NVLink/NVSwitch)
      |
Server Fabric Layer
(PCIe Gen5/Gen6)
      |
Cluster Fabric Layer
(InfiniBand/Ethernet)
      |
Storage Fabric
(GPFS/Lustre/Ceph)

Modern AI performance depends on every layer.


Section 1: The New AI Data Center Problem

Introduce the challenge.

Ten years ago:

CPU-Centric Computing

Today:

GPU-Centric Computing

Tomorrow:

GPU Fabric-Centric Computing

For training LLMs:

  • GPT

  • Llama

  • Granite

  • DeepSeek

  • Mistral

The model no longer fits in one GPU.

The model may span:

  • 8 GPUs

  • 64 GPUs

  • 512 GPUs

  • Thousands of GPUs

Communication becomes as important as computation.


Section 2: AI Infrastructure Architecture

Show a typical AI node.

             CPU
              |
        PCIe Gen5/6
              |
    --------------------
    |    |    |    |
   GPU  GPU  GPU  GPU
     \   |   |  /
      NVSwitch

Architectural message:

PCIe connects everything.

NVLink accelerates GPU communication.


Section 3: Why PCIe Alone Is No Longer Enough

Example:

A 70B parameter model.

During training:

GPU → GPU
Gradient Exchange
Tensor Sharing
Activation Movement
Checkpoint Sync

PCIe becomes the bottleneck.

Architects ask:

Can GPUs exchange data faster than through the CPU root complex?

This is where NVLink enters.


Section 4: NVLink Creates a GPU Memory Fabric

This is a very important architectural concept.

Instead of saying:

NVLink is faster.

Say:

NVLink transforms individual GPUs into a collective memory-compute fabric.

Illustration:

GPU1 <--> GPU2
 |          |
 |          |
GPU3 <--> GPU4

Benefits:

  • Faster tensor transfer

  • Reduced synchronization delay

  • Efficient model parallelism

  • Lower communication overhead

Architect Insight:

NVLink is not just a link, it is a distributed GPU memory architecture.


Section 5: Why AI Factories Need InfiniBand

Single server scaling eventually hits limits.

When training a trillion-parameter model:

Server 1
Server 2
Server 3
Server 4
...

Now communication must cross servers.

Requirements:

  • Microsecond latency

  • RDMA

  • Congestion control

  • Collective communication acceleration

This is InfiniBand's domain.

Architecture:

NODE A
  |
GPU Mesh
  |
InfiniBand
  |
NODE B
  |
GPU Mesh

Key architectural point:

NVLink = Scale-Up

InfiniBand = Scale-Out


Section 6: Scale-Up vs Scale-Out

This section will attract architects.

Design GoalPreferred Technology
8 GPU ServerNVLink
16 GPU ServerNVLink + NVSwitch
Multi-Rack ClusterInfiniBand
Enterprise AIPCIe + Ethernet
Exascale HPCNVLink + InfiniBand

Architects always evaluate:

  • How large can one node become?

  • How many nodes are required?

  • What is the communication pattern?


Section 7: Emerging AI Data Center Trends

This will make your blog more future-looking.

NVIDIA AI Factories

Components:

  • Blackwell GPUs

  • NVLink Switch

  • Quantum InfiniBand

  • Spectrum-X Ethernet

Goal:

Create a unified GPU fabric.


AMD Instinct Ecosystem

Using:

  • Infinity Fabric

  • ROCm

  • Ultra Ethernet Consortium standards


CXL Integration

Upcoming architectures may use:

  • CXL Memory Pools

  • Shared accelerator memory

  • Memory disaggregation

Future hierarchy:

HBM
 ↓
NVLink Fabric
 ↓
CXL Memory Pool
 ↓
Storage

This is where AI infrastructure is moving.


Section 8: Cost Architecture Perspective

Architects care about TCO.

A useful discussion:

ArchitectureCostComplexityScalability
PCIe OnlyLowLowLimited
NVLinkMediumMediumHigh
InfiniBandHighHighVery High
NVLink + InfiniBandHighestHighestAI Factory Scale

The objective is not maximum bandwidth.

The objective is:

Performance per Dollar and Training Time per Dollar


Section 9: Design Recommendations

Enterprise AI

  • PCIe Gen5

  • 4 to 8 GPUs

  • Ethernet

Suitable for:

  • RAG

  • Fine-tuning

  • Enterprise inference


AI Research Labs

  • NVLink

  • 8 to 16 GPU systems

Suitable for:

  • LLM training

  • Foundation model experimentation


National Labs / Hyperscalers

  • NVLink

  • NVSwitch

  • InfiniBand NDR/XDR

Suitable for:

  • Trillion-parameter models

  • Exascale AI

  • Scientific HPC


Conclusion

A powerful ending would be:

In modern AI systems, GPUs are no longer isolated compute devices. They are nodes within a hierarchy of interconnected fabrics. PCIe provides system connectivity, NVLink builds high-speed GPU memory fabrics, and InfiniBand extends that fabric across thousands of servers. Architects designing next-generation AI factories must think beyond GPU specifications and focus on the data movement architecture that ultimately determines scalability, efficiency, and time-to-insight. 

Hybrid Computing: Why the Future of HPC Is No Longer CPU vs GPU, But CPU + GPU + QPU + DPU

For decades, CPUs were the center of computing. Then GPUs revolutionized AI, simulation, and scientific computing. Today, HPC systems are evolving even further, integrating specialized processors such as Quantum Processing Units (QPUs), Data Processing Units (DPUs), Tensor Processing Units (TPUs), Neural Processing Units (NPUs), and Language Processing Units (LPUs).

This model is known as Hybrid Computing, where different processors collaborate, each handling the tasks they perform best.


What Is Hybrid Computing in HPC?

Hybrid computing refers to combining multiple types of processors within the same system to accelerate complex workloads efficiently.

Instead of forcing a CPU to perform every task, modern systems distribute work intelligently across specialized accelerators:

ProcessorPrimary Role
CPUControl, orchestration, scheduling, preprocessing
GPUMassively parallel computation for AI, simulation, and analytics
QPUQuantum algorithms for optimization and quantum simulations
DPUNetworking, storage, security, and infrastructure offload
TPUTensor operations optimized for AI training and inference
NPUEnergy-efficient AI inference at the edge
LPUUltra-low latency Large Language Model inference

The result is a system where every processor focuses on its strengths rather than competing for the same workload.


The Evolution of HPC

Traditional HPC architecture looked like this:

Application
     |
    CPU
     |
  Memory

Modern HPC evolved to:

Application
     |
 CPU + GPU

The emerging architecture now looks more like:

CPU + GPU + DPU + QPU + Specialized Accelerators

This transformation is driven by growing demands from:

  • Large Language Models (LLMs)

  • Scientific simulations

  • Climate modeling

  • Drug discovery

  • Financial risk analysis

  • Genomics

  • Material science

  • Digital twins

Each of these domains benefits from a different type of accelerator.


Hybrid Quantum-Classical Computing

One of the most exciting developments in HPC is the integration of Quantum Processing Units (QPUs).

A QPU is a specialized processor designed to perform quantum computations using qubits rather than classical bits.

However, quantum systems are not designed to replace classical HPC systems. Instead, they complement them.

A hybrid quantum-classical workflow typically operates as follows:

Problem Definition
         |
         v
 CPU/GPU Cluster
(Prepare Data)
         |
         v
 Quantum Algorithm
      on QPU
         |
         v
 Results Returned
         |
         v
 CPU/GPU Post Processing

The classical HPC environment manages:

  • Workflow orchestration

  • Data preparation

  • Job scheduling

  • Result verification

  • Large-scale analytics

The QPU accelerates the specific portion of the problem where quantum mechanics offers potential advantages.


Where QPUs Can Help

Quantum computing excels at exploring massive solution spaces simultaneously.

Potential applications include:

Combinatorial Optimization

Finding optimal solutions among millions of possibilities.

Examples:

  • Airline scheduling

  • Supply chain optimization

  • Manufacturing planning

  • Vehicle routing

Quantum Chemistry

Simulating molecular interactions that are extremely difficult for classical computers.

Examples:

  • Drug discovery

  • Material science

  • Catalyst design

Financial Modeling

Risk analysis requiring exploration of huge probability spaces.

Examples:

  • Portfolio optimization

  • Derivative pricing

  • Market simulations


Why Hybrid Systems Matter

Despite the excitement around quantum computing, today's quantum processors face limitations:

  • Limited qubit counts

  • Noise and error rates

  • Short coherence times

  • Scalability challenges

As a result, quantum computers are currently most effective when integrated alongside classical HPC systems.

Hybrid architectures provide:

  • The scalability of classical computing

  • The computational advantages of quantum algorithms

  • Practical deployment in real-world workloads

This is why vendors such as IBM, NVIDIA, Microsoft, Google, and Intel are investing heavily in hybrid quantum-classical infrastructure.


How CPUs, GPUs, DPUs, and QPUs Work Together

Building a powerful AI model is only part of the challenge. Equally important is choosing the processor architecture that best matches the workload. While CPUs and GPUs receive most of the attention, modern AI platforms increasingly rely on a diverse set of processors, each optimized for specific tasks.

  • CPU serves as the control center, handling orchestration, scheduling, operating systems, and application management.

  • GPU delivers massive parallel processing capabilities for AI training, inference, and scientific computing.

  • TPU is specialized for tensor operations and large-scale machine learning workloads.

  • NPU brings power-efficient AI inference to edge devices such as laptops, smartphones, and IoT systems.

  • LPU is optimized for ultra-low-latency Large Language Model (LLM) inference and real-time AI interactions.

  • DPU accelerates networking, storage, security, and infrastructure services, allowing CPUs and GPUs to focus on computation.


Consider a modern AI-driven scientific computing workflow.

CPU

The CPU acts as the coordinator.

Responsibilities include:

  • Managing jobs

  • Scheduling resources

  • Workflow orchestration

  • Data preprocessing

Think of the CPU as the project manager of the system.


GPU

The GPU performs large-scale parallel computation.

Examples:

  • AI training

  • Deep learning

  • Matrix operations

  • HPC simulations

Think of the GPU as a team of thousands of workers performing calculations simultaneously.


DPU

The DPU manages infrastructure tasks.

Examples:

  • RDMA networking

  • Security isolation

  • Firewall functions

  • Encryption

  • Storage services

  • NVMe-over-Fabrics

Think of the DPU as the operations and security team that keeps the data center running efficiently.

By offloading these responsibilities, GPUs spend more time computing and less time waiting for data.


QPU

The QPU tackles highly specialized quantum algorithms.

Examples:

  • Quantum chemistry

  • Optimization

  • Quantum machine learning

Think of the QPU as a specialist consultant brought in to solve specific problems that classical systems struggle with.


Real-World Hybrid Workflow

Imagine a pharmaceutical company searching for a promising drug molecule.

Step 1: Data Preparation

CPU nodes collect and prepare molecular datasets.

Step 2: AI Screening

GPUs process millions of candidate molecules using AI models.

Step 3: Quantum Simulation

A QPU evaluates a small set of promising molecular interactions using quantum algorithms.

Step 4: Result Analysis

CPU and GPU resources analyze outcomes and rank candidates.

The entire workflow uses multiple processors working together, each contributing its unique strengths.


The Rise of Specialized Accelerators

The future of computing will not be built using only CPUs and GPUs.

New accelerators are emerging rapidly.

TPU (Tensor Processing Unit)

Designed specifically for tensor operations used in AI.

Commonly associated with Google Cloud AI workloads.

NPU (Neural Processing Unit)

Focused on efficient AI inference.

Commonly found in:

  • Laptops

  • Smartphones

  • Edge devices

LPU (Language Processing Unit)

Optimized for low-latency generative AI and LLM inference.

Designed to reduce response times for conversational AI systems.

DPU (Data Processing Unit)


Handles infrastructure services such as:

  • Networking

  • Storage

  • Security

  • Telemetry

Allowing CPUs and GPUs to focus on applications and computation.


An Architect's Perspective

The key question is no longer:

Which processor should I choose?

The better question is:

Which processor should perform each task?

A successful architecture balances:

  • Compute performance

  • Memory capacity

  • Network bandwidth

  • Latency requirements

  • Security requirements

  • Power consumption

  • Scalability targets

Modern HPC systems increasingly resemble an orchestra rather than a single instrument.

Each processor plays a different role, but together they create a far more powerful system.

A common misconception is that these processors compete with one another. In reality, they complement each other by accelerating different layers of the AI stack. Modern AI and HPC systems often combine several of these processors within the same architecture to maximize efficiency and performance.

The key architectural questions are not "Which processor is fastest?" but rather:

  • What type of workload am I running?

  • Is the workload training or inference?

  • How much parallelism is required?

  • What latency targets must be achieved?

  • What are the power, memory, networking, and infrastructure constraints?

Choosing the appropriate processor architecture can significantly improve performance, efficiency, scalability, and cost effectiveness, while selecting the wrong one can become an expensive infrastructure mistake.


Final Thoughts

The future of HPC is not about replacing CPUs with GPUs or replacing GPUs with QPUs.

It is about combining the right processors for the right tasks.

CPUs orchestrate, GPUs accelerate computation, DPUs manage infrastructure, and QPUs solve specialized quantum problems. Together, they form a hybrid computing platform capable of tackling problems that would be impractical for any single processor architecture.

The future of AI infrastructure is not built around a single processor. It is built around heterogeneous computing, where CPUs, GPUs, TPUs, NPUs, LPUs, DPUs, and even QPUs work together, each accelerating the tasks they were designed to perform best.

Friday, November 6, 2026

The Fast Lane to AI and HPC: Direct GPU Communication with Storage and Network

Introduction

For years, architects have focused on adding more CPU cores, faster GPUs, and larger memory capacities. However, in modern AI training, HPC workloads, and large-scale analytics, the biggest bottleneck is often not computation.

It is data movement.

A modern NVIDIA H100 or Blackwell GPU can process massive amounts of data, but if every byte must travel through CPU memory before reaching the GPU, the CPU becomes a traffic controller rather than a compute engine.

This is where GPUDirect RDMA and GPUDirect Storage (GDS) change the game.

The core idea is simple:

Let the GPU communicate directly with storage and network devices without making unnecessary stops in CPU memory.


Traditional Data Path


Before GPUDirect, data traveled through several stages:

NVMe Storage
      |
      v
System Memory
      |
      v
CPU
      |
      v
GPU Memory

For network communication:

Remote Node
     |
     v
NIC
     |
     v
CPU Memory
     |
     v
CPU
     |
     v
GPU Memory

Every transfer requires:

  • Additional memory copies

  • CPU interrupts

  • Context switches

  • Higher latency

  • Lower throughput

The CPU becomes a bottleneck.


Understanding the Diagram

The diagram shows two hosts connected through:

  • RoCE (RDMA over Converged Ethernet)

  • InfiniBand

Each host contains:

  • CPU

  • System Memory

  • NVIDIA GPU

  • NVMe Storage

  • RDMA-capable NIC

The interesting part is that data bypasses the CPU.


GPUDirect Storage Explained


Look at the storage path in the diagram.

Instead of:

NVMe
 -> CPU Memory
 -> GPU Memory

GPUDirect Storage allows:

NVMe
 -> GPU Memory

directly through DMA.

What happens?

The NVMe drive performs a DMA transfer directly into GPU memory.

The CPU simply initiates the transfer.

The actual data never flows through CPU memory.

Benefits

  • Lower latency

  • Higher storage throughput

  • Reduced CPU utilization

  • Fewer memory copies

This is especially useful for:

  • AI training datasets

  • GenAI model training

  • Scientific simulations

  • Video analytics

  • Climate modeling


GPUDirect RDMA Explained


Now look at the bottom of the diagram.

A traditional network transfer follows:

GPU
 -> CPU Memory
 -> NIC
 -> Network
 -> NIC
 -> CPU Memory
 -> GPU

Several copies occur during transmission.

With GPUDirect RDMA:

GPU Memory
      |
      v
Remote GPU Memory

through the RDMA-capable NIC.

The CPU sets up the transaction but is not involved in the actual data movement.


Why RDMA Matters


RDMA stands for:

Remote Direct Memory Access

It allows one system to read or write memory on another system without involving the remote CPU.

In the diagram:

GPU Memory (Host 1)
         |
      RDMA
         |
GPU Memory (Host 2)

This creates an extremely low-latency communication channel.


Why AI Clusters Need This


Consider a large language model training job.

Thousands of GPUs exchange:

  • Gradients

  • Weights

  • Activations

  • Checkpoints

every second.

Without RDMA:

GPU -> CPU -> NIC

With RDMA:

GPU -> NIC

The difference becomes enormous when scaled across:

  • 512 GPUs

  • 2048 GPUs

  • 10000+ GPUs

This is why modern AI supercomputers use:

  • NVIDIA Quantum InfiniBand

  • NVIDIA Spectrum Ethernet

  • Mellanox ConnectX NICs

  • RoCE v2

along with GPUDirect RDMA.


DMA: The Unsung Hero


The diagram highlights DMA.

DMA means:

Direct Memory Access

DMA allows hardware devices to move data without CPU intervention.

Without DMA:

CPU copies every byte

With DMA:

Device transfers data directly

GPUDirect uses DMA extensively to create direct data paths.


Real-World Example


Imagine training a 70-billion-parameter LLM.

Dataset size:

50 TB

Traditional path:

NVMe
 -> RAM
 -> CPU
 -> GPU

Millions of memory copies occur.

Using GPUDirect Storage:

NVMe
 -> GPU Memory

The CPU is freed to manage orchestration rather than moving data.

This improves:

  • GPU utilization

  • Training time

  • System efficiency


Architectural Benefits


1. Lower Latency

Fewer software layers.

Fewer memory copies.

Less waiting.


2. Reduced CPU Overhead

CPU cycles become available for:

  • Scheduling

  • Orchestration

  • Monitoring

  • Control-plane tasks

instead of moving data.


3. Better GPU Utilization

GPUs spend more time computing.

Less time waiting for data.


4. Higher Bandwidth

RDMA networks can deliver:

  • 200 Gbps

  • 400 Gbps

  • 800 Gbps

with very low CPU involvement.


5. Improved Scalability

As cluster size grows, CPU bottlenecks become increasingly expensive.

GPUDirect allows architectures to scale more efficiently.


Why Architects Should Care


Many architects focus on:

  • CPU speed

  • Core count

  • GPU count

The real performance challenge often lies elsewhere:

Data movement costs more than computation.

A GPU capable of 100+ TFLOPS is useless if it waits for data.

The purpose of GPUDirect is not to make the GPU faster.

The purpose is to ensure the GPU is never idle.


Evolution in system architecture.

Traditional systems were designed around the CPU being the center of all communication.

Modern AI and HPC systems are shifting to a new model:

  • Storage talks directly to GPU memory.

  • Network talks directly to GPU memory.

  • CPU becomes the coordinator, not the data mover.

This architectural shift is one of the key reasons why modern AI supercomputers achieve unprecedented scalability and efficiency.

Thursday, November 5, 2026

Understanding the Building Blocks of Persistent Memory in Linux : PMEM DAX Folios

  

Introduction

When debugging DAX-related issues, engineers often encounter terms like:

  • PMEM

  • devdax

  • fsdax

  • folio

  • compound page

  • page order

  • MAX_FOLIO_ORDER

  • vmemmap_shift

This blog explains how these concepts fit together and why a seemingly simple operation such as creating a DAX namespace can trigger complex memory-management behavior inside the kernel.


What is Persistent Memory (PMEM)?





Persistent Memory (PMEM) is storage that retains data across power cycles while being directly addressable like memory.

Traditionally:

TechnologyFastPersistent
DRAM✅ Yes❌ No
SSD/HDD❌ No✅ Yes
PMEM✅ Yes✅ Yes

PMEM combines characteristics of both memory and storage.

Examples include:

  • Intel Optane Persistent Memory

  • NVDIMM

  • Future CXL-attached persistent memory devices




Where is PMEM Located?




Think of PMEM as being physically attached to the memory subsystem rather than accessed through traditional storage controllers.

Traditional storage path:

Application
      ↓
Page Cache
      ↓
SSD / NVMe
----------------------

PMEM path:

Application
      ↓
CPU Memory Bus
      ↓
Persistent Memory

On Power systems and x86 systems, PMEM is exposed to Linux as a memory region that can be mapped directly into applications.

You can view PMEM devices using:

ndctl list

or

daxctl list

What is DAX?

DAX stands for Direct Access.

DAX allows applications to bypass the page cache and directly access persistent memory.

Without DAX:

Application
      ↓
Page Cache
      ↓
Storage Device

With DAX:

Application
      ↓
Persistent Memory

This removes:

  • Extra memory copies

  • Page cache overhead

  • Additional latency

Benefits:

  • Faster databases

  • Lower latency

  • Better HPC performance

  • Reduced memory consumption


Types of DAX

1. Filesystem DAX (fsdax)

Uses a filesystem such as:

  • XFS

  • ext4

Example:

PMEM
 └── XFS (DAX enabled)
      └── File

Applications access files directly from persistent memory.


2. Device DAX (devdax)

Provides direct raw access to the PMEM device.

Example:

/dev/dax0.0

Applications mmap() the region directly.

No filesystem is involved.

Creation example:

ndctl create-namespace \
    --mode=devdax

What is a Folio?

A folio is a Linux kernel memory object used to represent one or more physically contiguous pages.

Before folios:

struct page

represented every individual page.

Now:

struct folio

can represent:

  • 1 page

  • 2 pages

  • 512 pages

  • 16384 pages

as a single unit.

Benefits:

  • Less memory-management overhead

  • Better performance

  • Improved page-cache scalability


Understanding Page Order

Linux measures compound page size using order.

Formula:

Size = PAGE_SIZE × 2^order

For ppc64le:

PAGE_SIZE = 64 KB

Examples:

OrderSize
064 KB
1128 KB
2256 KB
41 MB
52 MB
1064 MB
141 GB

For example:

64 KB × 2^14
= 1 GB

Therefore:

Order 14 = 1 GB object

What is a Compound Page?

A compound page is a collection of many base pages treated as a single memory object.

Example:

Compound Page
     │
 ├── Page 0
 ├── Page 1
 ├── Page 2
 ├── ...
 └── Page N

Instead of handling thousands of pages individually, Linux manages them as a single object.

This improves:

  • TLB efficiency

  • Memory management

  • Mapping performance


What is a DAX Folio?

A DAX folio is simply a folio backing DAX memory.

Normal memory:

RAM
 ↓
Folio

DAX memory:

Persistent Memory
      ↓
ZONE_DEVICE
      ↓
DAX Folio

A DAX folio represents PMEM pages that can be directly mapped into applications.


How a 1 GB DAX Folio Gets Created

Suppose we create:

ndctl create-namespace \
    --mode=devdax \
    -a 1073741824

Notice:

1073741824 = 1 GB

The kernel interprets this alignment requirement and calculates:

PAGE_SHIFT = 16

because:

64 KB pages

and

1 GB = 2^30

Therefore:

Order = 30 - 16
       = 14

Linux now attempts to create:

Order-14 folio

which represents:

16384 pages

or:

1 GB DAX folio

Why Does MAX_FOLIO_ORDER Matter?

The kernel limits how large a folio can be.

Inside memory management code:

folio_set_order(folio, order);

Linux verifies:

VM_WARN_ON_ONCE(order > MAX_FOLIO_ORDER);

This prevents creation of excessively large folios.

If:

Order = 14

but

MAX_FOLIO_ORDER < 14

then Linux generates:

VM_WARN_ON_ONCE()

indicating the folio exceeds supported limits.


Real Example

User creates:

ndctl create-namespace \
    --mode=devdax \
    -a 1073741824

Kernel receives:

1 GB aligned PMEM range

Kernel computes:

Order 14 folio

During initialization:

memremap_pages()
    ↓
memmap_init_zone_device()
    ↓
memmap_init_compound()
    ↓
prep_compound_head()
    ↓
folio_set_order()

Kernel checks:

order > MAX_FOLIO_ORDER

and generates a warning if the architecture does not support such a large folio.


Commands Useful for DAX Investigation

Commands Useful for DAX Investigation

List PMEM devices

ndctl list

Show namespaces

ndctl list -N

Show DAX devices

daxctl list

View PMEM regions

ndctl list -R

Check memory block size

getconf PAGE_SIZE

Check DAX device

ls -l /dev/dax*

View DAX mappings

dmesg | grep dax

View compound page warnings

dmesg | grep -i folio

Key Takeaways

  • PMEM is persistent memory that combines storage persistence with memory-like access.

  • DAX allows direct access to PMEM without page cache involvement.

  • devdax exposes PMEM as a raw memory device.

  • Folios are Linux memory objects representing one or more pages.

  • Order defines folio size as PAGE_SIZE × 2^order.

  • On ppc64le with 64 KB pages, a 1 GB DAX alignment translates to order 14.

  • The kernel creates a DAX folio to represent that PMEM region.

  • If the calculated order exceeds MAX_FOLIO_ORDER, Linux triggers a warning during DAX initialization.

Final One-Line Summary DAX device

ls -l /dev/dax*

View DAX mappings

dmesg | grep dax

View compound page warnings

dmesg | grep -i folio

Key Takeaways

  • PMEM is persistent memory that combines storage persistence with memory-like access.

  • DAX allows direct access to PMEM without page cache involvement.

  • devdax exposes PMEM as a raw memory device.

  • Folios are Linux memory objects representing one or more pages.

  • Order defines folio size as PAGE_SIZE × 2^order.

  • On ppc64le with 64 KB pages, a 1 GB DAX alignment translates to order 14.

  • The kernel creates a DAX folio to represent that PMEM region.

  • If the calculated order exceeds MAX_FOLIO_ORDER, Linux triggers a warning during DAX initialization.

DAX enables direct access to persistent memory, and a DAX folio is the kernel's representation of that PMEM region; large DAX alignments can create large compound folios whose order may exceed architecture-specific folio limits.

CUDA Meets DOCA : One Trains the AI, The Other Keeps It Running


Introduction

When people talk about AI infrastructure, the conversation almost always starts with GPUs. Terms like LLM training, inference acceleration, CUDA cores, and Tensor Cores have become common across the industry. As a result, many engineers are familiar with CUDA, NVIDIA's software platform that unlocks the power of GPU computing.

However, there is another NVIDIA technology quietly becoming essential in modern AI factories, hyperscale data centers, and HPC environments: DOCA.

While CUDA accelerates the computation performed by GPUs, DOCA accelerates the networking, storage, security, and infrastructure services that keep thousands of GPUs efficiently connected and utilized.

To understand the future of AI infrastructure, it's important to understand both.


What is CUDA?

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform and programming model that allows software developers to leverage GPUs for general-purpose computing.

Before CUDA, graphics processors were primarily used for rendering images. CUDA transformed GPUs into powerful computing engines capable of executing thousands of parallel threads.

Key CUDA Use Cases

  • AI and Deep Learning Training

  • Large Language Model (LLM) Training

  • Model Inference

  • Scientific Simulations

  • Computational Physics

  • Weather Forecasting

  • Financial Modeling

  • High Performance Computing (HPC)

Frameworks such as:

  • PyTorch

  • TensorFlow

  • RAPIDS

  • cuDNN

  • NCCL

all rely heavily on CUDA underneath.

Think of CUDA as:

The software layer that tells GPUs how to perform massive parallel computations.

Without CUDA, modern AI training at scale would be dramatically slower and more difficult.


What is DOCA?

DOCA (Data Center Infrastructure-on-a-Chip Architecture) is NVIDIA's software framework for programming BlueField DPUs (Data Processing Units).

As AI clusters continue growing from tens of GPUs to thousands of GPUs, infrastructure challenges become significant:

  • Network congestion

  • Storage bottlenecks

  • Security overhead

  • Packet processing

  • Telemetry collection

  • Multi-tenant isolation

Traditionally these tasks consumed valuable CPU resources.

DOCA allows these services to be offloaded to the DPU, freeing CPU cores for application workloads while improving overall data center efficiency.

Key DOCA Use Cases

  • RoCE and RDMA acceleration

  • Network packet processing

  • Zero-trust security

  • Firewall services

  • Storage virtualization

  • Data path acceleration

  • Network telemetry

  • Service isolation

  • SmartNIC applications

Think of DOCA as:

The software layer that tells DPUs how to accelerate infrastructure services.


Understanding the Difference

A simple way to understand the distinction is by looking at where the work happens.

TechnologyHardwareFocus Area
CUDAGPUCompute acceleration
DOCADPUInfrastructure acceleration
CPUProcessorGeneral purpose processing

CUDA Accelerates

  • Matrix operations

  • Neural network training

  • Deep learning inference

  • Scientific computation

DOCA Accelerates

  • Network traffic

  • Storage access

  • Data movement

  • Security processing

  • Infrastructure management


The AI Factory Analogy

Imagine a modern AI factory with 8,000 GPUs training a trillion-parameter model.

CUDA's Job

CUDA helps every GPU perform billions of mathematical operations per second.

It powers:

  • Tensor operations

  • Backpropagation

  • Gradient updates

  • Model inference

In short:

CUDA helps the AI think.

DOCA's Job

DOCA ensures data reaches GPUs efficiently and securely.

It handles:

  • GPU-to-GPU communication

  • RDMA transfers

  • Storage connectivity

  • Flow monitoring

  • Security enforcement

In short:

DOCA keeps the AI factory running smoothly.


Why DOCA Matters More Than Ever

In early HPC clusters, compute was the primary bottleneck.

Today, many AI environments struggle more with:

  • Data movement

  • Network latency

  • Storage throughput

  • Security overhead

When thousands of GPUs communicate simultaneously, inefficient infrastructure can leave expensive accelerators sitting idle.

For example:

A cluster with 1,024 GPUs may deliver only a fraction of its potential performance if networking becomes saturated.

This is where DOCA and BlueField DPUs become critical.

They move infrastructure functions away from CPUs and closer to the network, reducing latency and improving scalability.


CUDA + DOCA: Better Together

Modern AI systems need both technologies.

CUDA Provides

  1. GPU acceleration
  2. AI training performance
  3. AI inference performance
  4. Scientific computing acceleration

DOCA Provides

  1. Networking acceleration
  2. Storage acceleration
  3. Security offload
  4. Infrastructure isolation
  5. Telemetry and monitoring

Together they create a complete AI platform:

Application Layer
       │
AI Frameworks (PyTorch, TensorFlow)
       │
CUDA → GPU Acceleration
       │
High-Speed Fabrics (RoCE, InfiniBand)
       │
DOCA → DPU Infrastructure Acceleration
       │
Storage, Network, Security Services

Why Infrastructure Engineers Should Care

For years, infrastructure teams focused on servers, networks, and storage while AI teams focused on GPUs.

That separation is rapidly disappearing.

Modern AI deployments require engineers who understand:

  • GPUs

  • DPUs

  • RDMA

  • RoCE

  • InfiniBand

  • Storage fabrics

  • Cluster networking

Understanding CUDA alone explains how AI workloads run.

Understanding DOCA explains how AI infrastructure scales.


-----------

Restaurant Example

Imagine a restaurant:

  • GPU = Expert Chef

    • Cooks food very fast

    • Focuses only on cooking

  • CPU = Manager

    • Handles orders

    • Coordinates staff

    • Manages operations

  • DPU = Support Staff

    • Takes orders

    • Handles billing

    • Manages security

    • Delivers food

Without support staff, the chef keeps getting interrupted.

Chef cooking
↓
Customer arrives
↓
Chef answers phone
↓
Chef processes payment
↓
Chef serves food
↓
Back to cooking

The chef becomes less productive.

A DPU does the same thing for a server.


In a Server

Traditionally the CPU does:

  • Networking

  • Security

  • Firewall

  • Encryption

  • Storage I/O

  • Virtual networking

  • Kubernetes networking

  • Monitoring

and also runs applications.

CPU
├─ Application
├─ Networking
├─ Encryption
├─ Storage
├─ Security
└─ Monitoring

CPU becomes overloaded.


What a DPU Does

A DPU takes infrastructure tasks away from the CPU.

CPU
└─ Runs applications

GPU
└─ Runs AI / HPC

DPU
├─ Networking
├─ Security
├─ Storage
├─ Telemetry
└─ Virtualization

Now each processor focuses on its specialty.


Networking Example

Suppose your AI cluster has:

1000 GPUs

During training, GPUs exchange gradients every few milliseconds.

Without DPU:

GPU
 ↓
CPU
 ↓
NIC
 ↓
Network

CPU must handle:

  • Packet creation

  • Packet checking

  • Interrupts

  • Network queues

Result:

CPU utilization = High
Latency = Higher
GPU waiting = More

With BlueField DPU

GPU
 ↓
DPU
 ↓
Network

The DPU handles:

  • RDMA

  • RoCE

  • Packet processing

  • Network scheduling

CPU stays free.

Benefits:

  • Lower latency

  • Faster AI training

  • Better GPU utilization


Security Example

Consider a cloud provider.

You rent a server.

You should never be able to access another customer's data.

The DPU acts like a security guard.

Without DPU:

CPU runs applications
CPU runs firewall
CPU runs security
CPU runs encryption

If CPU is compromised, security may also be compromised.


With DPU

Application
    ↓
   CPU

Security
Firewall
Isolation
Encryption
    ↓
   DPU

Even if somebody compromises the operating system:

Linux OS hacked

the DPU can still enforce:

  • Network isolation

  • Firewall rules

  • Encryption

  • Tenant separation

This concept is known as:

Infrastructure Isolation


Real AI Cluster Example

Imagine an NVIDIA DGX SuperPOD.

AI training traffic:

100s of GB/s

Without DPU:

GPU → CPU → Network

With DPU:

GPU → DPU → Network

The CPU no longer becomes the bottleneck.

This is why hyperscalers use DPUs.

Examples:

  • NVIDIA BlueField

  • AWS Nitro

  • Microsoft Azure SmartNIC

  • Google Infrastructure Processing Unit (IPU)

All are variations of the same concept.


Suppose a 500 TB training dataset is stored in NVMe storage.

Traditional flow:

NVMe
 ↓
CPU
 ↓
GPU

CPU copies data.

With DPU + GPUDirect Storage:

NVMe
 ↓
DPU
 ↓
GPU Memory

Benefits:

  • Less copying

  • Lower latency

  • Reduced CPU overhead

  • Faster training startup


Simple Analogy

Think of a modern AI server as:

GPU = Scientist
CPU = Project Manager
DPU = Operations Team

You do not want your scientist:

  • Checking badges

  • Managing security

  • Dispatching trucks

  • Moving files

You want the scientist doing science.

  • GPU should do computation

  • CPU should coordinate

  • DPU should run infrastructure


A DPU is to a data center what a dedicated operations and security team is to a company: it offloads networking, storage, and security services from CPUs so GPUs can spend more time computing and less time waiting for data.



What is GPU and GPU Memory: 

ComponentHuman AnalogyPurpose
GPUBrainPerforms calculations
GPU Memory (HBM/GDDR)Short-term memoryStores data that the GPU is currently working on

Simple Example

Imagine you're solving a large Sudoku puzzle.

CPU World

  • CPU = You

  • RAM = Table where papers are placed

GPU World

  • GPU = 10,000 assistants solving parts of the puzzle simultaneously

  • GPU Memory = The desk space available directly beside those assistants

The assistants (GPU) need a place to keep the numbers they're processing.

That place is GPU Memory.


Physical View

A modern NVIDIA H100 looks roughly like this:

+-----------------------------------+
|                                   |
|  HBM Memory   GPU   HBM Memory    |
|                                   |
|  HBM Memory       HBM Memory      |
|                                   |
+-----------------------------------+

The large center chip is:

GPU

Surrounded by:

HBM Memory Chips

connected through extremely fast links.


What Does GPU Do?

The GPU performs:

  • AI training

  • AI inference

  • Matrix multiplication

  • Scientific computing

  • Image rendering

  • HPC simulations

Example:

2 + 2 = 4

CPU may perform a few calculations quickly.

GPU performs:

Millions or billions

of such calculations simultaneously.


What Does GPU Memory Do?

GPU memory stores:

  • AI model weights

  • Training data

  • Activations

  • Tensors

  • Intermediate results

For example, during LLM training:

Llama
GPT
Granite

model parameters reside in GPU memory.

The GPU continuously reads and updates them.


Why Not Use System RAM?

System RAM is much slower.

Example:

Memory TypeBandwidth
DDR5 System RAM~50-100 GB/s
HBM3 GPU Memory~3-5 TB/s

That's about:

3000-5000 GB/s

for HBM.

A modern GPU can consume data much faster than CPU RAM can provide.


Data Flow Example

Without GPUDirect:

NVMe
  ↓
System RAM
  ↓
CPU
  ↓
GPU Memory
  ↓
GPU

With GPUDirect Storage:

NVMe
  ↓
GPU Memory
  ↓
GPU

Notice:

The GPU still computes.

The GPU Memory still stores data.

GPUDirect simply moves data into GPU Memory more efficiently.


Real NVIDIA H100 Example

An NVIDIA H100 contains roughly:

GPU Compute Engine
+
80 GB HBM3 Memory

The GPU and memory are separate hardware blocks but packaged together.

Think of:

GPU = Processor
HBM = Local Memory

just like:

CPU = Processor
RAM = System Memory

Architect View

In the diagram you were studying:

GPU Memory ←→ GPU

The GPU Memory holds:

  • AI model parameters

  • Training batches

  • Input tensors

The GPU performs:

  • Matrix operations

  • Tensor operations

  • AI computations

The GPU is the worker.

The GPU Memory is the workspace.


Conclusion

The future of AI is not just about faster GPUs. It is equally about building efficient infrastructure capable of feeding thousands of GPUs with data, securely and at scale.

CUDA accelerates the compute.

DOCA accelerates the infrastructure.

As AI factories continue to grow, engineers who understand both technologies will be better equipped to design, optimize, and troubleshoot next-generation AI, HPC, and cloud environments.