H200 GPU Glossary: HBM3e, Tensor Cores, NVLink and Other Terms in Simple Words

When you hear about GPUs there are a lot of technical terms thrown at you along with it. Some of the most common words are: HBM3e, Tensor Cores, NVLink, CUDA, PCIe, and memory bandwidth. But what are they? Why do they matter? And why is it important for you to know them?

Let’s talk about some most common GPU glossary that you should know while you are working in in the industry, be it for your business, start-up, or whether you are an employee, knowing about this is not just a “good-to-know” thing but actually crucial as they are directly related to your workloads and each of them matters while training your models or any other kind of workloads you are using it for, especially for data center GPUs like H200.

In fact, let’s take H200 as a benchmark for this blog, although most of the terms are not just for H200, but any kind of GPUs.

Let’s begin.

Let’s suppose you run a factory. Knowing the inside out of that factory is very crucial, correct?

When you understand what kind of workers you want and how many, how much space for storage you want and how your machines are connected to each other, working becomes easier. You know exactly what to do, what is your factory capable of and where are you lacking and you can make improvements or amends accordingly. Understanding these components helps you choose the right GPU infrastructure for AI training, inference, and high-performance computing.

Now with the same analogy, let’s consider the Nvidia H200 GPU cloud is a factory. The processing units of the H200 GPU are the workers of the factory, memory is the storage area, and interconnects are the roads connecting different machines.

In this blog I will help you understand your factory, so you know exactly what you are dealing with.

These are the terms we will be covering in this blog:

• HBM3e

• Tensor Cores

• NVLink

• CUDA

• Memory bandwidth

• FP8, FP16 and BF16

• PCIe

• MIG

• TDP

• TFLOPS

1. HBM3e: High-speed GPU memory

HBM3e stands for High Bandwidth Memory, third-generation enhanced.

It is a specialized type of memory designed to move enormous amounts of data quickly.

Imagine a warehouse where thousands of workers need materials simultaneously. A traditional storage system might struggle to supply everyone quickly. HBM3e works like a warehouse with multiple high-speed delivery lanes.

The NVIDIA H200 SXM provides 141 GB of HBM3e memory and approximately 4.8 TB/s of memory bandwidth.

Why does this matter?

Large AI models need substantial memory to store their parameters and temporary data. HBM3e helps the GPU access this information quickly, reducing delays during training and inference.

In simple words: HBM3e is extremely fast memory that helps GPUs handle large AI workloads.

2. Tensor Cores: Specialized AI calculation engines

Tensor Cores are dedicated processing units inside NVIDIA GPUs designed to accelerate matrix calculations.

AI models perform billions or trillions of mathematical operations involving matrices.

Imagine calculating thousands of multiplication problems. A regular processor handles many types of calculations, while Tensor Cores are specially optimized for the mathematical operations used heavily in AI.

The H200 uses fourth-generation Tensor Cores and supports precision formats such as FP8, FP16, and BF16.

These capabilities make it useful for deep learning, large language models, and scientific computing.

In simple words: Tensor Cores are specialized engines that make AI calculations faster.

3. NVLink: A high-speed connection between GPUs

When multiple GPUs work together, they need to exchange information.

NVLink is NVIDIA’s high-speed GPU interconnect technology, designed to move data between compatible GPUs efficiently.

Imagine two offices sharing large files. A slow network connection forces employees to wait, while a high-speed connection allows information to move much faster.

The H200 SXM supports fourth-generation NVLink with up to 900 GB/s of aggregate bidirectional bandwidth per GPU.

This becomes especially important when training large AI models across multiple GPUs.

In simple words: NVLink is a high-speed communication highway connecting GPUs.

4. CUDA: Software that lets applications use GPUs

CUDA stands for Compute Unified Device Architecture.

It is NVIDIA’s parallel computing platform and programming model that allows developers to run calculations on NVIDIA GPUs.

For example, frameworks such as PyTorch can use CUDA to perform AI computations on an H200 instead of relying entirely on the CPU.

In simple words: CUDA is the software foundation that helps applications use NVIDIA GPU processing power.

5. Memory bandwidth: How fast data moves

Memory capacity and memory bandwidth are different.

Capacity tells you how much data can fit inside GPU memory. Bandwidth tells you how quickly that data can be transferred.

Imagine a water tank. Its size represents memory capacity, while the width of the pipe represents memory bandwidth.

The H200’s 4.8 TB/s bandwidth allows extremely large amounts of information to move every second under ideal conditions.

In simple words: Memory bandwidth is the speed at which the GPU can access its stored information.

6. FP8, FP16 and BF16: Different ways to represent numbers

AI models perform calculations using numerical formats called precision formats.

FP16 and BF16 use 16 bits to represent numbers, while FP8 uses only eight bits.

Smaller formats can reduce memory usage and accelerate supported operations, but they may also reduce numerical precision.

For example, using FP8 for suitable AI calculations can improve performance while reducing the amount of data being processed.

However, not every operation can safely use lower precision.

In simple words: Precision formats determine how numbers are stored and calculated, affecting speed, memory consumption, and accuracy.

7. PCIe: The connection between the GPU and server

PCIe stands for Peripheral Component Interconnect Express.

It connects components such as GPUs, NVMe storage devices, and network adapters to the server.

The H200 supports PCIe Gen5 connectivity.

Think of PCIe as the main road connecting the GPU to the rest of the server, while NVLink provides specialized high-speed communication between compatible GPUs.

In simple words: PCIe connects the GPU to other server hardware.

8. MIG: Dividing one GPU into smaller GPUs

MIG stands for Multi-Instance GPU.

It allows supported NVIDIA GPUs to be divided into smaller, isolated GPU instances.

For example, an H200 SXM can be configured with up to seven MIG instances, depending on the selected profiles.

Imagine a large office building divided into separate offices. Different customers can use their own space without requiring an entire building.

MIG is particularly useful for cloud providers offering smaller GPU allocations to multiple customers.

In simple words: MIG allows one physical GPU to serve multiple isolated workloads.

9. TDP: Understanding GPU power requirements

TDP stands for Thermal Design Power.

It describes the thermal power level that the cooling system must be designed to handle.

The H200 SXM has a configurable maximum TDP of up to 700 watts.

This does not mean the GPU constantly consumes 700 watts. Actual consumption depends on utilization, workload, and power configuration.

For hosting providers, TDP is important when planning power supplies, rack density, electricity requirements, and cooling capacity.

In simple words: TDP helps determine how much power and cooling a GPU server may require.

10. TFLOPS: Measuring computing performance

TFLOPS means trillions of floating-point operations per second.

It is a measurement of theoretical computational throughput.

For example, a GPU with higher TFLOPS capability can potentially perform more mathematical operations every second.

However, TFLOPS figures must be compared using the same precision format and conditions.

Higher TFLOPS does not guarantee faster AI training because memory bandwidth, communication, and software efficiency also matter.

In simple words: TFLOPS measures potential computing power, not guaranteed application speed.

Quick-reference glossary