SkillsAdd

nvidia/skills · Official

doca-gpunetio-ib-write-lat

Use this skill when the user is measuring GPU-kernel-initiated RDMA WRITE latency through doca-gpunetio — building and running the `gpunetio_ib_write_lat` client + server pair under `doca/tools/gpunetio_ib_write_lat/`, checking GPU-NIC pairing, reading the half-iter / full-iter / CUDA-side usec columns, characterizing median / p99 / jitter for a real-time control loop, picking GPUNetIO vs GPI vs CPU-initiated `perftest`, or weighing the latency-vs-batching trade-off. Trigger even without 'GPUNe…

Hot #1042 First seen Jul 18, 2026

Installation

$ npx skills add nvidia/skills --skill doca-gpunetio-ib-write-lat
View on skills.sh ↗ GitHub repository ↗

Also in this package

Other skills from nvidia/skills · top by installs.

npx skills add nvidia/skills
rag-blueprint 2.9K accelerated-computing-cudf 2.8K aiq-research 2.8K cuopt-developer 2.5K aiq-deploy 2.4K cuopt-install 2.4K cudaq-guide 2.4K data-designer 2.2K deepstream-dev 2.2K omniverse-cad-to-simready 2.2K

Browse all from nvidia/skills

More details

Agent compatibility

Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.

Claude Code Not declared
Cursor Not declared
Codex Not declared
GitHub Copilot Not declared
Windsurf Not declared
Gemini CLI Not declared
Cline Not declared
OpenCode Not declared

Repository health

Stars 3.2K
License LICENSE-APACHE
Default branch main
Open issues 5
Status Active

Skill metadata

Parsed from SKILL.md frontmatter.

LicenseApache-2.0
CompatibilityRequires DOCA SDK installed at /opt/mellanox/doca on Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with an InfiniBand-capable ConnectX or BlueField RNIC. Requires NVIDIA GPU with CUDA Toolkit and `nvidia_peermem` loaded; client and server hosts each need a GPU-NIC pair on a common PCIe / NVLink fabric. Reads pkg-config doca-gpunetio / doca-rdma / doca-common and builds from the source tree at /opt/mellanox/doca/tools/gpunetio_ib_write_lat against the installed DOCA.
More metadata
kind
tool

Package contents

Files included with this skill beyond the listing page.

  • skill md SKILL.md 15,174 B
  • docs SUMMARY.md 900 B

History

  1. First seen on skills.sh
  2. First recorded snapshot · 121 installs

SKILL.md

DOCA GPUNetIO ibwritelat

Where to start: This is a tool skill for the GPUNetIO- flavored ibwritelat benchmark shipped under doca/tools/gpunetioibwrite_lat/ (a client + server pair, built from source against the installed DOCA via meson). It measures the latency of an RDMA WRITE work request when the WR is posted from a CUDA kernel through the doca-gpunetio device-side surface, in a ping-pong cadence. Open [TASKS.md](TASKS.md) and start at [## configure](TASKS.md#configure) for the GPU-NIC pairing precondition and the build pattern; jump to [## run](TASKS.md#run) for the single-iteration smoke flow. Open [CAPABILITIES.md](CAPABILITIES.md) when the question is what this tool actually measures, how it differs from the GPI sister tool on the same physical operation, or how to interpret the half-iter / full-iter / CUDA-side usec output and the median / p99 / jitter characterization. If DOCA is not installed yet, route to [doca-setup](../../doca-setup/SKILL.md) first; if the user is still deciding between GPUNetIO and GPI as a programming surface, the picture in [../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes](../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes) and [../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes](../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes) is the first stop.

Example questions this skill answers well

The CLASSES of doca-gpunetio-ib-write-lat questions this skill is built to answer, each with one worked example. The class is the load-bearing piece; the worked example is one instance.

  • **"What GPU-init RDMA-WRITE latency / jitter can the

GPUNetIO path deliver for a real-time / control-loop workload?"** — worked example: "measure per-iteration WRITE latency between two hosts with an H100 + ConnectX-7 on each side, target the median and the p99 separately". Answered by the GPU-NIC pairing precondition in [CAPABILITIES.md ## Capabilities and modes](CAPABILITIES.md#capabilities-and-modes) + the bring-up flow in [TASKS.md ## configure](TASKS.md#configure) + [TASKS.md ## run](TASKS.md#run).

  • **"This is the GPUNetIO tool — how does the latency

number differ from the GPI programming surface?"** — worked example: "the team is using GPI; should I expect GPUNetIO to beat / tie / lose vs GPI?". Answered by the "same physical operation, different runtime framework" rule in [CAPABILITIES.md ## Capabilities and modes](CAPABILITIES.md#capabilities-and-modes) + the cross-link to the GPI library skill [../../libs/doca-gpi/CAPABILITIES.md](../../libs/doca-gpi/CAPABILITIES.md) (note: doca/tools/ ships no GPI ibwritelat benchmark binary — GPI is a programming surface, not a shipped benchmark tool).

  • **"Median vs p99 vs jitter — which one is the actual

answer for a real-time control loop?"** — worked example: "my control loop has a deadline; the median is well under the budget but p99 spikes; do I quote the median or the p99?". Answered by the median-vs-p99-vs-jitter rule in [CAPABILITIES.md ## Observability](CAPABILITIES.md#observability) + the eval-loop overlay in [TASKS.md ## test](TASKS.md#test).

  • **"What is the latency-vs-batching trade-off specific

to GPU-init RDMA?"** — worked example: "my CUDA kernel could batch multiple WRs to amortize the GPU-side overhead; what does that buy me on latency vs what does it cost me?". Answered by the latency-vs-batching trade-off in [CAPABILITIES.md ## Capabilities and modes](CAPABILITIES.md#capabilities-and-modes).

  • **"What version of DOCA + CUDA Toolkit do I need for

this binary to build and run?"** — worked example: "my install has DOCA at one semver and CUDA at another; will the ToT-shipped gpunetioibwrite_lat even link?". Answered by the version overlay in [CAPABILITIES.md ## Version compatibility](CAPABILITIES.md#version-compatibility).

  • **"How do I read the half-iter / full-iter / CUDA-side

usec columns?"** — worked example: "the binary printed half-iter, full-iter, and a CUDA-side number — what is the right column to quote for one-way latency vs round-trip vs cross-check?". Answered by the column- semantics rule in [CAPABILITIES.md ## Observability](CAPABILITIES.md#observability).

Audience

This skill serves external developers and performance engineers who need a reproducible measurement of the latency of an RDMA WRITE WR when the WR is posted from a CUDA kernel through doca-gpunetio, on the user's actual install and GPU-NIC pair. Concretely:

  • A developer designing a GPU-resident real-time control

loop and deciding whether the GPUNetIO path's tail latency fits the deadline.

  • A platform operator validating a tuning change (NUMA

pinning, GPU PCIe placement, IB device choice, GID index, NIC firmware burn) by re-running this benchmark against the new state.

  • An SRE / performance engineer producing a *"this is the

GPUNetIO-driven WRITE latency on this GPU-NIC pair today, with median + p99 + jitter"* artifact downstream consumers can cite.

  • An AI agent answering *"is the doca-gpunetio latency

budget acceptable for this real-time workload class"* honestly — with measured numbers, the build + invocation that produced them, and the GPU + NIC + DOCA version that scopes them — rather than guessing.

It is not for users debugging the doca-gpunetio library itself (route to [../../libs/doca-gpunetio/SKILL.md](../../libs/doca-gpunetio/SKILL.md)), and not a substitute for the perftest upstream ibwritelat (which measures CPU-initiated WRITE latency).

Language scope

The doca-gpunetio-ib-write-lat tool is shipped as C plus CUDA .cu translation units under doca/tools/gpunetioibwrite_lat/, split into a client/ subtree, a server/ subtree, and a common/ subtree shared between them (per the verified file layout: client/{main.c,perftest.{c,h},meson.build}, server/{main.c,perftest.{c,h},meson.build}, common/{common.c,common.h,kernel.cu}). The host-side build is meson against the installed DOCA pkg-config modules (doca-gpunetio, doca-rdma, doca-common, plus the CUDA Toolkit dependency); the device-side build is nvcc against the DOCA GPU NetIO device-side header set. There is no Python / Rust / Go binding — the tool is a pair of CLI binaries.

When to load this skill

Load this skill when the user is — or the agent needs to — build and run the gpunetioibwrite_lat client + server on real hosts with DOCA installed plus a CUDA Toolkit matched to the DOCA install, and a GPU + IB device pair on each host's PCIe topology. Concretely:

  • Measuring kernel-initiated RDMA WRITE latency between

two hosts (or a host and a BlueField DPU) with the GPUNetIO surface.

  • Characterizing tail latency (p99 / p99.9) and jitter

for a real-time / control-loop workload class.

  • Deciding whether the GPUNetIO path is the right runtime

surface for a class of workload vs the GPI programming surface (the [doca-gpi](../../libs/doca-gpi/SKILL.md) library — doca/tools/ ships no GPI benchmark binary) or the classic CPU-initiated perftest path.

  • Capturing a documented baseline (build + invocation +

DOCA version + GPU + NIC + as-deployed environment + numbers) for later regression hunts.

  • Diagnosing a build / link / run failure that surfaces

the GPUNetIO + RDMA bring-up sequence under this tool's shipped scaffolding.

Do not load this skill for general DOCA orientation, library API work, or installation. For those, use [doca-public-knowledge-map](../../doca-public-knowledge-map/SKILL.md), [../../libs/doca-gpunetio/SKILL.md](../../libs/doca-gpunetio/SKILL.md), or [doca-setup](../../doca-setup/SKILL.md). Do not load it for application-level real-time deadline analysis — this benchmark measures the WR latency through GPUNetIO, not the user's full pipeline.

What this skill provides

This is a thin loader. Substantive material lives in two companion files:

  • CAPABILITIES.md — what the tool measures (the

ping-pong WRITE latency primitive driven by both sides' CUDA kernels through doca-gpunetio), the runtime-surface selection rule (GPUNetIO vs GPI vs CPU-initiated), the GPU-NIC pairing precondition, the latency-vs-batching trade-off intrinsic to GPU-init RDMA, the median / p99 / jitter reporting taxonomy, the version overlay (DOCA .pc PLUS CUDA Toolkit), the layered error taxonomy, the observability surface (stdout report including the timeout knob the gpunetiordmawritelat* kernel functions surface per the verified common.h), and the safety overlay.

  • TASKS.md — step-by-step workflows for the in-scope

task verbs: install, configure, build, modify, run (smoke-before-bulk; single-iteration verification; reading the report columns), test (the eval loop — median / p99 / jitter / steady-state), debug (walk the error taxonomy layer by layer), use (how a latency result feeds a real-time class-of-workload decision), plus a Deferred task verbs block.

The skill assumes a host where DOCA is already installed, a CUDA Toolkit matched to the install is present, and the operator has whatever privileges the public install profile expects for binding a docadev, a docagpu, and an OOB TCP socket.

What this skill deliberately does not ship

This skill is agent guidance, not a samples or scripts bundle. It deliberately does not contain — and pull requests should not add:

  • Specific flag strings or expected latency numbers

beyond what the tool's shipped --help and main.c ARGP registration establish. The flag surface is small (device name, GPU PCIe address, GID index, server IP on the client side); the agent re-reads the binary's --help on the installed version.

  • **Pre-written DOCA GPUNetIO or CUDA kernel source

code** that would compete with the shipped tool tree. The shipped client/, server/, and common/ subtrees are the verified worked example.

  • Wrappers, parsers, or scripts in any language that

consume the tool's stdout. The output format is small and documented in [CAPABILITIES.md ## Observability](CAPABILITIES.md#observability).

  • A samples/, bindings/, or reference/ subtree.

This is a thin loader for a shipped tool tree.

Loading order

  1. Read this SKILL.md first to confirm the user's

question is in scope (the user actually wants to measure kernel-initiated WRITE latency through GPUNetIO, not the GPI variant, not the CPU-initiated variant, and not a library API question).

  1. **For what the tool measures, the surface-selection

rule against the GPI sister tool and the CPU-initiated perftest, the latency-vs-batching trade-off, the median / p99 / jitter reporting taxonomy, the version overlay, the error taxonomy, the observability surface, and the safety overlay, see [CAPABILITIES.md](CAPABILITIES.md).**

  1. **For step-by-step workflows — install,

configure, build, modify, run, test, debug, use — see [TASKS.md](TASKS.md).**

Related skills

  • [../../libs/doca-gpunetio/SKILL.md](../../libs/doca-gpunetio/SKILL.md) —

the library this tool wraps. The per-GPU docagpu context, the GPU-visible RDMA handles, the CUDA-side persistent-kernel pattern, the dual capability- discovery rule (DOCA cap-query AND cudaGetDeviceProperties), and the env preconditions (nvidiapeermem loaded, CUDA buffers registered with DOCA) live there.

  • [../../libs/doca-rdma/SKILL.md](../../libs/doca-rdma/SKILL.md) —

the underlying RDMA library. The RDMA queue this tool binds is created and connected via doca-rdma; the queue lifecycle, the transport type (RC vs UC vs UD), the permission matrix, and the connection method are owned there.

  • [../../libs/doca-verbs/SKILL.md](../../libs/doca-verbs/SKILL.md) —

the raw-verbs escape hatch beneath doca-rdma / doca-gpunetio. This tool stays on the higher-level surfaces.

  • [../doca-gpunetio-ib-write-bw/SKILL.md](../doca-gpunetio-ib-write-bw/SKILL.md) —

bandwidth analog of this tool on the same runtime framework. Same physical operation; different metric class (latency vs BW). The two together carry the full GPUNetIO-side latency / throughput picture.

  • [doca-gpi](../../libs/doca-gpi/SKILL.md) — the GPI

programming surface (CUDA-kernel-initiated RDMA). The alternative runtime framework for the same physical operation; doca/tools/ ships no GPI ibwritelat benchmark binary, so the GPI comparison is against the library surface, not a sibling tool. The selection rule in [CAPABILITIES.md ## Capabilities and modes](CAPABILITIES.md#capabilities-and-modes) is the decision aid; the agent's job is to teach when to pick which.

  • [doca-version](../../doca-version/SKILL.md) — the

canonical version-detection chain, four-way match rule. The ## Version compatibility section here is a thin overlay.

  • [doca-setup](../../doca-setup/SKILL.md) — env

preparation, install verification, GPU + CUDA Toolkit pairing, nvidia_peermem load, hugepages, NUMA, and the NGC DOCA container path.

  • [doca-public-knowledge-map](../../doca-public-knowledge-map/SKILL.md) —

routing to the public DOCA documentation set and the CUDA Toolkit pointer.

  • [doca-debug](../../doca-debug/SKILL.md) — the

cross-cutting debug ladder.

  • [doca-hardware-safety](../../doca-hardware-safety/SKILL.md) —

the bundle-wide hardware-safety meta-policy.