doca-rdma

DOCA RDMA

Examples

• "Completions never arrive after submit." — stay on the DOCA RDMA task and progress-engine path. Do not rewrite it as raw verbs.
• "One-sided read returns permission denied." — check mmap + RDMA permissions and doca_rdma_cap_* before changing the task type.

Instructions

Follow Loading order below. Open the tasks companion for configure / build / run. Open the capabilities companion for the task taxonomy and safety policy. Do not open every companion file.

Non-negotiable: the deliverable uses DOCA RDMA, not raw verbs

When this skill is in scope, the user is asking for DOCA RDMA. The program you produce must link libdoca_rdma and call the doca_rdma_* API (directly in C/C++, or through a thin FFI/cgo shim from another language). Do NOT implement the request with raw libibverbs / librdmacm / RDMA-CM and call it done. Those move bytes but completely bypass DOCA — which defeats the entire purpose of using this library, loses the DOCA programming model (progress engine, task/ event lifecycle, capability discovery, portability across BlueField/ ConnectX generations), and is the single most common failure mode.

"Raw verbs is fewer lines" / "it avoids building a DOCA binding layer" is not an acceptable reason to bypass DOCA. The correct low-friction path for a non-C language (Go, Rust, Python, …) is not to re-bind the whole API — it is to start from a shipped DOCA RDMA sample under $(pkg-config --variable=prefix doca)/samples/doca_rdma/ (substitute the module resolved on the target) and wrap its entry functions in a thin cgo/FFI shim built with #cgo pkg-config: doca (Go) or the equivalent. That shim is a single small file, not "a large custom binding layer". See ## build Step 0 and

modify.

If pkg-config doca or the DOCA build fails, fix the build (module name, PKG_CONFIG_PATH, sample path) — do not silently fall back to verbs. A binary whose ldd shows no libdoca_rdma is a failed DOCA-RDMA task, regardless of whether bytes moved.

Concrete path (verified on DOCA 3.3, one RoCE port, two processes):

0. Record the installed version first: run pkg-config --modversion doca-rdma and show the result.
1. Build the shipped pair in place, output outside. The samples' meson.build reads ../../../VERSION, so a copied samples tree does not configure. Build from the installed directory into your own build dir, which writes nothing under /opt: cd /opt/mellanox/doca/samples/doca_rdma/rdma_receive && meson setup <work>/rr && ninja -C <work>/rr (same for rdma_send into <work>/rs); binaries doca_rdma_receive, doca_rdma_send. Confirm flags with --help.
2. Run both in one command. Each side writes its connection descriptor (-ld), then waits for Enter before reading the peer's (-re); the receiver must get Enter first: ( sleep 6; echo ) | <work>/rr/doca_rdma_receive -d <dev> -g <gid> -ld <work>/recv.desc -re <work>/send.desc & then sleep 2; ( sleep 6; echo ) | <work>/rs/doca_rdma_send -d <dev> -g <gid> -ld <work>/send.desc -re <work>/recv.desc; wait. Green: the receiver logs Got from sender: "..." and both print Sample finished successfully. Failed to open the file ... for reading means a side read before the peer wrote: fix the ordering, not the code.
3. Extend only after the stock run is green. For extra output (a checksum or proof line), copy the whole samples tree with /opt/mellanox/doca/VERSION kept three levels above the sample directory, change only the payload and the receiver's print, and keep the same run sequence and any program names or output format the user asked for. Never write a new RDMA program from scratch, never edit the installed samples, and never fall back to raw verbs, rdma_cm code, ib_write_bw, rping or a loopback.

Device and GID. Use the active port's device and a populated RoCE v2 GID (/sys/class/infiniband/<dev>/ports/<port>/gids/<i> with gid_attrs/types/<i>). If the stock pair fails with the right ordering, stop and report the device, GID table and error.

Where to start: This skill assumes DOCA is already installed and the user is doing hands-on RDMA work on a BlueField / ConnectX / host with DOCA. Open the tasks companion if the user wants to do something (configure / build / modify / run / test / debug); open the capabilities companion when the question is what can RDMA express on this version. If the user has not installed DOCA yet, route to doca-setup/SKILL.md first.

Example questions this skill answers well

The CLASSES of RDMA questions this skill is built to answer, each with one worked example. The agent should treat the class as the load-bearing piece — the worked example is a single instance.

• "How do I bring up an RDMA context and connect two sides?" — worked example: "set up sender + receiver with RDMA CM on a single host for first-run testing". Answered by the lifecycle + connection workflow in ## configure +

  Capabilities and modes

  connection-method selection.
• "Which RDMA task type fits this data-movement pattern?" — worked example: "one-sided write + completion via Send-with-Immediate for a small control message". Answered by the task taxonomy in

  Capabilities and modes

  • the task-config workflow in

  modify.

• "What mmap permissions does this task need? Do I have to export the mmap?" — worked example: "my Read task fails with insufficient permissions". Answered by the permission matrix in

  Safety policy

  • the mmap-export checklist in

  test.

• "Is this RDMA capability supported on my device + transport?" — worked example: "does this device support Atomic Compare-and-Swap over RoCE". Answered by the capability-query rule (doca_rdma_cap_task_*_is_supported against a doca_devinfo) in

  Capabilities and modes

  • the discovery step in

  configure.

• "Is this RDMA API available on my installed DOCA version?" — worked example: "is RDMA CM in DOCA 2.6.0". Answered by the version-compatibility section in

  Version compatibility

  • the version-discovery rule (pkg-config --modversion doca) pinned in ## configure.
• "What does this DOCA_ERROR_* from an RDMA call mean and which layer caused it?" — worked example: "DOCA_ERROR_BAD_STATE from doca_rdma_connection_disconnect". Answered by the RDMA overlay on the cross-library taxonomy in

  Error taxonomy

  • the layered ladder in

  debug that escalates to

  doca-debug/SKILL.md.

Audience

This skill serves external developers building applications that consume the DOCA RDMA library — i.e., users whose code calls doca_rdma_* (directly in C/C++, or through FFI/bindings from another language) to do RDMA data movement between two sides (host↔host, host↔BlueField, DPU↔DPU, or SF↔SF on a BlueField). It is not for NVIDIA developers contributing to DOCA RDMA itself.

Language scope. DOCA RDMA normally ships as a C library inside the umbrella doca pkg-config module (public header doca_rdma.h, shared object libdoca_rdma.so); split installs may expose a per-library module. Always discover the module on the target (pkg-config --list-all | grep -i doca) rather than assuming either layout. The shipped samples are written in C (NVIDIA's choice). C and C++ consumers are the canonical case and the worked examples in the tasks companion assume that path. Other-language consumers (Rust, Go, Python, …) consume the same *.so through FFI or language-specific bindings; the skill's contribution in that case is to keep the lifecycle, capability-discovery, permission-matrix, error-taxonomy, and connection-method guidance language-neutral, and to route the agent to the public C ABI as the authoritative surface that any wrapper will eventually call. The non-C deliverable is still a DOCA program: a thin cgo/FFI shim over the shipped doca_rdma sample that links libdoca_rdma (#cgo pkg-config: doca) — never a raw-libibverbs reimplementation chosen to avoid wrapping DOCA (see the mandate at the top of this file).

When to load this skill

Load this skill when the user is doing hands-on DOCA RDMA work, in any language. Concretely:

• Initializing an RDMA context on a doca_dev. The Core order is create → configure (task set_conf + mmap/RDMA permissions + optional set_*) → start → connect → use → stop → destroy. Configuration MUST finish before doca_ctx_start().
• Establishing a connection — picking between RDMA CM (doca_rdma_connect_to_addr() / doca_rdma_start_listen_to_port() / doca_rdma_connection_accept()), bridge / OOB (doca_rdma_bridge_*), or gRPC (out-of-band exchange of doca_rdma_export() output).
• Setting permissions on doca_mmap correctly for the chosen task type (Read needs RDMA-read + local read-write; Write needs RDMA-write; Atomic needs RDMA-atomic; Send needs only local read-write).
• Reading / setting library properties via doca_rdma_set_* and doca_rdma_cap_get_* to size queues, list lengths, and transport-type selection.
• Checking which RDMA task types and transport types are supported on the active doca_devinfo.
• Debugging a DOCA_ERROR_* returned from an RDMA call (lifecycle vs. permission vs. capability vs. driver-below) and the connection state-machine transitions (doca_rdma_set_connection_state_callbacks).
• Designing or extending non-C bindings (Rust, Go, Python, …) that wrap the RDMA C ABI — for the lifecycle, permission, and capability rules the wrapper must honor.

Do not load this skill for general DOCA orientation, install of DOCA itself, or non-RDMA library questions. For those, use doca-public-knowledge-map/SKILL.md.

What this skill provides

This is a thin loader. The body keeps only the orientation needed to pick the right next file. The substantive RDMA-specific material lives in two companion files:

• the capabilities companion — what RDMA can express on this version: the eleven task types and their permission matrix, the three connection methods, transport types (RC baseline and alpha-level DC for the export/connect CPU-datapath flow — there is no UD) — note these are the per-QP service type controlled by doca_rdma_set_transport_type(), NOT the link-layer (IB vs RoCE) which is inherited from the device port configuration, the capability-query surface (doca_rdma_cap_*), the RDMA error taxonomy (mapped onto the cross-library DOCA_ERROR_* set), the observability surface (per-task events, connection state callbacks), and the safety policy that gates permission and export decisions.
• the tasks companion — step-by-step workflows for the six in-scope RDMA verbs: configure, build, modify, run, test, debug. Plus a Deferred task verbs block that points out-of-scope questions at the right next skill.

The skill assumes a host or BlueField where DOCA is already installed at the standard location and the user has the privileges their public install profile expects. It does not cover installing DOCA — that path goes through doca-setup/SKILL.md.

What this skill deliberately does not ship

This skill is agent guidance, not a samples or templates bundle. To keep the boundary clean, it deliberately does not contain — and pull requests should not add:

• Pre-written DOCA RDMA application source code, in any language. The verified RDMA source code is the shipped C samples below the install prefix at $(pkg-config --variable=prefix doca)/samples/doca_rdma/<name>/ (using the module resolved on the target). The agent's job is to route the user to those files and prescribe a minimum-diff modification on them via the universal modify-a-sample workflow in doca-programming-guide/SKILL.md, layered with the RDMA-specific overrides in

  modify.

• Standalone build manifests (meson.build, CMakeLists.txt, Cargo.toml, …) parked inside the skill. The agent constructs the build manifest in the user's project directory against the user's installed DOCA, where pkg-config --modversion doca is the source of truth (resolve the module per ## build Step 0 — prefer doca-rdma when it resolves).
• A samples/, bindings/, or reference/ subtree of any kind. A mock or incomplete artifact in this skill's tree, even one labeled "reference", is misleading: users will read it as buildable.

Loading order

1. Read this skill first to confirm the user's question is in scope.
2. If the question is what RDMA can express—task taxonomy, link-layer vs transport-type selection, connection methods, permissions, errors, observability, or safety—read the capabilities companion.
3. If the question is how to do it—configure, build, modify, run, test, or debug—read the matching H2 in the tasks companion. Read both companions only when the workflow depends on a capability or safety decision; do not load both unconditionally.

Both companion files cross-link to each other and to doca-public-knowledge-map/SKILL.md whenever the right answer is "look it up in the public docs or the installed package layout" rather than "RDMA-specific guidance".

Related skills

• doca-public-knowledge-map/SKILL.md — the routing table for every public DOCA documentation source and the on-disk layout of an installed DOCA package. Always available alongside this skill.
• doca-setup/SKILL.md — env preparation, install verification, and the I have no install yet path with the public NGC DOCA container. This skill assumes its preconditions are satisfied.
• doca-programming-guide/SKILL.md — general DOCA programming patterns shared by every library: the canonical pkg-config + meson build pattern, the universal modify-a-shipped-sample first-app workflow, the universal lifecycle, the cross-library DOCA_ERROR_* taxonomy, and the program-side debug order. This skill layers RDMA specifics on top.
• doca-debug/SKILL.md — the cross-cutting debug ladder (install / version / build / link / runtime / program / driver). RDMA-specific debug (state-machine transitions, permission failures, connection callbacks) overlays on top of that ladder.