rocprofiler-systems-discovering-profiling-options

Discovering Profiling Options

Maps a "what's available" or "how do I configure X" question to the exact rocprof-sys-avail / rocprof-sys-run --help=<topic> command that answers it. rocprof-sys-avail is self-updating and authoritative — prefer running it over reciting static documentation.

Prerequisites

A working rocprof-sys-avail binary, built/installed alongside rocprof-sys-run/rocprof-sys-sample. If it is not on PATH, run source <install-prefix>/share/rocprofiler-systems/setup-env.sh first.

Discovery command reference

User question	Command	Notes
What settings/env vars exist, with descriptions?	rocprof-sys-avail -S -bd	-b suppresses the current-value/availability column, -d adds the description column
What hardware counters are available on my system?	rocprof-sys-avail -H -bd -A	-A/--available restricts to counters this machine can actually use; add -c CPU or -c GPU to split by device
What ROCm APIs can be traced?	rocprof-sys-avail -bd -r ROCM_DOMAINS	Regex-filters settings to the ROCPROFSYS_ROCM_DOMAINS row, listing every valid domain (hip_runtime_api, hsa_api, kernel_dispatch, rccl_api, kfd_events, ...)
What operations/API calls exist within one ROCm domain?	rocprof-sys-avail --list-domains then rocprof-sys-avail --list-operations <domain>	e.g. --list-operations hip_runtime_api lists individual HIP calls. Only domains printed by --list-domains are accepted: composites must be expanded (hip_api → hip_runtime_api, hip_compiler_api; hsa_api → hsa_core_api, hsa_amd_ext_api, hsa_image_ext_api, hsa_finalize_ext_api; kfd_events → the kfd_* domains; roctx → marker_api), and kernel_dispatch has no operation list
What Timemory components can I collect?	rocprof-sys-avail -C -bd	add -A to restrict to components usable on this build
What settings exist for one topic (tracing, sampling, output, ...)?	rocprof-sys-avail --list-categories then rocprof-sys-avail -S -bd -c settings::<category>	category names don't always match plain English — it's settings::trace, not settings::tracing — so always check --list-categories first
What GPU power/temp/utilization metrics can I sample?	rocprof-sys-avail -bd -r AMD_SMI_METRICS	surfaces ROCPROFSYS_AMD_SMI_METRICS choices (busy, temp, power, mem_usage, ...)
What CLI flags does rocprof-sys-run have for topic X?	rocprof-sys-run --help=<topic>	topics: preset, general, tracing, profiling, output, sampling, process, counters, backend, execution, debug, misc, gpu, cpu, rocm, parallel
How do I persist chosen settings across runs?	rocprof-sys-avail -G ~/.rocprof-sys.cfg [--all] then export ROCPROFSYS_CONFIG_FILE=~/.rocprof-sys.cfg	--all includes descriptions/categories in the generated file; formats are txt (default), json, xml via -F

Terminology

"Trace" and "tracing" overlap in the output, so map the name you see to what it controls before answering:

Name you see	What it controls
ROCPROFSYS_TRACE, settings::trace and --help=tracing (its tuning options)	Perfetto trace output
ROCPROFSYS_USE_ROCPD	rocPD output
ROCPROFSYS_PROFILE, --help=profiling	timemory profile output
ROCPROFSYS_USE_SAMPLING, --help=sampling	call-stack sampling
ROCPROFSYS_ROCM_DOMAINS, --help=rocm	ROCm API tracing
ROCPROFSYS_USE_MPIP, _RCCLP, _OMPT, _KOKKOSP	MPI, RCCL, OpenMP and Kokkos API tracing

"Tracing" means API tracing (recording the calls of one API); "trace output" means the Perfetto file. A question like "what tracing options are there?" usually means the ROCm domains, so ask or answer both if it is unclear.

How to answer a discovery question

1. Identify which axis the question is about: settings/env-vars, Timemory components, hardware counters, ROCm API domains, GPU metrics, or CLI flags.
2. Pick the matching row from the table above.
3. Run the command (or hand it to the user) and report the relevant rows — don't dump the entire table if the question was narrow; combine with -r <regex> / -c <category> / -A to narrow first.
4. If the result is still broad, suggest adding -A/--available (installed/ supported only) or a category/regex filter.

Persisting configuration

Once the right settings are identified, freeze them into a config file instead of re-typing environment variables every run:

rocprof-sys-avail -G ~/.rocprof-sys.cfg --all   # --all adds descriptions/categories
export ROCPROFSYS_CONFIG_FILE=~/.rocprof-sys.cfg

Select the format with -F txt|json|xml; they load differently:

• txt (default) and xml: load with ROCPROFSYS_CONFIG_FILE.
• json: this is a hierarchical preset file. It does not load through ROCPROFSYS_CONFIG_FILE (the run fails with "missing the expected 'rocprofiler-systems' root object"); load it with rocprof-sys-run --preset=<file>.json -- <app>.

When the same setting is given in more than one place, the order of priority is: config file, then environment variable, then command-line parameter, each overriding the one before. For example, with ROCPROFSYS_SAMPLING_FREQ = 50 in the file, ROCPROFSYS_SAMPLING_FREQ=77 in the environment beats it, and rocprof-sys-run --sampling-freq=100 -- <app> beats both.

Using what you discover

Names found with the commands above are applied through these settings:

• CPU counters (-H -c CPU): set ROCPROFSYS_PAPI_EVENTS, ROCPROFSYS_PROFILE=ON and add papi_array (or papi_vector) to ROCPROFSYS_TIMEMORY_COMPONENTS, e.g. wall_clock,papi_array; results are written to papi_array*.txt/json (or papi_vector*). Without the component the run succeeds but writes no counter files. All events must come from one namespace — mixing PAPI_* presets with perf::* events only prints a "Failure to add named event" warning and drops the counters, the run still exits 0.
  • Host CPU counters read through Linux perf_event need /proc/sys/kernel/perf_event_paranoid <= 2. That limit does not apply to metrics read from other sources, such as the PAPI net::: network events, which read /proc/net/dev.
• CPU overflow sampling: sample on a hardware event instead of a timer. Set ROCPROFSYS_USE_SAMPLING=ON, ROCPROFSYS_SAMPLING_OVERFLOW=ON, ROCPROFSYS_SAMPLING_OVERFLOW_EVENT=<event> (list events with rocprof-sys-avail -H -c CPU -r overflow) and a large ROCPROFSYS_SAMPLING_OVERFLOW_FREQ (events per sample, e.g. 10000000). Setting only the event does nothing (it falls back to CPU-time sampling), and leaving the interval at its default of 300 events aborted the run when tested. It also needs perf_event_paranoid <= 2.
• GPU counters (-H -c GPU): names are listed per device, e.g. FETCH_SIZE:device=0. Set ROCPROFSYS_ROCM_EVENTS for kernel-dispatch counters (writes rocprof-device-<N>-<counter>.txt/json), or ROCPROFSYS_GPU_PERF_COUNTERS for polled (PMC) sampling. A name without the :device=N suffix is collected on every device.
• GPU metrics (power, temperature, utilization): collected via AMD SMI. ROCPROFSYS_AMD_SMI_METRICS selects which metrics to enable.
  • By default, busy,temp,power,mem_usage are enabled.
  • The value all collects everything.
  • ROCPROFSYS_USE_AMD_SMI=OFF disables this backend entirely.
• ROCm API tracing: set ROCPROFSYS_ROCM_DOMAINS to a comma-separated list, e.g. hip_runtime_api,kernel_dispatch,memory_copy.
  • Copy names exactly from rocprof-sys-avail -bd -r ROCM_DOMAINS: one unknown name will fail produce a failure
  • kfd_events also needs HSA_XNACK=1 and an XNACK-capable GPU; check with rocminfo | grep -i xnack (a xnack-/xnack+ suffix on the GPU target means capable). On a GPU without XNACK it records nothing and prints nothing.

Common Mistakes

Mistake	Fix
Guessing a category name for -c (e.g. settings::tracing)	Run rocprof-sys-avail --list-categories first — category names don't always match the obvious English word (it's settings::trace)
Dumping the full -S/-C/-H table for a narrow question	Combine with -r <regex> or -c <category>/-A to filter before reporting
Confusing this skill with picking a profiling preset	Preset selection is choosing-a-profiling-preset; this skill is for exploring/fine-tuning individual options, including on top of a chosen preset
Typing a ROCm domain name from memory	An unknown ROCPROFSYS_ROCM_DOMAINS name silently disables the whole list; copy names from rocprof-sys-avail -bd -r ROCM_DOMAINS
Forgetting -A/--available	Without it, -H/-C list counters/components the current build or hardware doesn't actually support, which can overwhelm the answer

rocprof-sys-avail is the source of truth for names and descriptions. For topics beyond option discovery (unified memory profiling, causal profiling), see the online docs: https://rocm.docs.amd.com/projects/rocprofiler-systems/en/develop/how-to/configuring-runtime-options.html