data-collection-sim

Data Collection Simulation

Purpose

Produce annotated static-scene synthetic data with Replicator writers (RGB, depth, segmentation, pose, Kitti/COCO variants) in headless batch runs.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Error / symptom	Cause	Solution
Extension or import not found	Wrong $ISAAC_SIM_DIR or stale build	Point env vars at _build/linux-x86_64/release or rebuild
Black or empty frames	Missing lights or non-RTX render mode	Add dome/key light; confirm RTX / PathTracing settings
Hang on stage load or first render	MDL compile or oversized stage	Follow isolation steps in isaac-sim-troubleshooting

Build headless SDG pipelines using Isaac Sim 6.0 Replicator. Outputs annotated frames (RGB, depth, segmentation, bbox, pose) to disk via writers.

Available Scripts

Script	Purpose	Arguments
scripts/minimal_sdg_pipeline.py	Minimal static-scene SDG pipeline using Isaac Sim Replicator	see script --help
scripts/shelf_pose_grid_capture.py	Shelf pose grid capture for static-scene SDG	CLI flags via argparse (see script --help)
scripts/validate_sdg_output.sh	Validate sdg output	positional args per script header (see script)
scripts/warehouse_sdg.py	Warehouse sdg	CLI flags via argparse (see script --help)

Running scripts

From agent runtimes that expose skill execution helpers, invoke helpers with run_script():

run_script("scripts/minimal_sdg_pipeline.py", args=["--help"])

From a built Isaac Sim tree, run the same file with ./python.sh (Linux) or python.bat (Windows) from _build/*/release, or execute shell helpers directly when they do not require the simulator.

Related Skills

• mobility-gen — mobile-robot trajectory recording then replay+render (two-phase, robot-mounted sensors)
• isaac-sim-sensor — sensor primitives (camera, LiDAR, IMU, contact)
• occupancy-map — produces occupancy maps for spawn placement

Architecture

Config (YAML/JSON) → SimulationApp (headless) → Scene Setup → Randomizers → Capture Loop → Writer → Disk

Required Imports

from isaacsim import SimulationApp
simulation_app = SimulationApp({"renderer": "RealTimePathTracing", "headless": True})

import carb.settings
import omni.replicator.core as rep
import omni.usd
import isaacsim.core.experimental.utils.stage as stage_utils
from isaacsim.core.experimental.utils.semantics import add_labels, remove_all_labels
from isaacsim.storage.native import get_assets_root_path

stage_utils.open_stage() / stage_utils.add_reference_to_stage() / stage_utils.define_prim() / stage_utils.get_current_stage() are the Kit 110 replacements for the legacy isaacsim.core.utils.stage.* and isaacsim.core.utils.prims.create_prim flow. Keep omni.replicator.core for SDG primitives.

Migration: for the full omni.isaac.* → isaacsim.* mapping, see Renaming Extensions.

Writers

Resolve writers via rep.WriterRegistry.get(name) (or rep.writers.get(name), same registry).

omni.replicator.core built-in writers (omni.replicator.core/scripts/writers_default/):

Writer	Use case
BasicWriter	RGB, bbox_2d (tight/loose), bbox_3d, semantic/instance/instance_id segmentation, depth (image-plane / camera), normals, occlusion, motion vectors, camera params, pointcloud, skeleton
KittiWriter	KITTI-format datasets
CocoWriter	COCO-format datasets (instance/bbox)
CosmosWriter	Cosmos warehouse video clips (PNG sequences + MP4 per modality: rgb, shaded_seg, segmentation, depth, edges). Requires /app/omni.graph.scriptnode/opt_in = True
FPSWriter	Frame-rate / capture-time telemetry
Custom Writer subclass	direct access to annotator tensors (subclass omni.replicator.core.Writer, register with rep.writers.register_writer)

isaacsim.replicator.writers adds Isaac-specific writers:

Writer	Use case
PoseWriter	6-DoF object pose estimation (optional write_debug_images=True)
DataVisualizationWriter	debug / overlay visualization

Deprecated and not for new work: DOPEWriter, YCBVideoWriter, PytorchWriter, PytorchListener (also OgnPose node). They will be removed in a future major release.

Writer initialization patterns

Two equivalent patterns are supported. Recent (Isaac Sim 6.0+) examples favor the explicit-backend form:

backend = rep.backends.get("DiskBackend")
backend.initialize(output_dir="/tmp/sdg_output")
writer = rep.writers.get("BasicWriter")
writer.initialize(backend=backend, rgb=True, bounding_box_2d_tight=True)
writer.attach(rp)

Legacy short form (still works; backend created implicitly from output_dir):

writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir="/tmp/sdg_output", rgb=True, bounding_box_2d_tight=True)
writer.attach(rp)

Annotators

Core annotators available via rep.annotators.get(name):

• rgb — RGBA uint8
• distance_to_image_plane — depth float32
• semantic_segmentation — per-pixel class labels
• instance_segmentation — per-pixel instance IDs
• bounding_box_2d_tight — tight 2D bboxes
• bounding_box_2d_loose — loose 2D bboxes
• bounding_box_3d — 3D bboxes in world coords
• camera_params — intrinsics, extrinsics, resolution
• occlusion — visibility ratio per instance
• normals — surface normals
• pointcloud — 3D point cloud from depth

Minimal Pipeline

run_minimal_sdg_pipeline(output_dir, num_frames, rt_subframes) — open a warehouse stage, tag a prop with semantic labels, create a camera, attach a BasicWriter, and run the capture loop.

See scripts/minimal_sdg_pipeline.py.

Full warehouse pipeline

For a config-driven, headless warehouse capture (YAML config, --num-frames, --output-dir), see scripts/warehouse_sdg.py. Validate the produced dataset with scripts/validate_sdg_output.sh:

bash scripts/validate_sdg_output.sh <output_dir> [expected_frames]

Domain Randomization

Use rep.functional API for randomization each frame:

# Scatter objects on surface
rep.functional.randomizer.scatter_2d(prims=objects, surface_prims=plane, check_for_collisions=True, rng=rng)

# Randomize camera pose
rep.functional.modify.pose(cam, position_value=rng.uniform(pos_min, pos_max),
                           look_at_value=target_prim, look_at_up_axis=(0, 0, 1))

# Randomize lights via OmniGraph events
rep.utils.send_og_event(event_name="randomize_lights")

Headless Execution

The canonical Isaac Sim Python launcher is python.sh (use python.bat on Windows). isaac-sim.sh launches the full editor app and is not the right entry point for standalone SDG scripts.

# $ISAAC_SIM_DIR is either the install root or <repo>/_build/linux-x86_64/release
"$ISAAC_SIM_DIR/python.sh" "$WORKSPACE_DIR/data_collection.py" --config config.yaml

Headlessness is controlled by SimulationApp({"headless": True}) inside the script, not by a launcher flag.

Or via the Isaac Lab runner ($ISAAC_LAB_DIR is your Isaac Lab checkout):

"$ISAAC_LAB_DIR/isaaclab.sh" -p data_collection.py --config config.yaml

Configuration Pattern

Use YAML config to parameterize everything:

resolution: [1280, 720]
rt_subframes: 32
num_frames: 100
headless: true
env_url: "/Isaac/Environments/Simple_Warehouse/full_warehouse.usd"
writer: BasicWriter
output_dir: /tmp/sdg_output
annotations:
  rgb: true
  bounding_box_2d_tight: true
  semantic_segmentation: true
  distance_to_image_plane: true
  bounding_box_3d: true
objects:
  - url: "/Isaac/Props/YCB/Axis_Aligned/003_cracker_box.usd"
    label: cracker_box
    count: 5
  - url: "/Isaac/Props/YCB/Axis_Aligned/008_pudding_box.usd"
    label: pudding_box
    count: 3

Validation Checklist

1. Output directory contains expected number of frames
2. RGB images are non-black (mean RGB > 30)
3. Annotation files match frame count
4. Semantic labels appear in segmentation maps
5. Bounding boxes have non-zero area
6. No NaN in depth maps

Key Rules

• Set rep.orchestrator.set_capture_on_play(False) for manual step control.
• rt_subframes: render the same frame multiple times to reduce ghosting from large pose deltas and to let materials/textures converge. Tune for your renderer: small (4-8) is often enough for RTX Real-Time + DLSS Quality; 16-32 is typical for path tracing or scenes with heavy material streaming.
• DLSS Quality: carb.settings.get_settings().set("rtx/post/dlss/execMode", 2). Recommended for SDG; default Performance mode can produce edge artifacts below ~600x600.
• Tag every prim you want annotated via add_labels(...) (taxonomy-aware) or rep.functional.modify.semantics(...). Both write the UsdSemantics.LabelsAPI schema.
• For static-scene SDG, pass delta_time=0.0 to rep.orchestrator.step so the timeline does not advance between captures.
• Call rep.orchestrator.wait_until_complete() before cleanup so the background backend has flushed everything to disk.
• Use rng = np.random.default_rng(seed) and rep.set_global_seed(seed) for reproducible randomization.
• Performance knobs for high-throughput captures (see sdg_getting_started_05.py):
  • rep.orchestrator.step(wait_for_render=False) decouples capture from render completion. Data may correspond to a previous frame; only use when strict frame-to-data correspondence is not required.
  • carb.settings.get_settings().set("/exts/omni.replicator.core/enableWriteToFabric", True) writes randomization deltas directly to Fabric instead of going through USD first. Faster, but transient — changes are not persisted in the USD stage.

Sibling SDG workflows

Workflow	Module / example
Mobile-robot trajectory record + replay	mobility-gen skill + isaacsim.replicator.mobility_gen
Grasp dataset generation	isaacsim.replicator.grasping (GraspingManager, GraspPhase) — source/standalone_examples/api/isaacsim.replicator.grasping/grasping_workflow_sdg.py
Episode record + replay	isaacsim.replicator.episode_recorder
Teleop record + replay	isaacsim.replicator.teleop
Reusable randomization behavior scripts (attached to prims, USD-persistent)	isaacsim.replicator.behavior
Sensor primitives (LiDAR, IMU, camera)	isaac-sim-sensor, isaac-camera
Spawn placement from a map	occupancy-map

Reference examples

Paths relative to $ISAAC_SIM_DIR (install root or <this-repo> for a source build) and $ISAAC_LAB_DIR:

• API examples: $ISAAC_SIM_DIR/source/standalone_examples/api/isaacsim.replicator.examples/sdg_getting_started_0[1-5].py, sdg_workflow_0[12].py, multi_camera.py, motion_blur_raytracing.py, motion_blur_pathtracing.py, cosmos_writer_simple.py, simready_assets_sdg.py, sdg_deformables.py, sdg_geomsubset.py, subscribers_and_events.py, custom_event_and_write.py, custom_fps_writer_annotator.py.
• Scene-based SDG: $ISAAC_SIM_DIR/source/standalone_examples/replicator/scene_based_sdg/
• Object-based SDG: $ISAAC_SIM_DIR/source/standalone_examples/replicator/object_based_sdg/
• Augmentation pipelines: $ISAAC_SIM_DIR/source/standalone_examples/replicator/augmentation/
• Cosmos warehouse writer: $ISAAC_SIM_DIR/source/standalone_examples/replicator/cosmos_writer_warehouse.py
• Infinigen SDG: $ISAAC_SIM_DIR/source/standalone_examples/replicator/infinigen/
• Grasping SDG: $ISAAC_SIM_DIR/source/standalone_examples/api/isaacsim.replicator.grasping/grasping_workflow_sdg.py
• Isaac Lab imitation learning: $ISAAC_LAB_DIR/scripts/imitation_learning/