isaac-camera

Camera (Isaac Sim 6 / Kit 110)

Purpose

Create and configure USD cameras, render products, intrinsics, AOVs, and lens distortion models for Isaac Sim 6 sensor pipelines.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Error / symptom	Cause	Solution
Extension or import not found	Wrong $ISAAC_SIM_DIR or stale build	Point env vars at _build/linux-x86_64/release or rebuild
Black or empty frames	Missing lights or non-RTX render mode	Add dome/key light; confirm RTX / PathTracing settings
Hang on stage load or first render	MDL compile or oversized stage	Follow isolation steps in isaac-sim-troubleshooting

Two layers stack:

1. UsdGeomCamera — the USD camera prim (focal length, apertures, clipping, transform).
2. isaacsim.sensors.experimental.rtx — the runtime sensor wrapper (authoring class + runtime sensor class). Use this for any new code that captures frames, attaches annotators, or applies lens distortion.

The legacy isaacsim.sensors.camera.Camera class still works but the extension itself is a stub (no Python). The implementation moved to isaacsim.sensors.experimental.rtx; new code should use that.

Migration: see the Migration from isaacsim.sensors.camera.Camera table below, plus the official camera migration guide for the full API change set.

When to use

• Create or configure camera prims in USD stages.
• Attach RGB / depth / segmentation / bbox annotators for capture.
• Apply lens distortion (OpenCV pinhole, OpenCV fisheye, LUT).
• Convert real-world calibration (OpenCV / ROS) to Omniverse units.
• Build stereo, multi-sensor, or tiled multi-camera rigs.
• Animate cameras (keyframes, motion paths, orbit, fly-through).
• Generate synthetic data with Replicator randomization.
• Set up Metropolis cameras for MEGA scenarios.

Fundamentals

Unit system

OpenUSD expresses optical properties in tenths of a scene unit.

Property	Units	Notes
Focal length	tenths of scene unit	35 = 35 mm when scene units are cm
Horizontal aperture	tenths of scene unit	sensor / film width
Vertical aperture	tenths of scene unit	sensor / film height
Focus distance	scene units	perfectly-sharp distance
Clipping range	scene units	near / far clip planes

Meter stage: a 25 mm focal length is 0.025.

Coordinate system

Camera looks down -Z; +Y is up, +X is right. Same regardless of stage up-axis.

Creating and capturing — RtxCamera + CameraSensor

RtxCamera + CameraSensor is the way to create a camera and capture from it in Isaac Sim 6. RtxCamera wraps a UsdGeom.Camera prim with OmniSensorAPI applied; CameraSensor builds a Replicator render product on top and exposes annotators. Both extend the isaacsim.core.experimental.prims.XformPrim family, so the standard pose / transform methods apply.

from isaacsim.sensors.experimental.rtx import RtxCamera, CameraSensor
import isaacsim.core.experimental.utils.app as app_utils

cam = RtxCamera(
    "/World/cam",
    tick_rate=30.0,                       # 0 = autotrigger
    aux_output_level="NONE",              # RtxCamera supports NONE only
)
cam.camera.set_focal_lengths(24.0)
cam.camera.set_clipping_ranges(0.01, 1000.0)

sensor = CameraSensor(
    cam,
    resolution=(480, 640),                # (height, width)
    annotators=["rgb", "distance_to_image_plane"],
)
app_utils.play(commit=True)
data = sensor.get_data("rgb")

Author the pose directly on RtxCamera (it's an XformPrim): pass positions= / translations= / orientations= (quaternion wxyz) to the constructor, or call set_world_poses(...) afterwards.

cam.set_world_poses(positions=[[2.0, 2.0, 2.0]],
                    orientations=[[1.0, 0.0, 0.0, 0.0]])   # wxyz

TiledCameraSensor batches many cameras into one render call; useful for multi-view RL data collection. SingleViewDepthCameraSensor simulates stereo depth via OmniSensorDepthSensorSingleViewAPI.

Lens distortion (OpenCV fisheye / pinhole)

Set via API schemas + USD attributes at authoring time. The schema and attribute names are stable; use them through RtxCamera's schemas and attributes constructor args, or Apply them on the camera prim directly.

from isaacsim.sensors.experimental.rtx import RtxCamera
from pxr import Gf

cam = RtxCamera(
    "/World/fisheye",
    schemas=["OmniLensDistortionOpenCvFisheyeAPI"],
    attributes={
        "omni:lensdistortion:opencvFisheye:fx": 500.0,
        "omni:lensdistortion:opencvFisheye:fy": 500.0,
        "omni:lensdistortion:opencvFisheye:cx": 640.0,
        "omni:lensdistortion:opencvFisheye:cy": 360.0,
        "omni:lensdistortion:opencvFisheye:k1": 0.05,
        "omni:lensdistortion:opencvFisheye:imageSize": Gf.Vec2i(1280, 720),
    },
)

Equivalent OpenCV-pinhole schema: OmniLensDistortionOpenCvPinholeAPI (attributes omni:lensdistortion:opencvPinhole:{fx,fy,cx,cy,k1,k2,p1,p2,...}). LUT-based distortion uses OmniLensDistortionLutAPI.

Animation

from pxr import Usd, Gf

for frame, pos in enumerate(camera_positions):
    cam.GetPrim().GetAttribute("xformOp:translate").Set(
        Gf.Vec3d(*pos), Usd.TimeCode(frame)
    )

Replicator randomization

import omni.replicator.core as rep

with rep.new_layer():
    cameras = rep.get.prims(path_pattern="/World/Camera.*")
    with rep.trigger.on_frame(num_frames=200):
        with cameras:
            rep.modify.pose(
                position=rep.distribution.uniform((-5, -5, 2), (5, 5, 8)),
                look_at="/World/Target",
            )

Sensor checker / supported configs

For asset-driven lidar / camera configs and validation helpers, see isaacsim.sensors.experimental.rtx.SUPPORTED_LIDAR_CONFIGS and the sensor_checker module bundled with the extension. The Replicator AOV annotator names match the standard registry: rgb, distance_to_image_plane, distance_to_camera, normals, semantic_segmentation, instance_segmentation, bounding_box_2d_tight, bounding_box_2d_loose, bounding_box_3d, camera_params, pointcloud, occlusion.

Migration from isaacsim.sensors.camera.Camera

Full guide: isaacsim.sensors.camera → isaacsim.sensors.experimental.rtx.

Old	New
isaacsim.sensors.camera.Camera("/World/cam", resolution=(1280, 720))	RtxCamera("/World/cam") + CameraSensor(cam, resolution=(720, 1280), annotators=["rgb"])
camera.set_focal_length(24)	cam.camera.set_focal_lengths(24.0)
camera.set_clipping_range(0.01, 1000)	cam.camera.set_clipping_ranges(0.01, 1000)
camera.initialize() then camera.get_current_frame()	app_utils.play(commit=True) then sensor.get_data(annotator)
Carb-settings distortion (/<path>/distortionModel)	OmniLensDistortion*API schemas on the camera prim