isaac-sim-validator

Isaac Sim Validator

WARNING: Level 3 executes the target script with your full user privileges. There is no container or sandbox isolation. Only use --confirm-execute on scripts you fully trust.

Purpose

Reject common delivery defects at the owning boundary: standalone library or Kit application/extension.

Levels 1–2 are static checks only. Level 3 runs the target Python script under timeout with the calling user's privileges. Level 3 requires an explicit --confirm-execute (or CONFIRM_EXECUTE=1) opt-in; it is not a sandbox.

Prerequisites

• Standalone libraries need their configured build or installed test environment.
• Kit checks need a built Isaac Sim tree; a GPU is required only for GPU-backed checks.

Limitations

• Levels 1–2 cannot prove runtime correctness for every workload.
• Level 3 executes untrusted script content only after --confirm-execute; there is no container/firejail isolation.
• Thresholds target common demo/SDG failures, not all product features.

Troubleshooting

Error / symptom	Cause	Solution
Kit extension not found	Wrong $ISAAC_SIM_DIR or stale build	Point env vars at _build/linux-x86_64/release or rebuild
Black or empty frames	Missing lights or non-RTX render mode	Add dome/key light; confirm RTX / PathTracing settings
Hang on stage load or first render	MDL compile or oversized stage	Follow isolation steps in isaac-sim-troubleshooting

Final gate before delivery. Reject outputs that fail the checks below; do not soften results.

For a standalone library, validate a normal-process public import, declared package/native closure, relocation, SDK consumer when applicable, and isaacsim.common.logging.Logger diagnostics. Do not require SimulationApp, rendering, or a GPU.

Kit-only validation helper

Script	Purpose	Arguments
scripts/validate_sim.sh	Validate sim	positional args per script header (see script)

Running scripts

From agent runtimes that expose skill execution helpers, invoke helpers with run_script():

run_script("scripts/validate_sim.sh", args=[])

From a built Isaac Sim tree, run the same file with ./python.sh (Linux) or python.bat (Windows) from _build/*/release, or execute shell helpers directly when they do not require the simulator.

Kit simulation checklist

Check	Pass	On fail
Script path	exists, readable	reject: Script path invalid: <path>
File size	> 1 KB	reject: Script empty or corrupted
isaacsim import	uses isaacsim.*, not omni.isaac.core	reject: Use isaacsim.* namespace; omni.isaac.core is deprecated
SimulationApp config	explicit width and height	reject: SimulationApp without window args causes swapchain-capture size mismatch
Lights present	DomeLight (intensity >= 100) and DistantLight (>= 500)	reject: No lighting; render will be black
Render mode	RayTracedLighting for iteration; PathTracing only for hero shots	reject: PathTracing too slow for iteration; use RayTracedLighting
ACES tonemap	carb rtx.post.tonemap.op = 4; filmIso 200 default / 600 deep-aisle / 400 aerial	suggest: Enable ACES (op=4) for contrast
Final render	>= 150 KB and mean_RGB > 30	reject: Render is black or low-energy; check lighting
User paths	no /home/<user>/, hardcoded C:\path\to\..., or per-agent home paths; prefer %USERPROFILE%, env vars, or argparse	reject: Hardcoded user path; use $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR, or argparse
Network endpoints	no bare IPs (192.168.x.x, 10.x.x.x) or non-localhost hosts	reject: Hardcoded endpoint; parameterize via env var or config
Output dir	configurable (env var, CLI, or function param)	reject: Output dir hardcoded; accept as parameter or read $OUTPUT_DIR
VRAM headroom	nvidia-smi headroom > 20% after settle	warn at >90% utilization; likely leak or scale issue
pkill -f "kit/kit" used	flag for kill+relaunch bug	warn: Verify zero kit processes before next launch
Sim duration	>= 3 s of physics	reject: Sim ran < 3 s; physics has not settled

File structure validation (USD scenes)

Item	Valid	Invalid	Action
$ISAAC_LAB_DIR/scripts/	read-only reference	writing outputs there	redirect to $OUTPUT_DIR or $WORKSPACE_DIR/outputs/
get_assets_root_path() -> s3://...	runtime OK	baked into delivered output	cache locally; reference via configurable path
Melting / Splat MDL-only shaders	OK in raw form	MDL-only at export	filter out; MDL-only causes black render
MakeInstanceable for RL	required (make_instanceable: true)	absent	reject
usdPreviewSurface on surfaces	required	absent	reject; suggest adding UsdPreviewSurface
convex_decompose_mesh	false (with clean_sdf_dc)	true	recommend false for stability
mpu=0.01	OK for warehouse shell (13 m tiles)	on a robot	reject

Render validation (frame analysis)

Metric	Good	Bad	Action
File size	1–4 MB	~82 KB	82 KB = black, reject
Mean RGB	80–160	< 20	reject: Too dark; add lights
Max RGB	200–250	0	reject: No light in scene
Pixel variance	> 15	< 5	flat = missing shadows; add accents
Color histogram	multi-modal	single peak	overexposed or flat lighting; adjust intensity
Object visibility	>= 3 robots or 50+ assets	< 10 visible	suggest: Scene too sparse; add set dressing
Shadow consistency	consistent shadows	random black blobs	reject: Shadow artifacts; check light cone bounds

Video validation

For any delivered MP4/GIF, extract and inspect at least start/middle/end frames. Reject or recut the presentation copy if the first frames show render warm-up, the subject is occluded, or the camera loses the task. Keep the complete source render and metrics when a shorter presentation cut is made.

RL training validation

Issue	Action
AppLauncher not first	reject: AppLauncher must precede any isaaclab imports
Task __init__.py non-empty	reject: Task __init__.py must be empty; do not import configs
omni.isaac.lab import	reject: Use isaaclab.*; omni.isaac.lab is deprecated
PhysX backend (when Newton applies)	reject: Use Newton; it is the default on Kit 110
Missing PYTHONUNBUFFERED=1	warn: Set PYTHONUNBUFFERED=1 to see logs in real time
No GPU memory logging	suggest: Log torch.cuda.memory_allocated()/1e6 to W&B
Batch size exceeds VRAM	suggest: Reduce num_envs (try 2048)
Wrong checkpoint name	reject: Must be model_XXXX.zip, not policy.pt
Single seed	suggest: Run seeds 42, 123, 456 for valid comparison

Asset & shell validation

Input	Check	Action
docker run command	-v mount for /workspace/output	reject: Output folder not mounted
Sub-agent spawn	workdir=$WORKSPACE_DIR	reject: Sub-agent workdir hardcoded; use $WORKSPACE_DIR
Blender USD export	correct mpu, dual-shader (UsdPreviewSurface + MDL), correct export type	reject if any missing
Scene complexity	renders within 10 settle steps	reject: Scene too complex; simplify assets
Ghost artifacts	cleanup of /dev/shm/carb-*	reject: Use workspace_cleanup.py to clear /dev/shm/carb-*

Kit decision tree

graph TD
    A[Received Output] --> B{Has script?}
    B -->|No| C[Reject: "No script provided"]
    B -->|Yes| D{File size >1KB?}
    D -->|No| E[Reject: "Script empty or corrupted"]
    D -->|Yes| F{Python?}
    F -->|Yes| G{isaacsim.* imports?}
    G -->|No| H[Reject: "Use isaacsim.* not omni.isaac.core"]
    G -->|Yes| I{SimulationApp or --exec?}
    I -->|Neither| J[Reject: "Must use SimulationApp with window args OR isaac-sim.sh --exec"]
    I -->|SimulationApp| K{width and height set?}
    K -->|No| L[Reject: "SimulationApp requires explicit width and height"]
    K -->|Yes| M{Dome + Distant lights?}
    M -->|No| N[Reject: "Add DomeLight and DistantLight (min intensity 100 and 500)"]
    M -->|Yes| O{RayTracedLighting?}
    O -->|No| P[Reject: "Use RayTracedLighting for iteration"]
    O -->|Yes| Q{Render output exists?}
    Q -->|No| R[Reject: "No render output"]
    Q -->|Yes| S{>= 150 KB and mean_RGB > 30?}
    S -->|No| T[Reject: "Render is black, too dark, or corrupted"]
    S -->|Yes| U{Hardcoded user paths or bare IPs?}
    U -->|Yes| V[Reject: "Replace hardcoded user paths/IPs with env vars or args"]
    U -->|No| Y[VALID - deliver]

Training output

Item	Action
policy_only_XXXX.pt	accept
model_XXXX.zip	accept
log.csv or wandb URL	accept
results.json with episode data	accept
No reward curve / metrics	reject: Log episode/reward_mean and train/value_loss

Reply format

• Pass: Valid: <one-line summary>
• Fail: Invalid: <list of failed checks>
• Suggest (only when otherwise valid): Could be improved: <tip>

Do not hedge ("I think", "maybe"). Do not deliver a black frame.

Input sources

• isaac-sim-orchestrator, isaac-sim-rendering, other producer skills, and sub-agent outputs.
• Invoked on Isaac Sim deliverables: "validate this Isaac Sim script", "check this Isaac Sim render", "review this sim output".

Automated validation

Run the checks above from the CLI with scripts/validate_sim.sh:

# Levels 1–2: static only
bash scripts/validate_sim.sh --level <1|2> <script.py> [--output-dir <dir>] [--timeout <seconds>]

# Level 3: EXECUTES the script — requires explicit opt-in
bash scripts/validate_sim.sh --level 3 --confirm-execute <script.py> [--output-dir <dir>] [--timeout <seconds>]
# or: CONFIRM_EXECUTE=1 bash scripts/validate_sim.sh --level 3 <script.py> ...

Feedback loop

• Rejections feed back into the producing skill's failure-mode catalog.
• Recurring failures become new validator rules.
• Standing rubric: references/quality-criteria.md. Record each iteration in references/feedback-log.md.