mcpField
I
isaac-sim-remote

isaac-sim-remote

github.com/isaac-simDevelopment4.4k46 installsUpdated 5d ago
Repository

About isaac-sim-remote

A skill for controlling an already-running Isaac Sim through its local code-editor TCP server, for developers and agents iterating on simulation scenes. Its scripts handle stage, prim, simulation, camera, and screenshot operations without restarting the app.

Key features

  1. 01Local TCP control of Isaac Sim
  2. 02Stage and prim operations
  3. 03Simulation and camera control
  4. 04Screenshots, video, and logs
  5. 05Command dispatch with dry-run support

Use cases

  1. 01Inspect or edit a running stage
  2. 02Adjust prims during iteration
  3. 03Capture screenshots or viewport video
  4. 04Check app health and console logs

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/isaac-sim/isaacsim --skill isaac-sim-remote

After installing, type/isaac-sim-remotein Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

references/
api-reference.md
pitfalls.md
workflows.md
scripts/
app_screenshot.py
camera_control.py
capture_annotator.py
console_log.py
execute_command.py
health_check.py
isaacsim_send.py
open_stage.py
prim_properties.py
prim_transform.py
select_prim.py
set_asset_root.py
shutdown_app.py
simulation_control.py
stage_info.py
verify_asset.py
viewport_screenshot.py
viewport_video.py

Details

From the project README

Isaac Sim Remote

Purpose

Drive an already-running Isaac Sim over the code-editor TCP socket so agents can iterate on stages, prims, physics, and screenshots without restarting the app.

Security

This skill is a local developer control plane, not a remote-execution service.

  • Bind: clients default to 127.0.0.1:8226. Do not publish that port off-host.
  • Trust boundary: equal to the OS user who launched Isaac Sim on this machine.
  • Design (intentional): isaacsim_send.py delivers Python source to the local code-editor server; execute_command.py dispatches any name already present in omni.kit.commands.get_commands() after an identifier check and audit log. An allowlist would defeat live Kit iteration — treat registry-wide dispatch as the product, not a hole.
  • Mitigations: loopback-only default; --dry-run prints payloads without send; CLI --arg values use ast.literal_eval (literals only).

Prerequisites

  • Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
  • NVIDIA GPU with a current driver (nvidia-smi).
  • Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

  • Requires Isaac Sim launched with the Python code-editor server enabled.
  • Remote scripts cannot import arbitrary repo modules unless paths are injected.

Troubleshooting

Error / symptomCauseSolution
Connection refused on 8226Python server disabledLaunch Isaac Sim with code-editor server enabled
NameError in remote cellModule not on sys.pathInline helpers or inject repo paths before import
Stale stage statePrior remote editsReset prims or reopen stage before repro steps

Related: debug-with-local-kit (when behavior depends on a Kit-from-source build), profile-isaac-sim (to attach Tracy to the running process), isaac-sim-validator (final QA gate on any rendered output).

Upstream isaac-sim-ui (menu/widget OmniUIQuery automation) and isaac-sim-recording (cursor tracking, tutorial video capture) are not imported. The inline UI patterns here (Play-button click via OmniUIQuery, full-app vs viewport screenshots) cover the common cases.

Every helper in scripts/ is invoked through scripts/isaacsim_send.py. From agent runtimes that expose skill execution helpers, call e.g. run_script("scripts/app_screenshot.py", args=["--help"]); from a built tree run the file with ./python.sh (Linux) / python.bat (Windows) under _build/*/release. Command recipes live in references/workflows.md.

Available Scripts

ScriptPurposeArguments
scripts/app_screenshot.pyApp screenshotsee script --help
scripts/camera_control.pyCamera controlsee script --help
scripts/capture_annotator.pyCapture annotatorsee script --help
scripts/console_log.pyConsole logsee script --help
scripts/execute_command.pyDispatch a registered omni.kit.commands entry (loopback trust; see Security)see script --help
scripts/health_check.pyHealth checksee script --help
scripts/isaacsim_send.pyLocalhost python_server IPC client (--dry-run supported)CLI flags via argparse (see script --help)
scripts/open_stage.pyOpen stagesee script --help
scripts/prim_properties.pyPrim propertiessee script --help
scripts/prim_transform.pyPrim transformsee script --help
scripts/select_prim.pySelect primsee script --help
scripts/set_asset_root.pySet asset rootsee script --help
scripts/simulation_control.pySimulation controlsee script --help
scripts/stage_info.pyStage infosee script --help
scripts/verify_asset.pyVerify assetsee script --help
scripts/viewport_screenshot.pyViewport screenshotsee script --help
scripts/viewport_video.pyViewport videosee script --help

Launching Isaac Sim

cd _build/linux-x86_64/release
bash isaac-sim.sh --no-window --no-ros-env \
    --enable isaacsim.code_editor.python_server

Wait for app ready. TCP server: 127.0.0.1:8226. Use --enable isaacsim.code_editor.python_server (not carb /enabled=true). Full launch variants and health/asset/send/screenshot/control workflows: references/workflows.md. Pitfalls: references/pitfalls.md. APIs: references/api-reference.md.

Important Notes

  • Lighting in headless mode: No lights = black image. Always add a DomeLight with intensity 1000–5000.
  • Renderer warm-up: After new stage, call await app_utils.update_app_async(steps=120) before rendering.
  • Connection errors: Ensure Isaac Sim shows app ready and python_server is --enabled.
  • Server hangs: Crashed examples can wedge the server. Restart Isaac Sim. Use health_check.py to verify.
  • Assets not found: Configure S3 fallback with set_asset_root.py.

Comments

Markdown supported