mcpField
P
physicalai-runtime-running-policy-on-robot

physicalai-runtime-running-policy-on-robot

github.com/openvinotoolkitDevelopment4081 installsUpdated today
Repository

About physicalai-runtime-running-policy-on-robot

A development skill for running policies on robots with physicalai-runtime. It explains Python and YAML entry points, execution modes, callbacks, and validation using fake hardware.

Key features

  1. 01Python and YAML setup guidance
  2. 02Sync and RTC execution modes
  3. 03Runtime callback guidance
  4. 04Fake robot and camera testing
  5. 05Configuration and action validation

Use cases

  1. 01Running a policy in a notebook
  2. 02Deploying through physicalai run
  3. 03Testing a runtime without hardware

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/openvinotoolkit/physicalai --skill physicalai-runtime-running-policy-on-robot

After installing, type/physicalai-runtime-running-policy-on-robotin Claude Code to confirm it loaded.

Details

From the project README

Running a Policy on a Robot

PolicyRuntime (src/physicalai/runtime/runtime.py) owns the control loop; InferenceModel owns policy math. Execution strategy lives under src/physicalai/runtime/execution.py and src/physicalai/runtime/rtc_execution.py. CLI: physicalai run in src/physicalai/cli/run.py instantiates from YAML via jsonargparse.

Workflow

  1. Choose API vs config: Python for notebooks/tests; YAML + physicalai run for reproducible deployment.

    • Done when: entry point matches the user's task.

  2. Python minimal loop (see docs/how-to/runtime/run-policy-on-robot.md):

    from physicalai.runtime import PolicyRuntime, SyncExecution
from physicalai.inference import InferenceModel
from physicalai.robot import SO101
from physicalai.capture import UVCCamera

runtime = PolicyRuntime(
    fps=30,
    robot=SO101(port="/dev/ttyACM0"),
    model=InferenceModel("./exports/act_policy"),
    cameras={"wrist": UVCCamera(device="/dev/video0", width=640, height=480)},
    execution=SyncExecution(),
)
with runtime:
    runtime.run(duration_s=60)
    • Done when: components connect and the loop runs in a test or dry-run with fakes.

  3. YAML config — nest class_path / init_args for robot, model, cameras, execution; run:

    physicalai run --config runtime.yaml --run.duration_s=60

  4. Execution mode — pick sync vs RTC per docs/how-to/runtime/use-execution-modes.md; do not build ad-hoc timing around InferenceModel.select_action when PolicyRuntime should own the queue.

  5. Callbacks — register via runtime callback APIs (docs/how-to/runtime/add-runtime-callbacks.md) for telemetry/latency, not inside inference adapters.

Validation loop

uv run pytest tests/unit/runtime/ -q

Use fake robots/cameras from runtime tests when hardware is unavailable.

Required checks

  • fps and camera read rates are consistent.
  • Action dimensions match robot send_action expectations.
  • Config class_path targets are importable without training packages.
  • Document breaking changes to runtime config schema in docs/reference/config-schema.md.

References

  • docs/how-to/runtime/run-policy-on-robot.md
  • docs/how-to/config/write-runtime-config.md

Comments

Markdown supported