About compare-models
compare-models is a skill for narrowing down and testing Replicate models. It guides users who need to choose by cost, latency, quality, and capabilities through a workflow from shortlist to test predictions.
Key features
- 01Search models and build a shortlist
- 02Inspect model input and output schemas
- 03Check pricing and prediction time
- 04Test batches and compare outputs
- 05Balance cost, latency, and quality
Use cases
- 01Shortlist models for an image task
- 02Compare inference time and model costs
- 03Check supported model inputs and outputs
- 04Evaluate results using the same prompts
MCP support
The README mentions a Replicate MCP server but does not explain how this skill uses or connects to it.
Install
Copy the command and run it in a terminal.
$
npx skills add https:/After installing, type/compare-modelsin Claude Code to confirm it loaded.
Details
From the project READMEDocs
- Reference: https://replicate.com/docs/llms.txt
- OpenAPI schema: https://api.replicate.com/openapi.json
- MCP server: https://mcp.replicate.com
- Per-model docs:
https://replicate.com/{owner}/{model}/llms.txt - Set
Accept: text/markdownwhen requesting docs pages for Markdown responses.
Workflow
- Search or browse collections to build a shortlist of candidate models.
- Fetch each model's schema to compare inputs, outputs, and capabilities.
- Check pricing from model metadata or the Replicate website.
- Run a small batch of test predictions to compare output quality.
- Pick the model that best fits your constraints (cost, latency, quality).
What to compare
- Speed: Check
metrics.predict_timeon completed predictions for actual inference time. Official models are always warm. Community models can cold-boot. - Cost: Official models have predictable per-run pricing. Community models charge by compute time (GPU-seconds). Run a few predictions and check the
metricsfield for actual cost. - Quality: Run the same prompts through each model and compare outputs. Quality is subjective. Match it to your use case, not a leaderboard.
- Capabilities: Compare input schemas for supported features (reference images, masks, aspect ratios, streaming, multi-image input). Check output formats.
Key tradeoffs
- Lowest cost: smaller/distilled models. Accept slower inference and lower quality.
- Lowest latency: official models or schnell/turbo variants. Accept higher cost per run.
- Highest quality: pro/max/quality variants. Accept slower inference and higher cost.
- Most control: models with ControlNet, masks, or reference images. Accept more complex input setup.
Official vs community models
- Official models: always warm, stable APIs, predictable pricing, maintained by Replicate.
- Community models: may cold-boot, require version pinning, maintained by the author.
- If a community model meets your needs and an official model doesn't, consider creating a deployment for consistent uptime.
Prompting guidance
For prompting techniques and task-specific guidance:
- Image generation and editing: see the prompt-images skill.
- Video generation: see the prompt-videos skill.
Comments
Markdown supported