mcpField
T
together-batch-inference

together-batch-inference

github.com/togethercomputerData & databases39101 installsUpdated 2w ago
Repository

About together-batch-inference

A skill for developers running large offline inference workloads through Together AI's Batch API. It suits independent requests where latency is not critical and cost matters, covering JSONL input, batch submission, polling, and result reconciliation.

Key features

  1. 01Prepare JSONL batch requests
  2. 02Submit and poll batch jobs
  3. 03Download result and error files
  4. 04Reconcile outputs by custom_id

Use cases

  1. 01Classify independent text requests in bulk
  2. 02Generate synthetic data or transform datasets
  3. 03Run large summarization and enrichment jobs asynchronously

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/togethercomputer/skills --skill together-batch-inference

After installing, type/together-batch-inferencein Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

agents/
openai.yaml
references/
api-reference.md
scripts/
batch_workflow.py
batch_workflow.ts

Details

From the project README

Together Batch Inference

Overview

Use Together AI's Batch API for large offline workloads where latency is not the primary concern.

Typical fits:

  • bulk classification
  • synthetic data generation
  • dataset transformations
  • large summarization or enrichment jobs
  • low-cost asynchronous inference

When This Skill Wins

  • The user has many independent requests to run
  • A JSONL request file is acceptable
  • Turnaround time can be minutes or hours instead of seconds
  • Lower cost matters more than immediate interactivity

Hand Off To Another Skill

  • Use together-chat-completions for real-time requests or tool-calling apps
  • Use together-evaluations for managed LLM-as-a-judge workflows
  • Use together-embeddings for retrieval-specific vector generation

Quick Routing

Workflow

  1. Build a JSONL file where each line contains custom_id and body.
  2. Upload the file with purpose="batch-api".
  3. Create the batch with input_file_id=... and the target endpoint.
  4. Poll until the job is terminal.
  5. Download output and error files, then reconcile by custom_id.

High-Signal Rules

  • Python scripts require the Together v2 SDK (together>=2.0.0). If the user is on an older version, they must upgrade first: uv pip install --upgrade "together>=2.0.0".
  • Use input_file_id, not legacy file parameters.
  • Keep custom_id stable and meaningful so result reconciliation is easy.
  • Batch is for independent requests. If the workload depends on shared conversation state, it is probably the wrong tool.
  • Always inspect the error file in addition to the success output.
  • client.batches.create() returns a wrapper; access the batch object via response.job (e.g., response.job.id). client.batches.retrieve() returns the batch object directly.
  • For classification or labeling workloads, set max_tokens low (e.g., 4), use temperature: 0, and constrain the system prompt to return only the label. This minimizes output tokens and cost.
  • Small batches (under 1K requests) typically complete in minutes. The 24-hour completion window is a maximum, not typical.

Resource Map

Official Docs

Comments

Markdown supported