mcpField
K
kafka

kafka

github.com/dobroslavradosavljevicData & databases339 installsUpdated 1d ago
Repository

About kafka

kafka is a TypeScript/Node skill for Kafka producer, consumer, and administration work, centered on `@platformatic/kafka`. It covers broker behavior, delivery semantics, topic design, and operational checks.

Key features

  1. 01Producer, consumer, and admin workflows
  2. 02Topic and partition design
  3. 03Delivery guarantees and commits
  4. 04Serialization and transaction guidance
  5. 05Operational verification

Use cases

  1. 01Producing and consuming Kafka messages from Node
  2. 02Planning partitions, keys, and retention
  3. 03Checking delivery semantics and consumer lag

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/dobroslavradosavljevic/skills --skill kafka

After installing, type/kafkain Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

agents/
openai.yaml
references/
clients-ecosystem.md
fundamentals.md
patterns-ops.md
platformatic-kafka.md
source-map.md
usage-guide.md

Details

From the project README

Apache Kafka (TypeScript)

Use this skill for Kafka produce/consume/admin work driven from TypeScript/Node: prefer @platformatic/kafka, plus broker semantics, topic design, delivery guarantees, and ops that change app code.

Workflow

  1. Inspect the local surface:
    • Package: @platformatic/kafka (snapshot 2.13.0). Engines: Node >=22.22 or >=24.6. Kafka brokers 3.5–4.2 CI-tested (upstream Apache Kafka is at 4.3.x; KRaft-only on 4.0+).
    • Clients: Producer / Consumer / Admin with bootstrapBrokers (not KafkaJS brokers).
    • Topics: partitions, keys, RF / min.insync.replicas, cleanup policy.
    • Semantics: at-least-once vs EOS; auto vs manual commit; ProduceAcks.
  2. For day-to-day how-to, follow usage-guide.md first.
  3. Refresh docs when versions drift. Start from source-map.md.
  4. Route deeper detail:
  5. Prefer @platformatic/kafka over kafkajs (unmaintained) and over native node-rdkafka unless librdkafka is required. Do not invent KafkaJS connect/subscribe/run APIs on Platformatic.
  6. Verify with focused produce/consume smokes, Admin topic create when needed, lag awareness, and close() on shutdown.

Core Judgment

  • Kafka is an append-only partitioned log. Ordering is per partition only; parallelism ceiling for a group = partition count.
  • Default app semantic: at-least-once + idempotent handlers. Use transactions/READ_COMMITTED only when you need Kafka↔Kafka EOS.
  • Durability triad: RF=3, min.insync.replicas=2, producer ProduceAcks.ALL (+ idempotent producer when available).
  • Keys: stable entity ids for order/partitioning. Compaction requires keys + tombstones (null value).
  • Prefer outbox/CDC over dual-write (DB + Kafka in one request).
  • Poison pills: retry/DLQ then commit — never block a partition forever.
  • Platformatic: long-lived clients; lazy connect; close(); consume via consume() → stream (for await / data / concurrent forEach); offsets are bigint; headers on consume are a Map.
  • Pass serializers / deserializers (e.g. stringSerializers) — default expects Buffer.
  • Classic groups rebalance stop-the-world by default; opt in to KIP-429 cooperative-sticky (protocols: [{ name: COOPERATIVE_STICKY_ASSIGNOR, version: 3 }], all members together) or use KIP-848 (groupProtocol: "consumer") on Kafka 4.0+. Watch consumer:heartbeat:stalled for stuck members.
  • Kafka share groups (KIP-932 queues, production-ready since Kafka 4.2) are not implemented in @platformatic/kafka — do not promise queue semantics from it.
  • Kafka 4.0+ is KRaft-only (no ZooKeeper). Local/dev: fix advertised.listeners.
  • Prefer bun / bunx in command examples; runtime must still satisfy Node engines for @platformatic/kafka.

Verification

Prefer repository-owned commands. For meaningful Kafka work, cover the relevant subset:

  • bun pm ls @platformatic/kafka and Node ≥22.22 (or ≥24.6).
  • Smoke: Admin createTopics (or existing topic), Producer.send, Consumer.consume + process + close().
  • Durability: ProduceAcks.ALL; idempotence/transactionalId when claiming EOS.
  • Consumer: correct groupId, mode (EARLIEST/LATEST/COMMITTED), commit-after-process when autocommit: false.
  • Security: TLS + SASL (or mTLS); no PLAINTEXT on public networks.
  • Ops: consumer lag, under-replicated partitions, hot-key skew, rebalance storms (max.poll / processing time).

Report which checks ran, which did not, and broker/version assumptions that remain.

Comments

Markdown supported