di-agent-knowledge-engine-streamsets

Name: di-agent-knowledge-engine-streamsets
Author: ibm

github.com/ibmCloud & DevOps160 installsUpdated 3d ago

About di-agent-knowledge-engine-streamsets

A StreamSets Engine operations skill for people configuring, deploying, and troubleshooting StreamSets environments. It focuses on containerized engines, job lifecycle, high availability, and resource management rather than flow authoring.

Key features

01Docker/Podman engine deployment
02Environment and compute configuration
03Job execution and lifecycle management
04Failover and offset management
05Engine monitoring and resource limits

Use cases

01Troubleshoot StreamSets engine communication
02Deploy data processing engines in a network
03Configure job monitoring and failover

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/ibm/ibm-watsonx-data-integration-skills --skill di-agent-knowledge-engine-streamsets

After installing, type/di-agent-knowledge-engine-streamsetsin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

engines.md

environments.md

jobs.md

reference.md

Details

From the project README

StreamSets Engine

Flow authoring: This skill covers engine infrastructure and operations only. For building or modifying StreamSets flows, use your path's flow authoring skill.

When to Use This Skill

Configuring or troubleshooting StreamSets environments and engines
Engine deployment (Docker/Podman), communication methods (tunneling vs. direct)
Job lifecycle: execution, monitoring, failover, offset management
High availability setup and resource management

Engine Characteristics

Container-based: Runs as Docker/Podman containers in your network
Data ownership: All processing occurs within your infrastructure
Continuous processing: Jobs run continuously, processing data as it arrives
High availability: Multiple engines support failover and load distribution
Flexible communication: Tunneling (default) or direct connection methods

Key Concepts

Environments: Configure engines and compute resources for projects
Engines: Data Collector containers that execute flows
Jobs: Flow executions that run continuously on engines
Offsets: Track processing progress for resumable execution
VPCs: Virtual CPUs allocated to engine containers

Architecture

Browser ↔ watsonx.data integration ↔ Engine (in your network)
Engines process data locally, send metrics to watsonx.data integration
Jobs fail over automatically between engines (6.4+)
Resource thresholds prevent engine overload

Quick Start

Create environment → environments.md
Run engine → engines.md
Execute job → jobs.md

References

Environments - Environment creation and configuration
Engines - Engine deployment, management, and monitoring
Jobs - Job execution and lifecycle management
Technical Reference - System requirements, commands, and troubleshooting

Comments

Markdown supported

Details

Author: ibm
Repository: ibm/ibm-watsonx-data-integration-skills
Category: Cloud & DevOps
Installs: 60
GitHub stars: 1

Report

More skills

All skills

lark-apps

open.feishu.cn

739.5k

Develop and host Miaoda (Spark/Miaoda) apps: app creation, local full-stack development, cloud generation and iteration, creative design (UI mockups / interactive prototypes / wireframes / landing pages / dashboards / slide decks / visual exploration), integration of AI-related capabilities, Feishu platform capabilities, or other external capabilities; log/Trace/monitoring metric/PV/UV queries; environment variable management; app collaborators and collaboration permissions; app roles and members; and automation triggers (scheduled / record changes / Webhook / Feishu approvals). Use when users want to develop or create a system, tool, platform, or app; develop locally or in the cloud, modify, deploy, publish, launch, or get a shareable link; create a page or website with HTML and deploy it to Miaoda; design, create a mockup, prototype, wireframe, PPT, deck, or visual exploration; or mention Miaoda/Spark/Miaoda apps (links like `/app/app_xxx` or `/page/<token>`, runtime domains like *.aiforce.cloud), app databases, app file storage, open API Keys, visibility, app collaborators/developer permissions, app roles/role members, production logs, API request volume, error volume, latency, traffic, environment variables, or setting up automation tasks, scheduled triggers, or approval-triggered automation for Miaoda apps. Not for regular cloud drive file uploads (lark-drive), Feishu document editing (lark-doc), or native slide creation (lark-slides).

wizard

github.com/mattpocock

576.6k

Generate an interactive bash wizard that walks a human through steps only they can perform. Use when provisioning infrastructure, setting up credentials or CI secrets, walking an unfamiliar third-party dashboard, or running a one-off migration or cutover. Don't invoke this for steps the agent can perform itself.

azure-prepare

github.com/microsoft

573.4k

Prepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wants to use azd as the deployment tool, or the project already has an azure.yaml file. DO NOT USE FOR: non-azd deployments, Python App Service code-only deploys (use python-appservice-deploy), or cross-cloud migration (use azure-cloud-migrate). WHEN: prepare app for azd, create azure.yaml, set up azd infrastructure, modernize app for Azure with azd, deploy with azd, function app, timer trigger, service bus trigger, event-driven function, managed identity, generate Bicep, generate Terraform, create and deploy to Azure.

azure-ai

github.com/microsoft

572.8k

Use for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, OCR, convert text to speech.