di-agent-knowledge-engine-datastage

Name: di-agent-knowledge-engine-datastage
Author: ibm

github.com/ibmData & databases1118 installsUpdated 5d ago

About di-agent-knowledge-engine-datastage

A skill reference for IBM DataStage’s parallel engine, aimed at teams designing large-scale ETL and data warehouse jobs. It explains partitioning, pipeline parallelism, scalability, and performance factors.

Key features

01Partitioned parallel processing
02Pipeline parallelism
03Multi-node scalability
04Performance tuning factors

Use cases

01Designing large-volume batch ETL
02Planning parallel jobs across nodes
03Loading data warehouses
04Assessing job performance factors

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/ibm/ibm-watsonx-data-integration-skills --skill di-agent-knowledge-engine-datastage

After installing, type/di-agent-knowledge-engine-datastagein Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

ConcurrentJobExecution.md

ConfigurationManagement.md

DataSetPerformance.md

DataStageEngineDetails.md

DiskAndResourceOptimization.md

RestartAndRecovery.md

optimization/

MemoryManagementGuidelines.md

overview.md

PartitioningGuidelines.md

SortingGuidelines.md

SortingTroubleshooting.md

SortMergeCollector.md

SortStageAdvancedConfig.md

stages/

AddressVerificationStage.md

AggregatorStage.md

ApacheKafkaStage.md

BloomFilterStage.md

BuildopStage.md

ChangeApplyStage.md

ChangeCaptureStage.md

ChecksumStage.md

ColumnExportStage.md

ColumnGeneratorStage.md

ColumnImportStage.md

CombineRecordsStage.md

CompareStage.md

ComplexFlatFileStage.md

CompressStage.md

CopyStage.md

DataRulesStage.md

Details

From the project README

DataStage Parallel Engine

When to Use DataStage

Batch ETL processing of large data volumes
Parallel processing across multiple nodes
Complex transformations with high throughput requirements
Integration with enterprise databases and file systems
Data warehouse loading and CDC operations

Engine Characteristics

Parallel processing: Divides data into partitions processed simultaneously
Pipeline parallelism: Multiple stages process different data concurrently
Scalable: Add nodes to increase throughput
High performance: Optimized for large-scale data movement

Key Concepts

Partitioning: Data divided across processing nodes
Nodes: Physical or logical processing units
Partitions: Subsets of data processed independently
Configuration file: Defines nodes and resources

Performance Factors

Job design (stage selection, partitioning, data flow)
Configuration (node count, partition count, resources)
Infrastructure (disk I/O, network bandwidth, CPU)

References

Engine Details
Concurrent Job Execution
Configuration Management
Data Set Performance
Disk and Resource Optimization
Restart and Recovery
Flow optimization (partitioning, sorting, memory) → optimization/overview.md
Per-stage semantics, requirements, best practices, and properties → stages/
Transformer expressions

Comments

Markdown supported

Details

Author: ibm
Repository: ibm/ibm-watsonx-data-integration-skills
Category: Data & databases
Installs: 118
GitHub stars: 1

Report

More skills

All skills

hyperframes-core

github.com/heygen-com

930.7k

The HyperFrames composition contract — build one renderable project. Use for composition structure, the `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, deterministic-render rules, and validation. Also covers Tailwind projects and the STORYBOARD.md / SCRIPT.md plan formats. Read before writing composition HTML.

lark-base

open.feishu.cn

837.7k

Operate on Feishu Base: create tables, fields, records, and views; handle statistics, formulas/lookups, forms, dashboards, app mode (BaseApp/AppMode pages and components), Workspace directories, workflows, role permissions, and the template center (Base template categories, lists, and search). Use when encountering Base/multidimensional tables/bitable, BaseApp/AppMode, or /base/ or /app/ links. Do not use lark-apps for BaseApp; route file import/export to lark-drive and authentication/authorization to lark-shared.

lark-sheets

open.feishu.cn

836.2k

Feishu Spreadsheets: create and operate spreadsheets. Supports worksheets and row/column structures (add/delete/merge/size/hide/freeze/group), cell read/write (values/formulas/styles/comments/cell images), range copy/move/sort/fill, find and replace, bulk updates, and creating and maintaining objects such as charts, pivot tables, conditional formatting, filters and filter views, dropdown lists, sparklines, and floating images. Also supports formula validation, restoring historical versions, and importing/exporting local Excel/CSV files and Feishu spreadsheets. Use when users need to create or edit spreadsheets, summarize and visualize data, format spreadsheets, calculate formulas (including migrating Excel formulas), or build financial models (DCF, three statements, budgets, Sensitivity, etc.). For multidimensional tables (Base/bitable), use lark-base. If users want to search for spreadsheet files in cloud storage (Drive/cloud storage) by name or keyword, use lark-drive's drive +search to locate resources first. When users provide a doubao.com /sheets/ URL/token, use this skill directly; do not fall back to WebFetch because the domain is not Feishu. Routing is based on the URL path pattern and token, not the domain.

remotion-best-practices

github.com/remotion-dev

720.7k

Best practices and domain knowledge for building videos programmatically with Remotion (videos in React/TypeScript, rendered to MP4). Use whenever writing or editing Remotion code — scaffolding a video project, animating with interpolate/spring over useCurrentFrame(), sequencing scenes and transitions, captions/subtitles, audio, GIFs/Lottie/3D, charts, visual effects, dynamic duration via calculateMetadata, or rendering. Consult before reaching for a Remotion API, since its components and props evolve (e.g. @remotion/media).