creating-backstage-catalog-provider

Creating a Backstage Catalog Provider

When to use this skill

• Building a module that ingests catalog entities from an external source (Azure DevOps, GitHub topic search, internal asset registry, JSON file, etc.)
• Editing an existing catalog-backend-module-* plugin
• The ingestion runs periodically (provider model) — for one-shot processing of an entity definition file, write a CatalogProcessor instead

What you build

A catalog provider is a backend module (createBackendModule({ pluginId: 'catalog', ... })) that registers a provider against an extension point. There are two provider models — pick based on data volume.

Choose the model first: normal vs incremental

Use normal EntityProvider when...	Use IncrementalEntityProvider when...
Total entity count is small (~hundreds at most)	Total entity count is large (thousands+)
The full set fits in memory comfortably	Holding the full set in memory is impractical
The source returns the full snapshot in one round-trip (or a handful)	The source supports pagination — cursor / page-token / offset / etc.
You're prototyping or building a POC	You're shipping for production-scale catalogs
Refresh frequency is low	High write rate or scale; incremental handles batching, backoff, retry

The two models register against different extension points and have different runtime semantics:

• Normal registers against catalogProcessingExtensionPoint from @backstage/plugin-catalog-node. Adopters need no extra install.
• Incremental registers against incrementalIngestionProvidersExtensionPoint from @backstage/plugin-catalog-backend-module-incremental-ingestion. Adopters must also install catalogModuleIncrementalIngestionEntityProvider (the framework module that orchestrates batches and persists cursors via the catalog database).

If you don't know upfront, default to normal. Migrating to incremental later is mechanical — same domain fetch logic, different framework wiring.

If the data volume is genuinely unknown, ask the user. Rough thresholds: <500 entities → normal is fine; >10,000 → incremental is required; in between → ask about source pagination support and growth trajectory.

Choose the right pattern

Building...	Reference
A normal EntityProvider class (default for small/medium volume)	references/entity-provider.md (Normal section)
An IncrementalEntityProvider class (large volume)	references/entity-provider.md (Incremental section)
The module that wires either kind in	references/module-registration.md

Two invocation modes

• Standalone: editing an existing catalog-backend-module-* plugin.
• Post-scaffold (called by creating-backstage-plugin for catalog-provider-module): scaffold both files together — provider class + module registration.

Common gotchas

• The provider's getProviderName() must be unique across the catalog (same rule for both normal and incremental). Use a stable name — emitted entities are linked to the provider by it; renaming orphans them.
• Normal provider: applyMutation({ type: 'full', entities }) replaces the whole set the provider owns. applyMutation({ type: 'delta', added, removed }) adds/removes individual entries — useful when you only learn about changes incrementally. Note this is a delta mutation, not the same thing as the IncrementalEntityProvider model.
• Normal provider: the connection isn't available until connect() runs — guard this.connection against undefined in any method that might run before connect.
• Normal provider: don't fetch from an external system in connect() directly — receive a SchedulerServiceTaskRunner and call this.schedule.run({ id, fn }) inside connect() so the work runs periodically and survives restarts.
• Incremental provider: the framework owns the iteration loop. Your next(context, cursor?) returns one page; the framework calls it repeatedly with the cursor you returned previously, and persists state in the catalog database. Don't try to schedule your own refresh.
• Incremental provider: adopters MUST install catalogModuleIncrementalIngestionEntityProvider (from @backstage/plugin-catalog-backend-module-incremental-ingestion) in addition to your module. Document this prerequisite in your module's README.
• catalogProcessingExtensionPoint comes from @backstage/plugin-catalog-node (the library package). incrementalIngestionProvidersExtensionPoint comes from @backstage/plugin-catalog-backend-module-incremental-ingestion. Don't depend on @backstage/plugin-catalog-backend from a module — that's the host plugin.