dm-limits-and-best-practices

CDF 数据建模：限制、并发与最佳实践

这是一份参考技能。在编写或审查调用 CDF Data Modeling API 的代码时，请采用以下模式。

---

DMS 限制参考

如需了解最新的并发限制、资源限制和属性值限制，请参阅官方文档： https://docs.cognite.com/cdf/dm/dm_reference/dm_limits_and_restrictions

需要注意的事项：

• 实例的 apply、delete 和 query 操作各自有独立的并发请求限制
• 超出这些限制会返回 429 Too Many Requests
• 转换会占用大量并发额度，因此留给其他客户端的额度会减少
• instances.list 有最大分页大小（请使用分页获取完整结果）
• instances.query 的表表达式各自有独立的项目数量限制
• instances.upsert 每次调用最多接受 1000 个项目
• in 筛选器每个表达式最多接受 1000 个值；更大的集合必须拆分为多个批次

---

搜索与筛选：何时使用哪种方式

instances.search — 对文本属性进行全文搜索

当你需要对字符串字段（名称、描述等）进行模糊／文本匹配时，请使用 instances.search。它支持 operator 参数：

• AND（默认）— 缩小搜索范围。所有词都必须匹配。适用于用户提供了具体查询的情况。
• OR — 范围宽泛的“撒网式”搜索。任意词都可以匹配。适用于探索式／输入提示式搜索，以尽可能提高召回率。

// Narrow search: find a specific cell by name (AND — all terms must match)
const exactResults = await client.instances.search({
  view: { type: 'view', ...PROCESS_CELL_VIEW },
  query: 'reactor tank A',
  properties: ['name'],
  operator: 'AND',
  limit: 10,
});

// Broad search: typeahead/autocomplete (OR — any term can match)
const broadResults = await client.instances.search({
  view: { type: 'view', ...BATCH_VIEW },
  query: 'BUDE completed',
  properties: ['name', 'description', 'batchStatus'],
  operator: 'OR',
  limit: 10,
});

你可以将 search 与 filter 结合使用，通过精确匹配条件进一步限制结果：

// Text search + exact filter: search for "pump" but only in active nodes
const filtered = await client.instances.search({
  view: { type: 'view', ...PROCESS_CELL_VIEW },
  query: 'pump',
  properties: ['name', 'description'],
  filter: {
    equals: {
      property: getContainerProperty(MY_CONTAINER, 'status'),
      value: 'active',
    },
  },
  limit: 20,
});

使用 filter 的 instances.list / instances.query — 精确匹配筛选

当你需要精确、确定性的匹配（相等、范围、in、hasData 等）时，请使用 filter。不支持模糊匹配，值必须完全匹配。

// Exact match: get all completed batches
const completedBatches = await client.instances.list({
  instanceType: 'node',
  sources: [{ source: { type: 'view', ...BATCH_VIEW } }],
  filter: {
    equals: {
      property: getContainerProperty(BATCH_CONTAINER, 'batchStatus'),
      value: 'completed',
    },
  },
  limit: 1000,
});

选择指南

需求	使用
用户在搜索框中输入内容	使用 OR 的 instances.search
按名称查找特定项目	使用 AND 的 instances.search
按状态、日期范围、枚举值筛选	list/query 上的 filter
文本搜索 + 精确约束	instances.search + filter

in 筛选器的值数量限制（1000）与分批处理

CDF in 筛选器在单个筛选表达式中最多支持 1000 个值。如果需要按超过 1000 个 ID 进行筛选，请将值拆分为多个批次并发出多个请求，然后合并结果。

const IN_FILTER_BATCH_SIZE = 1000;
// Reuse the Chunking Utility defined in the Batching Write Operations section.

async function listByExternalIds(
  client: CogniteClient,
  externalIds: string[],
): Promise<NodeOrEdge[]> {
  const idBatches = chunk(externalIds, IN_FILTER_BATCH_SIZE);
  const responses = await Promise.all(
    idBatches.map((batch) =>
      cdfTaskRunner.schedule(() =>
        client.instances.list({
          instanceType: 'node',
          sources: [{ source: { type: 'view', ...MY_VIEW } }],
          filter: {
            in: {
              property: ['node', 'externalId'],
              values: batch,
            },
          },
          limit: 1000,
        })
      )
    )
  );

  return responses.flatMap((r) => r.items);
}

---

QueuedTaskRunner（信号量）

始终使用全局 cdfTaskRunner 封装 CDF API 调用。它会限制并发请求，避免 429 错误和死锁。

源代码

如果项目尚无信号量工具，请创建 src/shared/utils/semaphore.ts，并使用以下实现：

/**
 * AbortError thrown when a queued task is cancelled
 */
export class AbortError extends Error {
  public constructor(message: string = 'Aborted') {
    super(message);
    this.name = 'AbortError';
  }
}

type PendingTask<AsyncFn, AsyncFnResult> = {
  resolve: (result: AsyncFnResult) => void;
  reject: (error: unknown) => void;
  fn: AsyncFn;
  key?: string;
};

const DEFAULT_MAX_CONCURRENT_TASKS = 15;

/**
 * QueuedTaskRunner for controlling concurrent operations
 * Used to limit concurrent CDF API requests to avoid rate limiting and deadlocks
 * Essentially a semaphore that allows a limited number of tasks to run at once.
 */
export default class QueuedTaskRunner<
  AsyncFn extends () => Promise<AsyncFnResult>,
  AsyncFnResult = Awaited<ReturnType<AsyncFn>>,
> {
  private pendingTasks: PendingTask<AsyncFn, AsyncFnResult>[] = [];
  private currentPendingTasks: number = 0;
  private readonly maxConcurrentTasks: number = 1;

  public constructor(
    maxConcurrentTasks: number = DEFAULT_MAX_CONCURRENT_TASKS
  ) {
    this.maxConcurrentTasks = maxConcurrentTasks;
  }

  public schedule(
    fn: AsyncFn,
    options: { key?: string } = {}
  ): Promise<AsyncFnResult> {
    this.startTrackingTime();

    return new Promise((resolve, reject) => {
      if (options.key !== undefined) {
        // Cancel existing tasks with the same key (deduplication)
        this.pendingTasks
          .filter((task) => task.key === options.key)
          .forEach((task) => task.reject(new AbortError()));

        this.pendingTasks = this.pendingTasks.filter(
          (task) => task.key !== options.key
        );
      }

      this.pendingTasks.push({
        resolve,
        reject,
        fn,
        key: options.key,
      });

      this.attemptConsumingNextTask();
    });
  }

  public async attemptConsumingNextTask(): Promise<void> {
    if (this.pendingTasks.length === 0) return;
    if (this.currentPendingTasks >= this.maxConcurrentTasks) return;

    const pendingTask = this.pendingTasks.shift();
    if (pendingTask === undefined) {
      throw new Error('pendingTask is undefined, this should never happen');
    }

    this.currentPendingTasks++;
    const { fn, resolve, reject } = pendingTask;

    try {
      const result = await fn();
      resolve(result);
    } catch (e) {
      reject(e);
    } finally {
      this.currentPendingTasks--;
      this.tick();
      this.attemptConsumingNextTask();
    }
  }

  public clearQueue = (): void => {
    this.pendingTasks = [];
  };

  private startTime: number | null = null;

  private startTrackingTime = (): void => {
    if (this.startTime === null) {
      this.startTime = performance.now();
    }
  };

  private tick = (): void => {
    if (this.pendingTasks.length === 0) {
      this.startTime = null;
    }
  };
}

/**
 * Global task runner for CDF API requests
 * Limits concurrent requests to avoid 429 rate limiting and deadlocks
 */
export const cdfTaskRunner = new QueuedTaskRunner(DEFAULT_MAX_CONCURRENT_TASKS);

使用模式

始终使用 cdfTaskRunner.schedule() 封装 CDF 调用：

import { cdfTaskRunner } from '../../../../shared/utils/semaphore';

// Single query
export async function fetchBatches(client: CogniteClient): Promise<CDFBatch[]> {
  return cdfTaskRunner.schedule(async () => {
    const response = await client.instances.query({
      with: { /* ... */ },
      select: { /* ... */ },
    });
    return response.items?.nodes || [];
  });
}

// Multiple parallel queries (safe — the semaphore limits concurrency)
export async function enrichBatch(
  client: CogniteClient,
  batch: CDFBatch
): Promise<BatchEnrichment> {
  const [currentOp, lastOp, cells, material] = await Promise.all([
    fetchCurrentOperation(client, batch.space, batch.externalId),
    fetchLastCompletedOperation(client, batch.space, batch.externalId),
    fetchProcessCells(client, batch.space, batch.externalId),
    fetchMaterial(client, batch.space, batch.externalId),
  ]);
  return { currentOp, lastOp, cells, material };
}

// Each of the above functions internally uses cdfTaskRunner.schedule(),
// so Promise.all is safe — the semaphore prevents exceeding concurrency limits

使用键进行去重

当同一查询再次触发时（例如，用户快速更改筛选条件），使用 key 选项取消过时的请求：

const result = await cdfTaskRunner.schedule(
  async () => client.instances.query({ /* ... */ }),
  { key: `batch-flow-${batchId}` }
);
// If another call with the same key arrives before this completes,
// the previous pending call is rejected with AbortError

---

分页

DMS instances.list 最多返回 limit 个项目，并通过 nextCursor 提供下一页游标。 DMS instances.query 使用以表表达式名称为键的 cursors 对象。

instances.list 分页

async function fetchAllNodes(client: CogniteClient): Promise<CDFNodeResponse[]> {
  const allItems: CDFNodeResponse[] = [];
  let cursor: string | undefined = undefined;

  do {
    const response = await client.instances.list({
      instanceType: 'node',
      sources: [{ source: { type: 'view', ...MY_VIEW } }],
      filter: {
        equals: {
          property: getContainerProperty(MY_CONTAINER, 'status'),
          value: 'active',
        },
      },
      limit: 1000,
      cursor,
    });

    allItems.push(...response.items);
    cursor = response.nextCursor;
  } while (cursor);

  return allItems;
}

instances.query 分页

query 端点以 Record<string, string> 的形式返回 nextCursor（每个表表达式对应一个游标）。请通过 cursors 参数使用它：

import { isEmpty } from 'lodash';

async function fetchAllResults(
  client: CogniteClient
): Promise<{ results: CDFResult[]; edges: EdgeDefinition[] }> {
  const QUERY_LIMIT = 10_000;

  const fetchPage = async (
    nextCursors?: Record<string, string>
  ): Promise<{ results: CDFResult[]; edges: EdgeDefinition[] }> => {
    const { items, nextCursor } = await client.instances.query({
      with: {
        results: {
          limit: QUERY_LIMIT,
          nodes: {
            filter: {
              hasData: [{ type: 'view', ...RESULT_VIEW }],
            },
          },
        },
        relatedEdges: {
          limit: QUERY_LIMIT,
          edges: {
            from: 'results' as const,
            maxDistance: 1,
            direction: 'outwards' as const,
            filter: {
              equals: {
                property: ['edge', 'type'],
                value: MY_EDGE_TYPE,
              },
            },
          },
        },
      },
      cursors: nextCursors, // Pass cursors from previous page
      select: {
        results: {
          sources: [
            { source: { type: 'view', ...RESULT_VIEW }, properties: ['*'] },
          ],
        },
        relatedEdges: {},
      },
    });

    const results = (items?.results || []) as CDFResult[];
    const edges = (items?.relatedEdges || []).filter(
      (e) => e.instanceType === 'edge'
    );

    // Recurse if more pages exist
    if (!isEmpty(nextCursor)) {
      const next = await fetchPage(nextCursor);
      return {
        results: [...results, ...next.results],
        edges: [...edges, ...next.edges],
      };
    }

    return { results, edges };
  };

  return fetchPage();
}

分页与 QueuedTaskRunner 结合使用

始终使用信号量封装分页获取操作，避免耗尽并发额度：

export async function fetchAllWithPagination(
  client: CogniteClient
): Promise<CDFNodeResponse[]> {
  return cdfTaskRunner.schedule(async () => {
    const allItems: CDFNodeResponse[] = [];
    let cursor: string | undefined = undefined;

    do {
      const response = await client.instances.list({
        instanceType: 'node',
        sources: [{ source: { type: 'view', ...MY_VIEW } }],
        filter: { /* ... */ },
        limit: 1000,
        cursor,
      });

      allItems.push(...response.items);
      cursor = response.nextCursor;

      // Optional: break early if you have enough data
      if (allItems.length >= 500) break;
    } while (cursor);

    return allItems;
  });
}

---

写入操作分批处理

当要 upsert 大量实例时，请将其分块，以符合 apply 并发限制。每次 instances.upsert 调用最多接受 1000 个项目。

分块工具

function chunk<T>(arr: T[], size: number): T[][] {
  const chunks: T[][] = [];
  for (let i = 0; i < arr.length; i += size) {
    chunks.push(arr.slice(i, i + size));
  }
  return chunks;
}

使用 QueuedTaskRunner 批量 Upsert

const UPSERT_BATCH_SIZE = 1000;

async function batchUpsertNodes(
  client: CogniteClient,
  nodes: NodeOrEdgeCreate[]
): Promise<void> {
  const chunks = chunk(nodes, UPSERT_BATCH_SIZE);

  // Process chunks through the semaphore — safe even with Promise.all
  await Promise.all(
    chunks.map((batch) =>
      cdfTaskRunner.schedule(async () => {
        await client.instances.upsert({
          items: batch,
        });
      })
    )
  );
}

使用 QueuedTaskRunner 批量删除

实例删除有更严格的并发限制。请使用单独且限制更严格的任务运行器：

import QueuedTaskRunner from '../../../../shared/utils/semaphore';

// Dedicated runner for deletes (stricter concurrency — check docs for current limit)
const deleteTaskRunner = new QueuedTaskRunner(2);

async function batchDeleteNodes(
  client: CogniteClient,
  nodeIds: { space: string; externalId: string }[]
): Promise<void> {
  const chunks = chunk(nodeIds, 1000);

  for (const batch of chunks) {
    await deleteTaskRunner.schedule(async () => {
      await client.instances.delete(
        batch.map((id) => ({
          instanceType: 'node' as const,
          ...id,
        }))
      );
    });
  }
}

---

常见陷阱

1. 嵌套信号量调用导致死锁

如果函数 A 占用一个信号量槽位并调用也需要槽位的函数 B，当所有槽位都被占用时，就可能发生死锁。**请将信号量放在最外层调用，**或确保内部调用不会经过同一个信号量。

// BAD: Nested semaphore — can deadlock
async function fetchAndEnrich(client: CogniteClient) {
  return cdfTaskRunner.schedule(async () => {
    const batches = await fetchBatches(client); // This also calls cdfTaskRunner.schedule!
    // If all slots are held by fetchAndEnrich callers, fetchBatches will never run
  });
}

// GOOD: Let inner functions own the semaphore
async function fetchAndEnrich(client: CogniteClient) {
  const batches = await fetchBatches(client); // Has its own semaphore call
  const enriched = await Promise.all(
    batches.map((b) => enrichBatch(client, b)) // Each has its own semaphore call
  );
  return enriched;
}

2. 忘记分页

DMS 最多返回 limit 个项目。如果不分页，数据会在不知不觉中丢失。请务必检查 nextCursor：

// BAD: May miss data
const response = await client.instances.list({ limit: 1000, /* ... */ });
const items = response.items; // Could be incomplete!

// GOOD: Paginate
const allItems = [];
let cursor;
do {
  const response = await client.instances.list({ limit: 1000, cursor, /* ... */ });
  allItems.push(...response.items);
  cursor = response.nextCursor;
} while (cursor);

3. 使用不受信号量限制的 Promise.all

并行发起大量 API 调用会立即触发 429 限制：

// BAD: Too many simultaneous requests
await Promise.all(batchIds.map((id) => client.instances.query({ /* ... */ })));

// GOOD: Each call goes through the semaphore
await Promise.all(
  batchIds.map((id) =>
    cdfTaskRunner.schedule(() => client.instances.query({ /* ... */ }))
  )
);

4. 每个表表达式的查询限制

instances.query 中的每个表表达式都有自己的 limit。如果遍历结果可能超过单个表达式的限制，就必须使用 cursors 参数分页。

5. in 筛选条件过大

每个表达式的 in 筛选条件最多可包含 1000 个值。在单个 in 筛选条件中传入超过 1000 个值，可能会失败，也可能因端点或版本而产生不完整的结果。请务必将值分批处理并运行批量请求。

---

总结检查清单

• 使用 cdfTaskRunner.schedule() 包装所有 CDF API 调用
• 使用 cursor / nextCursor 对 instances.list 调用分页
• 当数据可能超过限制时，使用 cursors / nextCursor 对 instances.query 调用分页
• 将写入操作分批，每次 instances.upsert 调用处理 1000 个项目
• 对删除操作使用单独且限制更严格的任务运行器
• 避免嵌套调用 cdfTaskRunner.schedule()，以防止死锁
• 将 Promise.all 与由信号量包装的函数搭配使用，切勿与原始 API 调用搭配使用
• 文本匹配使用 instances.search，精确匹配查询使用 filter
• 将 in 筛选条件的值拆分为每批最多 1000 个，并合并响应
• 有关当前限制，请参阅 https://docs.cognite.com/cdf/dm/dm_reference/dm_limits_and_restrictions