data-quality

CMDB 数据质量管理

概述

CMDB 的价值取决于其数据质量。数据质量差会导致变更失败、影响分析错误，并削弱信任。本技能涵盖：

• 衡量数据质量 KPI（完整性、准确性、一致性）
• 检测并解决重复 CI
• 识别和管理过时记录／孤立记录
• 实施协调流程
• 建立持续的数据治理

适用场景： 进行 CMDB 健康状况评估期间、重大变更之前、定期维护过程中，或 CMDB 可信度较低时。

价值主张： 高质量的 CMDB 数据可将变更失败率降低 50%，将事件解决时间缩短 30%，并提高审计合规性。

前置条件

• 角色： 完全访问权限需要 cmdb_admin，只读评估需要 itil
• 访问权限： cmdb_ci、cmdb_rel_ci、sys_audit 表
• 知识： CMDB 结构、识别规则
• 相关技能： cmdb/ci-discovery、cmdb/relationship-mapping

数据质量维度

维度	定义	目标	衡量方式
完整性	必填字段已填写	>95%	所有必填字段均已填写的记录百分比
准确性	数据与实际情况相符	>90%	经权威来源验证的记录百分比
一致性	不存在冲突数据	>98%	不存在冲突的记录百分比
及时性	数据为最新数据	<30 天	在阈值期限内更新的记录百分比
唯一性	不存在重复项	>99%	唯一记录的百分比
有效性	值在允许范围内	>99%	通过验证规则的记录百分比

流程

第 1 步：评估完整性

完整性用于衡量必填字段是否已填写。

按 CI 类定义必填字段：

CI 类	必填字段
cmdb_ci_server	name、ip_address、os、operational_status、support_group
cmdb_ci_appl	name、version、operational_status、support_group、install_status
cmdb_ci_service	name、operational_status、business_criticality、owned_by
cmdb_ci_database	name、type、version、operational_status

检查服务器的完整性：

Tool: SN-Query-Table
Parameters:
  table_name: cmdb_ci_server
  query: operational_status=1^nameISEMPTY^ORip_addressISEMPTY^ORsupport_groupISEMPTY
  fields: sys_id,name,ip_address,support_group
  limit: 100

全面完整性审计：

Tool: SN-Execute-Background-Script
Parameters:
  description: CMDB Completeness Audit
  script: |
    var mandatoryFields = {
      'cmdb_ci_server': ['name', 'ip_address', 'os', 'operational_status', 'support_group'],
      'cmdb_ci_appl': ['name', 'version', 'operational_status', 'support_group'],
      'cmdb_ci_service': ['name', 'operational_status', 'business_criticality', 'owned_by'],
      'cmdb_ci_database_instance': ['name', 'type', 'version', 'operational_status']
    };

    var results = {};

    for (var ciClass in mandatoryFields) {
      var fields = mandatoryFields[ciClass];
      var total = 0;
      var complete = 0;
      var incomplete = [];

      var gr = new GlideRecord(ciClass);
      gr.addQuery('operational_status', '1'); // Only operational CIs
      gr.query();

      while (gr.next()) {
        total++;
        var isComplete = true;
        var missingFields = [];

        for (var i = 0; i < fields.length; i++) {
          if (gr.getValue(fields[i]) === null || gr.getValue(fields[i]) === '') {
            isComplete = false;
            missingFields.push(fields[i]);
          }
        }

        if (isComplete) {
          complete++;
        } else {
          incomplete.push({
            sys_id: gr.sys_id.toString(),
            name: gr.name.toString(),
            missing: missingFields
          });
        }
      }

      results[ciClass] = {
        total: total,
        complete: complete,
        completeness: total > 0 ? Math.round((complete / total) * 100) : 100,
        incomplete: incomplete.slice(0, 10) // First 10 examples
      };
    }

    gs.info('=== COMPLETENESS AUDIT RESULTS ===');
    for (var cls in results) {
      var r = results[cls];
      gs.info(cls + ': ' + r.completeness + '% complete (' + r.complete + '/' + r.total + ')');
      if (r.incomplete.length > 0) {
        gs.info('  Examples of incomplete records:');
        r.incomplete.forEach(function(inc) {
          gs.info('    ' + inc.name + ' - Missing: ' + inc.missing.join(', '));
        });
      }
    }

第 2 步：检测重复项

重复项会削弱对 CMDB 的信任并引发混淆。

按名称查找潜在重复项：

Tool: SN-Execute-Background-Script
Parameters:
  description: Find duplicate CIs by name
  script: |
    var duplicates = new GlideAggregate('cmdb_ci');
    duplicates.addQuery('operational_status', '1');
    duplicates.addAggregate('COUNT');
    duplicates.groupBy('name');
    duplicates.addHaving('COUNT', '>', 1);
    duplicates.orderByAggregate('COUNT', 'DESC');
    duplicates.query();

    gs.info('=== DUPLICATE CI NAMES ===');
    var count = 0;
    while (duplicates.next() && count < 20) {
      var name = duplicates.name.toString();
      var dupCount = duplicates.getAggregate('COUNT');
      gs.info('Name: "' + name + '" - Count: ' + dupCount);

      // Show details
      var details = new GlideRecord('cmdb_ci');
      details.addQuery('name', name);
      details.addQuery('operational_status', '1');
      details.query();
      while (details.next()) {
        gs.info('  sys_id=' + details.sys_id + ', class=' + details.sys_class_name + ', ip=' + details.ip_address + ', created=' + details.sys_created_on);
      }
      count++;
    }
    gs.info('Total duplicate name groups: ' + duplicates.getRowCount());

按 IP 地址查找重复项：

Tool: SN-Execute-Background-Script
Parameters:
  description: Find duplicate CIs by IP address
  script: |
    var duplicates = new GlideAggregate('cmdb_ci_computer');
    duplicates.addQuery('operational_status', '1');
    duplicates.addNotNullQuery('ip_address');
    duplicates.addAggregate('COUNT');
    duplicates.groupBy('ip_address');
    duplicates.addHaving('COUNT', '>', 1);
    duplicates.query();

    gs.info('=== DUPLICATE IP ADDRESSES ===');
    while (duplicates.next()) {
      var ip = duplicates.ip_address.toString();
      var count = duplicates.getAggregate('COUNT');
      gs.info('IP: ' + ip + ' - Count: ' + count);

      var details = new GlideRecord('cmdb_ci_computer');
      details.addQuery('ip_address', ip);
      details.addQuery('operational_status', '1');
      details.query();
      while (details.next()) {
        gs.info('  ' + details.name + ' (' + details.sys_class_name + ')');
      }
    }

按序列号查找重复项：

Tool: SN-Query-Table
Parameters:
  table_name: cmdb_ci_hardware
  query: serial_numberISNOTEMPTY^operational_status=1
  fields: serial_number,name,sys_class_name,sys_id

然后分析重复项：

Tool: SN-Execute-Background-Script
Parameters:
  description: Find duplicate serial numbers
  script: |
    var duplicates = new GlideAggregate('cmdb_ci_hardware');
    duplicates.addQuery('operational_status', '1');
    duplicates.addNotNullQuery('serial_number');
    duplicates.addAggregate('COUNT');
    duplicates.groupBy('serial_number');
    duplicates.addHaving('COUNT', '>', 1);
    duplicates.query();

    gs.info('=== DUPLICATE SERIAL NUMBERS ===');
    gs.info('Found: ' + duplicates.getRowCount() + ' duplicate serial numbers');
    while (duplicates.next()) {
      gs.info('Serial: ' + duplicates.serial_number + ' Count: ' + duplicates.getAggregate('COUNT'));
    }

第 3 步：识别过时记录

过时记录指未在可接受的时间范围内更新的记录。

定义过时阈值：

CI 类型	过时阈值	原因
服务器	30 天	应定期扫描
应用程序	60 天	动态变化少于基础设施
服务	90 天	业务服务变化缓慢
网络	14 天	网络扫描频繁

查找过时的服务器记录：

Tool: SN-Query-Table
Parameters:
  table_name: cmdb_ci_server
  query: operational_status=1^sys_updated_on<javascript:gs.daysAgo(30)
  fields: name,ip_address,sys_updated_on,discovery_source
  limit: 100

全面过时记录审计：

Tool: SN-Execute-Background-Script
Parameters:
  description: CMDB Staleness Audit
  script: |
    var thresholds = {
      'cmdb_ci_server': 30,
      'cmdb_ci_computer': 30,
      'cmdb_ci_appl': 60,
      'cmdb_ci_service': 90,
      'cmdb_ci_network_gear': 14,
      'cmdb_ci_database_instance': 30
    };

    var results = {};
    var cutoffDate = new GlideDateTime();

    for (var ciClass in thresholds) {
      var days = thresholds[ciClass];
      cutoffDate.setValue(new GlideDateTime());
      cutoffDate.addDays(-days);

      var gr = new GlideRecord(ciClass);
      gr.addQuery('operational_status', '1');
      gr.query();
      var total = gr.getRowCount();

      var stale = new GlideRecord(ciClass);
      stale.addQuery('operational_status', '1');
      stale.addQuery('sys_updated_on', '<', cutoffDate);
      stale.query();
      var staleCount = stale.getRowCount();

      results[ciClass] = {
        total: total,
        stale: staleCount,
        threshold: days,
        freshness: total > 0 ? Math.round(((total - staleCount) / total) * 100) : 100
      };

      // Sample stale records
      var samples = [];
      stale.setLimit(5);
      stale.query();
      while (stale.next()) {
        samples.push({
          name: stale.name.toString(),
          last_updated: stale.sys_updated_on.getDisplayValue()
        });
      }
      results[ciClass].samples = samples;
    }

    gs.info('=== STALENESS AUDIT RESULTS ===');
    for (var cls in results) {
      var r = results[cls];
      gs.info(cls + ': ' + r.freshness + '% fresh (' + r.stale + ' stale of ' + r.total + ', threshold: ' + r.threshold + ' days)');
      if (r.samples.length > 0) {
        r.samples.forEach(function(s) {
          gs.info('  Stale: ' + s.name + ' (last updated: ' + s.last_updated + ')');
        });
      }
    }

第 4 步：检查关系健康状况

关系是数据质量的关键维度。

查找孤立 CI（没有关系）：

Tool: SN-Execute-Background-Script
Parameters:
  description: Find CIs with no relationships
  script: |
    var ciClasses = ['cmdb_ci_server', 'cmdb_ci_appl', 'cmdb_ci_database_instance'];
    var orphans = {};

    ciClasses.forEach(function(ciClass) {
      orphans[ciClass] = [];

      var gr = new GlideRecord(ciClass);
      gr.addQuery('operational_status', '1');
      gr.query();

      while (gr.next()) {
        var hasRel = new GlideRecord('cmdb_rel_ci');
        hasRel.addQuery('parent', gr.sys_id);
        hasRel.addOrCondition('child', gr.sys_id);
        hasRel.setLimit(1);
        hasRel.query();

        if (!hasRel.hasNext()) {
          orphans[ciClass].push({
            sys_id: gr.sys_id.toString(),
            name: gr.name.toString()
          });
        }
      }
    });

    gs.info('=== ORPHAN CI REPORT ===');
    for (var cls in orphans) {
      gs.info(cls + ': ' + orphans[cls].length + ' orphans');
      orphans[cls].slice(0, 5).forEach(function(o) {
        gs.info('  ' + o.name + ' (' + o.sys_id + ')');
      });
    }

查找无效关系：

Tool: SN-Query-Table
Parameters:
  table_name: cmdb_rel_ci
  query: parent.operational_status!=1^ORchild.operational_status!=1
  fields: parent.name,child.name,parent.operational_status,child.operational_status
  limit: 50

第 5 步：验证数据准确性

将 CMDB 数据与权威来源进行比较。

根据发现数据进行验证：

Tool: SN-Execute-Background-Script
Parameters:
  description: Compare CMDB to discovery data
  script: |
    // Find CIs where CMDB data differs from discovery
    var mismatches = [];

    var gr = new GlideRecord('cmdb_ci_server');
    gr.addQuery('operational_status', '1');
    gr.addNotNullQuery('discovery_source');
    gr.query();

    while (gr.next()) {
      var issues = [];

      // Check if last discovery was recent
      if (gr.last_discovered) {
        var lastDisc = new GlideDateTime(gr.last_discovered);
        var now = new GlideDateTime();
        var daysSince = gs.dateDiff(lastDisc, now, true) / (24 * 60 * 60 * 1000);

        if (daysSince > 30) {
          issues.push('Not discovered in ' + Math.round(daysSince) + ' days');
        }
      }

      // Check for empty critical fields that discovery should populate
      if (!gr.os || gr.os.toString() === '') {
        issues.push('OS not set');
      }
      if (!gr.cpu_count || gr.cpu_count.toString() === '0') {
        issues.push('CPU count not set');
      }
      if (!gr.ram || gr.ram.toString() === '0') {
        issues.push('RAM not set');
      }

      if (issues.length > 0) {
        mismatches.push({
          name: gr.name.toString(),
          sys_id: gr.sys_id.toString(),
          issues: issues
        });
      }
    }

    gs.info('=== ACCURACY VALIDATION ===');
    gs.info('CIs with potential accuracy issues: ' + mismatches.length);
    mismatches.slice(0, 20).forEach(function(m) {
      gs.info('  ' + m.name + ': ' + m.issues.join(', '));
    });

第 6 步：生成数据质量仪表板

创建综合质量评分：

Tool: SN-Execute-Background-Script
Parameters:
  description: Generate CMDB Data Quality Score
  script: |
    var dashboard = {
      generatedAt: new GlideDateTime().getDisplayValue(),
      totalCIs: 0,
      operationalCIs: 0,
      scores: {
        completeness: 0,
        uniqueness: 0,
        freshness: 0,
        relationships: 0
      },
      overallScore: 0,
      grade: '',
      topIssues: []
    };

    // Count total CIs
    var totalGr = new GlideAggregate('cmdb_ci');
    totalGr.addAggregate('COUNT');
    totalGr.query();
    if (totalGr.next()) {
      dashboard.totalCIs = parseInt(totalGr.getAggregate('COUNT'));
    }

    // Count operational CIs
    var opGr = new GlideAggregate('cmdb_ci');
    opGr.addQuery('operational_status', '1');
    opGr.addAggregate('COUNT');
    opGr.query();
    if (opGr.next()) {
      dashboard.operationalCIs = parseInt(opGr.getAggregate('COUNT'));
    }

    // Completeness Score (check name and class are set)
    var completeGr = new GlideAggregate('cmdb_ci');
    completeGr.addQuery('operational_status', '1');
    completeGr.addNotNullQuery('name');
    completeGr.addNotNullQuery('sys_class_name');
    completeGr.addAggregate('COUNT');
    completeGr.query();
    if (completeGr.next()) {
      var complete = parseInt(completeGr.getAggregate('COUNT'));
      dashboard.scores.completeness = Math.round((complete / dashboard.operationalCIs) * 100);
    }

    // Uniqueness Score (inverse of duplicate rate)
    var dupGr = new GlideAggregate('cmdb_ci');
    dupGr.addQuery('operational_status', '1');
    dupGr.addAggregate('COUNT');
    dupGr.groupBy('name');
    dupGr.addHaving('COUNT', '>', 1);
    dupGr.query();
    var dupGroups = dupGr.getRowCount();
    dashboard.scores.uniqueness = Math.max(0, 100 - Math.round((dupGroups / dashboard.operationalCIs) * 100 * 10));

    // Freshness Score (updated in last 30 days)
    var cutoff = new GlideDateTime();
    cutoff.addDays(-30);
    var freshGr = new GlideAggregate('cmdb_ci');
    freshGr.addQuery('operational_status', '1');
    freshGr.addQuery('sys_updated_on', '>', cutoff);
    freshGr.addAggregate('COUNT');
    freshGr.query();
    if (freshGr.next()) {
      var fresh = parseInt(freshGr.getAggregate('COUNT'));
      dashboard.scores.freshness = Math.round((fresh / dashboard.operationalCIs) * 100);
    }

    // Relationship Score (CIs with at least one relationship)
    var withRelGr = new GlideAggregate('cmdb_rel_ci');
    withRelGr.addAggregate('COUNT', 'DISTINCT', 'parent');
    withRelGr.query();
    var withRel = 0;
    if (withRelGr.next()) {
      withRel = parseInt(withRelGr.getAggregate('COUNT', 'DISTINCT', 'parent'));
    }
    dashboard.scores.relationships = Math.min(100, Math.round((withRel / dashboard.operationalCIs) * 100));

    // Overall Score (weighted average)
    dashboard.overallScore = Math.round(
      (dashboard.scores.completeness * 0.3) +
      (dashboard.scores.uniqueness * 0.25) +
      (dashboard.scores.freshness * 0.25) +
      (dashboard.scores.relationships * 0.2)
    );

    // Grade
    if (dashboard.overallScore >= 90) dashboard.grade = 'A';
    else if (dashboard.overallScore >= 80) dashboard.grade = 'B';
    else if (dashboard.overallScore >= 70) dashboard.grade = 'C';
    else if (dashboard.overallScore >= 60) dashboard.grade = 'D';
    else dashboard.grade = 'F';

    // Top Issues
    if (dashboard.scores.completeness < 90) {
      dashboard.topIssues.push('Completeness below 90%: Review mandatory field requirements');
    }
    if (dashboard.scores.uniqueness < 95) {
      dashboard.topIssues.push('Duplicate CIs detected: Run deduplication process');
    }
    if (dashboard.scores.freshness < 80) {
      dashboard.topIssues.push('Stale data: ' + (100 - dashboard.scores.freshness) + '% of CIs not updated in 30 days');
    }
    if (dashboard.scores.relationships < 70) {
      dashboard.topIssues.push('Orphan CIs: Many CIs lack relationships');
    }

    gs.info('=== CMDB DATA QUALITY DASHBOARD ===');
    gs.info('Generated: ' + dashboard.generatedAt);
    gs.info('');
    gs.info('CI Population:');
    gs.info('  Total CIs: ' + dashboard.totalCIs);
    gs.info('  Operational CIs: ' + dashboard.operationalCIs);
    gs.info('');
    gs.info('Quality Scores:');
    gs.info('  Completeness: ' + dashboard.scores.completeness + '%');
    gs.info('  Uniqueness: ' + dashboard.scores.uniqueness + '%');
    gs.info('  Freshness: ' + dashboard.scores.freshness + '%');
    gs.info('  Relationships: ' + dashboard.scores.relationships + '%');
    gs.info('');
    gs.info('OVERALL SCORE: ' + dashboard.overallScore + '% (Grade: ' + dashboard.grade + ')');
    gs.info('');
    if (dashboard.topIssues.length > 0) {
      gs.info('Top Issues to Address:');
      dashboard.topIssues.forEach(function(issue) {
        gs.info('  - ' + issue);
      });
    }

第 7 步：修复问题

修复信息不完整的记录：

Tool: SN-Update-Record
Parameters:
  table_name: cmdb_ci_server
  sys_id: [ci_sys_id]
  data:
    support_group: [group_sys_id]
    os: "Red Hat Enterprise Linux 8"
    operational_status: 1

合并重复记录：

Tool: SN-Execute-Background-Script
Parameters:
  description: Merge duplicate CIs (keep newer, retire older)
  script: |
    var keepSysId = 'NEWER_CI_SYS_ID';
    var retireSysId = 'OLDER_CI_SYS_ID';

    // Move relationships from retired CI to kept CI
    var rels = new GlideRecord('cmdb_rel_ci');
    rels.addQuery('parent', retireSysId);
    rels.query();
    while (rels.next()) {
      rels.parent = keepSysId;
      rels.update();
    }

    rels = new GlideRecord('cmdb_rel_ci');
    rels.addQuery('child', retireSysId);
    rels.query();
    while (rels.next()) {
      rels.child = keepSysId;
      rels.update();
    }

    // Retire the duplicate
    var retireCi = new GlideRecord('cmdb_ci');
    if (retireCi.get(retireSysId)) {
      retireCi.operational_status = 6; // Retired
      retireCi.install_status = 7; // Retired
      retireCi.comments = 'Merged into ' + keepSysId + ' on ' + new GlideDateTime().getDisplayValue();
      retireCi.update();
      gs.info('Retired duplicate CI: ' + retireCi.name);
    }

退役过时记录：

Tool: SN-Update-Record
Parameters:
  table_name: cmdb_ci_server
  sys_id: [stale_ci_sys_id]
  data:
    operational_status: 6
    install_status: 7
    comments: "Retired due to staleness - not discovered in 90+ days"

工具使用摘要

操作	MCP 工具	REST 端点
查询 CI	SN-Query-Table、SN-List-CmdbCis	GET /cmdb_ci
更新 CI	SN-Update-Record	PATCH /cmdb_ci/{id}
获取架构	SN-Get-Table-Schema	GET /sys_dictionary
复杂分析	SN-Execute-Background-Script	N/A
聚合查询	SN-Execute-Background-Script	GET /stats/cmdb_ci

最佳实践

• 自动化审计： 通过定时作业安排每周质量报告
• 先定义 KPI： 在开始衡量之前设定质量目标
• 优先处理关键 CI： 首先关注生产环境和关键基础设施
• 根因分析： 修复数据错误的根源，而不只是处理表面问题
• 治理流程： 为 CI 类别指定数据管理员
• 发现对齐： 确保发现模式支持质量目标
• 持续改进： 长期跟踪质量趋势

质量改进路线图

阶段	重点	目标	时间安排
1	完整性	95%	第 1-2 周
2	重复项	<1%	第 3-4 周
3	过时数据	<5%	第 5-6 周
4	关系	>80%	第 7-8 周
5	持续进行	维持	持续

故障排除

质量分数突然下降

症状： 仪表板显示质量显著下降 原因： 发现导入、批量数据加载或架构变更 解决方案： 检查 sys_import_log 中近期的批量操作；检查发现模式

找不到重复项

症状： 手动检查发现脚本未检测到的重复项 原因： 名称存在细微差异（空格、大小写、拼写错误） 解决方案： 使用模糊匹配或 Levenshtein 距离进行比较

过时记录不断重新出现

症状： 修正后，同一批 CI 仍反复被标记为过时 原因： 发现未运行，或 CI 不在发现范围内 解决方案： 确认发现计划和模式包含该 CI

已建立映射，但关系分数仍然较低

症状： 关系已存在，但分数仍然较低 原因： 只统计了父级关系，未统计子级关系 解决方案： 确保查询统计 CI 作为父级或子级出现的情况

示例

示例 1：每周质量报告

# Run weekly via scheduled job
Tool: SN-Execute-Background-Script
Parameters:
  description: Weekly CMDB Quality Report
  script: |
    // [Include dashboard script from Step 6]

    // Email results
    var email = new GlideEmailOutbound();
    email.setSubject('Weekly CMDB Quality Report - Score: ' + dashboard.overallScore + '%');
    email.setBody(JSON.stringify(dashboard, null, 2));
    email.addRecipient('[email protected]');
    email.send();

示例 2：批量去重

# 1. Identify duplicates
Tool: SN-Execute-Background-Script
Parameters:
  script: [duplicate detection script from Step 2]

# 2. Review and approve merge candidates
# (Manual review step)

# 3. Execute merges (batch)
Tool: SN-Execute-Background-Script
Parameters:
  description: Batch merge approved duplicates
  script: |
    var merges = [
      { keep: 'sys_id_1', retire: 'sys_id_2' },
      { keep: 'sys_id_3', retire: 'sys_id_4' }
    ];

    merges.forEach(function(m) {
      // [Merge logic from Step 7]
    });

示例 3：完整性修正专项活动

# 1. Export incomplete records
Tool: SN-Query-Table
Parameters:
  table_name: cmdb_ci_server
  query: operational_status=1^support_groupISEMPTY
  fields: sys_id,name,ip_address,os

# 2. Assign to teams for remediation
# (Route to support groups based on naming/IP patterns)

# 3. Batch update when teams provide data
Tool: SN-Update-Record
Parameters:
  table_name: cmdb_ci_server
  sys_id: [ci_sys_id]
  data:
    support_group: [group_sys_id]

# 4. Re-run completeness audit to verify

相关技能

• cmdb/ci-discovery - CI 创建和分类
• cmdb/relationship-mapping - 管理 CI 关系
• cmdb/impact-analysis - 使用高质量 CMDB 进行影响分析
• cmdb/service-graph-diagnosis - 诊断拓扑和发现质量差距

参考资料

• ServiceNow CMDB Health
• CMDB Data Manager
• ServiceNow Identification and Reconciliation
• ITIL Configuration Management
• Gartner CMDB Best Practices