implementing-cloud-dlp-for-data-protection

为数据保护实施云 DLP

何时使用

• 当合规框架（GDPR、HIPAA、PCI DSS）要求自动发现和保护敏感数据时
• 当构建跨云存储对数据进行分类和标记的数据治理计划时
• 当为基于云的数据管道实施数据丢失防护控制时
• 当审计云环境中未受保护的敏感数据（PII、PHI、财务数据）时
• 当将 DLP 扫描集成到 CI/CD 管道以防止敏感数据进入生产环境时

不要使用于端点 DLP（使用 Microsoft Purview 或 Symantec DLP Agent），不要用于电子邮件 DLP（使用 Microsoft 365 DLP 或 Google Workspace DLP），也不要用于网络级数据外泄防护（使用 VPC 端点策略和网络防火墙）。

先决条件

• 已启用 Amazon Macie 并具备适当的 S3 存储桶权限
• 已启用 Google Cloud DLP API（gcloud services enable dlp.googleapis.com）
• 已配置 Azure Information Protection 或 Microsoft Purview
• 具备用于 DLP 服务管理和数据访问的 IAM 权限
• 了解与组织相关的数据敏感度类别（PII、PHI、PCI、专有数据）

工作流

步骤 1：部署 Amazon Macie 以进行 S3 数据发现

启用 Macie 并为 S3 存储桶配置自动化敏感数据发现任务。

# Enable Amazon Macie
aws macie2 enable-macie

# List all S3 buckets Macie can scan
aws macie2 describe-buckets \
  --query 'buckets[*].[bucketName,classifiableSizeInBytes,unclassifiableObjectCount.total]' \
  --output table

# Create a classification job for specific buckets
aws macie2 create-classification-job \
  --job-type SCHEDULED \
  --name "weekly-pii-scan" \
  --schedule-frequency-details '{"weekly":{"dayOfWeek":"MONDAY"}}' \
  --s3-job-definition '{
    "bucketDefinitions": [{
      "accountId": "ACCOUNT_ID",
      "buckets": ["customer-data-bucket", "analytics-data-lake", "backup-bucket"]
    }],
    "scoping": {
      "includes": {
        "and": [{
          "simpleScopeTerm": {
            "key": "OBJECT_EXTENSION",
            "values": ["csv", "json", "parquet", "txt", "xlsx"],
            "comparator": "EQ"
          }
        }]
      }
    }
  }' \
  --managed-data-identifier-ids '["SSN","CREDIT_CARD_NUMBER","EMAIL_ADDRESS","AWS_CREDENTIALS","PHONE_NUMBER"]'

# Create custom data identifier for internal employee IDs
aws macie2 create-custom-data-identifier \
  --name "EmployeeID" \
  --regex "EMP-[0-9]{6}" \
  --description "Internal employee ID format"

# Check job status and results
aws macie2 list-classification-jobs \
  --query 'items[*].[name,jobStatus,statistics.approximateNumberOfObjectsToProcess]' \
  --output table

步骤 2：配置 Google Cloud DLP API 以进行数据检查

使用 Google Cloud DLP 检查并去标识化 GCP 资源中的敏感数据。

# Inspect a Cloud Storage bucket for sensitive data
gcloud dlp inspect-content \
  --content-type=TEXT_PLAIN \
  --min-likelihood=LIKELY \
  --info-types=PHONE_NUMBER,EMAIL_ADDRESS,CREDIT_CARD_NUMBER,US_SOCIAL_SECURITY_NUMBER \
  --storage-type=CLOUD_STORAGE \
  --gcs-uri="gs://sensitive-data-bucket/data/*.csv"

# Create an inspection job for BigQuery
cat > dlp-job.json << 'EOF'
{
  "inspectJob": {
    "storageConfig": {
      "bigQueryOptions": {
        "tableReference": {
          "projectId": "PROJECT_ID",
          "datasetId": "customer_data",
          "tableId": "transactions"
        },
        "sampleMethod": "RANDOM_START",
        "rowsLimit": 10000
      }
    },
    "inspectConfig": {
      "infoTypes": [
        {"name": "CREDIT_CARD_NUMBER"},
        {"name": "US_SOCIAL_SECURITY_NUMBER"},
        {"name": "EMAIL_ADDRESS"},
        {"name": "PHONE_NUMBER"},
        {"name": "PERSON_NAME"}
      ],
      "minLikelihood": "LIKELY",
      "limits": {"maxFindingsPerRequest": 1000}
    },
    "actions": [{
      "saveFindings": {
        "outputConfig": {
          "table": {
            "projectId": "PROJECT_ID",
            "datasetId": "dlp_results",
            "tableId": "findings"
          }
        }
      }
    }]
  }
}
EOF

gcloud dlp jobs create --project=PROJECT_ID --body-from-file=dlp-job.json

步骤 3：使用 Cloud DLP 实施数据去标识化

配置去标识化转换，以掩码、令牌化或脱敏敏感数据。

# deidentify_pipeline.py - De-identify sensitive data using Google Cloud DLP
from google.cloud import dlp_v2

def deidentify_data(project_id, text):
    """De-identify PII in text using Cloud DLP."""
    client = dlp_v2.DlpServiceClient()

    inspect_config = {
        "info_types": [
            {"name": "EMAIL_ADDRESS"},
            {"name": "PHONE_NUMBER"},
            {"name": "CREDIT_CARD_NUMBER"},
            {"name": "US_SOCIAL_SECURITY_NUMBER"},
        ],
        "min_likelihood": dlp_v2.Likelihood.LIKELY,
    }

    deidentify_config = {
        "info_type_transformations": {
            "transformations": [
                {
                    "info_types": [{"name": "EMAIL_ADDRESS"}],
                    "primitive_transformation": {
                        "character_mask_config": {
                            "masking_character": "*",
                            "number_to_mask": 0,
                            "characters_to_ignore": [
                                {"common_characters_to_ignore": "PUNCTUATION"}
                            ],
                        }
                    },
                },
                {
                    "info_types": [{"name": "CREDIT_CARD_NUMBER"}],
                    "primitive_transformation": {
                        "crypto_replace_ffx_fpe_config": {
                            "crypto_key": {
                                "kms_wrapped": {
                                    "wrapped_key": "WRAPPED_KEY_BASE64",
                                    "crypto_key_name": "projects/PROJECT/locations/global/keyRings/dlp/cryptoKeys/tokenization",
                                }
                            },
                            "common_alphabet": "NUMERIC",
                        }
                    },
                },
                {
                    "info_types": [{"name": "US_SOCIAL_SECURITY_NUMBER"}],
                    "primitive_transformation": {
                        "redact_config": {}
                    },
                },
            ]
        }
    }

    item = {"value": text}
    parent = f"projects/{project_id}/locations/global"

    response = client.deidentify_content(
        request={
            "parent": parent,
            "deidentify_config": deidentify_config,
            "inspect_config": inspect_config,
            "item": item,
        }
    )
    return response.item.value

步骤 4：配置 Azure Information Protection

在 Microsoft Purview 中为 Azure 资源设置敏感度标签和 DLP 策略。

# Connect to Microsoft Purview compliance
Connect-IPPSSession

# Create sensitivity labels
New-Label -DisplayName "Confidential - PII" \
  -Name "Confidential-PII" \
  -Tooltip "Contains personally identifiable information" \
  -ContentType "File, Email"

New-Label -DisplayName "Highly Confidential - Financial" \
  -Name "HighlyConfidential-Financial" \
  -Tooltip "Contains financial data subject to PCI DSS" \
  -ContentType "File, Email"

# Create auto-labeling policy for Azure Storage
New-AutoSensitivityLabelPolicy -Name "Auto-Label-PII" \
  -ExchangeLocation All \
  -SharePointLocation All \
  -OneDriveLocation All \
  -Mode Enable

New-AutoSensitivityLabelRule -Policy "Auto-Label-PII" \
  -Name "Detect-SSN" \
  -ContentContainsSensitiveInformation @{
    Name = "U.S. Social Security Number (SSN)";
    MinCount = 1;
    MinConfidence = 85
  } \
  -ApplySensitivityLabel "Confidential-PII"

# Azure: Configure DLP policy for Storage accounts
az security assessment create \
  --name "storage-sensitive-data" \
  --assessed-resource-type "Microsoft.Storage/storageAccounts"

# Enable Microsoft Defender for Storage with sensitive data threat detection
az security pricing create --name StorageAccounts --tier standard \
  --subplan DefenderForStorageV2 \
  --extensions '[{"name":"SensitiveDataDiscovery","isEnabled":"True"}]'

步骤 5：将 DLP 集成到数据管道

将 DLP 扫描添加到 ETL 和数据管道工作流中，以防止敏感数据泄漏。

# pipeline_dlp_gate.py - DLP gate for data pipelines
import boto3
import json

macie_client = boto3.client('macie2')
s3_client = boto3.client('s3')

def scan_pipeline_output(bucket, prefix):
    """Scan pipeline output data for sensitive content before promotion."""
    job_response = macie_client.create_classification_job(
        jobType='ONE_TIME',
        name=f'pipeline-scan-{prefix}',
        s3JobDefinition={
            'bucketDefinitions': [{
                'accountId': boto3.client('sts').get_caller_identity()['Account'],
                'buckets': [bucket]
            }],
            'scoping': {
                'includes': {
                    'and': [{
                        'simpleScopeTerm': {
                            'key': 'OBJECT_KEY',
                            'comparator': 'STARTS_WITH',
                            'values': [prefix]
                        }
                    }]
                }
            }
        },
        managedDataIdentifierSelector='ALL'
    )

    return job_response['jobId']

def check_scan_results(job_id):
    """Check if DLP scan found sensitive data."""
    response = macie_client.list_findings(
        findingCriteria={
            'criterion': {
                'classificationDetails.jobId': {'eq': [job_id]},
                'severity.description': {'eq': ['High', 'Critical']}
            }
        }
    )
    return len(response.get('findingIds', [])) > 0

def gate_decision(bucket, prefix):
    """DLP gate: block pipeline if sensitive data found."""
    job_id = scan_pipeline_output(bucket, prefix)
    has_sensitive_data = check_scan_results(job_id)

    if has_sensitive_data:
        return {
            'decision': 'BLOCK',
            'reason': 'Sensitive data detected in pipeline output',
            'action': 'Apply de-identification before promoting to production'
        }
    return {'decision': 'ALLOW', 'reason': 'No sensitive data detected'}

步骤 6：监控 DLP 发现结果并生成报告

汇总各云服务提供商的 DLP 发现结果，并生成合规报告。

# Macie: Get finding statistics
aws macie2 get-finding-statistics \
  --group-by "severity.description" \
  --finding-criteria '{"criterion":{"category":{"eq":["CLASSIFICATION"]}}}'

# Macie: List findings by sensitivity type
aws macie2 list-findings \
  --finding-criteria '{
    "criterion": {
      "classificationDetails.result.sensitiveData.category": {"eq": ["PERSONAL_INFORMATION"]},
      "severity.description": {"eq": ["High"]}
    }
  }' \
  --sort-criteria '{"attributeName": "updatedAt", "orderBy": "DESC"}'

# GCP DLP: List job results
gcloud dlp jobs list --project=PROJECT_ID --filter="state=DONE" \
  --format="table(name, createTime, inspectDetails.result.processedBytes, inspectDetails.result.totalEstimatedTransformations)"

# Export Macie findings to S3 for compliance reporting
aws macie2 create-findings-report \
  --finding-criteria '{"criterion":{"category":{"eq":["CLASSIFICATION"]}}}' \
  --sort-criteria '{"attributeName":"severity.score","orderBy":"DESC"}'

关键概念

术语	定义
数据丢失防护	检测并防止敏感数据从云环境未经授权披露的安全控制和技术
Amazon Macie	AWS 服务，使用机器学习发现、分类并保护存储在 S3 存储桶中的敏感数据
Google Cloud DLP	GCP API，用于检查、分类并去标识化 Cloud Storage、BigQuery 和 Datastore 中的敏感数据
数据去标识化	使用掩码、令牌化、加密或脱敏转换敏感数据，以移除可识别特征，同时保留其可用性
敏感度标签	应用于数据的分类标签（机密、高度机密），会触发 DLP 策略执行和访问控制
自定义数据标识符	添加到 DLP 服务的组织特定模式（正则表达式或关键词），用于检测专有敏感数据格式

工具与系统

• Amazon Macie：面向 S3 的 ML 驱动的敏感数据发现与分类，并自动生成发现结果
• Google Cloud DLP API：用于检查、分类、去标识化和脱敏敏感数据的可编程 API
• Microsoft Purview：数据治理平台，具备敏感度标记、自动分类和 DLP 策略执行
• Azure Information Protection：与 Microsoft 365 和 Azure 存储集成的数据分类与标记服务
• Nightfall AI：第三方云 DLP 工具，支持跨 SaaS 应用和云基础设施进行扫描

常见场景

场景：在未受保护的 S3 数据湖中发现 PII

背景：一次合规审计发现，分析团队的 S3 数据湖中，CSV 文件包含客户 PII（姓名、邮箱、SSN），且没有加密或访问控制。组织必须对所有数据进行分类，并实施 DLP 控制。

方法：

1. 启用 Macie，并针对数据湖存储桶创建一次性分类作业
2. 查看 Macie 发现结果，识别哪些对象包含 PII 以及包含哪些类型
3. 为组织特定格式（员工 ID、账号）创建自定义数据标识符
4. 实施每周计划的 Macie 作业，以持续进行发现
5. 构建数据管道门禁，在将新数据提升到数据湖之前进行扫描
6. 为分析用例应用去标识化转换（掩码 SSN、令牌化邮箱）
7. 配置 S3 存储桶策略，仅允许授权角色访问已分类数据

陷阱：Macie 按扫描的 GB 收费。大型数据湖可能产生显著成本。使用范围规则聚焦高风险对象类型（CSV、JSON、Parquet），并排除已知安全的格式（压缩归档、二进制文件）。去标识化必须为分析保留数据效用，同时消除再识别风险。

输出格式

Cloud DLP Compliance Report
==============================
Organization: Acme Corp
Scan Period: 2026-02-01 to 2026-02-23
Environments: AWS (12 buckets), GCP (3 datasets), Azure (5 storage accounts)

DATA DISCOVERY SUMMARY:
  Total objects/records scanned:    2,847,000
  Objects with sensitive data:        45,200 (1.6%)
  Unique sensitivity categories:      8

SENSITIVE DATA FINDINGS:
  PII (names, emails, phone):       23,400 objects
  Financial (credit cards, bank):     8,700 objects
  Health (PHI, medical records):      3,200 objects
  Credentials (API keys, tokens):     1,400 objects
  Government ID (SSN, passport):      5,800 objects
  Custom (employee ID, account):      2,700 objects

FINDINGS BY SEVERITY:
  Critical:    1,400 (exposed credentials)
  High:       14,200 (unprotected PII/PHI)
  Medium:     18,600 (standard PII)
  Low:        11,000 (non-sensitive patterns)

PROTECTION STATUS:
  Data with encryption at rest:       78%
  Data with access controls:          65%
  Data with sensitivity labels:       12%
  Pipeline data with DLP gates:       30%

REMEDIATION ACTIONS:
  Objects quarantined:                1,400
  De-identification applied:          8,200
  Access controls tightened:         14,200
  Sensitivity labels applied:        45,200