google-cloud-solution-multi-agent-security

Agent Gateway multi-agent security

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read and follow the instructions in Usage Attribution.

Critical Enforcement Rules & Rationale

• Gcloud Release Tracks: Always use the exact release tracks specified in the commands (e.g., gcloud beta network-services agent-gateways). Omitting these prefixes causes commands to fail because Agent Gateway features are located in specialized, non-default namespaces.
• API Enablement: Include modelarmor.googleapis.com in the API enablement list when setting up guardrails. Excluding it prevents Model Armor policies and filters from successfully attaching to the Gateway.
• Egress Verification: Egress policy verification requires using the Python script (scripts/verify_egress_policies.py), not curl. Egress gateways rely on runtime SDK lifecycle handling and JWT context that a standard curl command cannot simulate correctly.
• Model Armor Keys: In model-armor-config.yaml, always include both piAndJailbreakFilterSettings and sdpFilterSettings (filterEnforcement: ENFORCE). Invalid or missing filters cause deployment validation failures or lead to silent bypasses of the guardrails.
• Subnet Private Access: Any subnet hosting a Private Service Connect network attachment for Egress Gateways must have private_ip_google_access = true enabled in Terraform. Disabling this blocks connectivity to Google-managed endpoints, causing total routing failures for agents.
• Direct Delivery: Immediately provide the requested architecture, configuration files, CLI commands, scripts, and diagrams in full. Do not stop at a planning phase, do not generate a plan artifact, and do not ask for user confirmation before delivering outputs.
• No Infrastructure Execution: Do not attempt to run deployment or verification commands (such as gcloud, kubectl, terraform, or curl) against real cloud resources during design. You are generating plan configurations, not executing them.

Important

Just-In-Time (JIT) Resource Loading Protocol: Inspect template files in assets/ and executable scripts in scripts/ using view_file as needed for extended configurations, deployment scripts, and test suites.

---

Quick Reference: Required Filenames

Always generate files with these exact names when requested:

1. agw-ingress-config.yaml (assets/agw-ingress-config.yaml)
2. agw-egress-config.yaml (assets/agw-egress-config.yaml)
3. agw-authz-extension.yaml (assets/agw-authz-extension.yaml)
4. agw-authz-policy.yaml (assets/agw-authz-policy.yaml)
5. model-armor-config.yaml (assets/model-armor-config.yaml)
6. sgp-policy.yaml (assets/sgp-policy.yaml)
7. iap-policy.json (assets/iap-policy.json)
8. model-armor-payload.json (assets/model-armor-payload.json)

---

1. Dual Ingress & Egress Architecture Design (dual_ingress_egress_architecture_design)

• Ingress Pattern: CLIENT_TO_AGENT fronted by Ingress Control Plane (Agent Gateway, Model Armor).

• Egress Pattern: AGENT_TO_ANYWHERE utilizing Egress Control Plane (Agent Gateway, roles/iap.egressor CEL policies, Cloud DNS) and Egress Data Plane (PSC Interface, Cloud Run, PSC Google APIs Global Endpoint), coordinated via Agent Registry & Agent Engine runtime.

• Mermaid Diagram:

  graph TD
      Client["External Clients"] -->|HTTPS / MCP| GLB["Global Load Balancer"]
      GLB --> Ingress["Ingress Agent Gateway (CLIENT_TO_AGENT)"]
      Ingress --> MA["Model Armor (CONTENT_AUTHZ)"]
      MA --> Agent["Agent Engine Agents (BillingAgent, SupportAgent, FraudAgent)"]
      Agent --> Egress["Egress Agent Gateway (AGENT_TO_ANYWHERE)"]
      Egress --> PSC["Private Service Connect Network Attachment"]
      PSC --> Tools["Private MCP Tool Backends"]
  

---

2. Ingress & Egress Guardrail Policy Config (ingress_and_egress_guardrail_policy_config)

When requested for Ingress & Egress guardrail policy configs, you MUST generate and create all required files in the workspace:

• agw-ingress-config.yaml (assets/agw-ingress-config.yaml): Declares governedAccessPath: CLIENT_TO_AGENT with protocols HTTP and MCP.
• agw-egress-config.yaml (assets/agw-egress-config.yaml): Declares governedAccessPath: AGENT_TO_ANYWHERE with protocol MCP.
• agw-authz-extension.yaml (assets/agw-authz-extension.yaml): Configures AuthzExtension service for IAP authorization.
• agw-authz-policy.yaml (assets/agw-authz-policy.yaml): Configures AuthzPolicy action ALLOW targeting both Ingress and Egress gateways.
• iap-policy.json (assets/iap-policy.json): Binds roles/iap.egressor with CEL condition checking iap.googleapis.com/mcp.toolName == 'get_account_balance' && iap.googleapis.com/mcp.tool.isReadOnly == true.
• model-armor-config.yaml (assets/model-armor-config.yaml): Enables piAndJailbreakFilterSettings and sdpFilterSettings with filterEnforcement: ENFORCE.
• sgp-policy.yaml (assets/sgp-policy.yaml): Implements Natural Language Constraints blocking transactions > $1000 and sanitizing PII.

---

3. Ingress & Egress Infrastructure Deployment (ingress_and_egress_infrastructure_deployment)

Inspect and provide the step-by-step gcloud CLI commands from scripts/deploy_infrastructure.sh:

1. Enable Required APIs: compute, networkservices, networksecurity, modelarmor, iap, agentregistry, serviceextensions, and aiplatform.
2. Import Agent Gateways: Ingress (agw-ingress-config.yaml) and Egress (agw-egress-config.yaml) via gcloud alpha network-services agent-gateways import.
3. Import Authz Extension: agw-authz-extension.yaml via gcloud beta service-extensions authz-extensions import.
4. Import Authz Policy: agw-authz-policy.yaml via gcloud beta network-security authz-policies import.

---

4. Ingress & Egress Security Validation (ingress_and_egress_security_validation)

When validating security for Ingress and Egress:

1. Ingress 403 Unauthenticated Test: Provide the copy-pasteable verification curl command from scripts/validate_ingress_unauth.sh sending an unauthenticated POST request to the Reasoning Engine endpoint expecting HTTP 403 Forbidden.
2. Python Egress Verification Script (MUST use Python script snippet, NOT curl): Provide the Python verification script snippet from scripts/verify_egress_policies.py sending JSON-RPC tools/call requests (get_account_balance) through the Egress Gateway to verify HTTP 200 for allowed tools.
3. Model Armor Test Payload: Generate model-armor-payload.json (assets/model-armor-payload.json) containing prompt injection/jailbreak instructions.

---

5. Troubleshooting Ingress & Egress Failures (troubleshooting_ingress_and_egress_failures)

• Ingress 403 (Client-to-Agent):

  • Root Cause: Unauthenticated client requests or missing/invalid OAuth 2.0 / IAP identity tokens.
  • OAuth Configuration Steps:
    1. Configure OAuth 2.0 Client ID credentials in Google Cloud Console.
    2. Grant the client identity / service account roles/iap.httpsResourceAccessor permission.
    3. Exchange credentials with Google OAuth to acquire an OIDC / OAuth ID token.
    4. Pass the token in the Authorization: Bearer <TOKEN> header.
  • Verification Command: Provide the curl command from scripts/verify_ingress_auth.sh.

• Egress 403 (Agent-to-Anywhere):

  • Root Cause: Missing roles/iap.egressor IAM bindings on the Agent Identity, malformed principal ID, or mismatched CEL condition on tool metadata.
  • Fix Command: Provide the exact gcloud command from scripts/fix_egress_iap.sh.

---

6. Hybrid VPN Connectivity & Egress Routing (hybrid_vpn_connectivity_egress_routing)

• Terraform HCL: Refer to baseline Terraform config in assets/main.tf for VPC, subnets (private_ip_google_access = true), PSC network attachment, Cloud DNS private forwarding for aws.internal., and HA VPN gateway/router.
• Egress Gateway Config (agw-egress-config.yaml): Generate configuration declaring governedAccessPath: AGENT_TO_ANYWHERE, pointing to the PSC network attachment, and referencing aws.internal. in dnsPeeringConfig (see assets/agw-egress-config.yaml).
• Python SDK Deployment Script: Refer to scripts/hybrid_vpn_agent.py for the complete script initializing Vertex AI with agent_to_anywhere_config referencing the Egress Gateway, enabling telemetry, and deploying HybridAgent using types.IdentityType.AGENT_IDENTITY.

---

7. Private Egress GKE Internal Load Balancer (private_egress_gke_internal_load_balancer)

• Expose GKE internal MCP tool server via an Internal Load Balancer (ILB) at literal IP 10.0.1.50, connecting via Agent Gateway PSC Interface + Cloud DNS Private zone.
• Cloud DNS Record Mapping: Provide the command from scripts/create_gke_dns_record.sh mapping the private domain to GKE's private ILB IP 10.0.1.50.
• Explicit TLS Warning: Agent Gateway egress does not natively trust self-signed certificates or private enterprise CAs. You must use publicly trusted TLS certificates signed by a trusted Certificate Authority (e.g., Let's Encrypt).

---

8. Governance Controls Model Armor SGP (governance_controls_model_armor_sgp)

When configuring dual safety layers with Model Armor on Ingress and SGP on Egress:

1. Model Armor Config: Generate model-armor-config.yaml (assets/model-armor-config.yaml) with piAndJailbreakFilterSettings and sdpFilterSettings (filterEnforcement: ENFORCE).
2. Semantic Governance Policy: Generate sgp-policy.yaml (assets/sgp-policy.yaml) with Natural Language Constraints blocking transactions > $1000 and sanitizing PII.
3. Curl PATCH Command: Provide the curl command from scripts/enforce_sgp_patch.sh to update authzExtensions with sgpEnforcementMode set to ENFORCE.

---

9. Multi-Agent Cloud Run Egress Routing (multi_agent_cloud_run_egress_routing)

Do NOT produce a plan artifact or stop at planning. When configuring multi-agent Cloud Run egress routing, you MUST directly provide and generate ALL required components:

1. Egress Gateway Config (agw-egress-config-run.yaml): Generate configuration declaring governedAccessPath: AGENT_TO_ANYWHERE, PSC network attachment, and DNS peering for *.run.app (see assets/agw-egress-config-run.yaml).
2. Register Cloud Run Services in Agent Registry: Provide the registration commands from scripts/register_cloud_run_services.sh registering all 3 Cloud Run services (marketing-tool-service, sales-tool-service, support-tool-service) in the us-east4 Agent Registry.
3. iap-policy.json (Multi-Agent): Generate iap-policy.json (assets/iap-policy-multi-agent.json) containing all 3 principal:// bindings in the members list under roles/iap.egressor.
4. Python SDK Deployment Script: Refer to scripts/multi_agent_cloud_run.py for the complete GenAI SDK deployment script.

---

10. Advanced Model Armor Filtering (advanced_model_armor_filtering)

For custom keyword matching, configure userDefinedFilterSettings (see assets/model-armor-advanced.yaml).

---

11. Known Traps & Gotchas (known_traps_and_gotchas)

• network_attachment is ForceNew: Enabling Semantic Governance Policies (SGP) or modifying network attachments after the initial Terraform apply will force-recreate the gateway resource. If not managed carefully, this can cause dependency deadlocks during destroy operations. Plan infrastructure sequencing accordingly.
• Authz Policy Limit: An Agent Gateway allows at most 4 custom authorization policies attached concurrently. Ensure your security posture consolidates rules within this limit.