Support AI Human‑in‑the‑Loop: 30‑Day Enablement Plan

```yaml
policy_id: sup-hitl-2025-01
version: 1.3
owners:
  support_ops: "@maria.gonzalez"
  qa_lead: "@li.chen"
  legal_contact: "@alex.reid"
regions:
  - us-east-1
  - eu-central-1
services:
  ticketing: zendesk
  approvals: slack
  knowledge: confluence
  telemetry_sink: snowflake
models:
  low_risk: azure_openai:gpt-4o-mini (vpc:true, log_prompts:true)
  high_risk: mistral-large (vpc:true, log_prompts:true)
knowledge_sources:
  authoritative_spaces:
    - space: PROD_HELP
      freshness_days: 14
    - space: SECURITY
      freshness_days: 7
risk_tiers:
  low:
    intents: ["password_reset", "status_check", "invoice_copy"]
    pii_allowed: false
    auto_send: true
  medium:
    intents: ["feature_howto", "basic_troubleshoot"]
    pii_allowed: masked
    auto_send: false
    reviewer_role: team_lead
  high:
    intents: ["security_incident", "data_deletion", "billing_dispute"]
    pii_allowed: masked
    auto_send: false
    reviewer_role: sme
thresholds:
  min_confidence: 0.78
  min_retrieval_coverage: 0.85   # proportion of tokens sourced from authoritative KB
  max_generation_time_ms: 4500
  csat_guardrail_pred: 0.72       # below -> require review regardless of tier
approval_workflow:
  medium:
    route: slack
    channel: "#support-approvals"
    timeout_seconds: 900
    fallback: escalate_to: "@oncall-lead"
  high:
    route: slack
    channel: "#sme-approvals"
    timeout_seconds: 1200
    fallback: escalate_to: "@oncall-sme"
qa_sampling:
  low: 0.05
  medium: 0.25
  high: 1.0
rbac:
  approvers:
    team_lead: ["approve_medium", "reject"]
    sme: ["approve_high", "reject"]
  agents:
    can_send_low_without_review: true
telemetry:
  log_fields: [ticket_id, agent_id, reviewer_id, prompt, response, citations, confidence, retrieval_coverage, decision, latency_ms]
  destinations:
    - snowflake.schema.support_ai_logs
    - zendesk.tags
  retention_days: 365
slos:
  review_latency_p50_seconds: 420
  review_latency_p95_seconds: 1200
  aht_low_risk_seconds: 210
  reopen_rate_target: 0.11
error_budget:
  reopen_rate_budget: 0.02   # if exceeded, reduce auto-send coverage by 20%
rollout:
  wave1:
    queues: ["tech_troubleshooting"]
    team: ["emea_t1"]
    start: 2025-01-10
    end: 2025-01-24
  wave2:
    queues: ["billing", "how_to"]
    condition: "reopen_rate < 0.12 for 14 days"
review_rubric:
  link: https://confluence.example.com/display/SUP/HITL+Rubric
  sections: ["Source present", "Scope correct", "Tone compliant"]
```

{
  "title": "Support AI Human‑in‑the‑Loop: 30‑Day Enablement Plan",
  "published_date": "2025-12-09",
  "author": {
    "name": "David Kim",
    "role": "Enablement Director",
    "entity": "DeepSpeed AI"
  },
  "core_concept": "AI Adoption and Enablement",
  "key_takeaways": [
    "Human-in-the-loop isn’t a checkbox; it’s a design pattern that maps risk tiers to approval paths and telemetry.",
    "Start with confidence and retrieval coverage thresholds; route medium/high risk to humans with clear SLOs.",
    "Train agents on a short rubric and source-citation habits; measure reopens, AHT, and CSAT weekly.",
    "Instrument prompt logging and RBAC so Legal is comfortable and QA has evidence.",
    "Run a 30-day audit → pilot → scale motion with one queue, one team lead, and real SLOs before expanding."
  ],
  "faq": [
    {
      "question": "Will approvals slow agents down?",
      "answer": "For low-risk intents, auto-send remains. Medium/high-risk approval SLOs are under seven minutes p50, and AHT still decreased after removing rework from reopens."
    },
    {
      "question": "How do we pick thresholds?",
      "answer": "Start conservative. Use two weeks of telemetry to see where reopens correlate with low retrieval coverage or confidence, then raise auto-send coverage on safe intents."
    },
    {
      "question": "What if our KB is stale?",
      "answer": "HITL exposes exactly which pages fail retrieval. We prioritize those gaps and set freshness windows by space so bad sources can’t auto-ship."
    },
    {
      "question": "Which tools do you integrate?",
      "answer": "Zendesk and ServiceNow for ticketing; Confluence/Guru for knowledge; Slack/Teams for approvals; Snowflake/BigQuery for logs; Looker/Power BI for reporting."
    }
  ],
  "business_impact_evidence": {
    "organization_profile": "Mid-market B2B SaaS, 200 agents on Zendesk, Confluence KB, US/EU regions.",
    "before_state": "Agents used AI drafts without consistent thresholds; outdated articles caused wrong citations and reopens spiked.",
    "after_state": "Risk-tiered workflows with approvals, telemetry in Snowflake, and weekly KB refresh. Auto-send limited to low-risk with high retrieval coverage.",
    "metrics": [
      "Reopens reduced 27% on pilot queue (14 days).",
      "AHT down 14% on low-risk intents; no SLA breach during volume spike.",
      "CSAT up 2.1 points in EMEA; 40% fewer QA flags on hallucinations."
    ],
    "governance": "Legal approved because prompts/responses/citations are logged with reviewer IDs; RBAC restricts approvals; data stays in-region; and models never train on client data. Human-in-the-loop enforced by policy with documented thresholds and sampling."
  },
  "summary": "Head of Support playbook: coach human-in-the-loop design in 30 days. Cut reopens, protect CSAT, and ship governed copilot workflows with audit trails."
}