Human-in-the-Loop Support: 30-Day Enablement Plan

```yaml
playbook:
  name: "Support HITL Pilot - Billing & Tier2 Tech"
  version: "v1.3"
  owners:
    support_ops: "alex.robbins@company.com"
    qa_lead: "maria.nguyen@company.com"
    security_partner: "sec-gov@company.com"
    deepspeed_pm: "pilot@deepspeedai.com"
  regions:
    - us-east
    - eu-central
  data_residency:
    us-east: "us-only"
    eu-central: "eu-only"
  queues:
    - name: "Billing-T1"
      tool: "Zendesk"
      intents: ["refund", "invoice_error", "downgrade", "tax_id"]
    - name: "Tech-T2"
      tool: "ServiceNow"
      intents: ["login_lockout", "oauth_error", "org_migration"]
  hitl_thresholds:
    # Confidence bands from retrieval + model agreement
    accept_auto: 
      threshold: 0.88
      allowed_intents: ["refund", "login_lockout"]
      post_accept_sampling: 0.10   # 10% sampled to QA
    require_review:
      lower: 0.60
      upper: 0.88
      action: "agent_review_required"
    escalate:
      threshold: 0.60
      action: "macro_fallback + route_to_t2"
  trust_signals:
    display:
      - confidence_band
      - sources: ["KB", "changelog", "entitlements"]
      - policy_checks: ["pii_present", "tone_risk", "export_control"]
  reason_codes:
    accept:
      - "A1_correct_complete"
      - "A2_minor_tone_edit"
    reject:
      - "R1_incorrect_fact"
      - "R2_missing_source"
      - "R3_off_brand_tone"
      - "R4_policy_blocked"
    escalate:
      - "E1_complex_edge_case"
      - "E2_entitlement_conflict"
  qa_rubric:
    dimensions:
      - name: "factual_accuracy"
        weight: 0.4
      - name: "policy_compliance"
        weight: 0.3
      - name: "tone_alignment"
        weight: 0.2
      - name: "resolution_completeness"
        weight: 0.1
    pass_threshold: 0.85
    slo:
      review_time_hours: 48
      sample_rate:
        Billing-T1: 0.10
        Tech-T2: 0.15
  approvals:
    macro_changes:
      required_roles: ["SupportOps", "QA", "Security"]
      steps:
        - { step: "draft", owner: "SupportOps" }
        - { step: "qa_review", owner: "QA" }
        - { step: "security_check", owner: "Security" }
        - { step: "publish", owner: "SupportOps" }
    threshold_changes:
      change_window_cron: "0 17 * * FRI"
      approvers: ["SupportOps", "Security"]
  rbac:
    roles:
      Agent:
        actions: ["accept", "edit", "reject", "escalate"]
      Supervisor:
        actions: ["view_metrics", "approve_sampling_overrides"]
      SupportOps:
        actions: ["update_thresholds", "edit_macros"]
      Security:
        actions: ["view_prompt_logs", "approve_policy_checks"]
  observability:
    telemetry:
      - accept_rate
      - edit_rate
      - reject_rate
      - escalate_rate
      - reopen_rate
      - csat_delta
    sinks:
      - "snowflake.de_support.hitl_events"
      - "s3://support-hitl-pilot-logs/"
  pii_controls:
    redaction: true
    prompt_logging: true
    secrets_store: "AWS Secrets Manager"
  training:
    weekly_calibration:
      day: "Wednesday"
      duration_minutes: 30
      participants: ["Supervisors", "QA", "Top 5 agents"]
    office_hours:
      times: ["Tue 10:00", "Thu 15:00"]
      host: "DeepSpeed Enablement"
```

{
  "title": "Human-in-the-Loop Support: 30-Day Enablement Plan",
  "published_date": "2025-11-26",
  "author": {
    "name": "David Kim",
    "role": "Enablement Director",
    "entity": "DeepSpeed AI"
  },
  "core_concept": "AI Adoption and Enablement",
  "key_takeaways": [
    "HITL is a coaching system, not a brake—design accept/reject flows and reason codes that improve the model weekly.",
    "Instrument the copilot with confidence, policy, and source signals so agents can trust and act fast.",
    "Run a 30-day audit → pilot → scale motion with explicit gates, sampling, and QA rubrics tied to CSAT and reopens.",
    "Prove it with numbers: fewer reopens, faster responses, and audit-ready evidence for Legal and Security."
  ],
  "faq": [
    {
      "question": "Will HITL slow down my queue during spikes?",
      "answer": "No. We set thresholds so high-confidence intents can auto-accept or require only quick edits. Sampling focuses QA on the riskiest 10–15% while keeping throughput high."
    },
    {
      "question": "How do agents know when to trust the suggestion?",
      "answer": "Confidence bands, source citations, and policy badges are visible in the UI. Agents are trained to accept above threshold, edit in the middle band, and escalate below."
    },
    {
      "question": "What if Legal blocks AI output?",
      "answer": "We configure RBAC, prompt logging, redaction, and data residency on day one, plus approval steps for macro changes. Legal gets evidence and control—not surprises."
    },
    {
      "question": "Can we run this in our cloud?",
      "answer": "Yes. Deploy in your AWS/Azure/GCP VPC or on-prem. We integrate with Snowflake/BigQuery/Databricks and your ticketing systems without sending data to third parties for training."
    }
  ],
  "business_impact_evidence": {
    "organization_profile": "B2B SaaS company, 120-agent global support team on Zendesk + ServiceNow, KB in Confluence, data in Snowflake.",
    "before_state": "Copilot suggestions were sporadically used; agents lacked trust signals and over-edited or ignored outputs. Reopens and tone misfires spiked after releases.",
    "after_state": "HITL thresholds, reason codes, and weekly calibration in place. Agents see confidence, sources, and policy checks; supervisors coach using reason-code analytics.",
    "metrics": [
      "Reopens on billing intents down 28% within 30 days",
      "AHT down 14% on Tier 1 queues",
      "CSAT up 4.8 points on pilot queues",
      "40% of tickets assisted by the copilot with agent-in-the-loop"
    ],
    "governance": "Security approved rollout due to RBAC by queue, prompt logging with redaction, evidence pipeline to Snowflake, regional data residency, and a firm policy to never train foundation models on client data."
  },
  "summary": "Coach agents with human-in-the-loop design to lift CSAT and cut rework. A governed 30-day plan with audit trails, RBAC, and trust signals your team believes."
}