# GenAI.md - Generative AI Project Context --- ## How To Use This File This file carries no model names and no price figures. Both expire faster than documentation gets updated, and a stale number is worse than no number because people trust it. Two rules keep it accurate: **anything in square brackets is yours to fill in**, and **anything that can change under you gets a "measured on" date** - older than your re-measurement cadence means unknown, not true. Model identifiers live in environment variables and in the model card, never in prose, so there is one place to change when you rotate a tier. **Project**: Claro - customer support AI agent, drafting and triage **Environment**: Production - roughly 2,400 tickets/day across 3 product lines **Owner**: Support Platform team. On-call runbook: `docs/oncall.md` **Benchmark last re-measured**: [date] **Eval set version**: [eval-set-vN] --- ## Model Tiering Do not document "the model we use." Document the **tiers** the system routes between and the criteria that decide which tier handles a request. Tiers are stable; the models filling them are not. | Tier | Purpose | Selection criteria | Escalates to | |------|---------|--------------------|--------------| | Fast | Classification, triage, routing, extraction, cheap safety pre-checks | Lowest cost and latency that clears the accuracy floor on your eval set. Deterministic settings. | Reasoning | | Reasoning | Drafting, multi-step reasoning, anything a customer reads | Highest accuracy on your eval set within your latency budget. Cost is secondary. | Fallback | | Fallback | Keeps the product alive when the primary provider degrades | A different provider or a self-hosted option, so an outage is not correlated. Accepts lower accuracy. | Human | | Human | Anything the tiers above decline, or anything a guardrail rejects | Always available. Never optional. | - | **Routing for Claro**: every ticket enters at the Fast tier for category and confidence scoring. Above the routing threshold and in the automatable set -> Reasoning tier drafts. Below the threshold, or in the sensitive set (billing disputes, account security, legal, accessibility) -> Human tier, no draft generated. Reasoning tier unavailable -> Fallback drafts and the draft is marked `requires_human_review` regardless of confidence. Fallback unavailable -> Human tier, P1 alert. The Fast tier absorbs the volume, the Reasoning tier only sees what justifies it, and swapping either is a config change plus an eval run rather than a rewrite. --- ## Choosing What Fills Each Tier Published benchmarks measure someone else's task on someone else's data. They cannot tell you whether a candidate clears your accuracy floor on your tickets. Run the same harness against every candidate, on the same day, with the same prompts, and fill this in yourself: | Candidate | Tier considered | Accuracy on eval set | p95 latency | Cost per 1K units | Failure modes observed | Measured on | Decision | |-----------|-----------------|----------------------|-------------|-------------------|------------------------|-------------|----------| | [Candidate A] | [Fast/Reasoning/Fallback] | [%] | [seconds] | [amount] | [what it got wrong, and how] | [date] | [Selected/Rejected/Fallback] | | [Candidate B] | [tier] | [%] | [seconds] | [amount] | [failure modes] | [date] | [decision] | | [Candidate C] | [tier] | [%] | [seconds] | [amount] | [failure modes] | [date] | [decision] | | [Self-hosted option] | [tier] | [%] | [seconds] | [amount, incl. infrastructure] | [failure modes] | [date] | [decision] | **Column definitions** - be precise or the table is not comparable across runs: - **Accuracy**: your versioned eval set's primary metric, named by version. Not a vendor score. - **p95 latency**: end to end from your infrastructure, at your concurrency, including retries. Provider-reported latency excludes your network and your queueing. - **Cost per 1K units**: a unit is one piece of work the business recognizes - one ticket, one document. Not one token and not one API call, because those hide how many calls a unit takes. - **Failure modes observed**: the most valuable column and the one people skip. Two candidates with identical accuracy are different products if one fails by hedging and the other by inventing a refund policy. - **Measured on**: the date. Providers change models behind stable identifiers, and eval sets move. ### The cost formula ```python # scripts/cost_model.py # Per-token prices are INPUTS. Pull them from your provider's pricing page on the day you # measure and record that date beside the result. They are never constants in this repo. def cost_per_unit(avg_input_tokens, avg_output_tokens, input_price, output_price, calls_per_unit=1): """Cost of one unit of business work - one ticket, one document.""" return ((avg_input_tokens * input_price) + (avg_output_tokens * output_price)) * calls_per_unit def blended_monthly_cost(units_per_month, tier_mix, tier_costs): """tier_mix maps tier name -> share of traffic and must sum to 1.0.""" return sum(tier_mix[t] * tier_costs[t] for t in tier_mix) * units_per_month ``` Measure `avg_input_tokens` and `calls_per_unit` from production logs, never from estimates. Retries, tool calls, and reflection loops routinely make `calls_per_unit` two or three times the design assumption, and that is where an unexpected bill comes from. ### Re-measure on a cadence The frontier moves and so does your traffic. Put re-measurement on the calendar so it is routine rather than an emergency: | Trigger | Action | |---------|--------| | Scheduled - every quarter | Full re-run across all candidates, table updated, dates refreshed | | Provider announces a change behind an identifier you use | Re-run the eval set before it takes effect, not after | | Eval set version bumped | Re-run all deployed tiers so the table stays internally comparable | | Accuracy or drift alert fires | Re-run the affected tier immediately | | Cost per unit moves beyond your agreed tolerance | Re-run, and re-check the tier mix | If the table has not been touched in more than two cadences, the honest state is "we do not know what our options are," and that belongs in the risk register. --- ## Configuration ```python # config/model_config.py import os # Model identifiers come from the environment. Pin an EXACT identifier from your provider's # current documentation - never a moving alias like "latest" - and record the pinned value in # the model card. Rotating a tier is then a config change plus an eval run. TIERS = { "fast": { "model": os.environ["MODEL_ID_FAST"], # e.g. "/" "max_tokens": 256, "temperature": 0.0, # Deterministic classification }, "reasoning": { "model": os.environ["MODEL_ID_REASONING"], # e.g. "/" "max_tokens": 1024, "temperature": 0.3, "top_p": 0.9, "stop_sequences": ["---"], }, "fallback": { "model": os.environ["MODEL_ID_FALLBACK"], # Different provider - do not correlate outages "max_tokens": 1024, "temperature": 0.3, }, } ROUTING = { "confidence_threshold": 0.82, # Below this, route to the Human tier "automatable_categories": ["how_to", "feature_access", "technical_bug"], "always_human_categories": ["billing_dispute", "account_security", "legal", "accessibility"], } LIMITS = { "requests_per_minute": 1000, # From your provider agreement, not from this file "max_concurrent": 50, "retry_attempts": 3, "retry_backoff_base": 2, "retry_budget_per_unit": 2, # Hard cap - runaway retries are a cost incident "monthly_budget": os.environ["MODEL_BUDGET_MONTHLY"], # Per environment, in your currency "daily_alert_fraction": 0.05, # Alert when one day exceeds this share of the month "per_unit_ceiling_multiplier": 3.0, # Alert when measured cost per unit exceeds the benchmark } ``` **Cost controls that hold**, in order of effect: route at the Fast tier and escalate only what needs it, measuring the tier mix weekly because mix drift is the usual cause of a spike; cap `calls_per_unit` so one bad input cannot become a bill; cache identical requests and track the hit rate as a first-class metric; truncate history to the window your eval set proves is sufficient; and use stop sequences plus `max_tokens` so a verbose failure mode cannot run away. --- ## Prompt Architecture The system prompt is assembled from versioned components at request time, so any change is reviewable and revertible on its own. `PROMPT_VERSION` is bumped in the same PR as any component change and stamped into every request log. ```python # prompts/system_prompt.py PROMPT_VERSION = "v2.9" def build_system_prompt(product_line: str, ticket_category: str) -> str: return "\n\n".join([ ROLE_DEFINITION, # Who the system is, and its hard limits PRODUCT_CONTEXT[product_line], # Verified product facts only RESPONSE_GUIDELINES, # Tone and structure CATEGORY_SPECIFIC_INSTRUCTIONS[ticket_category], SAFETY_GUARDRAILS, # Mirrors the deterministic checks below DISCLOSURE_NOTICE, # Never claim to be human; never deny being AI ]) ROLE_DEFINITION = """ You are drafting customer support responses for Claro, a project management platform. - Empathetic but concise. Two to four paragraphs maximum. - If you are not confident, say so and escalate rather than guessing. - Never state a price, plan limit, or feature availability that is not in the supplied account context. Verified context only. - Never reference one customer's data in another customer's response. """ ``` **Few-shot examples** are the cheapest fix for output-format drift. Keep them versioned, keep them short, re-check them on every product release, and include the **refusal** cases - the shape of a correct escalation is as learnable as the shape of a correct answer. ```python # prompts/examples.py EXAMPLES = [ { "ticket": "I cannot export my project data as CSV. The button is grayed out.", "context": "Customer is on the entry tier. CSV export requires a higher tier.", "category": "feature_access", "response": "[approved draft - names the capability difference, quotes no price]", "quality": "approved", }, { "ticket": "Your app charged me twice this month. I want a refund immediately.", "context": "Billing shows two line items: a renewal and an add-on purchase.", "category": "billing_dispute", "response": None, "quality": "escalated", "note": "billing_dispute is in always_human_categories - no draft is generated.", }, ] ``` --- ## Eval Harness The eval set is a **first-class repo artifact**, versioned like code and reviewed like code. If it lives on a laptop or in a spreadsheet, you do not have an eval harness, you have anecdotes. ```python # data/eval/manifest.py EVAL_SET = { "name": "claro-support-eval", "version": "v3", # Bump on any change to items or labels "size": 500, "format": "JSONL", "location": "s3://claro-ml/datasets/eval/eval_v3.jsonl", "checksum": "[sha256 of the file]", # CI verifies this before running "categories": {"billing_dispute": 120, "technical_bug": 150, "feature_request": 80, "how_to": 100, "account_access": 50}, "labeled_by": "Senior support team, 3 annotators, majority vote", "inter_annotator_agreement": "[value]", # Below your floor means the LABELS are the problem "human_calibration_subset": 100, # Used to calibrate the judge - see below "known_gaps": ["Under-represents non-English tickets", "No examples of the import flow shipped after v3 was cut"], "last_refreshed": "2026-07-27", "refresh_cadence": "Monthly, owned by the support team lead", } ``` ### Regression gating on every PR Any change to a prompt, config, tier, or routing rule runs the eval set in CI and blocks the merge on regression: ```bash # .ci/eval-gate.sh set -euo pipefail python scripts/verify_eval_checksum.py --manifest data/eval/manifest.py python scripts/evaluate.py --eval-set data/eval/manifest.py \ --config config/model_config.py --output "results/PR-${PR_NUMBER}.json" python scripts/compare_results.py --baseline results/baseline-main.json \ --candidate "results/PR-${PR_NUMBER}.json" \ --tolerance-accuracy 0.005 --tolerance-latency-p95 0.10 --tolerance-safety 0.0 --comment-on-pr ``` Gate **safety at zero tolerance** and quality metrics at a stated tolerance. Zero tolerance on a noisy metric produces flaky builds that get disabled, which is worse than a tolerance nobody argues about. ### LLM-as-judge, and its limits Scoring open-ended output with a model is the only way to get coverage at useful volume. It is also a measurement instrument that drifts, so treat it like one: - **Keep a human-labelled calibration subset** and report judge agreement with it on every run. If agreement drops, the judge changed, not the system under test. - **Pin and version the judge separately** from the tiers under test, and never judge a system with a model from the same tier you are evaluating - shared blind spots score themselves well. - **Randomize position and presentation.** Judges show position bias in pairwise comparisons. Run both orders when the result is close. - **Judge one dimension at a time** against an explicit rubric. A single "quality" score is not reproducible across runs or reviewers. - **Never let the judge be the only gate on safety.** Safety is deterministic scanners plus human review. A model judging its own family's output is not a control. - **Report judge cost and latency.** A judge that costs more than the system it evaluates gets sampled down until it measures nothing. ### Metrics the harness reports ```python metrics = { "accuracy": exact_category_match + response_relevance_score, "consistency": cosine_similarity_between_repeated_runs, "safety_violations": pii_leaks + unauthorized_claims + fabricated_facts, "refusal_correctness": correct_escalations / total_should_escalate, "latency_p50": median_response_time_ms, "latency_p95": p95_response_time_ms, "calls_per_unit": total_model_calls / total_units, "cost_per_unit": measured_from_logs, "judge_agreement_with_humans": agreement_on_calibration_subset, } ``` **Human evaluation, monthly**: the support team lead scores 50 randomly sampled drafts on Accuracy (1-5, correct and complete), Tone (1-5, brand voice), Actionability (1-5, clear next step), and Safety (pass/fail). Targets 4.2 / 4.0 / 4.0 and 100% safety pass. Sampled items feed back into the eval set and the calibration subset. --- ## Experiment Tracking Every prompt change, tier swap, routing change, or parameter adjustment gets a row. **Failed experiments are the most valuable rows** - they are the only record of what the team already ruled out, and without them the next person re-runs the same dead end. | ID | Date | Change | Metric before | Metric after | Decision | |----|------|--------|---------------|--------------|----------| | EXP-042 | 2026-07-27 | Added product-specific context to the system prompt | 91.2% accuracy | 94.2% accuracy | Shipped | | EXP-041 | 2026-07-27 | Lowered temperature on the Reasoning tier | 89.8% consistency | 96.1% consistency | Shipped | | EXP-040 | 2026-07-27 | Rotated the Reasoning tier to a new candidate | [before] cost/unit | [after] cost/unit | Shipped, benchmark table updated | | EXP-038 | 2026-07-27 | Self-consistency: 3 samples plus majority vote on drafts | 94.2% accuracy | 94.4% accuracy | **Reverted** - triple cost for a noise-level gain. Do not retry without a cheaper voting scheme. | | EXP-037 | 2026-07-27 | Moved triage to the Reasoning tier to raise confidence scores | 88.1% triage accuracy | 93.0% triage accuracy | **Reverted** - gain real, but blended cost/unit and p95 both broke budget. Fast tier plus a lower threshold was cheaper for the same outcome. | | [EXP-ID] | [date] | [change] | [before] | [after] | [Shipped/Reverted/Iterating, and why] | ```bash # One experiment per branch. Change exactly one thing; bump PROMPT_VERSION if a component moved. git checkout -b exp/EXP-043-escalation-rules python scripts/evaluate.py --eval-set data/eval/manifest.py \ --config config/model_config.py --output results/EXP-043.json python scripts/compare_results.py --baseline results/EXP-042.json --candidate results/EXP-043.json # Write the row BEFORE you decide, and write it even if you revert. # A reverted experiment with no row is a lesson you will pay for twice. ``` **Hyperparameter versioning**: every setting that affects output is versioned in git and stamped into the request log - prompt version, tier identifiers, temperature, `top_p`, `max_tokens`, stop sequences, routing thresholds, retry budget, eval set version. If a production response cannot be traced back to the exact configuration that produced it, you cannot reproduce a complaint. --- ## Data Lineage | Dataset | Source | Consent basis | Retention | Known biases | |---------|--------|---------------|-----------|--------------| | Production tickets | Support tool, PII-redacted at ingest | Terms of service, support processing | 90 days | Skews to customers who file tickets; misses silent churn | | Eval set vN | Sampled production tickets, human-labelled | Same, plus internal review approval | Per version | Under-represents non-English; lags product changes by up to one refresh cycle | | Few-shot examples | Hand-authored from approved responses | Internal | Versioned with prompts | Reflects the product state at authoring time - re-check every release | | Judge calibration subset | Human-labelled sample of the eval set | Internal | Per version | Small sample; wide confidence interval on rare categories | Customer PII is redacted before storage, not after. No customer data is used for fine-tuning without explicit, recorded consent. Every dataset has a named owner and a refresh cadence. --- ## Deployment ``` 1. Change on a feature branch (one experiment per branch) 2. CI verifies the eval set checksum, then runs the eval harness 3. Regression gate: quality within tolerance, safety at zero tolerance 4. Results posted to the PR (accuracy, latency, cost/unit, calls/unit, judge agreement) 5. Human review of 10 sampled outputs - required, not advisory 6. Merge to main -> auto-deploy to staging 7. Staging soak: 24 hours of shadowed live traffic, no customer impact 8. Canary: 5% of production traffic, minimum 2 hours, auto-abort on any guardrail breach 9. Ramp 5% -> 25% -> 50% -> 100%, holding each step for one full traffic cycle 10. Promotion to 100% requires named approval, recorded in the deploy log ``` **Pre-deployment checklist** - [ ] Eval harness green, eval set version recorded in the PR - [ ] Safety metrics unchanged or better - zero tolerance - [ ] Cost per unit and calls per unit measured, within the budget ceiling - [ ] p95 latency within budget at production concurrency, not test concurrency - [ ] Prompt version bumped and the experiment row written - [ ] Model card updated - tiers, pinned identifiers, eval results, limitations - [ ] Rollback verified: the previous configuration is pinned and known-good - [ ] Canary abort conditions configured and tested - [ ] Human-tier capacity confirmed for the failure case - [ ] AI-generated disclosure present wherever output reaches a person **Rollback**: revert the config commit and push - CI redeploys, then verify `PROMPT_VERSION` in the request log. A tier rollback changes the pinned identifier in the environment, not in code. The chain degrades Fast -> Reasoning -> Fallback -> Human, and human-only is a supported operating state, not an outage. **Rollback is only real if it is rehearsed** - drill it on a schedule and record the measured time to restore in the runbook. --- ## Monitoring and Drift | Metric | Threshold | Channel | |--------|-----------|---------| | Error rate | above 2% over 5 min | Pager (P2) | | Latency p95 | above budget over 10 min | Slack, team channel | | Safety violations | any, per hour | Pager (P1) | | Daily spend | above the daily alert fraction | Slack, cost channel | | Cost per unit | above the benchmark by the ceiling multiplier | Slack, cost channel | | Calls per unit | above the retry budget | Pager (P2) - runaway loop | | Fabricated-fact rate | above 1% over 1 hour | Pager (P2) | | Tier mix shift | Reasoning-tier share beyond tolerance | Slack, cost channel | | Provider availability | below 99% over 15 min | Pager (P1) | | Confidence distribution shift | divergence above threshold vs the trailing 30 days | Slack, ML channel | | Judge agreement with humans | below the calibration floor | Slack, ML channel | ```python # Every model invocation is logged. This log IS the audit trail - see Provenance below. logger.info("model_invocation", { "request_id": request_id, "ticket_id": ticket_id, "tier": tier_name, # fast | reasoning | fallback "model_id": resolved_model_id, # The exact pinned identifier that served the request "prompt_version": PROMPT_VERSION, "eval_set_version": EVAL_SET["version"], "routing_reason": routing_reason, "input_tokens": usage.input_tokens, "output_tokens": usage.output_tokens, "calls_this_unit": calls_this_unit, "latency_ms": elapsed_ms, "cost": calculated_cost, "category": predicted_category, "confidence": confidence_score, "safety_check": "pass", # or "fail" plus the failing check "human_reviewer": reviewer_id, # null until reviewed "disclosed_to_user": True, }) ``` ### Retrain or roll back A drift alert does not tell you which. Work the tree before touching the model: ```mermaid flowchart TD A[Drift alert fires] --> B{Did a change land in the window?} B -- Yes --> C[Roll back the change, re-measure] B -- No --> D{Input distribution shifted?} D -- Yes --> E{Is the eval set still representative?} E -- No --> F[Refresh the eval set FIRST, then re-benchmark] E -- Yes --> G[Re-tune prompts or routing, gate on the eval set] D -- No --> H{Provider-side change suspected?} H -- Yes --> I[Pin the previous identifier, open a provider ticket, re-run the eval set] H -- No --> J[Escalate to the Human tier, investigate before any model change] ``` The common mistake is jumping to a model change when the eval set has gone stale. If the eval set no longer represents production traffic, every measurement downstream of it is unreliable and a "fix" chosen on that basis is a guess. --- ## Safety and Ethical Guardrails ```python # Post-processing checks on every output. Deterministic scanners, not a model judging itself. SAFETY_CHECKS = [ "pii_detection", # Emails, phone numbers, national IDs, card fragments "cross_customer_leak", # Any identifier not belonging to this ticket's customer "price_and_plan_claims", # Claims about price or plan limits absent from supplied context "feature_verification", # Named features must exist in the current feature registry "unauthorized_commitment", # Refunds, credits, deadlines, SLAs the system may not promise "regulated_advice", # Legal, medical, financial advice patterns "tone_check", "disclosure_present", ] # On any failure: the response is NOT sent, the ticket routes to the Human tier, the incident is # logged with the failing check and full request context, and an alert fires above threshold. ``` - **Human oversight is a control, not a courtesy.** Sensitive categories never get an automated response regardless of confidence, every automated response is reversible, and the reviewer is recorded. - **Disclosure.** Output is labelled AI-generated at the point a person reads it, not in a policy document. The system never claims to be human and never denies being an AI when asked. - **Contestability.** A customer can request human handling in one step, and that request is logged. - **Fairness.** Accuracy and escalation rates are reported per category and per customer segment. An aggregate number hides the segment the system serves badly. - **Data minimization.** Send the narrowest context that clears the accuracy floor, and prove a wider one is needed before sending it. --- ## Model Cards A model card is a versioned file in the repo, updated in the same PR as the change it describes. It is how you answer a documentation or traceability question without a scramble. ```markdown # Model Card - Claro Support Drafting **Card version**: [n] **Updated**: 2026-07-27 **Owner**: Support Platform team ## System Purpose: draft support responses for human review; classify and route tickets. Deployment: production, [N] units/day. Human review required before any response is sent. Tiers and pinned identifiers: config/model_config.py plus this deployment's environment. Not intended for: automated refunds, account security decisions, legal or financial advice, employment or credit decisions, or any use where output reaches a customer unreviewed. ## Data Training or tuning data: none - this is a prompted system, not a fine-tuned one. Eval set: [name] [version], [N] items, labelled by [who], agreement [value]. Known gaps and biases: restate them here. Do not link to Data Lineage and hope. ## Performance Primary metric on eval set [version]: [value], measured [date]. Latency p95 at production concurrency: [value]. Cost per unit: [value]. Per-segment breakdown: [table or link]. ## Limitations observed [Concrete failure modes seen in evaluation and production. "Sometimes wrong" is a disclaimer, not a limitation.] ## Human oversight Who reviews, what they can override, how a decision is contested, how long records are kept. ## Change log | Card version | Date | Change | Approved by | |--------------|------|--------|-------------| | [n] | 2026-07-27 | [what changed and why] | [name] | ``` --- ## Regulatory Documentation - EU AI Act If your system is placed on the market or used in the EU, the EU AI Act's main application date is **2 August 2026**. Two things become relevant then, and they are commonly conflated. **Article 50 transparency obligations apply broadly**, including to systems that are **not** high-risk: people must be informed when they are interacting with an AI system, and AI-generated or AI-manipulated content must be identifiable as such. This is the obligation most teams actually have. **Annex III high-risk requirements (Articles 6 to 15)** bring conformity assessment, technical documentation, logging and traceability, human oversight, accuracy and robustness measures, CE marking, and registration in the EU database. They apply to the **specific use cases listed in Annex III** - among them biometrics, critical infrastructure, education access, employment and worker management, access to essential services, law enforcement, migration, and administration of justice - plus AI acting as a safety component of a regulated product. **Do not overclaim, in either direction.** Ordinary AI developer assistance, and AI-generated code in general, do not by themselves make a system high-risk: Annex III regulates named use cases, not the presence of AI. Equally, "we are not high-risk" is not a conclusion you assert - it is one you document, because this same system pointed at worker evaluation instead of ticket drafting lands in a different category. Classification is a legal determination for your organization to make with counsel. What this file guarantees is that the evidence exists and is current when asked. The compliance questions are **documentation, traceability, and human oversight**: | Question an assessor asks | Where this repo answers it | |---------------------------|----------------------------| | What does the system do, and what is it not for? | Model card, System section | | What data was used, and what are its known biases? | Data Lineage, Model card Data section | | How was performance measured, and when? | Eval Harness, benchmark table with measurement dates | | What are the known limitations? | Model card, Limitations - stated concretely | | Can you reconstruct a specific past decision? | Request log - request id, tier, pinned identifier, prompt version | | Who reviewed it, and can a person override it? | Provenance, human review fields, Human tier routing | | Is AI-generated content identifiable to the user? | `disclosure_present` check, DISCLOSURE_NOTICE | | Who is accountable? | Named owners on the model card, datasets, and alerts | --- ## Provenance and Audit Trail Every artifact that reaches a person should be reconstructible: **what was generated, by which system, reviewed by whom, and when.** ```python # Written once per generated artifact, append-only, retained per the retention policy. provenance_record = { "artifact_id": artifact_id, "artifact_type": "support_response_draft", "generated_at": iso8601_utc, "generated_by": { "tier": tier_name, "model_id": resolved_model_id, # The exact pinned identifier "prompt_version": PROMPT_VERSION, "config_commit": git_sha, # Ties to the exact config in git }, "inputs_reference": { "ticket_id": ticket_id, "context_fields": ["account_tier", "recent_activity"], # Names sent, not values "input_hash": sha256_of_rendered_prompt, }, "safety_checks": {"pii_detection": "pass", "disclosure_present": "pass"}, "review": { "reviewed_by": reviewer_id, # null means not yet reviewed "reviewed_at": iso8601_utc, "action": "sent_unchanged", # sent_unchanged | sent_edited | rejected | escalated "edit_distance": 0, }, "disclosed_to_user": True, "retention_expires_at": iso8601_utc, } ``` - **Append-only.** A correction is a new record referencing the old one, never an in-place update. - **Record the pinned identifier, not just the tier name.** "Reasoning tier" does not tell a future investigator what actually ran. - **Reference inputs, do not copy them.** Hashes and field names, so the record is useful without becoming a second copy of customer data. - **Human review is a recorded event.** "A human looked at it" with no record is indistinguishable from no review. - **Ship these events to your central log store** under the same retention and access controls as other audit data. --- ## Troubleshooting **High latency**: provider status page first. Then `calls_per_unit` - a retry loop shows up as latency before it shows up as cost. Then prompt growth, which taxes every request. Then tier mix: more Reasoning-tier traffic raises aggregate p95 with nothing else changing. **Low output quality**: tier mix and routing threshold before blaming the model. Then whether the eval set still represents production traffic - stale eval, unreliable everything. Then judge agreement with the calibration subset, since the measurement may be what moved. Then recent experiment rows, including reverted ones, for a partial revert. Then few-shot examples against the current product state. **Cost spikes**, in order: `calls_per_unit` (retries and reflection loops), tier mix (a confidence-threshold change quietly moves volume to the expensive tier), cache hit rate, input length (one new context field multiplies across every request). Per-token prices last - and if those changed, re-run the benchmark and stamp a fresh measurement date on the table. **Suspected provider-side change**: compare the eval result against the last recorded run for the same pinned identifier. If the identifier is stable and results moved, record an experiment row with "provider-side change" as the cause - that row is the only evidence you will have later. Pin the previous known-good identifier if one is still available, and open a provider ticket.