Documentation That Makes Model Work Reproducible
A generative AI project accumulates far more than weights: dataset provenance, configuration history, evaluation results, deployment procedure. GenAI.md gives you structured files for all of it, so a teammate who was not there can rerun the experiment and land on the same number.
Record architecture choices, hyperparameter reasoning, dataset characteristics, and serving configuration in files that move through review. Every iteration then teaches the whole team rather than only the person who ran it.
The distance between a promising notebook and a system you can operate is almost entirely record-keeping. Close it with documentation that follows the work from raw data through to the rollback plan.
Generative AI Project Best Practices
Record the parts of model work that outlive the checkpoint: provenance, evaluation, thresholds, and the decisions behind each one.
Log the Failures Too
Record each run with its hypothesis, configuration, result, and conclusion, including the runs that went nowhere. Dead ends are expensive to rediscover and they map the boundaries of the space faster than successes do.
Justify Every Hyperparameter
Note why the learning rate moved, why the batch size changed, and what each adjustment did to the numbers. A config file states the value. Only the log states the reasoning behind it.
Trace the Data to Its Source
Follow each dataset from origin through cleaning, filtering, and augmentation, and name the biases you already know about. Provenance is the first thing you need when a model behaves strangely and the first thing any review asks for.
Define the Eval Suite in Writing
Name the metrics, the benchmarks, the held-out sets, and the passing bar, then explain why those and not others. Comparison across runs only means something if the measuring stick stayed still.
Write the Deployment Checklist
Packaging, serving setup, rollout stages, canary criteria, rollback steps. Production model failures are expensive and unusually repetitive, and a checklist catches the repetitive ones cheaply.
Set Drift Thresholds in Advance
Specify the performance floor, the input distribution checks, and the alert conditions. Define what counts as degraded before you are staring at a graph arguing about whether this counts.
Document the Guardrails
Capture fairness criteria, bias testing results, and safety constraints, and revisit them with every model update. These belong beside the model card, not in a separate review artifact nobody reopens.
Maintain a Model Card
Give every deployed model a short page: intended use, known limitations, training data summary, measured performance, update history. It is the interface between the people who built the system and everyone who has to rely on it.
Reproducibility Is Documentation
If an experiment cannot be reproduced from its written record alone, the record is incomplete, and the person who can fill the gap is a single point of failure. The test is simple: hand the file to somebody who was not involved and see whether they land on the same number. Passing it means the environment was captured too, right down to the random seed and the pinned dependency list, and not only the headline number. That discipline is not overhead wrapped around the real work. On any project that runs longer than a quarter, it is most of what makes the real work possible.
The GenAI Template
GenAI.md
# GenAI.md - Generative AI Project Context
<!-- Model tiering, eval harness, experiment tracking, deployment, drift, and compliance -->
<!-- Written to survive model churn: methods and tables you fill in, never pinned versions or prices -->
<!-- Last updated: 2026-07-27 -->
---
## How To Use This File
This file carries no model names and no price figures. Both expire faster than documentation gets
updated, and a stale number is worse than no number because people trust it. Two rules keep it
accurate: **anything in square brackets is yours to fill in**, and **anything that can change under
you gets a "measured on" date** - older than your re-measurement cadence means unknown, not true.
Model identifiers live in environment variables and in the model card, never in prose, so there is
one place to change when you rotate a tier.
**Project**: Claro - customer support AI agent, drafting and triage
**Environment**: Production - roughly 2,400 tickets/day across 3 product lines
**Owner**: Support Platform team. On-call runbook: `docs/oncall.md`
**Benchmark last re-measured**: [date] **Eval set version**: [eval-set-vN]
---
## Model Tiering
Do not document "the model we use." Document the **tiers** the system routes between and the
criteria that decide which tier handles a request. Tiers are stable; the models filling them are not.
| Tier | Purpose | Selection criteria | Escalates to |
|------|---------|--------------------|--------------|
| Fast | Classification, triage, routing, extraction, cheap safety pre-checks | Lowest cost and latency that clears the accuracy floor on your eval set. Deterministic settings. | Reasoning |
| Reasoning | Drafting, multi-step reasoning, anything a customer reads | Highest accuracy on your eval set within your latency budget. Cost is secondary. | Fallback |
| Fallback | Keeps the product alive when the primary provider degrades | A different provider or a self-hosted option, so an outage is not correlated. Accepts lower accuracy. | Human |
| Human | Anything the tiers above decline, or anything a guardrail rejects | Always available. Never optional. | - |
**Routing for Claro**: every ticket enters at the Fast tier for category and confidence scoring.
Above the routing threshold and in the automatable set -> Reasoning tier drafts. Below the
threshold, or in the sensitive set (billing disputes, account security, legal, accessibility) ->
Human tier, no draft generated. Reasoning tier unavailable -> Fallback drafts and the draft is
marked `requires_human_review` regardless of confidence. Fallback unavailable -> Human tier, P1
alert. The Fast tier absorbs the volume, the Reasoning tier only sees what justifies it, and
swapping either is a config change plus an eval run rather than a rewrite.
---
## Choosing What Fills Each Tier
Published benchmarks measure someone else's task on someone else's data. They cannot tell you
whether a candidate clears your accuracy floor on your tickets. Run the same harness against every
candidate, on the same day, with the same prompts, and fill this in yourself:
| Candidate | Tier considered | Accuracy on eval set | p95 latency | Cost per 1K units | Failure modes observed | Measured on | Decision |
|-----------|-----------------|----------------------|-------------|-------------------|------------------------|-------------|----------|
| [Candidate A] | [Fast/Reasoning/Fallback] | [%] | [seconds] | [amount] | [what it got wrong, and how] | [date] | [Selected/Rejected/Fallback] |
| [Candidate B] | [tier] | [%] | [seconds] | [amount] | [failure modes] | [date] | [decision] |
| [Candidate C] | [tier] | [%] | [seconds] | [amount] | [failure modes] | [date] | [decision] |
| [Self-hosted option] | [tier] | [%] | [seconds] | [amount, incl. infrastructure] | [failure modes] | [date] | [decision] |
**Column definitions** - be precise or the table is not comparable across runs:
- **Accuracy**: your versioned eval set's primary metric, named by version. Not a vendor score.
- **p95 latency**: end to end from your infrastructure, at your concurrency, including retries.
Provider-reported latency excludes your network and your queueing.
- **Cost per 1K units**: a unit is one piece of work the business recognizes - one ticket, one
document. Not one token and not one API call, because those hide how many calls a unit takes.
- **Failure modes observed**: the most valuable column and the one people skip. Two candidates with
identical accuracy are different products if one fails by hedging and the other by inventing a
refund policy.
- **Measured on**: the date. Providers change models behind stable identifiers, and eval sets move.
### The cost formula
```python
# scripts/cost_model.py
# Per-token prices are INPUTS. Pull them from your provider's pricing page on the day you
# measure and record that date beside the result. They are never constants in this repo.
def cost_per_unit(avg_input_tokens, avg_output_tokens, input_price, output_price, calls_per_unit=1):
"""Cost of one unit of business work - one ticket, one document."""
return ((avg_input_tokens * input_price) + (avg_output_tokens * output_price)) * calls_per_unit
def blended_monthly_cost(units_per_month, tier_mix, tier_costs):
"""tier_mix maps tier name -> share of traffic and must sum to 1.0."""
return sum(tier_mix[t] * tier_costs[t] for t in tier_mix) * units_per_month
```
Measure `avg_input_tokens` and `calls_per_unit` from production logs, never from estimates. Retries,
tool calls, and reflection loops routinely make `calls_per_unit` two or three times the design
assumption, and that is where an unexpected bill comes from.
### Re-measure on a cadence
The frontier moves and so does your traffic. Put re-measurement on the calendar so it is routine
rather than an emergency:
| Trigger | Action |
|---------|--------|
| Scheduled - every quarter | Full re-run across all candidates, table updated, dates refreshed |
| Provider announces a change behind an identifier you use | Re-run the eval set before it takes effect, not after |
| Eval set version bumped | Re-run all deployed tiers so the table stays internally comparable |
| Accuracy or drift alert fires | Re-run the affected tier immediately |
| Cost per unit moves beyond your agreed tolerance | Re-run, and re-check the tier mix |
If the table has not been touched in more than two cadences, the honest state is "we do not know
what our options are," and that belongs in the risk register.
---
## Configuration
```python
# config/model_config.py
import os
# Model identifiers come from the environment. Pin an EXACT identifier from your provider's
# current documentation - never a moving alias like "latest" - and record the pinned value in
# the model card. Rotating a tier is then a config change plus an eval run.
TIERS = {
"fast": {
"model": os.environ["MODEL_ID_FAST"], # e.g. "<provider>/<fast-tier-model-id>"
"max_tokens": 256, "temperature": 0.0, # Deterministic classification
},
"reasoning": {
"model": os.environ["MODEL_ID_REASONING"], # e.g. "<provider>/<reasoning-tier-model-id>"
"max_tokens": 1024, "temperature": 0.3, "top_p": 0.9, "stop_sequences": ["---"],
},
"fallback": {
"model": os.environ["MODEL_ID_FALLBACK"], # Different provider - do not correlate outages
"max_tokens": 1024, "temperature": 0.3,
},
}
ROUTING = {
"confidence_threshold": 0.82, # Below this, route to the Human tier
"automatable_categories": ["how_to", "feature_access", "technical_bug"],
"always_human_categories": ["billing_dispute", "account_security", "legal", "accessibility"],
}
LIMITS = {
"requests_per_minute": 1000, # From your provider agreement, not from this file
"max_concurrent": 50, "retry_attempts": 3, "retry_backoff_base": 2,
"retry_budget_per_unit": 2, # Hard cap - runaway retries are a cost incident
"monthly_budget": os.environ["MODEL_BUDGET_MONTHLY"], # Per environment, in your currency
"daily_alert_fraction": 0.05, # Alert when one day exceeds this share of the month
"per_unit_ceiling_multiplier": 3.0, # Alert when measured cost per unit exceeds the benchmark
}
```
**Cost controls that hold**, in order of effect: route at the Fast tier and escalate only what needs
it, measuring the tier mix weekly because mix drift is the usual cause of a spike; cap
`calls_per_unit` so one bad input cannot become a bill; cache identical requests and track the hit
rate as a first-class metric; truncate history to the window your eval set proves is sufficient; and
use stop sequences plus `max_tokens` so a verbose failure mode cannot run away.
---
## Prompt Architecture
The system prompt is assembled from versioned components at request time, so any change is
reviewable and revertible on its own. `PROMPT_VERSION` is bumped in the same PR as any component
change and stamped into every request log.
```python
# prompts/system_prompt.py
PROMPT_VERSION = "v2.9"
def build_system_prompt(product_line: str, ticket_category: str) -> str:
return "\n\n".join([
ROLE_DEFINITION, # Who the system is, and its hard limits
PRODUCT_CONTEXT[product_line], # Verified product facts only
RESPONSE_GUIDELINES, # Tone and structure
CATEGORY_SPECIFIC_INSTRUCTIONS[ticket_category],
SAFETY_GUARDRAILS, # Mirrors the deterministic checks below
DISCLOSURE_NOTICE, # Never claim to be human; never deny being AI
])
ROLE_DEFINITION = """
You are drafting customer support responses for Claro, a project management platform.
- Empathetic but concise. Two to four paragraphs maximum.
- If you are not confident, say so and escalate rather than guessing.
- Never state a price, plan limit, or feature availability that is not in the supplied account
context. Verified context only.
- Never reference one customer's data in another customer's response.
"""
```
**Few-shot examples** are the cheapest fix for output-format drift. Keep them versioned, keep them
short, re-check them on every product release, and include the **refusal** cases - the shape of a
correct escalation is as learnable as the shape of a correct answer.
```python
# prompts/examples.py
EXAMPLES = [
{
"ticket": "I cannot export my project data as CSV. The button is grayed out.",
"context": "Customer is on the entry tier. CSV export requires a higher tier.",
"category": "feature_access",
"response": "[approved draft - names the capability difference, quotes no price]",
"quality": "approved",
},
{
"ticket": "Your app charged me twice this month. I want a refund immediately.",
"context": "Billing shows two line items: a renewal and an add-on purchase.",
"category": "billing_dispute",
"response": None,
"quality": "escalated",
"note": "billing_dispute is in always_human_categories - no draft is generated.",
},
]
```
---
## Eval Harness
The eval set is a **first-class repo artifact**, versioned like code and reviewed like code. If it
lives on a laptop or in a spreadsheet, you do not have an eval harness, you have anecdotes.
```python
# data/eval/manifest.py
EVAL_SET = {
"name": "claro-support-eval",
"version": "v3", # Bump on any change to items or labels
"size": 500, "format": "JSONL",
"location": "s3://claro-ml/datasets/eval/eval_v3.jsonl",
"checksum": "[sha256 of the file]", # CI verifies this before running
"categories": {"billing_dispute": 120, "technical_bug": 150, "feature_request": 80,
"how_to": 100, "account_access": 50},
"labeled_by": "Senior support team, 3 annotators, majority vote",
"inter_annotator_agreement": "[value]", # Below your floor means the LABELS are the problem
"human_calibration_subset": 100, # Used to calibrate the judge - see below
"known_gaps": ["Under-represents non-English tickets",
"No examples of the import flow shipped after v3 was cut"],
"last_refreshed": "2026-07-27",
"refresh_cadence": "Monthly, owned by the support team lead",
}
```
### Regression gating on every PR
Any change to a prompt, config, tier, or routing rule runs the eval set in CI and blocks the merge
on regression:
```bash
# .ci/eval-gate.sh
set -euo pipefail
python scripts/verify_eval_checksum.py --manifest data/eval/manifest.py
python scripts/evaluate.py --eval-set data/eval/manifest.py \
--config config/model_config.py --output "results/PR-${PR_NUMBER}.json"
python scripts/compare_results.py --baseline results/baseline-main.json \
--candidate "results/PR-${PR_NUMBER}.json" \
--tolerance-accuracy 0.005 --tolerance-latency-p95 0.10 --tolerance-safety 0.0 --comment-on-pr
```
Gate **safety at zero tolerance** and quality metrics at a stated tolerance. Zero tolerance on a
noisy metric produces flaky builds that get disabled, which is worse than a tolerance nobody argues
about.
### LLM-as-judge, and its limits
Scoring open-ended output with a model is the only way to get coverage at useful volume. It is also
a measurement instrument that drifts, so treat it like one:
- **Keep a human-labelled calibration subset** and report judge agreement with it on every run. If
agreement drops, the judge changed, not the system under test.
- **Pin and version the judge separately** from the tiers under test, and never judge a system with
a model from the same tier you are evaluating - shared blind spots score themselves well.
- **Randomize position and presentation.** Judges show position bias in pairwise comparisons. Run
both orders when the result is close.
- **Judge one dimension at a time** against an explicit rubric. A single "quality" score is not
reproducible across runs or reviewers.
- **Never let the judge be the only gate on safety.** Safety is deterministic scanners plus human
review. A model judging its own family's output is not a control.
- **Report judge cost and latency.** A judge that costs more than the system it evaluates gets
sampled down until it measures nothing.
### Metrics the harness reports
```python
metrics = {
"accuracy": exact_category_match + response_relevance_score,
"consistency": cosine_similarity_between_repeated_runs,
"safety_violations": pii_leaks + unauthorized_claims + fabricated_facts,
"refusal_correctness": correct_escalations / total_should_escalate,
"latency_p50": median_response_time_ms, "latency_p95": p95_response_time_ms,
"calls_per_unit": total_model_calls / total_units,
"cost_per_unit": measured_from_logs,
"judge_agreement_with_humans": agreement_on_calibration_subset,
}
```
**Human evaluation, monthly**: the support team lead scores 50 randomly sampled drafts on Accuracy
(1-5, correct and complete), Tone (1-5, brand voice), Actionability (1-5, clear next step), and
Safety (pass/fail). Targets 4.2 / 4.0 / 4.0 and 100% safety pass. Sampled items feed back into the
eval set and the calibration subset.
---
## Experiment Tracking
Every prompt change, tier swap, routing change, or parameter adjustment gets a row. **Failed
experiments are the most valuable rows** - they are the only record of what the team already ruled
out, and without them the next person re-runs the same dead end.
| ID | Date | Change | Metric before | Metric after | Decision |
|----|------|--------|---------------|--------------|----------|
| EXP-042 | 2026-07-27 | Added product-specific context to the system prompt | 91.2% accuracy | 94.2% accuracy | Shipped |
| EXP-041 | 2026-07-27 | Lowered temperature on the Reasoning tier | 89.8% consistency | 96.1% consistency | Shipped |
| EXP-040 | 2026-07-27 | Rotated the Reasoning tier to a new candidate | [before] cost/unit | [after] cost/unit | Shipped, benchmark table updated |
| EXP-038 | 2026-07-27 | Self-consistency: 3 samples plus majority vote on drafts | 94.2% accuracy | 94.4% accuracy | **Reverted** - triple cost for a noise-level gain. Do not retry without a cheaper voting scheme. |
| EXP-037 | 2026-07-27 | Moved triage to the Reasoning tier to raise confidence scores | 88.1% triage accuracy | 93.0% triage accuracy | **Reverted** - gain real, but blended cost/unit and p95 both broke budget. Fast tier plus a lower threshold was cheaper for the same outcome. |
| [EXP-ID] | [date] | [change] | [before] | [after] | [Shipped/Reverted/Iterating, and why] |
```bash
# One experiment per branch. Change exactly one thing; bump PROMPT_VERSION if a component moved.
git checkout -b exp/EXP-043-escalation-rules
python scripts/evaluate.py --eval-set data/eval/manifest.py \
--config config/model_config.py --output results/EXP-043.json
python scripts/compare_results.py --baseline results/EXP-042.json --candidate results/EXP-043.json
# Write the row BEFORE you decide, and write it even if you revert.
# A reverted experiment with no row is a lesson you will pay for twice.
```
**Hyperparameter versioning**: every setting that affects output is versioned in git and stamped
into the request log - prompt version, tier identifiers, temperature, `top_p`, `max_tokens`, stop
sequences, routing thresholds, retry budget, eval set version. If a production response cannot be
traced back to the exact configuration that produced it, you cannot reproduce a complaint.
---
## Data Lineage
| Dataset | Source | Consent basis | Retention | Known biases |
|---------|--------|---------------|-----------|--------------|
| Production tickets | Support tool, PII-redacted at ingest | Terms of service, support processing | 90 days | Skews to customers who file tickets; misses silent churn |
| Eval set vN | Sampled production tickets, human-labelled | Same, plus internal review approval | Per version | Under-represents non-English; lags product changes by up to one refresh cycle |
| Few-shot examples | Hand-authored from approved responses | Internal | Versioned with prompts | Reflects the product state at authoring time - re-check every release |
| Judge calibration subset | Human-labelled sample of the eval set | Internal | Per version | Small sample; wide confidence interval on rare categories |
Customer PII is redacted before storage, not after. No customer data is used for fine-tuning without
explicit, recorded consent. Every dataset has a named owner and a refresh cadence.
---
## Deployment
```
1. Change on a feature branch (one experiment per branch)
2. CI verifies the eval set checksum, then runs the eval harness
3. Regression gate: quality within tolerance, safety at zero tolerance
4. Results posted to the PR (accuracy, latency, cost/unit, calls/unit, judge agreement)
5. Human review of 10 sampled outputs - required, not advisory
6. Merge to main -> auto-deploy to staging
7. Staging soak: 24 hours of shadowed live traffic, no customer impact
8. Canary: 5% of production traffic, minimum 2 hours, auto-abort on any guardrail breach
9. Ramp 5% -> 25% -> 50% -> 100%, holding each step for one full traffic cycle
10. Promotion to 100% requires named approval, recorded in the deploy log
```
**Pre-deployment checklist**
- [ ] Eval harness green, eval set version recorded in the PR
- [ ] Safety metrics unchanged or better - zero tolerance
- [ ] Cost per unit and calls per unit measured, within the budget ceiling
- [ ] p95 latency within budget at production concurrency, not test concurrency
- [ ] Prompt version bumped and the experiment row written
- [ ] Model card updated - tiers, pinned identifiers, eval results, limitations
- [ ] Rollback verified: the previous configuration is pinned and known-good
- [ ] Canary abort conditions configured and tested
- [ ] Human-tier capacity confirmed for the failure case
- [ ] AI-generated disclosure present wherever output reaches a person
**Rollback**: revert the config commit and push - CI redeploys, then verify `PROMPT_VERSION` in the
request log. A tier rollback changes the pinned identifier in the environment, not in code. The
chain degrades Fast -> Reasoning -> Fallback -> Human, and human-only is a supported operating
state, not an outage. **Rollback is only real if it is rehearsed** - drill it on a schedule and
record the measured time to restore in the runbook.
---
## Monitoring and Drift
| Metric | Threshold | Channel |
|--------|-----------|---------|
| Error rate | above 2% over 5 min | Pager (P2) |
| Latency p95 | above budget over 10 min | Slack, team channel |
| Safety violations | any, per hour | Pager (P1) |
| Daily spend | above the daily alert fraction | Slack, cost channel |
| Cost per unit | above the benchmark by the ceiling multiplier | Slack, cost channel |
| Calls per unit | above the retry budget | Pager (P2) - runaway loop |
| Fabricated-fact rate | above 1% over 1 hour | Pager (P2) |
| Tier mix shift | Reasoning-tier share beyond tolerance | Slack, cost channel |
| Provider availability | below 99% over 15 min | Pager (P1) |
| Confidence distribution shift | divergence above threshold vs the trailing 30 days | Slack, ML channel |
| Judge agreement with humans | below the calibration floor | Slack, ML channel |
```python
# Every model invocation is logged. This log IS the audit trail - see Provenance below.
logger.info("model_invocation", {
"request_id": request_id, "ticket_id": ticket_id,
"tier": tier_name, # fast | reasoning | fallback
"model_id": resolved_model_id, # The exact pinned identifier that served the request
"prompt_version": PROMPT_VERSION, "eval_set_version": EVAL_SET["version"],
"routing_reason": routing_reason,
"input_tokens": usage.input_tokens, "output_tokens": usage.output_tokens,
"calls_this_unit": calls_this_unit, "latency_ms": elapsed_ms, "cost": calculated_cost,
"category": predicted_category, "confidence": confidence_score,
"safety_check": "pass", # or "fail" plus the failing check
"human_reviewer": reviewer_id, # null until reviewed
"disclosed_to_user": True,
})
```
### Retrain or roll back
A drift alert does not tell you which. Work the tree before touching the model:
```mermaid
flowchart TD
A[Drift alert fires] --> B{Did a change land in the window?}
B -- Yes --> C[Roll back the change, re-measure]
B -- No --> D{Input distribution shifted?}
D -- Yes --> E{Is the eval set still representative?}
E -- No --> F[Refresh the eval set FIRST, then re-benchmark]
E -- Yes --> G[Re-tune prompts or routing, gate on the eval set]
D -- No --> H{Provider-side change suspected?}
H -- Yes --> I[Pin the previous identifier, open a provider ticket, re-run the eval set]
H -- No --> J[Escalate to the Human tier, investigate before any model change]
```
The common mistake is jumping to a model change when the eval set has gone stale. If the eval set no
longer represents production traffic, every measurement downstream of it is unreliable and a "fix"
chosen on that basis is a guess.
---
## Safety and Ethical Guardrails
```python
# Post-processing checks on every output. Deterministic scanners, not a model judging itself.
SAFETY_CHECKS = [
"pii_detection", # Emails, phone numbers, national IDs, card fragments
"cross_customer_leak", # Any identifier not belonging to this ticket's customer
"price_and_plan_claims", # Claims about price or plan limits absent from supplied context
"feature_verification", # Named features must exist in the current feature registry
"unauthorized_commitment", # Refunds, credits, deadlines, SLAs the system may not promise
"regulated_advice", # Legal, medical, financial advice patterns
"tone_check", "disclosure_present",
]
# On any failure: the response is NOT sent, the ticket routes to the Human tier, the incident is
# logged with the failing check and full request context, and an alert fires above threshold.
```
- **Human oversight is a control, not a courtesy.** Sensitive categories never get an automated
response regardless of confidence, every automated response is reversible, and the reviewer is
recorded.
- **Disclosure.** Output is labelled AI-generated at the point a person reads it, not in a policy
document. The system never claims to be human and never denies being an AI when asked.
- **Contestability.** A customer can request human handling in one step, and that request is logged.
- **Fairness.** Accuracy and escalation rates are reported per category and per customer segment.
An aggregate number hides the segment the system serves badly.
- **Data minimization.** Send the narrowest context that clears the accuracy floor, and prove a
wider one is needed before sending it.
---
## Model Cards
A model card is a versioned file in the repo, updated in the same PR as the change it describes. It
is how you answer a documentation or traceability question without a scramble.
```markdown
# Model Card - Claro Support Drafting
**Card version**: [n] **Updated**: 2026-07-27 **Owner**: Support Platform team
## System
Purpose: draft support responses for human review; classify and route tickets.
Deployment: production, [N] units/day. Human review required before any response is sent.
Tiers and pinned identifiers: config/model_config.py plus this deployment's environment.
Not intended for: automated refunds, account security decisions, legal or financial advice,
employment or credit decisions, or any use where output reaches a customer unreviewed.
## Data
Training or tuning data: none - this is a prompted system, not a fine-tuned one.
Eval set: [name] [version], [N] items, labelled by [who], agreement [value].
Known gaps and biases: restate them here. Do not link to Data Lineage and hope.
## Performance
Primary metric on eval set [version]: [value], measured [date].
Latency p95 at production concurrency: [value]. Cost per unit: [value].
Per-segment breakdown: [table or link].
## Limitations observed
[Concrete failure modes seen in evaluation and production. "Sometimes wrong" is a disclaimer,
not a limitation.]
## Human oversight
Who reviews, what they can override, how a decision is contested, how long records are kept.
## Change log
| Card version | Date | Change | Approved by |
|--------------|------|--------|-------------|
| [n] | 2026-07-27 | [what changed and why] | [name] |
```
---
## Regulatory Documentation - EU AI Act
If your system is placed on the market or used in the EU, the EU AI Act's main application date is
**2 August 2026**. Two things become relevant then, and they are commonly conflated.
**Article 50 transparency obligations apply broadly**, including to systems that are **not**
high-risk: people must be informed when they are interacting with an AI system, and AI-generated or
AI-manipulated content must be identifiable as such. This is the obligation most teams actually have.
**Annex III high-risk requirements (Articles 6 to 15)** bring conformity assessment, technical
documentation, logging and traceability, human oversight, accuracy and robustness measures, CE
marking, and registration in the EU database. They apply to the **specific use cases listed in Annex
III** - among them biometrics, critical infrastructure, education access, employment and worker
management, access to essential services, law enforcement, migration, and administration of justice
- plus AI acting as a safety component of a regulated product.
**Do not overclaim, in either direction.** Ordinary AI developer assistance, and AI-generated code in
general, do not by themselves make a system high-risk: Annex III regulates named use cases, not the
presence of AI. Equally, "we are not high-risk" is not a conclusion you assert - it is one you
document, because this same system pointed at worker evaluation instead of ticket drafting lands in a
different category. Classification is a legal determination for your organization to make with
counsel. What this file guarantees is that the evidence exists and is current when asked.
The compliance questions are **documentation, traceability, and human oversight**:
| Question an assessor asks | Where this repo answers it |
|---------------------------|----------------------------|
| What does the system do, and what is it not for? | Model card, System section |
| What data was used, and what are its known biases? | Data Lineage, Model card Data section |
| How was performance measured, and when? | Eval Harness, benchmark table with measurement dates |
| What are the known limitations? | Model card, Limitations - stated concretely |
| Can you reconstruct a specific past decision? | Request log - request id, tier, pinned identifier, prompt version |
| Who reviewed it, and can a person override it? | Provenance, human review fields, Human tier routing |
| Is AI-generated content identifiable to the user? | `disclosure_present` check, DISCLOSURE_NOTICE |
| Who is accountable? | Named owners on the model card, datasets, and alerts |
---
## Provenance and Audit Trail
Every artifact that reaches a person should be reconstructible: **what was generated, by which
system, reviewed by whom, and when.**
```python
# Written once per generated artifact, append-only, retained per the retention policy.
provenance_record = {
"artifact_id": artifact_id, "artifact_type": "support_response_draft",
"generated_at": iso8601_utc,
"generated_by": {
"tier": tier_name,
"model_id": resolved_model_id, # The exact pinned identifier
"prompt_version": PROMPT_VERSION,
"config_commit": git_sha, # Ties to the exact config in git
},
"inputs_reference": {
"ticket_id": ticket_id,
"context_fields": ["account_tier", "recent_activity"], # Names sent, not values
"input_hash": sha256_of_rendered_prompt,
},
"safety_checks": {"pii_detection": "pass", "disclosure_present": "pass"},
"review": {
"reviewed_by": reviewer_id, # null means not yet reviewed
"reviewed_at": iso8601_utc,
"action": "sent_unchanged", # sent_unchanged | sent_edited | rejected | escalated
"edit_distance": 0,
},
"disclosed_to_user": True, "retention_expires_at": iso8601_utc,
}
```
- **Append-only.** A correction is a new record referencing the old one, never an in-place update.
- **Record the pinned identifier, not just the tier name.** "Reasoning tier" does not tell a future
investigator what actually ran.
- **Reference inputs, do not copy them.** Hashes and field names, so the record is useful without
becoming a second copy of customer data.
- **Human review is a recorded event.** "A human looked at it" with no record is indistinguishable
from no review.
- **Ship these events to your central log store** under the same retention and access controls as
other audit data.
---
## Troubleshooting
**High latency**: provider status page first. Then `calls_per_unit` - a retry loop shows up as
latency before it shows up as cost. Then prompt growth, which taxes every request. Then tier mix:
more Reasoning-tier traffic raises aggregate p95 with nothing else changing.
**Low output quality**: tier mix and routing threshold before blaming the model. Then whether the
eval set still represents production traffic - stale eval, unreliable everything. Then judge
agreement with the calibration subset, since the measurement may be what moved. Then recent
experiment rows, including reverted ones, for a partial revert. Then few-shot examples against the
current product state.
**Cost spikes**, in order: `calls_per_unit` (retries and reflection loops), tier mix (a
confidence-threshold change quietly moves volume to the expensive tier), cache hit rate, input length
(one new context field multiplies across every request). Per-token prices last - and if those
changed, re-run the benchmark and stamp a fresh measurement date on the table.
**Suspected provider-side change**: compare the eval result against the last recorded run for the
same pinned identifier. If the identifier is stable and results moved, record an experiment row with
"provider-side change" as the cause - that row is the only evidence you will have later. Pin the
previous known-good identifier if one is still available, and open a provider ticket.
Why Markdown Matters for AI-Native Development
Reproducible Model Runs
A result you cannot reproduce is a rumor. GenAI.md records the seed, the library versions, the data snapshot, the hardware, and the exact configuration next to the number that came out. Six months later, when somebody asks whether that improvement was real, the file answers instead of the person who has since moved teams.
Eval Harnesses as the Real Test Suite
Model behavior does not have unit tests. It has an eval set, and an eval set is only as honest as its documentation. Write down what each case checks, why it was added, what score counts as passing, and which failures you have consciously decided to accept. An undocumented suite drifts into a scoreboard nobody trusts.
Drift, Monitoring, and Rollback
Deployed models degrade quietly as the input distribution moves underneath them. GenAI.md documents the thresholds that trigger an alert, the difference between retraining and rolling back, and who decides. Written on a calm afternoon that decision tree takes minutes. Written during an incident it takes the whole night.
"The model is the part everyone photographs and the smallest part of the work. What decides whether a system stays useful is the surrounding record: what it was trained on, what it was measured against, and what you agreed to do when the numbers start moving."
Frequently Asked Questions
What is GenAI.md?
GenAI.md is a structured markdown template for documenting generative AI projects. It covers model configuration, experiment tracking, data lineage, evaluation metrics, and deployment pipelines in a format that keeps ML teams aligned and experiments reproducible.
How does GenAI.md help with experiment tracking?
GenAI.md provides structured templates for logging every experiment with its hypothesis, configuration, results, and learnings. By capturing experiments in versioned markdown, your team avoids re-running dead ends and builds on prior discoveries.
Can GenAI.md replace MLflow or Weights & Biases?
GenAI.md complements experiment tracking tools rather than replacing them. It captures the human-readable context - rationale, trade-off decisions, and institutional knowledge - that automated trackers miss. Use both for complete coverage.
What model documentation does GenAI.md support?
GenAI.md includes templates for model cards, selection rationale tables, hyperparameter decision logs, and API configuration documentation. Each template captures both the technical specification and the business reasoning behind model choices.
How does GenAI.md handle data lineage documentation?
GenAI.md provides templates for tracing training data from source to processed dataset, including cleaning steps, filtering criteria, augmentation techniques, and known biases. This lineage documentation is critical for debugging model behavior and compliance.
Is GenAI.md free to use?
Yes. GenAI.md is completely free. Download the markdown template, adapt it to your ML project, and version it in your repository alongside your model code and training pipelines.
How does GenAI.md support production ML deployments?
GenAI.md includes deployment checklists, monitoring criteria templates, rollback procedures, and canary release documentation. These structured runbooks prevent the costly failures that occur when ML deployment knowledge lives only in senior engineers' heads.
About GenAI.md
Our Mission
GenAI.md is an RJL template for the operational side of model work - the parts that outlive any single checkpoint.
Checkpoints get replaced constantly. The eval set, the data provenance, and the decision log do not, and they are what makes each replacement quick rather than exploratory. Keeping them in files that live in the repository attaches them to the project instead of to a notebook on somebody's laptop.
Model cards deserve a specific mention here. A short, honest page covering intended use, known limitations, measured performance, and update history is the only artifact most of an organization will ever read about a system, and increasingly it is the one people ask for when they need to explain what a deployment does and who checked it.
Why Markdown Matters
AI-Native
LLMs parse markdown better than any other format. Fewer tokens, cleaner structure, better results.
Version Control
Context evolves with code. Git tracks changes, PRs enable review, history preserves decisions.
Human Readable
No special tools needed. Plain text that works everywhere. Documentation humans actually read.
Shipping models into production? Trade notes on evals, drift, and the documentation that survives a handoff.