This page compiles 9 files from the repository, verbatim, in reading order. The living version: this folder on GitHub.
3-expert/shared/decision-record-template.md
AI Decision Record Template
Decision title: Status: proposed / accepted / superseded / retired Date / owner / approvers: Decision expiry or review date:
Context
Describe the customer problem, production constraints, and decision boundary. Include the affected workflows, properties or tenants, data classes, dependencies, security and safety considerations, expected volume, latency target, and unit-cost target. Link the relevant system card, evaluations, incident history, and vendor terms.
Options considered
Option
Benefits
Risks / constraints
Evidence
Estimated cost
Reversibility
Do not compare only model quality. Include operational ownership, data residency, permission model, observability, failure modes, migration effort, and customer impact. For example, an open-weight deployment may lower marginal cost while increasing on-call and security burden.
Decision
State exactly what will be used, by whom, for which cohort, and under which controls. Record the acceptance criteria: offline evaluation threshold, staged rollout plan, SLOs, guardrails, cost cap, and rollback condition. Name the person authorized to stop or reverse the decision.
Consequences and follow-up
List accepted tradeoffs, unresolved risks, documentation updates, migrations, training, and time-bound follow-up actions. Link implementation pull requests, dashboards, feature flags, and launch evidence. A decision record should let a future operator understand why a configuration exists and when it must be reconsidered.
3-expert/shared/incident-post-mortem-template.md
AI Incident Post-Mortem Template
Title: Incident ID: Severity / start / end: Incident lead / communications lead / technical lead: Affected feature, properties, tenants, and channels:
Executive summary
Describe what users experienced, how many were affected, the worst plausible harm, and the final state. Use concrete language: “The maintenance assistant showed one tenant another tenant’s ticket summary for 17 minutes,” not “there was an issue with retrieval.” State whether any unsafe tool action, legal obligation, or data notification process was triggered.
Timeline
Time
Event
Evidence / decision
Detection
Containment
Customer communication
Recovery
Impact and scope
List confirmed impact, uncertain impact, and population checked. Segment by property, tenant boundary, model/prompt/retrieval version, provider, region, and release cohort. Include customer contacts, financial effect, support load, and safety or privacy implications.
Technical and control analysis
What changed? What condition allowed the failure? Which controls detected it, failed to detect it, or limited blast radius? Link traces, logs, deployment records, policy decisions, eval evidence, and the exact configuration. Do not attribute root cause to “the model hallucinated”; name the missing validation, ambiguous contract, bad data, unsafe permission, or untested dependency.
Corrective actions
Action
Type: containment / corrective / preventive
Owner
Due date
Verification evidence
Lessons and follow-up
Record what should change in the runbook, eval suite, dashboard, release gate, vendor strategy, or training. The incident is closed only after actions are verified, affected users receive required follow-up, and the learning is shared with the relevant team.
3-expert/shared/model-card-template.md
Model Card Template
Model identity
Name/version/date:
Base model and license:
Owner/contact:
Weights or endpoint identifier:
Intended use
Describe target users, supported tasks, operating languages, input limits, and expected environment. State prohibited or unsupported uses plainly.
Training and adaptation
Training objective and method
Data sources, rights, dates, filtering, and known gaps
Fine-tuning/preference data description
Hyperparameters, tokenizer, compute, and run identifier
Evaluation
Evaluation
Dataset/slices
Metric
Result
Limitations
Include comparison baseline, contamination controls, human-evaluation protocol, calibration, safety/refusal tests, and representative failures.
Risks and mitigations
Describe hallucination, bias, privacy, harmful-content, security, misuse, and distribution-shift risks. Pair every mitigation with its scope and remaining limitation.
Record changed data, weights, behavior, evaluations, and known regressions for every release.
This lesson belongs in a practitioner’s operating system, not a collection of facts to recite. The point is to make a better decision under uncertainty: define the claim, identify the evidence that could change it, name the failure mode, and record the consequence of being wrong. Read it with a live initiative in mind—an internal workflow, customer-facing product, training run, or research bet—and turn the ideas into an explicit test.
Start from the outcome rather than the technology. Specify the user or stakeholder, the task boundary, the data and permissions involved, the success measure, and the unacceptable result. Establish a baseline before changing anything. Then make the smallest reversible move that can distinguish competing explanations. A plausible demo is evidence of possibility, not evidence of reliability, value, or safety.
Keep an evidence log. Separate observations from interpretations, measured performance from anecdotes, and known risks from assumptions. Review representative failures by hand; aggregate metrics can hide the one pattern that matters. For high-impact work, assign a clear owner, predefine an escalation path, and decide what will cause a pause or rollback. Do not outsource accountability to a model, vendor, benchmark, or committee.
The professional standard is legibility. Another capable person should be able to understand why this approach was chosen, rerun the evaluation, find its limits, and improve it without guessing. Build reusable artifacts—datasets, decision records, checklists, incident notes, and release criteria—so each project leaves the next one stronger.
Working exercise
Write a one-page decision memo for a current initiative. State the hypothesis, baseline, evaluation, threshold, owner, risks, and next action. If any of these cannot be stated plainly, the work is not ready to scale.
3-expert/shared/production-ai-checklist.md
Production AI Checklist
Use this checklist as a release gate. Mark each item yes, not applicable, or blocked, with evidence and an owner. A launch is not complete because every box is checked; it is complete when the remaining risk is understood and explicitly accepted.
Product and scope
[ ] User problem is written in one sentence.
[ ] Target user and excluded users are named.
[ ] High-impact decisions are identified.
[ ] Non-AI alternative is documented.
[ ] Success metric is measurable.
[ ] Harm metric is measurable.
[ ] Customer-facing limitations are defined.
[ ] Human escalation path exists.
[ ] Feature owner is named.
[ ] Domain approver is named.
Data and privacy
[ ] Data-flow diagram is current.
[ ] Every source has an owner.
[ ] Tenant isolation is tested.
[ ] Authorization is enforced before retrieval.
[ ] PII inventory is complete.
[ ] Minimum-necessary fields are used.
[ ] Sensitive fields are redacted from logs.
[ ] Retention periods are defined.
[ ] Deletion path is tested.
[ ] Consent requirements are documented.
[ ] Vendor data terms are reviewed.
[ ] Regional residency is enforced.
[ ] Backups follow retention policy.
[ ] Dataset provenance is recorded.
[ ] Data freshness is visible.
Model and prompting
[ ] Model version is pinned.
[ ] Provider fallback is defined.
[ ] Prompt version is pinned.
[ ] System instructions state scope.
[ ] Untrusted content is labeled.
[ ] Context budget is bounded.
[ ] Structured output schema is validated.
[ ] Temperature and limits are intentional.
[ ] Model card is reviewed.
[ ] Known limitations are recorded.
[ ] Fine-tuning data is approved.
[ ] Retrieval index version is recorded.
[ ] Citation behavior is specified.
[ ] Refusal behavior is specified.
[ ] Model change procedure exists.
Retrieval and tools
[ ] Chunking strategy is documented.
[ ] Retrieval access is tenant-scoped.
[ ] Metadata filters are tested.
[ ] Stale documents have expiry rules.
[ ] Reindex plan is reversible.
[ ] Tool list is allowlisted.
[ ] Tools use least privilege.
[ ] Tool parameters are typed.
[ ] Irreversible actions require confirmation.
[ ] Idempotency keys protect writes.
[ ] Rate limits are set.
[ ] Tool retries are bounded.
[ ] Tool results are sanitized.
[ ] Audit events exist for writes.
[ ] Manual fallback works.
Safety and security
[ ] Threat model is approved.
[ ] Prompt injection tests pass.
[ ] Jailbreak tests pass.
[ ] Exfiltration tests pass.
[ ] Cross-tenant tests pass.
[ ] Identity spoofing tests pass.
[ ] Retrieval poisoning tests pass.
[ ] Abuse rate limits are tested.
[ ] Policy decisions have reason codes.
[ ] Guardrail bypass is monitored.
[ ] Secrets are not in prompts or logs.
[ ] Credentials are short-lived.
[ ] Vulnerability owner is named.
[ ] Red-team findings are tracked.
[ ] Security incident path is rehearsed.
Evaluation and release
[ ] Offline eval set is versioned.
[ ] Eval set is representative.
[ ] Hard negatives are included.
[ ] Safety evals are included.
[ ] Quality threshold is predeclared.
[ ] Cost threshold is predeclared.
[ ] Latency threshold is predeclared.
[ ] Results are segmented.
[ ] Human review rubric exists.
[ ] Statistical uncertainty is considered.
[ ] Shadow run is completed.
[ ] Canary cohort is defined.
[ ] Feature flag exists.
[ ] Dark-launch behavior is safe.
[ ] Rollback has been rehearsed.
Reliability and operations
[ ] SLOs are documented.
[ ] Error budget policy exists.
[ ] Availability is measured at the user edge.
[ ] Time to first useful response is measured.
[ ] Quality outcome is sampled.
[ ] Traces propagate across services.
[ ] Logs are structured.
[ ] Dashboard links to traces.
[ ] Alerts have runbooks.
[ ] Alert owner is named.
[ ] On-call rotation is staffed.
[ ] Incident severity model exists.
[ ] Status communication is prepared.
[ ] Queues have backpressure limits.
[ ] Provider outage behavior is tested.
[ ] Regional failover is tested.
[ ] Capacity is measured.
[ ] Cost per outcome is tracked.
[ ] Drift baseline exists.
[ ] Drift alerts have response actions.
UX, governance, and follow-through
[ ] Loading state is honest.
[ ] Cancellation works.
[ ] Error message offers a next step.
[ ] User corrections are captured.
[ ] Confidence language matches evidence.
[ ] Citations appear where decisions happen.
[ ] Accessible interaction is tested.
[ ] Admin controls are documented.
[ ] Change approval is recorded.
[ ] Decision record is stored.
[ ] Training is provided to operators.
[ ] Support team has a playbook.
[ ] Post-launch review is scheduled.
[ ] Success and harm metrics are reviewed.
[ ] Unused data is removed.
A failed item is useful information. Do not paper over it with an exception; convert it into a conscious launch decision, a compensating control, a named owner, and an expiry date.
3-expert/shared/red-team-report-template.md
Red-Team Report Template
Assessment name / date / owners: Systems and versions assessed: Authorization and rules of engagement: Test data classification and retention:
Objective and scope
State the harm hypotheses: cross-tenant disclosure, prompt injection through listing documents, unsafe maintenance escalation, tool misuse, identity spoofing, policy bypass, or denial of service. Name the in-scope model, retrieval sources, tools, identities, environments, and explicit exclusions. A red-team report without scope cannot be safely reproduced or prioritized.
Method
Describe attack families, test cases, automation, human review, success criteria, and stop conditions. Preserve exact prompts, documents, identities, configuration, timestamps, and traces in a restricted evidence store. Separate reproducible facts from the tester’s interpretation.
Findings
ID
Severity
Attack path
Preconditions
Impact
Reproduction
Evidence
Owner
Critical / high / medium / low
Score severity using exploitability, blast radius, sensitivity of data or action, likelihood, detection time, and effectiveness of existing controls. Demonstrate impact without accessing real customer data or causing irreversible action.
Remediation and retest
For each finding, specify immediate containment, durable fix, compensating control, due date, and retest criteria. A model-prompt adjustment alone is rarely sufficient for a permission-boundary problem. Close a finding only after the exact attack fails, regression tests are committed, telemetry can detect recurrence, and risk acceptance is recorded if any residual exposure remains.
3-expert/shared/safety-case-template.md
Safety Case Template
A safety case is an evidence-backed argument that the residual risk of a specific system and use is acceptable. It is not a policy statement.
1. System and decision
System/version:
Owner and approver:
Intended users and context:
Decision or action affected:
Explicitly excluded uses:
2. Claim
State the claim narrowly: “For [use], under [conditions], the system can be operated with acceptable risk because [controls and evidence].”
3. Hazard register
For each hazard, record the harmed party, severity, likelihood, signals, prevention, mitigation, and residual risk.
Hazard
Scenario
Preventive controls
Detection
Response
Residual risk
4. Evidence
Offline evaluation results and slice coverage
Human-review method and agreement
Red-team findings and closure status
Privacy/security assessment
Reliability, latency, and availability data
Provider/model-change assumptions
5. Operating controls
Describe access control, human approvals, runtime policy, logging, monitoring, incident ownership, rollback, and user recourse.
6. Limits and review
List untested conditions, assumptions, expiry date, next review trigger, and the named authority who can suspend the system. Link raw evidence; do not summarize away uncertainty.
This lesson belongs in a practitioner’s operating system, not a collection of facts to recite. The point is to make a better decision under uncertainty: define the claim, identify the evidence that could change it, name the failure mode, and record the consequence of being wrong. Read it with a live initiative in mind—an internal workflow, customer-facing product, training run, or research bet—and turn the ideas into an explicit test.
Start from the outcome rather than the technology. Specify the user or stakeholder, the task boundary, the data and permissions involved, the success measure, and the unacceptable result. Establish a baseline before changing anything. Then make the smallest reversible move that can distinguish competing explanations. A plausible demo is evidence of possibility, not evidence of reliability, value, or safety.
Keep an evidence log. Separate observations from interpretations, measured performance from anecdotes, and known risks from assumptions. Review representative failures by hand; aggregate metrics can hide the one pattern that matters. For high-impact work, assign a clear owner, predefine an escalation path, and decide what will cause a pause or rollback. Do not outsource accountability to a model, vendor, benchmark, or committee.
The professional standard is legibility. Another capable person should be able to understand why this approach was chosen, rerun the evaluation, find its limits, and improve it without guessing. Build reusable artifacts—datasets, decision records, checklists, incident notes, and release criteria—so each project leaves the next one stronger.
Working exercise
Write a one-page decision memo for a current initiative. State the hypothesis, baseline, evaluation, threshold, owner, risks, and next action. If any of these cannot be stated plainly, the work is not ready to scale.
3-expert/shared/slo-worksheet.md
AI SLO Worksheet
Use one worksheet per customer-facing journey. An SLO is an internal target for reliable, valuable behavior. It should be strict enough to protect trust and practical enough that the team can measure and improve it.
Avoid a vague “accuracy SLO.” For a leasing assistant, define whether the answer was based on an authorized source, whether it disclosed uncertainty, whether it avoided protected or sensitive inference, and whether the next action was correct. Quality often needs sampled human labels, so state sampling rate, reviewer rubric, lag, and the operational response to a bad result.
Operational plan
Containment action when fast burn occurs:
Owner and escalation path:
Feature flag or rollback control:
Customer communication path:
Manual fallback:
Dependencies whose failures count toward the SLO:
Dependencies whose failures are excluded, with reason:
Dashboard and trace links:
Review monthly and after every material incident. If a target cannot drive a decision, it is a report, not an SLO.
3-expert/shared/system-card-template.md
System Card Template
A model card describes a model. A system card describes the complete deployed product: model, prompts, retrieval, tools, users, policies, interfaces, and operations.
System definition
Name/version/owner:
Users and business purpose:
Model providers and component versions:
Data flows and trust boundaries:
High-impact decisions or actions:
Architecture
Describe input handling, identity/authorization, retrieval, model routing, tool calls, output validation, approval steps, logs, and data retention. Include a diagram link where available.
Behavior and boundaries
List supported tasks, refusal conditions, mandatory escalations, prohibited actions, localization requirements, and user disclosures. Explain what the system must not infer or decide.
Evidence
Claim
Evaluation/method
Result
Owner
Expiry
Include real-world monitoring and known failure examples, not only offline scores.
Risk operations
State red-team coverage, policy-engine controls, alert thresholds, incident playbook, rollback method, provider-outage plan, audit trail, and review cadence.
Change log
Every change to model, prompt, retrieval corpus, tools, permissions, or safety policy should have a linked evaluation and approval record.
This lesson belongs in a practitioner’s operating system, not a collection of facts to recite. The point is to make a better decision under uncertainty: define the claim, identify the evidence that could change it, name the failure mode, and record the consequence of being wrong. Read it with a live initiative in mind—an internal workflow, customer-facing product, training run, or research bet—and turn the ideas into an explicit test.
Start from the outcome rather than the technology. Specify the user or stakeholder, the task boundary, the data and permissions involved, the success measure, and the unacceptable result. Establish a baseline before changing anything. Then make the smallest reversible move that can distinguish competing explanations. A plausible demo is evidence of possibility, not evidence of reliability, value, or safety.
Keep an evidence log. Separate observations from interpretations, measured performance from anecdotes, and known risks from assumptions. Review representative failures by hand; aggregate metrics can hide the one pattern that matters. For high-impact work, assign a clear owner, predefine an escalation path, and decide what will cause a pause or rollback. Do not outsource accountability to a model, vendor, benchmark, or committee.
The professional standard is legibility. Another capable person should be able to understand why this approach was chosen, rerun the evaluation, find its limits, and improve it without guessing. Build reusable artifacts—datasets, decision records, checklists, incident notes, and release criteria—so each project leaves the next one stronger.
Working exercise
Write a one-page decision memo for a current initiative. State the hypothesis, baseline, evaluation, threshold, owner, risks, and next action. If any of these cannot be stated plainly, the work is not ready to scale.
3-expert/shared/training-run-checklist.md
Training Run Checklist
Use this before any fine-tune, preference-optimization job, or material retraining. A run that cannot be explained, reproduced, evaluated, and rolled back is not ready.
Decision and scope
[ ] State the user outcome, hypothesis, and baseline.
[ ] Name the model, version, license, and deployment target.
[ ] Define success metrics, safety metrics, and a ship/no-ship threshold.
[ ] Identify the accountable owner and the person authorized to stop the run.
[ ] Produce a model-card update and release decision record.
[ ] Stage deployment with monitoring, rollback, and user escalation paths.
Afterward
Write what changed, what did not, what surprised you, and which next experiment this result justifies. Negative results are assets when their conditions are preserved.
This lesson belongs in a practitioner’s operating system, not a collection of facts to recite. The point is to make a better decision under uncertainty: define the claim, identify the evidence that could change it, name the failure mode, and record the consequence of being wrong. Read it with a live initiative in mind—an internal workflow, customer-facing product, training run, or research bet—and turn the ideas into an explicit test.
Start from the outcome rather than the technology. Specify the user or stakeholder, the task boundary, the data and permissions involved, the success measure, and the unacceptable result. Establish a baseline before changing anything. Then make the smallest reversible move that can distinguish competing explanations. A plausible demo is evidence of possibility, not evidence of reliability, value, or safety.
Keep an evidence log. Separate observations from interpretations, measured performance from anecdotes, and known risks from assumptions. Review representative failures by hand; aggregate metrics can hide the one pattern that matters. For high-impact work, assign a clear owner, predefine an escalation path, and decide what will cause a pause or rollback. Do not outsource accountability to a model, vendor, benchmark, or committee.
The professional standard is legibility. Another capable person should be able to understand why this approach was chosen, rerun the evaluation, find its limits, and improve it without guessing. Build reusable artifacts—datasets, decision records, checklists, incident notes, and release criteria—so each project leaves the next one stronger.
Working exercise
Write a one-page decision memo for a current initiative. State the hypothesis, baseline, evaluation, threshold, owner, risks, and next action. If any of these cannot be stated plainly, the work is not ready to scale.