RFC-032: Post-Hoc Evaluation Loop — 8. Calibration Requirements

AIGP Specification › RFC-032: Post-Hoc Evaluation Loop › 8. Calibration Requirements

← 7. Governance Stage Extension · Section index · 9. Second-Order Cybernetics: Self-Learning Governance →

8. Calibration Requirements

8.1 The Eval-the-Evaluator Problem

A VERIFY grader that agrees with human judgment 70% of the time produces verdicts with 30% uncertainty. Without calibration, every MISMATCH verdict may be a grader error rather than an agent failure.

8.2 Calibration Protocol

Organizations deploying MODEL graders MUST maintain a calibration dataset:

{
  "calibration_dataset_id": "cal-2026-q2",
  "created_at": "2026-06-01T00:00:00Z",
  "scenarios": 200,
  "human_labeled": true,
  "grader_agreement_rate": 0.89,
  "false_positive_rate": 0.04,
  "false_negative_rate": 0.07,
  "cohens_kappa": 0.82,
  "last_recalibrated": "2026-06-15T00:00:00Z"
}

Metric	Minimum Threshold	Action if Below
grader_agreement_rate	0.80	Grader marked UNCALIBRATED; verdicts are advisory only
false_positive_rate	0.10	Investigate grader bias toward MISMATCH
false_negative_rate	0.10	Investigate grader bias toward MATCH
cohens_kappa	0.70	Retrain or replace grader

8.3 Progressive Automation

The calibration framework enables a maturity progression:

graph LR
    L0[Level 0: No post-hoc evaluation] --> L1[Level 1: HUMAN graders only]
    L1 --> L2[Level 2: COMPOSITE]
    L2 --> L3[Level 3: MODEL calibrated >0.85]
    L3 --> L4[Level 4: CODE derived from MODEL]

Each level reduces human review burden while maintaining evaluation quality — provided calibration metrics remain above threshold.

---

---

← 7. Governance Stage Extension · Section index · 9. Second-Order Cybernetics: Self-Learning Governance →