RFC-032: Post-Hoc Evaluation Loop — 5. Grader Types
AIGP Specification › RFC-032: Post-Hoc Evaluation Loop › 5. Grader Types
← 4. VERIFY Message · Section index · 6. Protocol Flow →
5. Grader Types
5.1 CODE Grader (Deterministic)
Evaluates assertions as executable expressions against the RECORD/TRACE/STEP_COMPLETE evidence chain. No ambiguity. No judgment. Boolean pass/fail per criterion.
Suitable for:
- State verification (does resource X exist?)
- Count constraints (step_count <= N)
- Negative assertions (tool Y was not invoked)
- Budget assertions (tokens_used <= anticipated)
Implementation: The governance server executes assertion expressions against a structured evidence payload. Assertions use a restricted expression language (no side effects, no network access, read-only against evidence).
5.2 MODEL Grader (LLM-as-Judge)
Evaluates criteria that require semantic understanding — quality of output, appropriateness of reasoning, compliance with intent rather than literal instruction.
Suitable for:
- Output quality assessment (was the response helpful?)
- Reasoning quality (did the agent’s intermediate reasoning follow sound logic?)
- Tone and style compliance (was the output appropriate for the context?)
- Ambiguous criteria where deterministic evaluation is not possible
Implementation: A separate model invocation (isolated from the agent being evaluated) receives the ANTICIPATE criteria, the RECORD evidence, and a grading prompt. Its output is parsed into per-criterion scores.
Calibration requirement: MODEL graders MUST be calibrated against human labels (see §8). Uncalibrated MODEL graders MUST be marked as confidence: "UNCALIBRATED" in grader_metadata.
5.3 HUMAN Grader
Routes the ANTICIPATE + RECORD bundle to a human reviewer who provides per-criterion pass/fail judgments.
Suitable for:
- High-stakes decisions where automated evaluation is insufficient
- Calibration exercises (building ground truth for MODEL grader training)
- Novel scenarios without established grading criteria
- Regulatory contexts requiring human oversight (EU AI Act Art. 14)
Implementation: The governance server emits an escalation-style notification containing the anticipation, evidence, and a grading form. The reviewer’s responses are captured as the VERIFY result.
5.4 COMPOSITE Grader
Mixes grader types per criterion. Some criteria are CODE-graded, others MODEL-graded, others HUMAN-graded. The VERIFY result aggregates across all grader types.
Suitable for:
- Complex tasks with both deterministic and semantic requirements
- Production systems where human review is reserved for ambiguous cases
- Progressive automation (start HUMAN, calibrate MODEL, graduate to CODE)