RFC-033: Quantitative Outcome Evaluation Model — 6. Evaluation Pipeline

AIGP Specification › RFC-033: Quantitative Outcome Evaluation Model › 6. Evaluation Pipeline

← 5. Protocol Integration · Section index · 7. Governance Actions →

6. Evaluation Pipeline

6.1 Architecture

RECORD arrives at GOV_APP
    │
    ├── [Deterministic] Compute Qs, Qe immediately
    │
    ├── [LLM-as-Judge] Sample 10% of invocations → grade Qc, Qk, Qr
    │   └── Uses Haiku (fast/cheap) as grading model
    │
    ├── [Statistical] Batch: compute Qn from embedding similarity
    │
    └── Compose OS → attach to RECORD → update baseline → check AR rules

6.2 Sampling Strategy

Not every invocation is fully evaluated (LLM grading has cost):

Tier	Sampling Rate	Trigger
Critical use_cases	100%	Always grade
Standard	10%	Random sample
Low-risk	1%	Random sample
Regression detected	100%	Temporary escalation
User feedback negative	100%	Reactive grade

6.3 Grading Prompt Template

You are an AI output quality evaluator. Score the following response on a scale of 0.0 to 1.0.

CONTEXT:
- User query: {query}
- System prompt intent: {system_prompt_summary}
- Retrieved context provided: {context_summary}

RESPONSE TO EVALUATE:
{response}

Score each dimension:
- Correctness (0.0-1.0): Is the response factually accurate given the context?
- Completeness (0.0-1.0): Does it fully address the query?
- Relevance (0.0-1.0): Is everything in the response relevant? No hallucination?

Return JSON: {"correctness": X, "completeness": Y, "relevance": Z}

---

---

← 5. Protocol Integration · Section index · 7. Governance Actions →