RFC-033: Quantitative Outcome Evaluation Model — 3. Outcome Score Model
AIGP Specification › RFC-033: Quantitative Outcome Evaluation Model › 3. Outcome Score Model
← 2. Motivation · Section index · 4. Temporal Model →
3. Outcome Score Model
3.1 Dimensions
Every invocation outcome is scored across six orthogonal dimensions:
| Dimension | Symbol | What it measures | Weight (default) |
|---|---|---|---|
| Correctness | Qc | Did the output satisfy the stated intent? | 0.30 |
| Completeness | Qk | Was the response thorough and sufficient? | 0.20 |
| Relevance | Qr | Was the response on-topic, no hallucination? | 0.20 |
| Safety | Qs | Were guardrails respected, no harmful content? | 0.15 |
| Efficiency | Qe | Was cost/latency proportional to value? | 0.10 |
| Consistency | Qn | Is this response consistent with prior responses? | 0.05 |
3.2 Composite Score
OS = Σ(wi × Qi) where Σwi = 1.0
OS = 0.30·Qc + 0.20·Qk + 0.20·Qr + 0.15·Qs + 0.10·Qe + 0.05·QnWeights are configurable per contract (AR rules). A compliance-critical app might weight Safety at 0.40; a creative app might weight Completeness at 0.35.
3.3 Dimension Scoring Methods
Each dimension has a defined scoring method, falling into three categories:
Deterministic (computed from evidence):
| Dimension | Method | Source |
|---|---|---|
| Safety (Qs) | 1.0 - (guardrail_triggers / total_invocations) |
RECORD.guardrail |
| Efficiency (Qe) | 1.0 - clamp((latency - target) / target, 0, 1) |
RECORD.duration_ms |
LLM-as-Judge (evaluated by a grading model):
| Dimension | Method | Source |
|---|---|---|
| Correctness (Qc) | Grading LLM compares output vs. intent + ground truth | RECORD.prompt + response |
| Completeness (Qk) | Grading LLM assesses coverage of required elements | RECORD.response vs. schema |
| Relevance (Qr) | Grading LLM detects hallucination / off-topic content | RECORD.response vs. context |
Statistical (derived from population):
| Dimension | Method | Source |
|---|---|---|
| Consistency (Qn) | Embedding similarity vs. mean response for same query class | Response embeddings |
3.4 Score Bands
| Band | Range | Interpretation | Governance Action |
|---|---|---|---|
| Excellent | 0.90–1.00 | Exceeds expectations | Expand autonomy, reduce oversight |
| Good | 0.75–0.89 | Meets expectations | Normal operation |
| Marginal | 0.60–0.74 | Below expectations | Increase sampling, alert |
| Poor | 0.40–0.59 | Significant quality issues | Restrict scope, human review |
| Failing | 0.00–0.39 | Unacceptable | Suspend, circuit break |