Skip to content
AIGP

RFC-033: Quantitative Outcome Evaluation Model — 10. Open Questions

AIGP Specification › RFC-033: Quantitative Outcome Evaluation Model › 10. Open Questions

← 9. Relationship to Existing Components · Section index · 11. References →

10. Open Questions

  1. Who grades the grader? — How do we evaluate the LLM-as-Judge’s accuracy? (Meta-evaluation loop)
  2. Ground truth bootstrapping — For new use_cases with no baseline, what’s the initial threshold?
  3. Cross-model comparability — Is a 0.85 from Sonnet equivalent to 0.85 from Haiku?
  4. Adversarial gaming — Can an agent optimize for the grading prompt rather than actual quality?
  5. Cultural bias in grading — Does the grading model have consistent quality standards across languages?

← 9. Relationship to Existing Components · Section index · 11. References →