RFC-033 Quantitative Outcome Evaluation — References

PRIVATE AND PROPRIETARY. Owned by Kanjani AI Research & Causum. See NOTICE.md.

Primary Sources

Industry Frameworks

Anthropic (2025). “Demystifying evals for AI agents.” Anthropic Engineering Blog. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Taxonomy of agent evaluation types (unit, integration, end-to-end)
- Scoring methodology: code-based, model-based, and human graders
- Key insight: “Evals make problems visible before they affect users”
Anthropic (2024). “A statistical approach to model evaluations.” Anthropic Research. https://www.anthropic.com/research/statistical-approach-to-model-evals
- Score computation: observed average across independently scored questions
- Statistical significance in evaluation results
Anthropic (2024). “Empirical performance evaluations.” Claude Documentation. https://docs.anthropic.com/claude/docs/empirical-performance-evaluations
- Quantitative measures as integral to production lifecycle
- Data-driven decision making from eval scores
AWS (2025). “Build reliable AI agents with Amazon Bedrock AgentCore Evaluations.” https://aws.amazon.com/blogs/machine-learning/build-reliable-ai-agents-with-amazon-bedrock-agentcore-evaluations/
- Managed evaluation across multiple quality dimensions
- Online evaluation with configurable sampling rates
AWS (2025). “LLM-as-a-judge on Amazon Bedrock Model Evaluation.” https://aws.amazon.com/blogs/machine-learning/llm-as-a-judge-on-amazon-bedrock-model-evaluation/
- LLM-as-Judge methodology for automated scoring
- Dimensions: correctness, completeness, faithfulness, relevance
- RAG evaluation integration
AWS (2025). “Evaluating AI agents: Real-world lessons from building agentic systems at Amazon.” https://aws.amazon.com/blogs/machine-learning/evaluating-ai-agents-real-world-lessons-from-building-agentic-systems-at-amazon/
- Agentic evaluation beyond single-model benchmarks
- Tool selection accuracy, multi-step reasoning coherence, task completion rates
AWS (2025). “Beyond the basics: A comprehensive foundation model selection framework.” https://aws.amazon.com/blogs/machine-learning/beyond-the-basics-a-comprehensive-foundation-model-selection-framework-for-generative-ai/
- Multi-dimensional evaluation beyond accuracy/latency/cost
- Real-world model performance factors

Standards Bodies

NIST (2023). “AI Risk Management Framework (AI RMF 1.0).” National Institute of Standards and Technology. https://www.nist.gov/itl/ai-risk-management-framework
- MEASURE function: empirical evidence base for AI risk management
- Four core functions: GOVERN → MAP → MEASURE → MANAGE
- Metrics vs. measures distinction
NIST (2024). “AI RMF Generative AI Profile (AI 600-1).” https://airc.nist.gov/airmf-resources/airmf/5-sec-core/
- GenAI-specific measurement categories
- Trustworthiness characteristics as evaluation dimensions
ISO/IEC 42001:2023. “Artificial Intelligence — Management System.”
- Quality management system requirements for AI
- Continuous monitoring and measurement obligations

Academic Research

Zheng et al. (2024). “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” NeurIPS 2023. https://arxiv.org/abs/2306.05685
- Foundational paper on LLM-as-Judge methodology
- Position bias, verbosity bias, self-enhancement bias
Li et al. (2024). “Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge.” https://arxiv.org/abs/2410.02736
- Systematic analysis of LLM judge biases
- Mitigation strategies for scoring reliability
Chen et al. (2025). “Evaluating Scoring Bias in LLM-as-a-Judge.” https://arxiv.org/abs/2506.22316
- Scoring bias under pointwise evaluation
- Impact of rubric design on score distribution
Liu et al. (2025). “Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge.” https://arxiv.org/abs/2602.02219
- Rubric position effects on scoring
- Recommendations for rubric construction
Wang et al. (2025). “Decomposed Criteria-Based Evaluation of LLM Responses (DeCE).” https://arxiv.org/abs/2509.16093
- Separating precision (factual accuracy) from recall (coverage)
- Instance-specific criteria extraction from gold answers
Zhang et al. (2024). “A Practical Guide for Evaluating LLMs and LLM-Reliant Systems.” https://arxiv.org/abs/2506.13023
- Production evaluation pipeline design
- Handling infinite prompt/response space
Kumar et al. (2025). “Evaluation of Large Language Models: Review of Metrics, Applications, and Methodologies.” https://www.preprints.org/manuscript/202504.0369
- Survey of 70+ studies on LLM evaluation
- Critical limitations in open-ended reasoning evaluation

Tools and Platforms

MLflow (2025). “LLM Evaluation Frameworks Explained for AI Practitioners.” https://mlflow.org/articles/llm-evaluation-frameworks-explained-for-ai-practitioners/
- Standardized evaluation architectures
- Reproducibility and auditability requirements
Braintrust (2025). “Evaluating agents.” https://www.braintrust.dev/blog/evaluating-agents
- Metrics vs. scorers distinction
- Code-based and LLM-as-Judge scoring functions
Arize Phoenix. “LLM Evaluation Models.” https://arize.com/docs/phoenix/api/evaluation-models
- Faithfulness, correctness, relevance evaluation tooling
- Production monitoring integration

Relationship to RFC-033 Dimensions

RFC-033 Dimension	Supporting References
Correctness (Qc)	[1] Anthropic evals, [5] AWS LLM-as-Judge, [15] DeCE precision
Completeness (Qk)	[5] AWS faithfulness, [15] DeCE recall, [6] task completion
Relevance (Qr)	[5] AWS relevance, [11] MT-Bench grading, [20] Arize
Safety (Qs)	[8] NIST MEASURE, [9] GenAI Profile, [4] AgentCore Evaluations
Efficiency (Qe)	[7] AWS model selection framework, [6] Amazon agent eval
Consistency (Qn)	[2] Anthropic statistical approach, [17] output consistency
LLM-as-Judge bias	[12][13][14] bias quantification and mitigation
Scoring methodology	[1][2][3] Anthropic eval methodology
Governance integration	[8] NIST GOVERN→MAP→MEASURE→MANAGE cycle