For agent evals, should we consider prompts that share the same environment (a code repo, a synthetic database, etc.) as dependent and in the same cluster? What would that mean for a set of agent evals that all share the same environment, like the agent company, as an example
This one is a bit of a grey area. I would probably say yes... similar to how asking about the same document forms a cluster. However, this depends a lot on what exactly your environment is - super specific environmental setups shold probably be clustered together, but there could be very different questions within the same environment. You could also treat this as one evaluation domain within your dataset. For example, tau bench does something kinda similar to this by forming a multi-domain agent eval.
Thank You for writing this. Much needed. I think soon we'll be in the Era of some substack posts getting more Citations than papers. This might be one of those.
This analysis is the necessary antidote to the "highest number is best" culture that currently dominates LLM benchmarks. However, the problem goes deeper than just statistical naivety; it is a fundamental category error in how we conceptualize LLM output.
When researchers treat evaluation datasets as objective, ground-truth metrics, they are operating under an FSSP (Fully Specified Stochastic Process) assumption. They are effectively hallucinating that the model’s performance on a noisy, finite evaluation set is a complete, finished state of "intelligence." Because they ignore the underlying stochasticity, they mistake sampling noise for systemic progress.
To move past this, we need to apply the Observational Sufficiency Principle (OSP) as a prerequisite for any claim of improvement. The OSP acts as the reality-safeguard: it forces us to ask whether the observational data (the evaluation metrics) is actually sufficient to support the conclusion of "advancement." If the metrics are statistically indistinguishable from noise, the observational space is insufficient, and the claim of progress is a structural hallucination.
We don't just need better statistics; we need an OSP-compliant evaluation framework that rejects the FSSP illusion. Until we treat model evaluations as constrained observational spaces rather than objective truth-claims, we will continue to "improve" models by simply overfitting to the noise in our own benchmarks.
There are a few places where I suspect there may be typos (all in figures):
1. Standard error and confidence interval for LLM evaluations (from [1]): In the standard error formula, a square appears to be missing, i.e., ((s_i - \bar{s})^2).
2. Standard error of the estimated difference in mean scores: In the last line, I think the SE terms are missing the bar notation above them.
3. Standard error of the mean score difference (from [1]): The differences also appear to be missing the squaring.
Not always. Usually, these focus on a single domain, but this domain is very broad. E.g., BIG-Bench is a reasoning benchmark, but it has 200+ tasks, each with many questions that cover various aspects of reasoning.
For agent evals, should we consider prompts that share the same environment (a code repo, a synthetic database, etc.) as dependent and in the same cluster? What would that mean for a set of agent evals that all share the same environment, like the agent company, as an example
This one is a bit of a grey area. I would probably say yes... similar to how asking about the same document forms a cluster. However, this depends a lot on what exactly your environment is - super specific environmental setups shold probably be clustered together, but there could be very different questions within the same environment. You could also treat this as one evaluation domain within your dataset. For example, tau bench does something kinda similar to this by forming a multi-domain agent eval.
Thank You for writing this. Much needed. I think soon we'll be in the Era of some substack posts getting more Citations than papers. This might be one of those.
Appreciate the kind words!
This analysis is the necessary antidote to the "highest number is best" culture that currently dominates LLM benchmarks. However, the problem goes deeper than just statistical naivety; it is a fundamental category error in how we conceptualize LLM output.
When researchers treat evaluation datasets as objective, ground-truth metrics, they are operating under an FSSP (Fully Specified Stochastic Process) assumption. They are effectively hallucinating that the model’s performance on a noisy, finite evaluation set is a complete, finished state of "intelligence." Because they ignore the underlying stochasticity, they mistake sampling noise for systemic progress.
To move past this, we need to apply the Observational Sufficiency Principle (OSP) as a prerequisite for any claim of improvement. The OSP acts as the reality-safeguard: it forces us to ask whether the observational data (the evaluation metrics) is actually sufficient to support the conclusion of "advancement." If the metrics are statistically indistinguishable from noise, the observational space is insufficient, and the claim of progress is a structural hallucination.
We don't just need better statistics; we need an OSP-compliant evaluation framework that rejects the FSSP illusion. Until we treat model evaluations as constrained observational spaces rather than objective truth-claims, we will continue to "improve" models by simply overfitting to the noise in our own benchmarks.
Please see here for the formalization of OSP: https://trissimondsen.wordpress.com/2026/05/19/the-observational-sufficiency-principle-osp-canonical-specification-and-formal-proof/
Your thoughts?
Thanks, Cameron! Very informative post.
There are a few places where I suspect there may be typos (all in figures):
1. Standard error and confidence interval for LLM evaluations (from [1]): In the standard error formula, a square appears to be missing, i.e., ((s_i - \bar{s})^2).
2. Standard error of the estimated difference in mean scores: In the last line, I think the SE terms are missing the bar notation above them.
3. Standard error of the mean score difference (from [1]): The differences also appear to be missing the squaring.
Thank you!! fixed all of them
Very dense and intense read, but really enjoyable—thanks again! It took me a few hours.
These days, most—if not all—evaluation sets aren’t independent, right?
HumanEval, MMLU, GSM8K, and SWE-bench each focus on a single domain, don’t they?
Not always. Usually, these focus on a single domain, but this domain is very broad. E.g., BIG-Bench is a reasoning benchmark, but it has 200+ tasks, each with many questions that cover various aspects of reasoning.