Statistical measurement · Exploratory evaluation and study design
LLM Text Measurement: How Errors Change Conclusions
From labels to proportions, trends and uncertainty.
Compare human labels, LLM labels and correction through limited human audits, tracing classification error into proportions, group differences, trends and intervals.
Research question
A close overall proportion can hide very different errors across groups. This study asks how changing the labeling method changes downstream statistical conclusions, and how much information a limited human-review budget can recover.
Approach
- Separate overall and group-level errors to uncover mistakes that cancel each other out.
- Calculate correction risk and conservative interval performance exactly on a fixed 500-item exploratory set, auditing 50 items in each of two groups.
- Prepare independent material and comparisons with existing estimators under the same budget, accounting for method selection and new labels.
Where it stands
The exploratory evaluation is complete. On this fixed population, conservative intervals had high coverage but an average width of 76.19 percentage points, showing the gap between coverage and useful information. This does not rule out other methods under the same budget. The next study design has converged, but the formal panel, new predictions, human labels and main simulation have not begun.