Small-model training · Judgment and abstention
Shiba: How Can a Small Model Learn a New Judgment?
Learn a new judgment. Know when to abstain.
Teach small models new text and visual judgment tasks while studying when they should say “I can’t tell.”
Research question
Growing out of Jevado’s text-judgment experiments, Shiba brings text and vision models into one research project. The current focus is what happens to a 0.6B model’s existing abilities when it learns a new judgment task, and how it should decide whether to answer a task it has not learned.
Approach
- Train lightweight add-ons for emotion classification and intent selection, then compare performance on fresh questions and unlearned tasks.
- Calibrate answer thresholds by task, recording both the share answered and the error rate among answers.
- Compare labeling needs across trained versions to understand the costs of learning a task and abstaining reliably.
Where it stands
Several rounds of task learning and abstention experiments are complete. The latest comparisons did not establish a consistent advantage for add-ons over the frozen model across unlearned tasks. Answer thresholds borrowed from one task also failed to transfer reliably to another. Work continues on task-specific calibration and handing off judgments; general reliability has not been established.