Multimodal analysis · Small exploratory study
Reading Emotion: Image, Sound and Subtitles
Observe the signals. Question the interpretation.
Compare movement, facial expression, audio and subtitles to study how models move from observable signals to interpretations, and how text changes their reading of an image.
Research question
Body movement can be measured, but a pose does not directly reveal intent. Using short interview clips, I combine movement, facial, audio and text signals to trace what each interpretation is actually based on.
Approach
- Combine MediaPipe, Py-Feat, Whisper and emotion2vec signals across four 30-second clips.
- Improve movement measurements with shoulder-width normalization and inspect direction and change frame by frame.
- Compare subtitles with visual signals to see whether text leads the model to reinterpret body movement.
Where it stands
Three rounds of analysis and review are complete. Interpretations of intent and subtext were driven mainly by subtitles. Movement thresholds and scores have not been calibrated by human reviewers; the results cannot establish a person’s actual mental state or general emotion-recognition accuracy.
Interview footage belongs to the original program. Shown here as signal-analysis figures from personal research.