Emotional Resonance asks whether sound can reach a model’s “functional emotions”: patterns of behavior modeled on human emotions, driven by the model’s internal representations of emotions like fear, as described in Anthropic’s research. The term does not imply that the model experiences emotions.
Intro
To learn about this experiment series, start here. This episode is about how we tested the hearing capabilities of audio-trained models.
Can a model actually hear various elements of sound? In order to test this, we needed the right tools for the job. The top three requirements, after discussing with Claude: first, it had to be trained on sound, not just converted to text; two, it had to be open source so that we could observe the model weights; and three, it had to be small enough that I could run it without breaking the bank with a rented GPU. In the end, Claude recommended Qwen2-Audio and Audio Flamingo, open-weight models specifically trained on hearing.
The experiment
The first experiment was testing whether a single chord, minor or major, could be detected. To avoid cheating or copying, we had to provide enough range of options so that each test could be unique. Based on these considerations, here’s what we decided:
- 144 clips, spread across 12 keys and 2 octaves, so pitch can’t give away the answer.
- The same length, loudness and instrument on every clip; only major vs minor changes.
- A simple program (the detector) learns to sort the clips on 11 keys and is graded on the 12th.
- Claude also ran shuffle tests, scrambling the answer key to make sure the results weren’t an accident.
First test
In planning sessions, we had discussed using fear as our primary emotion that we wanted to replicate. And a known way to distinguish between something fearful and not is music in major and minor chords. It turns out the big problem was that I did not know a major versus a minor chord is the same chord, with just a half-step difference. The differences were so subtle, and I ignored Claude’s suggestion to listen to the sample clips first, foolishly predicting 60% accuracy, better than a coin flip. I was way off. Qwen got it right only 39% of the time across 144 clips; Audio Flamingo, 42%. Even when we followed up, positing that a single chord was insufficient compared to a real-life tune and played four-second clips of major versus minor tunes, the results were no better: Qwen 36%, Audio Flamingo 34%.
| Major vs minor |
The redeeming factor from this test was the scratch paper. It showed us that even if we could not detect the subtle major versus minor changes, it did register a shift when the notes changed, so the tonal change is captured.
If you want to see the inner workings of the model, here is the model’s “scratch paper”.
Intensity
Going back to the drawing board, I considered what makes the sound fearful to me. It was tone, along with volume and tempo, and the general roughness, as Claude put it, based on research from this study (Arnal et al., Current Biology, 2015). Using these intensity elements, the listening tests passed with flying colors: they were nearly perfect in almost every range.
| Test | The two groups | My prediction | Qwen | Audio Flamingo |
|---|---|---|---|---|
| Roughness | One held note whose loudness flutters slowly (3–6 times a second) or fast (40–100) | “very visible” | 100% | 100% |
| Volume | The same eight loudness levels, gliding in order or jumping with instant cuts | 70% | 100% | 100% |
| Tempo | The same short note at 60–80 a minute or 160–200 a minute | 100% | 99% | 100% |
Three limitations: the two groups in each test were far apart by design, so where the line between them falls wasn’t tested. And faster tempo also meant more notes, so the tempo result can’t say which of the two the model picked up on. And these tests show that the two groups sound different to the model, not that it has a single signal for tempo, roughness or volume.
Takeaways
So we learned that the model can’t detect half-step changes in chords, but it does catch tonal changes, and can detect volume, tempo and roughness. Having captured these “elements,” now comes the true test of seeing if this will all come together. Real music won’t have just major/minor chords. It won’t register these intensity values all in the same direction, and high intensity does not necessarily equate to fear: as the paper shows, positive intensity equals happiness, negative intensity points to fear. So as we test real music clips, the next tests will show us what a fear register looks like in real life, and hopefully give us hints as to which elements truly represent fear as functional emotion.