Beyond Chorus / audio-sync proof of concept
A song is not
a timestamp.
I built a proof of concept that matches concert audio to a position in a studio recording, using frozen audio embeddings and evidence across windows. The hard part was keeping a repeated chorus from moving the lyrics to the wrong place.
The band didn’t bring
the studio recording.
I built Beyond Chorus because I wanted the words in front of me while listening to music, in the car or on smart glasses. I’ve listened to metal for years. Understanding the vocals is a separate engineering problem.
The app could recognize recordings, but the concert recordings I wanted to follow were where Shazam and ACRCloud fingerprint recognition let me down. I started experimenting with neural audio representations to see if I could follow those too.
A live performance has a different sound, room, audience and pace. Even when I know the song, the lyrics can still be in the wrong place. A convincing match to the first chorus is no help during the second.
That became the question behind 40–45 days in September–October 2026: could I understand the machinery well enough to build a better way of following the performance?
I needed to understand
what I was asking AI to build.
Before I’d finished Zero to Hero, I had already tried building this with AI-generated code in an earlier project. In my own experiments, none of those versions matched what I later built by hand in Jupyter.
I worked through Andrej Karpathy’s Neural Networks: Zero to Hero, in notebooks. I wanted to understand what the layers, gradients and training loop were doing. Reading about it wasn’t enough; I needed something I could break.
- 01
Build the intuition
Bigram models, MLPs, initialization and BatchNorm. Then manual backpropagation: make the gradients work before trusting autograd.
- 02
Follow the sequence
WaveNet-style hierarchies, CNNs, causal attention and byte-pair tokenization. Small models I could take apart and inspect.
- 03
Give it a real problem
Compare pretrained audio representations. Find a song in a library, then locate the live performance within its studio reference.
- 04
Learn from the wrong answer
Repeated sections, drifting positions, uncertain matches. Keep evidence across windows instead of letting each new match move the lyrics.
Writing the experiments myself changed the work. I could inspect the representations, change one assumption, and understand why a result got better or worse. Once I had a method I understood, I used AI to run hyperparameter sweeps and explore variations. I knew what to ask it to test, and how to judge the answer.
Those small language models were how I learned. The applied audio work uses pretrained encoders, including comparisons with MERT, MuQ and PupuM2D. I kept the encoders frozen and wrote the retrieval, alignment and tracking experiments around their outputs.
Find the song.
Then stay with it.
Averaging an audio representation is useful for finding a likely song. It throws away the sequence you need to find a position. I kept short segments for alignment, then compared incoming windows against the studio reference.
Center
Subtract the shared library mean so common audio characteristics don’t dominate the match.
Align
Compare ordered segments to candidate positions, keeping the timing information.
Accumulate
Shift prior evidence forward, let it decay, then add the next window’s scores.
Decide
Use a score threshold and a margin for jumps. The score isn’t a calibrated probability; a strong match can still be the wrong chorus.
The tracker became as interesting as the encoder. Each new window is a piece of evidence, not permission to forget everything the previous windows said.
Recognizing the right song can still give you the wrong lyrics.
Timing-grid diagnostic
Twelve frames at 25 Hz give me one 0.48-second segment. I retain twenty segments from each ten-second reference window: 9.6 seconds of features. A flat segment index therefore isn’t quite a clock. My conversion keeps the window boundaries.
| Index | Time (seconds) |
|---|---|
| 19 | 9.12 |
| 20 | 10.00 |
| 21 | 10.48 |
Six segments can advance 2.88 seconds within a window or 3.28 across its boundary. That matters when shifting old evidence forward. This is timing-grid arithmetic, not measured concert performance.
The wrong turns
are part of the work.
- A smaller search can protect a wrong answer.
- Restricting the next match to a radius around the current position looked sensible. Once the tracker chose the wrong repeated section, it could stay trapped there.
- More frequent guesses didn’t automatically help.
- I compared different window lengths and scan intervals. Faster scans did not meaningfully improve the sampled shows. More activity was not the same as more useful evidence.
- A smooth result is not ground truth.
- Accumulating evidence reduced large position jumps in my five-show offline comparison. I still had to look at whether the tracker was following the right repeated section.
- The product has to know when it doesn’t know.
- Confidence and recovery matter as much as a good match. When the signal is uncertain, keeping the lyrics steady can be more useful than making another confident-looking jump.
This is a working audio-sync proof of concept being productized for Beyond Chorus. The offline comparison used fitted reference timings and recordings also used for tuning. Fewer jumps don’t establish that the displayed verse was correct.
Next: a performance I haven’t tuned against.
I need independently marked positions on unseen performances. Then I can check whether the tracker follows the right repeated section, loses its place, and recovers. That’s the next test before I put this into Beyond Chorus.
mert@mertyentur.com