I switched to fftw – it’s a tiny library compared with openframeworks, with super-fast execution, plus it handles n-dimensional transforms.

The idea of jumping up an abstraction level to the methods used in facial recognition – intriguing. I’ll hafta look into that.

“…the identifier for a time segment is the difference between the highest and lowest amplitude.”

Another interesting idea, I’ll add it to my test case list. First, I gotta get my code working under fftw.

Another cool thing about fftw, it does runtime precompilation of an analysis – the initial compilation takes hundreds of millis (depending on CPU), and then the actual FFT transforms on live samples take microseconds after that.

I’m gonna keep chipping away at this til I got some answers… just curious.

I love the idea of identifying locations in a song via a frequency table mapped to some window.

Though, I think that the endpoint of this line of dev is to search for an arbitrary acoustic fingerprint within a song and call the location wherever it’s found the timecode.

The concept would be: here’s an acoustic fingerprint for a five second snippet, including patterns in frequency and amplitude and anything else you can find.

One way to identify frequency and amplitude patterns is to treat the acoustic data as if it were a face and use face recognition algorithms, e.g. principle components analysis.

A simpler method is to measure variance in some dimension over a fixed time window. So say that the identifier for a time segment is the difference between the highest and lowest amplitude.

It’s fun hacking. Makes me wish, yet again, that I did comp sci grad school.

Maybe this is an all-around silly idea that those with more digital audio experience would get right away, but I do enjoy experimenting.

http://www.openframeworks.cc/forum/viewtopic.php?f=10&t=2184&view=unread

Quite possibly worth a quick hack, just to know if there’s anything to this!

It seems that existing fingerprinting methods split between identifying an audio source as a whole, and just an audio fragment. Gonna look deeper into this…

To recollect some of this, the binding of lyrics and other metadata to timed points in a song is complicated by encoding and editing changes to a track. There is no one-dimensional identifier that isn’t subject to morphing when a track is ripped, re-encoded or otherwise messed about with.

Discarding timecode as a bind-point, the next thing that comes to mind is the fingerprinting of concurrent frequencies within a frame or time period. Frequency analysis is something that players already do, for equalization and visualization and other things.

Say there was a way to abstract the frequencies in a given frame or snippet of audio, such that we could reduce a 0.1 second sound slice to a few numbers:

[881.2hz, 220hz, 4150hz, 338hz]

Adding in the amplitude of each frequency as a percent of the source medium’s dynamic range:

[881.2hz@25%, 220hz@40%, 4150hz50%, 338hz@76%]

Would that be enough information to provide a unique location fingerprint for a point in a song? Only experimentation could tell just how unique these characterizations are. I know of libraries and applications which provide frequency analysis methods (e.g. Squeak.)

Say that frequency spans 0Hz-16KHz, a range that fits nicely into 14 bits. Amplitude, expressed in decibels or % of max, another 6 bits. So, three bytes and change to describe a given frequency sample.

With a pipeline of frequency@amplitude sets, a plugin would need lyrics to be indexed by unique set, so it would make sense to sort the freq@amp values in each set coming down the pipe:

[220hz, 338hz, 881hz, 4150hz]
[105hz, 240hz, 881hz, 2300hz, 4150hz]
[95hz, 262hz, 881hz, 2300hz, 4150hz]
…

The first thing to do is experiment, reduce test tracks to streams of freq@amp sets, and evaluate whether uniqueness exists.

Just some ideas… Hey, I know a developer who toys with digital audio for a living, he introduced me to Squeak. Gonna have a word with that boy and see what he knows.