GLSL & Sound

The AutoCorrelation Function

Autocorrelation is used to compare a signal with a time-delayed version of itself. If a signal is periodic, then the signal will be perfectly correlated with a version of itself if the time-delay is an integer number of periods. That fact, along with related experiments, has implicated autocorrelation as a potentially important part of signal processing in human hearing.

ACF is a simple method to visualize music that produces surprisingly good results. Perhaps the most unexpected property of ACF is that it accurately transfers the subjective “harmony level” from music to images. It’s almost an unreasonable property, if you think about it. Images below are ACF height maps in polar coordinates.

Female vocal David Parsons Piano Bird

More examples: soundshader.github.io/gallery (beware of large images).

Live demo: soundshader.github.io

Why ACF matches our perception of sound

Contrary to what you might think, our ears don’t seem to rely on an FFT-like process to extract isolated frequencies. Instead, our ears detect periodic parts in the signal, although in most cases those periodic parts closely match the FFT frequencies. There is a simple experiment that proves this point:

As can be clearly seen on the FFT image, the A signal is a pure sinusoidal tone, while B is a mix of tones. Despite each tone in B is higher than A, our ears perceive B as a lower tone. If we plot both waveforms, we’ll see that A has about 9 peaks in a 20 ms window, while B has only 5. The definition of “peak” is moot, but it doesn’t stop our ears from counting them and using the “number of peaks per second” as a proxy to the tone height.

ACF detects those peaks. ACF sees that there are 5 equally spaced time shifts where B[t] * B[t + shift] reaches the maximum, so on the ACF output we’ll see those 5 peaks.

Given that I’ve shamelessly stolen the experiment’s illustration above, I feel obligated to recommend the book where the illustration came from: Auditory Neuroscience.

One downside of ACF is that it drops the phase component of the input signal, and thus ACF is not reversible. This means that images that only render ACF, lose about 50% of the information from the sound and those 50% are important, e.g. dropping the phase from recorded speech makes that speech indiscernible. Real world sounds, such as voice, heavily use nuanced amplitude and phase modulation. ACF captures the former, but ignores the latter.

Visualizing ACF

ACF of a sound sample X[0..N-1] can be computed with two FFTs:

S = |FFT[X]|^2
ACF[X] = FFT[S]

And thus ACF contains exactly the same information as the spectral density S (the well known spectrogram).

If you’re familiar with the ACF definition, you’ll notice that I should’ve used the inverse FFT in the last step. There is no mistake. The inverse FFT can be computed as FFT[X*]*, where X* is complex conjugate, but since S[i] is real-valued (and positive, in fact), the conjugate has no effect on it, and since ACF is also real valued in this case, the second conjugate has no effect either.

ACF is a periodic and even function and so it can be naturally rendered in polar coordinates. In most cases, ACF has a very elaborate structure. Below are some examples, where red = ACF > 0 and blue = ACF < 0.

conventional music a bird song

Looking at the first example, we can tell that there are 5 prominent peaks in a 20 ms sound sample, which corresponds to 250 Hz. This means that our ears would necesserarily perceive this sound as a 250 Hz tone, regardless of what its spectrogram says. If it was a pure 250 Hz tone, we’d see perfectly round shapes of the r = cos(250Hz * t) line, but it’s not the case here: we see that the 5 peaks are modulated with small wavelets: there is one big wavelet in the middle (which consists of 3 smaller wavelets) and 4 smaller wavelets. Our ears would hear the big wavelet as the 2nd harmonic of the 250 Hz tone (i.e. it would be a 500 Hz tone with a smaller amplitude) and the 4 small wavelets as the 5th harmonic (1000 Hz) at barely discernible volume. In addition to that, the 500 Hz harmonic is also modulated by the 3 tiny wavelets, which means we’d hear a 1500 Hz tone, almost inaudible. We can say all this without even looking at the spectrogram or hearing the sound.

How I came up with this idea

Music is a temporal ornament. There are many types of ornaments, e.g. the 17 types of wallpaper tesselations, but few of them look like music. However there is one particular type of ornament that resembles music a lot - I mean those “mandala” images. I don’t know how and why those are produced, but I noticed a connection between those images and music:

Putting these observations together we naturally arrive with the ACF idea.

Questions?

Open an issue on github or shoot me a email at ssgh@aikh.org

License

AGPLv3