Results of Roberto's 32Kbps multi-codec public listening test

August 3, 2004

Roberto has finally released the results of its public 32kbps multi-audio-codec
listening test. Nero's AAC-Plus
won with MP3Pro closely behind.
In the next place came WMA, Qdesign, Real and Lowpass all closely
tied with Vorbis slightly further behind.
Lame came in last place.

It looks like SBR plays a big role in helping compress the high end
frequencies as MP3Pro and AAC-Plus both use this technology. However, as both LAME and Vorbis use
22kHz instead of 44.1kHz sampling rate at this low bitrate as well as low pass
filters, this may explain why these two performed rather poorly.

Overall Ratings: The results for each sample were
grouped together, without modifications.

Then I performed a
Tukey Parametric HSD analysis. The results are graphed below. Nero Digital
Audio wins, tied to CodingTechnologies' MP3pro. Ogg Vorbis, WMA Std., 7kHz
lowpass, Real Audio and QDesign Music Codec come in second place, with
Vorbis a little below the others. Lame loses.

There are some impressive findings in
this test. For starters, how well codecs evolved since mid-2000, when EBU
conduced their acclaimed MUSHRA formal
listening test. You can compare the results of this test with that one
using the 7kHz lowpass. Back then, the anchor clearly won over all other
codecs. This time, it often lost to most of them in each individual
sample.

It's also interesting to notice what a
wonderful technology SBR is, since it managed to bring the deprecated MP3
format to compare well against the state-of -the-art in audio
coding.

QDesign also was a pleasant surprise.
Considering it's an encoder that hasn't been developed since mid-1999, it
managed to compete very well against modern encoders

Although Vorbis didn't perform well,
almost getting under the big tie at second place, part of it can be
credited to the fact that it was being tested at 22kHz. The severe lowpass
at this bitrate seems to have introduced serious quality issues in the
samples tested. One can hope Xiph enables 32kHz sampling for 32kbps, and
further tunes such small bitrates.

Important note:
These plots represent group preferences (for the particular group of
people who participated in the test). Individual preferences will vary
somewhat. The best codec for a person is dependent on his own preferences
and the type of music he/she prefers.

Some other
important notes:

The built-in resampler in Oggenc wasn't
used. It has been brought to the test conducer's attention that it doesn't
work optimally compared to other resampling solutions (E.G, SSRC, used in
this test).
Thanks to dev0 for pointing this issue out.

The QDesign encoder used in this test is
the demo Pro encoder available at ReallyRareWares
from 1999.

It's worth reminding that Lame is the
bottom anchor in this test, and the Lowpass at 7kHz is the top
anchor

Overall Ratings of 32Kbps Public Listening
Test

See the full
results here.

While most users prefer high bitrates for their portable players, low
bitrates are still widely used to allow streaming to modem users, will be used
in upcoming mobile phone devices, saving bandwidth on websites and over-the-air
digital broadcasts. OD2 is an
example which uses WMA at 32Kbps when a user chooses to sample or stream
music.

Digital Radio
Mondiale (DRM, not to be confused with Digital Rights
Management) is a digital radio broadcast technology that uses AAC+SBR (Same
as Nero's AAC-Plus) to distribute near FM stereo
quality sound over shortwave radio.
While DRM radios may be rather expensive at present, one problem they may
face is users avoiding them by falsely assuming the 'DRM' feature means 'Digital
Rights Management'.

Feel
free to discuss and find out more about audio codecs on our Audio
Forum.

Source: Roberto's Listening Tests