Forum member Turing has made an interesting post in the Club CD Freaks Nero SDK Forum. Turing has been working on a project to measure the quality of an audio codec with a computer. He uses a method in which he compresses the original with a certain codec, convert it back to a WAVE file (uncompressed audio) and compares the output of the several codecs with the original file.
Although there are still various issues to solve, it would of course be great if someone would ever be able to create a reliable tool to compare codecs. It's still not finished by the study of Turing is certainly interesting and food for discussion. Due his experience Turing was able to conclude the following:
1. Use the Games-Howell test: The Tukey HSD test should not be used for either listening tests or mathematical based codec comparisons. This is due to the wide range of mean-squared error in different codec scores. The Games-Howellsignificance test offers the similar high degree of confidence and accounts for the difference in mean-squared error. 2. Compare Residual Scores: Statistical comparisons of scores should not be based on the raw scores from different audio clips. Even the Games-Howell test assumes that each codec has a constant mean. This assumption is not true because different clips produce different quality for both listening tests and mathematical comparisons. Much more discrimination can be inferred if the residual scores are compared (i.e. subtract the average score for each clip from the individual scores). Unless you think that all listeners are the same the same procedure should be applied in multi-listener tests on individual clips as well. 3. Tests results are volatile: Codec quality is rapidly changing with time. If you compare these results with the Amorim AAC tests the results are different. Nero won here, but iTunes was superior in the past. If you look at the history of codec comparisons the tests results have reversed many times. I do not have access to the older versions of Nero and iTunes in the Amorim AAC tests in order to confirm it, but I believe that the different result is not due to the method. |
More information with also some test results can be found in this thread on Club CD Freaks.com
Source: Club CDFreaks
