KpeX
12th September 2003, 22:38
Comparing codecs, codec features, and filters has always been a hot topic here at Doom9. I occasionally do some reading over at Hydrogenaudio, where ABX is the most preferred blind testing method for audio. ABX is a double blind comparison method in which a you try to determine through playback if an unknown source X is known source A or B. Although audio artifacts are (IMO) usually much more subtle than video artifacts I always wondered if ABX tests would be effective for video. So, I put together a batch script that does the process for any two AviSynth scripts. Therefore any two video sources that can be opened by AviSynth can be compared, which is pretty versatile :).
Download here: (right click - save target as)
www.freewebs.com/kpex/VidABX_v0.1.zip
Usage:
The script uses Gabest's excellent Media Player Classic for CLI playback, the mplayerc.exe file must be either in the path or in the same directory as the batch file.
The batch file should be called from command line with two AviSynth scripts to be compared as arguments, i.e. "VidABX.bat "path\script1.avs" "path\script2.avs"". You will then be presented with a menu that looks like this:
=========================================
= VideoABX
= 1. Play Video A "c:\dvd\encode.avs"
= 2. Play Video B "c:\dvd\movie.avs"
= 3. Play Unknown Video X
= 4. Make Current Trial Guess
= 5. End and log results
=========================================
Make your selection:
You can then choose 1-5 as shown, as you can see A and B are the known video sources, the idea is to watch the known A and B and compare them to the unknown X (you can watch any of the clips as many times as you want). When you think you know if X is A or B, use choice 4 and the script will prompt you for your guess. You can go through as many trials as you want, and when you're done there will be a log file in the same directory that looks something like this:
Video ABX Log 17:20:31.72 Fri 09/12/2003
===========================
1st Source Video AVS Script: "c:\dvd\encode.avs"
LoadPlugin("c:\PROGRA~1\AVISYN~1.5\plugins\MPEG2D~2\mpeg2dec3.dll")
MPEG2Source("C:\DVD\dvd.d2v")
Crop(0,60,-0,-64)
BicubicResize(608,256)
trim(40001,41000)-
End of 1st Source AVS Script
====================================
2nd Source Video AVS Script: "c:\dvd\movie.avs"
avisource("movie.avi").trim(40001,41000)-
End of 2nd Source AVS Script
====================================
ABX Results
====================================
Trial 1: Correct
Trial 2: Correct
Trial 3: Correct
Trial 4: Correct
===============================
Overall Data
Total Trials: 4
Correct Guesses: 4
Incorrect Guesses: 0
Percent Correct: 100
===============================
End of Logging: 17:22:52.16 Fri 09/12/2003
Requirements/Limitations:
1. Only works under win 2k or xp.
2. As noted above, mplayerc.exe must be in the path or current directory.
3. When using Xvid avi's with Bframes as one of the sources against any other source, make sure to trim at least the first frame, otherwise you'll get a bframe decoder error message at the start of playback, which kinda defeats the blindness ;).
4. In my testing so far it doesn't work so well for comparing formats, i.e. I can see the difference between MPEG2 artifacts and MPEG4 artifacts, but that doesn't necessarily mean one is higher quality.
(edit) 5. Forgot to mention: comparing AVS scripts will only be effective if the filters/functions are simple enough and your processor is fast enough to render the clip in real time. This usually isn't a problem for me because I don't like to use many filters especially on DVD source. For scripts that are more advanced you may have to save a temporary lossless avi, which may not be perfect either, and also doesn't work well for comparing long clips.
As I mentioned above, I started this mainly as an experiment, so let me know what you think: Is this a valid/useful video comparison method or not? Suggestions and bugs are welcome too. Enjoy,
Download here: (right click - save target as)
www.freewebs.com/kpex/VidABX_v0.1.zip
Usage:
The script uses Gabest's excellent Media Player Classic for CLI playback, the mplayerc.exe file must be either in the path or in the same directory as the batch file.
The batch file should be called from command line with two AviSynth scripts to be compared as arguments, i.e. "VidABX.bat "path\script1.avs" "path\script2.avs"". You will then be presented with a menu that looks like this:
=========================================
= VideoABX
= 1. Play Video A "c:\dvd\encode.avs"
= 2. Play Video B "c:\dvd\movie.avs"
= 3. Play Unknown Video X
= 4. Make Current Trial Guess
= 5. End and log results
=========================================
Make your selection:
You can then choose 1-5 as shown, as you can see A and B are the known video sources, the idea is to watch the known A and B and compare them to the unknown X (you can watch any of the clips as many times as you want). When you think you know if X is A or B, use choice 4 and the script will prompt you for your guess. You can go through as many trials as you want, and when you're done there will be a log file in the same directory that looks something like this:
Video ABX Log 17:20:31.72 Fri 09/12/2003
===========================
1st Source Video AVS Script: "c:\dvd\encode.avs"
LoadPlugin("c:\PROGRA~1\AVISYN~1.5\plugins\MPEG2D~2\mpeg2dec3.dll")
MPEG2Source("C:\DVD\dvd.d2v")
Crop(0,60,-0,-64)
BicubicResize(608,256)
trim(40001,41000)-
End of 1st Source AVS Script
====================================
2nd Source Video AVS Script: "c:\dvd\movie.avs"
avisource("movie.avi").trim(40001,41000)-
End of 2nd Source AVS Script
====================================
ABX Results
====================================
Trial 1: Correct
Trial 2: Correct
Trial 3: Correct
Trial 4: Correct
===============================
Overall Data
Total Trials: 4
Correct Guesses: 4
Incorrect Guesses: 0
Percent Correct: 100
===============================
End of Logging: 17:22:52.16 Fri 09/12/2003
Requirements/Limitations:
1. Only works under win 2k or xp.
2. As noted above, mplayerc.exe must be in the path or current directory.
3. When using Xvid avi's with Bframes as one of the sources against any other source, make sure to trim at least the first frame, otherwise you'll get a bframe decoder error message at the start of playback, which kinda defeats the blindness ;).
4. In my testing so far it doesn't work so well for comparing formats, i.e. I can see the difference between MPEG2 artifacts and MPEG4 artifacts, but that doesn't necessarily mean one is higher quality.
(edit) 5. Forgot to mention: comparing AVS scripts will only be effective if the filters/functions are simple enough and your processor is fast enough to render the clip in real time. This usually isn't a problem for me because I don't like to use many filters especially on DVD source. For scripts that are more advanced you may have to save a temporary lossless avi, which may not be perfect either, and also doesn't work well for comparing long clips.
As I mentioned above, I started this mainly as an experiment, so let me know what you think: Is this a valid/useful video comparison method or not? Suggestions and bugs are welcome too. Enjoy,