Log in

View Full Version : Encoder Blindtest


Gleitz
10th December 2004, 16:58
Up to the present moment encoder tests suffered from the fact that you had to rely on the subjective judgement of the person conducting those tests. The interested reader had no influence on the selection of the chosen scenes. So you could always ask: was this selection always done objectively?
Moreover, people perceive and judge details indivdually, e.g. certain influences are considered disturbing by some viewers whereas others don't even take notice of them.
That's why we decided to do a blind test and leave the visual judgement up to you, the users. Blind means that you watch & vote without knowing the encoder used for a certain clip.

For this session, we took some lowcost encoders and tried to find optimal settings for our test video clip. The encoded files can be downloaded and judged.

Our test clip is a PAL DV-Avi file saved using the Canopus dvsd DVC/DV Video Codec. The clip consists of 5 parts with different demands for the encoder.


Test settings
Bitrate: min. 3.000 kbps, avg. 6.000 kbps and max. bitrate 9.000 kbps.
2-pass VBR
Scene Change: on

The intention of using that bitrate is to fit a 99min DV avi on a DVD. At the same time we want to check if the encoder stays within the borders of the minimal and maximum bitrates set.
2-pass VBR was set as a least common denominator of the encoders used in the test even though some participants also support other possibilites.
Scene Change was activated to check whether the encoders correctly identify scene changes and set I-frames accordingly.
Another requirement was the flawless acception of the clips in professional authoring tools.


All other settings were set to achieve the best visual result. The exact settings for each participant will be published after the evaluation period is over since these settings shall not have an influence on your evaluation.


Watch & vote!
The visual evaluation will be done by you!
Download the test clips, put them on a PAL DVD (RW) and watch them on your TV.
When creating the DVD and TMPG DVD Author (TDA) reports an error you have to use a newer TDA version since the files aren't faulty.
Ulead shouldn't be used for authoring since it corrupts the files.
The current TDA version (1.6) and the current DVDLabPro are fine for authoring (but not older TDA versions and neither DVDLab Standard).

You can cast your vote on a provided form (see below). Feel free to post your opinion and your comments, too.

When enough votes and comments have been collected, the test will be terminated and we'll reveal the encoders (and their settings) used for the different clips. In a final review we'll cover the topic from a technical point of view.

The test clips are provided via the P2P file sharing tool BitTorrent. We can't offer a download via http or ftp since we can't afford paying the required bandwidth. eMule and other P2P tools don't allow us to control which files are shared through our Server (http://gleitz.info:6969) and thus aren't an alternative. BitTorrent is also the most effective way for such a kind of distribution.

You'll need a BitTorrent client to download the files. Popular clients are ABC [Yet Another Bittorrent Client] (http://pingpong-abc.sourceforge.net/) and Azureus (http://azureus.sourceforge.net/). You can restrict down- and upload speeds in both tools if you want to surf the internet while downloading.

Important notice
Please keep the files shared for a while after downloading - that's the whole point of a Peer2Peer network or else it won't work. Please wait until the files have been uploaded at least once or better twice. The first downloaders will possibly experience delays until the files have reached a decent distribution level.

If this first test of its kind fulfills our expectations more tests will certainly follow.

Credits
Special thanks go out to mb1 for his support. He has cut the test sources and furthermore created two files for the test on his own. Furthermore, the german DVD-SVCD-Forum (http://www.dvd-svcd-forum.de) deserves credit since its members provided the scenes for the test clip and supported the project in general.
Another "thank you" goes out to my team members that supported me actively in the realization.

And of course I also want to thank you - the hopefully numerous testers/voters - for your efforts!

-------------
Vote now (http://www.gleitz.org/index.php?id=94)

ronnylov
14th December 2004, 07:32
I have now downloaded the samples. Is it possible to download the source file too (probably in DV format)? It is interesting to have an original to compare with before I decide on how to vote.

wmansir
14th December 2004, 22:29
Do we get to know which encoders are being used?

If it is CCE, ProCoder and/or QueEnc I'll put a post in the DVD-RB forum to make people aware of this, since those encoders are all compatible with ReBuilder.

Gleitz
14th December 2004, 23:40
@ ronnylov
I does not have any rights at the source file. :(

@ wmansir
It is current only one Encoder in this case that can be used with DVD ReBuilder.
Even further tests will follow. Whether the QuEnc (avcodec.dll) is ever tested, I doubt. Maybe MEncoder (avcodec.so = avcodec.dll) with other Linux Encodern.

mb1
15th December 2004, 19:03
Is it possible to download the source file too (probably in DV format)? It is interesting to have an original to compare with before I decide on how to vote.

The source DV file is 255 MB (around 206 MB 7zipped). At the moment there is no possibility to download the source due to space and bandwidth resources.
If you have that amount of space available I`ll upload it. Drop a mail to mb_1(at)gmx.net

For a vote on quality about the seen clips there is no need for having the source material, too.

If this test will be succesful we plan to further test hardware, high priced software and freeware encoders to get a good quality basis and opinions for new encoders.

After this first test of course we will name the used encoders (and the settings).

ronnylov
15th December 2004, 23:05
It may not be necessary to have the source file to decide which clip is most appealing but to judge the quality of the encoder it is interesting to see how big the differencies are from the original (like colour differencies and artifacts that may be present already in the original). I could also back up my decision with PSNR and SSIM measurements and subtract the original from the encoded video to highlight differencies. But then it might not be as subjective any more.

I had to download over 212 MB encoded files and the source file is in similar sice. Can't you just use the same bittorrent tracker? I don't have the space for upload

Gleitz
31st December 2004, 13:17
I offer the Download of the test files for the next days over http.

Testfile 01 (http://gleitz.org/fileadmin/test01_file01.zip)
Testfile 02 (http://gleitz.org/fileadmin/test01_file02.zip)
Testfile 03 (http://gleitz.org/fileadmin/test01_file03.zip)
Testfile 04 (http://gleitz.org/fileadmin/test01_file04.zip)

ronnylov
12th January 2005, 14:01
When can we expect the results from the blindtest?
I have made my vote and want to see what the average opinion was and also some information of which encoder belonged to each file (I think I can guess).

Gleitz
12th January 2005, 22:15
This weekend we finish the first test. The experiences with the test were helpful for further tests which are already in the planning.

ronnylov
28th January 2005, 16:32
I have noticed that the results are available.
Click on the Vote Now link in the first post and then "Encodertest Teil 1" link. Very interesting results and a really nice work you have done!