Comparing the Transcoders

[quantum]

A favorite topic in this forum is comparing video quality between the various transcoders such as DVD Shrink, IC7, DVD2One, Recode, etc. I see a lot of general comments such as "the quality looks good to me". It's not at all easy to compare different dvd sources if you don't have a good technique. I often wonder how others go about comparing frames from different sources. DVD software players are not well suited to this.

My technique is to use one of these two programs:

VirtualDub-MPEG2-AC3 (http://fcchandler.home.comcast.net/)

VirtualDubMod (http://virtualdubmod.sourceforge.net/)

I generally use VirtualDub-mpeg2. VirtualDubMod works but is slightly less handy at playing in real time imo. However VirtualDubMod has a nice feature to save captures directly to a BMP. This takes extra steps in VirtualDub-mpeg2. Use either one or both.

To compare frames, open two instances of the program, and drag a VOB from each transoding job to a preview window. Do right mouse->2x size to zoom in for both instances. You can then use the slider to navigate between frames, or jump directly to a frame (ctrl+g). You can alt-tab between the two instances to compare frames. Simple and highly effective.

Some helpful keystrokes and mousestrokes while comparing:
========================================================
(left or right)Arrow = forward/back 1 frame
ALT+(left or right)Arrow = forward/back 50 frames
Ctrl+G = Jump to a frame
Ctrl+Left = First Frame
Ctrl+Right = Last Frame
spacebar = play/stop video from current position. This works best with Vdub-mpeg2. Vdubmod always seems to play from the beginning.

* Rightmouse (in video area) ->2x size (best for looking for compression artifacts)

* You can see a full screen at 2x if you have a desktop resolution of at least 1280x1024. You don't really need to see the whole frame to compare quality. I usually stay at 1024x768.

* It's best to have two instances maximized, then Alt-Tab between them.

* When the video is importing, you can abort at any time and you'll have a portion of the video up to the abort point

Why not use WinDVD, PowerDVD, or ZoomPlayer?
===========================================
WinDVD and PowerDVD will not allow multiple instances to be opened. Thus you cannot simultaneously compare the transcoded version to the original. Attempting to use one of these means you capture frames from random parts in the video, then open another instance, and attempt to duplicate this effort. Then open the frames in a viewer and compare. Not easy or exact.

ZoomPlayer does allow multiple instances, but the navigation is not as efficient, and you run into possible problems such as color differences with overlay windows. ZoomPlayer used to be my favorite comparison technique before I moved on to Virtualdub.

It's not enough to just randomly pick frames and take screen captures. Sometimes 90% of the frames may be identical, but the remaining 10% may have big differences. You need to be able to walk through both videos and simultaneously compare the "hard parts". This is easy with Virtualdub.

[Yannis]

I am not sure this is different from using, for example, PowerDVD to save frames as BMP's. There have been many threads here debating on the quality of various xyz-coders and in most of them posters talk about comparing stills saved as pictures.

In any case, some other comments that comparing stills is inadequate is totally correct, as motion quality cannot be evaluated this way.

At a third level, some other people say the quality is good for them; this is a diplomatic statement :D :D :D

[quantum]

It is very different from using PowerDVD. For one thing it's possible to interactively compare the two sources simultaneously. Something you can't do in PowerDVD. You would have to try it to see how much more efficient it is.

[kilg0r3]

I would suggest the psnr measurement function compare in avisynth. In order to do this you will have to create a project file with dvd2avi for each of your dvds and the load it via avisynth into a software player.

for more info search the xvid forum for "compare ()"

[JFerguson]

Originally posted by quantum
To compare frames, open two instances of the program, and drag a VOB from each transoding job to a preview window.

quantum, I'm giving this a try -- need help, though.

I opened up two instances. I dragged the VOBs into each instance.

When I try to preview (SPACE BAR or File | Preview input... ??), I get "No audio decompressor could be found to decompress the source audio format."

I guess I need a codec...?


p.s.: Update - I went to the Audio menu and selected "No audio", so now it will let me play the video. Still wondering what I need here, now that I got it installed, though. Thanks...

[quantum]

I don't know what the problem could be. It's always worked for me with audio without doing anything special. I wouldn't worry about it. In fact I'd recommend you leave it set to audio->no audio which is probably faster.

Update: Strange, I'm getting the same error now. Not sure what the pattern is, but I'm doing the audio->no audio thing also.

[jdobbs]

I agree with kilg0r3. Use the PSNR to settle the question. It isn't perfect -- but it is the only objective measure. But you have to make sure:

1. The exact same clip is used for comparisons.
2. The exact same compression ratio is used (final bitrate/size).
3. The exact same audio tracks are kept.
4. You compress it enough to show a difference.

Numbers don't lie. They don't have opinions. They don't change based upon the lighting. They don't have an agenda or really care who wrote the software. They also don't change their outcome based upon the number of beers they consume.

[quantum]

I agree with kilg0r3. Use the PSNR to settle the question. It isn't perfect -- but it is the only objective measureI didn't think there was a question to be settled. My point was to offer a relatively easy yet effective way to compare any two sources.

However with that said, the PSNR feature in Avisynth is interesting. I did the movie Traffic and ran about 50,000 frames through from Recode, IC7, and Shrink (using deep analysis), compared against the original of course.


REMOVED - Originally I had posted PSNR results here.
Further testing and visual frame comparisons
have demonstrated to me these numbers have little
relationship with obvious visual quality - results
removed since they seem useless and in fact can give
higher scores to bad frames and weaker transcoders.According to what little I've been able to figure out about PSNR, the higher the average, the better, so this shows Recode slightly in front with IC7 and Shrink close behind. It's interesting to see how close Recode and Shrink are in the minimum, maximum, and deviation columns.

From my own visual comparisons, I have noticed Recode and Shrink often have better frames, but then get very blocky in some areas, while IC7 may be slightly worse, but less blocky in those problem areas. This may be indicated in IC7's lower Mean Deviation.

[k2]

Numbers don't lie. They don't have opinions. They don't change based upon the lighting. They don't have an agenda or really care who wrote the software. They also don't change their outcome based upon the number of beers they consume.

Yes and No

I need to know the formula of PSNR before I decide if it is useful.

For example Deviation is a word I well understand, you do not want any if the case of comparison.

[quantum]

I'm not too sure about those numbers myself. I just ran a DVD2One project and it supposedly beat Shrink with 42.3207. I'm going to redo the whole thing making sure about the sizes and post the results in a new thread. I'm seeing more and more indication of strong similarity between Shrink and Recode. For instance the max value of 103 indicates a frame the transcoder did not touch. IC7 and DVD2One don't have any of these, while Shrink and Recode have several, and many are exactly the same frames. Hard to believe it's a coincidence.

[Zhnujm]

i know one thing for sure , if dvd2one beats instantcopy, this psnr value has nothing to do with quality.
also dvd2one leaves frames completely uncompressed, at least in variable ratio, maybe that changes the results.
dont know about the other transcoders but maybe its similar.
i try for myself if i get some time.

[quantum]

It didn't beat IC, it beat Shrink. I used the new constant setting in DVD2One. Again I'm going to redo the whole thing.

[mrbass]

Originally posted by Zhnujm
dvd2one leaves frames completely uncompressed, at least in variable ratio, maybe that changes the results.

so did dvdshrink 1.03 'thus the blink factor' If you blink you'll miss the macro blocks..lol

[jdobbs]

You have to be careful about average PSNR. I like to output to a file and create a line graph. For example, I've found that DVD2one can do pretty well in the average -- but when you look at it in a graph you'll see that it swings wildly in both directions. It makes for a good average but some of the frame differences found were huge. Usually it is a good thing when you don't see a large range between the minimum and maximum PSNR

[quantum]

I'm not confident I was careful enough in this first test with the sizes. I did a more careful and thorough test here (http://forum.doom9.org/showthread.php?s=&threadid=60006)

[jdobbs]

Interesting. I hadn't tried one of these since DVD2one included the "constant" mode.

[JFerguson]

Need more information on this PSNR analysis...

You use DVD2AVI to create .D2V file?

Then, what does the .AVS script look like for VirtualDub??

A little confused here...

[kheops]

answer of the last question is here

http://forum.doom9.org/showthread.php?s=&threadid=60006

+++

[monomer]

I have no idea what this PSNR thing is all about but I do know that in statistics the 'average' (or mean) is completely meaningless without a measurement of distibution (or variance).

Example: The Arizona desert may have the same average annual temperature as Hawaii but obviously one would be more comfortable in Hawaii. The desert just gets too darn hot in the day (especially during summer) and darn too cold at night (sometimes much below freezing). But by just knowing the average you'd probably pack the same clothes for both locations... a Big Mistake!

And 'still' captures by themselves can only tell part of the story. 'Stills' at best can only reveal certain artifacts (such as pixellation) but it does not show any motion artifacts. Example: Just last night I made a back-up copy of "Rosemary's Baby" using Shrink. In the scene where her husband is sitting on the bed talking to his agent on the phone, the features on his face are randomly 'floating' around... this is a motion artifact as his features do not float in the original. I re-did the movie with IC7 and in the same scene there are no floating noses and eyes and lips. How would you capture the floating facial features in a still or two? Would this PSNR be able to evaluate this type of motion artifact? I believe the best method for comparison of the various compression tools is to view the movies actually playing and side-by-side and if you cannot see any differences then for all intents and purposes there are none.

EDIT: I didn't mean to leave the impression that my set-up allows me to side-by-side compare movies... the best I can do is play one and quickly switch to the other in the same scene but obviously having them side-by-side in-sync would be better.

[MackemX]

the best transcoder for yourself is the one you find most pleasing on the eye during normal playback with the system you normally use to view your backups

all this working out, analyzing or still shots comparisions is pointless for me now as I can tell just by watching the backup on my setup as to which I would prefer

Do we all have the same viewing setups or the same pair of eyes and why are there settings for colour, brightness, contrast, sharpness etc?

Everyone has difference viewing setups and preferences and what's good for one person may not necessarily be suitable for someone else even if it is mathmatically proven to be better quality

I'm just glad I can make my mind up watching a backup sample and I've given up on trying to show others what I see as good quality for the reasons I've listed above

I didn't know this was one of the favourite discussions regarding comparisions. I always think the end up with people disputing each others choice of transcoding tool

I have a dual output on the PC and two monitors that are the same so I can do side by side analysis if I really wanted to but then again I don't really watch them on my PC

to each his own I guess :D

[gnipooldd]

Hi all. :)

It would be cool if someone could write a little proggy with two windows just to play and compare .VOB files, no? :D

[MackemX]

you could use your regular software player and IfoEdit's DVD play feature

you can either split the screen or you could make sure both are playing the same video then just make the windows the size of your screen and skip between the two using ALT+TAB or the taskbar

I just did :)

[gnipooldd]

Yeah, but I'm lazy. :D

[MackemX]

which two do you wanna compare?

I'll do it for you :D

in fact I'll save myself the bother as we all know Instantcopy is the best so just use that :p

[gnipooldd]

InstantCopy is a pain in the arse. :D
Oops, am I allowed to say that? :)
Please don't bother explaining me it isn't,
I have read it all. :p

[MackemX]

IC may be a pain in the arse but I like it!

oops, that don't sound right did it? :eek:

I wonder if IC8 will cause a few migrations from other tools as I think it's eaiser to use than IC and there's a few hidden extras ;)

I have a 100% foolproof method/solution to getting the best quality when doing a backup, just buy DVD5's all the time :D

[gnipooldd]

IC may be a pain in the arse but I like it!
oops, that don't sound right did it? :eek:
LOL. :D

I have a 100% foolproof method/solution to getting the best quality when doing a backup, just buy DVD5's all the time :D
He he :)

[MackemX]

you just know if you got a sample of 100 people, some would probably still say that X is better than quality Y when doing a DVD5 backup comparision

I bet if you put a frame by frame analysis up as a comparison, some people will think that one is better than the other. If you also played them back one after the other, some would probably say one was better than the other

it's impossible that everyone who was tested would say they are the same

amazing ain't it :D

[mrbass]

http://forum.doom9.org/showthread.php?&threadid=61128

better than PSNR. Thanks to DDogg for the heads up. Still though I agree with MackemX it's gonna be a field day still.

[monomer]

Originally posted by mrbass
http://forum.doom9.org/showthread.php?&threadid=61128


WOW! I went there and it was like going to another planet where the inhabitants speak an English-like language but I just can't quite make out what any of them are saying...

What I did glean from the thread was that these quantitative methods being used to assess the faithfulness of copies to the original use a weighted average or the average deviation of the copy from the original (which means variance is the thing being measured). Statistically, this is fine but I'm not sure exactly what data is being used... is it pixel for pixel color (how is that quantified?) or luminance? chroma? is it scaled to reflect human perceived biases? To tell you the truth, this is all way beyond my realm of understanding... but hey, mrbass thanks for trying to enlighten me. And thanks for all those wonderful guides! It's all greatly appreciated.