SSIM tests

[Kayaker]

Hi there.

I don't want to get flamed, so I'll be careful with this one...

I've made a quick test with DVD Shrink (AEC smooth) and DVD-rebuilder ( with QuEnc - high quality - 2 passes - No trellis )

The shrinking is about 80%

My SSIM (and PSNR ) tests showed a slightly anvantage of DVD Shrink.

Rebuild : SSIM: Structural Similarity Index Metric 0.23 : Average SSIM= 89.41

Shrink : SSIM: Structural Similarity Index Metric 0.23 : Average SSIM= 91.98

I know I _must_ be wrong somewhere.
BTW , I can't draw any conclusion for on test, one movie.


I know re-encoding should be better than transcoding.
And I suppose the greater the shrinking, the better the reencoding than trascoding.

Please if you can point me to some serious (and more thoroghly done that my very simple one ) tests I'd grateful.

Of course with more time I'll do my own.
But I want sone serious testing to start reading with.

BTW, I did notice the 100% equal first frames (around 1800 in my test) with DVDShrink and I did skip them in the test so an to not affect the SSIM average.
I wonder If DVD Shrink always do the same and don't drop any DCT coefficients in the first part so as to get exact frames.
As to fool some rapid test in the beggining... just speculating here....don't get me wrong.

[Sir Didymus]

Structural similarity metrics are not suitable for such type of comparisons: since the transcoder keeps unaltered many features of the the source (specifically the GOP structure is left unchanged), the similarity index tends to show very high values even in circumstances, i.e. blockiness or other artifacts, where your eyes (the right tool to perform such tests) would say the opposite.

On the other hand, encoders are always producing a different GOP structure than the source, so when you compare its output against the original, in average, it is very frequent to compare I frames of the source with B frames of the encoding output, or analog type of "apple with pear" comparisons, that are simpy misleading...

[Kayaker]

Well, maybe I'm in prehistoric times in quality metrics.

But if SSIM is not a good metric from the humay eye perception, what would be nowadays the right metric (if there is one, I know mathematics is one thing, eye perception another ) to test human eye perception ?

I also know that if there were a metric that catches perfectly what we see and what we don't, that model could be implemented in a codec and that codec could discard what we don't see. So I guess we are not that advanced in human eye perception modelling.

And of course MPEG2 uses some kind of human eye perception modelling.

[jdobbs]

Do a search of this thread. This subject has been discussed to death. The bottom line is that it is intuitively obvious to the casual observer that reencoding results in better output (substantiated by double-blind tests) except for very, very low reductions -- so (using common sense) any comparison that give a different result has to be flawed. Please don't take that as an off-the-wall flippant comment to "sell" DVD-RB (as it sounds) -- I'm saying it because it was proven true long before DVD-RB was ever written.

When a transcoder does a source at 80% it very probably doesn't touch the I frames and possibly not the P frames -- so the numbers are skewed upward... but the consistency of the video suffers because all of the reduction happens in B Frames...

Also, as you mentioned, any type of "difference" testing has problems... because of the way MPEG and other video codecs work... some "differences" are noticable to the human eye and others aren't...

[Kayaker]

OK. Where do I buy DVD-R ? right now !! ;)

This transcoding vs reencoding stuff is very interesting. I'll keep searching and testing.
I won't reinvent the wheel of course...

I guess for now with VobBlanker/PGCEdit and then DVD-R free version will be enough if I want reencode.
hehe what a cheap guy I am...

Thanks for the free program anyway !!!

[Sagittaire]

Do a search of this thread. This subject has been discussed to death. The bottom line is that it is intuitively obvious to the casual observer that reencoding results in better output (substantiated by double-blind tests) except for very, very low reductions -- so (using common sense) any comparison that give a different result has to be flawed. Please don't take that as an off-the-wall flippant comment to "sell" DVD-RB (as it sounds) -- I'm saying it because it was proven true long before DVD-RB was ever written.

When a transcoder does a source at 80% it very probably doesn't touch the I frames and possibly not the P frames -- so the numbers are skewed upward... but the consistency of the video suffers because all of the reduction happens in B Frames...

Also, as you mentioned, any type of "difference" testing has problems... because of the way MPEG and other video codecs work... some "differences" are noticable to the human eye and others aren't...

SSIM = 90 is very high quality : You use for this transcoding a very low reduction. In this particular SSIM say simply that here transcoding is better than reencoding for source preservation. With this quality level IMO you can't really see difference between source and trancoding/reencoding ...

[Kayaker]

@Sagittaire

You're right.
My eyes can't see the difference.
I have to zoom x2 a still frame and do some alt-tab to see slighly the differences.

I suppose with just 80% it's better to go the fast way and transcode.

Maybe the general belief (as I've seen) of go fast and trascode above 75-80 % and below that (more agressive compression) reencode has its reasons.

I'll do more testing with more aggressive compression.

[jdobbs]

It depends on a lot of things and you can't use percentages as a guide. If you have a source that was originally encoded as a low bitrate because it was very long, and then try to transcode at 90% it will look bad. If you have a source that was originally encoded at a very, very high bitrate you may be able to transcode at 75% and it will still look good.

Here's my view: Reecoding always does a better job -- but sometimes on easy sources it's hard to see the difference. It always seems to be one of those "easy" sources that is referenced when you hear people say "it looks the same to me." They are few and far between, though, especially when you do full-disc backups. I personally always reencode, just for the comfort of knowing it was done the best it could be.

But, as I've always said, DVD-RB is aimed at a different audience than transcoders... so I really don't think of it as a competing product. Shrink is an incredibly well written powerful program that was authored by one of the smartest guys around. For those who want to transcode it's about as good as it gets.

[Kayaker]

Ok.

Very clear.

Of course the only proble is time.
I don't have CCE, so I did QuEnc high quality. (just because I'm familiar with ffmpeg) and it took ages. 8 hours.
Stil have to test Hank compressor.


I'll test more agressive compression.
with DVD Shrink I can force more compression (lowering the 4.3 Gb size), but I suspect I can't do that with DVDR free version right ?
Or maybe with the addons... I have to check it out...

[Kayaker]

For those who want to transcode it's about as good as it gets.

So, it is pointless using DVDR with Rejig ?

[rahzel]

imo, anything above 80%, you're better off using a transcoder. anything less, then id use DVD-RB.

i myself have a hard time seeing a difference at 80%. the only time ive seen a difference with a transcoder, is at 75%.

[jdobbs]

imo, anything above 80%, you're better off using a transcoder. anything less, then id use DVD-RBI couldn't disagree more... but to each his own. As I said... percentages mean nothing:It depends on a lot of things and you can't use percentages as a guide. If you have a source that was originally encoded as a low bitrate because it was very long, and then try to transcode at 90% it will look bad. If you have a source that was originally encoded at a very, very high bitrate you may be able to transcode at 75% and it will still look good.Which would you rather have 10% of a dollar or 5% of 20...

So, it is pointless using DVDR with RejigOnly if you are really in a hurry.

[techmule]

I couldn't disagree more... but to each his own. As I said... percentages mean nothing:Which would you rather have 10% of a dollar or 5% of 20...

Only if you are really in a hurry.

That brings out another question, what would be an efficient way to determine the bitrate of a movie so as to determine what reduction in BR the final encode/transcode will do and then to decide if to go for transcoding or Encoding

[Sir Didymus]

Drag and drop some vobs of the decrypted files/iso into DgIndex and look at the bitrate values it gives.

Personally I never use transcoders - even for very limited reductions (but that's a personal choice: I have plenty of time to spend on playing with the backups... :) ). Into these circumstances I try to adopt other (manual) strategies: reducing just menus/extras or reencoding just the end credits at very low bitrates...

Cheers,
SD

[techmule]

I searched and found that ReMPEG2 also gives you the avrg bitrate used by your video for the whole movie in its info.

[jdobbs]

You could always determine the original video bitrate by dividing the DVD-RB calculated bitrate by the percentage. So if DVD-RB tells you the new bitrate is 3,725Kbs and it got to that value using a 63% reduction... the original bitrate was:

3725 / .63 = 5912

So the original encode was 5,912Kbs.

[Kayaker]

I've read old posts about the subject and now I know a lot more about it.

How transcoders "steal bitrate" first from B frames and all the reasons behind that.
Why that can generate the pulsating effect of touching some frames and not others. etc.etc.
However my tests with 80% DVDshrink DA, AEC greater smoothnes show that ( maybe because of I choose mamimum smoothnes ) that it drops DCT coefficients or "steals bits" from every frame more or less the same.

And why SSIM is not so good.
It takes into account only frame to frame comparison, don't make any temporal comparison.

Well all that are pros for reencoders.
So I guess JD didn't write DVD-R in vain after all ;)

Now I want to talk about the "percentage treshold" around which you can say that below that treshold the artifact introduced by a transcoder would be noticeable.

I agree with JD that that treshold percentage is not fixed at 75 or 80 %, and depend on type of video and original bitrate.

If the original bitrate is "on the edge" or has little "spare bits" left the quality may drop suddenly if you drop below let's say 90 %.

On the other hand if the original video was simple or too dark o was encoded at a high bitrate you got plenty "spare bits" and you could transcode at 60% and get good results.

Of course reencoding will use the better bitrate distribution for the new given bitrate no matter what, building new GOPs as the MPEG2 encoder seems it fits best.

To determine mathematically this treshold I guess we can recompress (NOT transcode. RECOMPRESS.) the original at a 90% , 80% , 70%

and below, and do an SSIM test against the original.

Below certain point the quality will drop significantly.
That point can give us un idea of how spare bits do we have in the original source.

Am I talking nonsense here ?
Please correct me if you think so.

[Voodoochild]

Do such tests, but I tell you this much. My friend did backup of DVD with shrink and he knows how to work with it. The DVD played it with jumps... I took it and did it with DVD-RB... :-) perfect. So beside quality there also matter of better DVD compliance. In that DVD-RB is much better. Apart from that I did tests of my own with Shrink VS DVD-RB, Once I used Shrink for 85% and up compression, I just use DVD-RB for everything even if no compression needed, you can guess what was the conclusions of my tests... :rolleyes: . Bottom line when I look on my TV DVD-RB gives better results, and if I need I can use avisynth filter to make it even more to my taste.

Elad

[Kayaker]

Don't worry.

I'm beyond comparing trascoding with reencoding already.
They are different things with both pros and cons.

I DID the homework and reserach.
I'm a converted man now ;)


The overall quality of reencoding (with a good encoder) is better.
But sure transcoding if fast and for low percentages you eye probably can't see the difference.

I've tested DVD-R with Hank encoder yesterday.
I wonder why the package cames with 0.17 and not the latest 0.18.
Maybe not well tested ?

Anyway I've tested with HC 0.18 with high compression ratio ( 50 % ) and it seems good (given the bitrate of course).

And I know all this testing if for fooling around only.

It's for us freaks who spend more time playing with videos than actually seeing them ;)

[Voodoochild]

The overall quality of reencoding (with a good encoder) is better.
But sure transcoding if fast and for low percentages you eye probably can't see the difference

I know what you mean by spending more time... on the machine instead of seeing them. Just wanted to add that it's the image quality that I see the differences between the 2, it's the overall motion and DVD flow.

I did 2 month ago "The 5th element" compression wasn't high, If I remember it was 83%. The outcome was that:
1. DVD Shrink in some scene the transition between frames wasn't smooth, there were pixilation for a split second in some places.
2. DVD-RB everything was smooth.

That the differences I'm talking about.
Any way, enjoy and try to spend more time seeing DVD's then encode them :-)

Elad

[Voodoochild]

image quality that I see the differences between the 2

I ment not in image quality I see the difference (unless using avisynth filters) But in the flow of the movie.

[Kayaker]

It'd be great to have some metric that takes into account temporal smoothnes or continuity also.

Maybe some kind of comparing the correlation between frames in the source (kinda of computing mpeg motion vectors ) and in the compressed stream and giving more points to the greater correlation of smoothnes between source and compressed.

Or something like SSIM with both spatial and temporal measure.

Of course, blind tests are the final judges.