Speed and quality of different denoisers

[Wilbert]

As a follow up to the Speed and compression of different denoisers (http://forum.doom9.org/showthread.php?s=&threadid=51181) thread, I decided to make a follow up test.

I made the test a bit differently, since I was not completely satisfied with the measured quantities in that thread.

The main goal of this test is to measure the quality and speed of denoisers at a given bitrate and compare them. Since this is often done if you do back ups. I made tests at 1000 kpbs, 750 kpbs and 600 kpbs, and used the movie 'From Hell' and 'an analog capture' as sources (the latter is not ready yet). For quality I measured the SSIM (http://forum.doom9.org/showthread.php?s=&threadid=61128) (100 is maximal: perfect clip).

Used plugins (all v2.5 plugins) and settings (XviD, CBR 1000 kpbs, 4000 frames). The script starts with

mpeg2source("F:\Temp\PluginsTests1000\from_hell.d2v")
Trim(0, 3999)
Crop(0, 72, 720, 432)
Plugin(...)

The results are given in the following table:


Plugin enc. time size (bytes) SSIM
None 1:29 20.146.176 79.36
Unfilter(-5,-5) 1:32 20.121.600 78.88
Undot() ´ 1:33 20.142.080 79.28
Mergechroma(blur(1.3)) 1:39 20.146.176 79.21
TemporalCleaner() 1:50 20.045.824 78.57
STMedianFilter(5,15,4,7) 1:53 20.109.312 78.38
STMedianFilter(8,15,4,7) 1:52 20.109.312 77.69
TemporalSoften(2,3,3,6,2) 1:53 20.062.208 79.82
Fluxsmooth() 2:03 20.076.544 79.21
Deen("c3d", 1) 2:07 19.888.128 76.68
Fluxsmooth(1,1) 2:09 20.137.984 79.53
TemporalSoften(3,5,5,10,2) 2:12 20.062.208 79.25
Convolution3d("moviehq") 2:16 20.127.744 79.61 # YV12 version
VagueDenoiser(0.8, 1, 6, true) 4:05 20.109.312 79.20
MipSmooth("moviehq") 4:14 20.043.776 78.88
NoMoSmooth() 4:23 20.064.256 79.27 # converted to YUY2


I put those values in a graph, where the encoding time is normalized on the unprocessed clip. For example, if the quantity is 2, that means that the encoding time of the clip will be twice as long as of the unprocessed one:

http://www.geocities.com/wilbertdijkhof/plugins_test/vob1000.png

Used plugins (all v2.5 plugins except Peach) and settings (XviD, CBR 750 kpbs).


Plugin enc. time size (bytes) SSIM
None 1:29 15.151.104 76.59
Unfilter(-5,-5) 1:31 15.149.056 76.13
Undot() ´ 1:32 15.151.104 76.51
Mergechroma(blur(1.3)) 1:39 15.153.152 76.47
TemporalCleaner() 1:50 15.142.912 75.80
STMedianFilter(5,15,4,7) 1:52 15.147.008 75.71
STMedianFilter(8,15,4,7) 1:52 15.147.008 75.05
TemporalSoften(2,3,3,6,2) 1:53 15.142.912 76.98
Peach(30, 40, false) 1:55 15.151.104 74.38
Fluxsmooth() 2:03 15.147.008 76.33
Deen("c3d", 1) 2:07 15.120.384 74.29
TemporalSoften(3,5,5,10,2) 2:08 15.134.720 76.45
Fluxsmooth(1,1) 2:09 15.151.104 76.72
Convolution3d("movielq") 2:15 15.147.008 76.48 # YV12 version
Convolution3d("moviehq") 2:16 15.153.152 76.81 # YV12 version
VagueDenoiser(0.8, 1, 6, true) 4:05 15.149.056 76.43
MipSmooth("moviehq") 4:15 15.140.864 76.26
MipSmooth("movielq") 4:15 15.126.528 75.52
NoMoSmooth() 4:24 15.144.960 76.51 # converted to YUY2


The graph:

http://www.geocities.com/wilbertdijkhof/plugins_test/vob750.png

Used plugins (all v2.5 plugins except Peach) and settings (XviD, CBR 600 kpbs).


Plugin enc. time size (bytes) SSIM
None 1:31 12.152.832 73.98
Unfilter(-5,-5) 1:32 12.144.640 73.57
Undot() ´ 1:40 12.148.736 73.96
Mergechroma(blur(1.3)) 1:42 12.148.736 73.92
TemporalCleaner() 1:51 12.152.832 73.35
STMedianFilter(5,15,4,7) 1:52 12.150.784 73.19
STMedianFilter(8,15,4,7) 1:52 12.150.784 72.66
TemporalSoften(2,3,3,6,2) 1:56 12.152.832 74.42
Peach(30, 40, false) 1:56 12.150.784 71.98
Fluxsmooth() 2:06 12.150.784 73.80
Deen("c3d", 1) 2:06 12.142.592 71.65
TemporalSoften(3,5,5,10,2) 2:11 12.148.736 73.97
Fluxsmooth(1,1) 2:09 12.148.736 74.13
Convolution3d("moviehq") 2:17 12.150.784 74.18 # YV12 version
Convolution3d("movielq") 2:18 12.148.736 73.88 # YV12 version
VagueDenoiser(0.8, 1, 6, true) 4:15 12.148.736 73.85
MipSmooth("moviehq") 4:27 12.150.784 73.75
MipSmooth("movielq") 4:17 12.144.640 72.87
NoMoSmooth() 4:27 12.148.736 74.02 # converted to YUY2


The graph:

http://www.geocities.com/wilbertdijkhof/plugins_test/vob600.png

I also measured the four "best" plugins at 1250 kbps, and put them in a graph with the other bitrates:

http://www.geocities.com/wilbertdijkhof/plugins_test/vobtop.png

[Lefungus]

Really nice work Wilbert !

Have you kept the csv files created by ssim for each denoiser ? Another nice graph could be created from it.
As even if the average SSIM is better, it's sometimes better to have a more constant ssim value, without peaks, but with a little lower average score.

Have you used lumimask=true for ssim comparisons ? i hope you have, as it should give better results.

Is cbr as accurate on filesizes as 2-pass ?

[Wilbert]

Have you kept the csv files created by ssim for each denoiser ? Another nice graph could be created from it.
Yes, I kept them. You mean a frame by frame comparison of different plugins?

Have you used lumimask=true for ssim comparisons ? i hope you have, as it should give better results.
Yes, but I don't understand this setting?

Is cbr as accurate on filesizes as 2-pass ?
No, I don't think so. The differences between the encodings can be 100 KB. But if I have to do 2 pass encodings, then it will take ages before I'm done :)

[Lefungus]

Originally posted by Wilbert
Yes, but I don't understand this setting?


SSIM with lumimask=true, assume that artifacts with low luminance are less visible. So each pixels is weighted. The VQEG tests have shown better results with it)

Could you post filesizes too, if you don't mind ?

[Lefungus]

About the results, it's interesting to see that only a few denoisers, does improve quality (if ssim is correct)

[Piper]

Thanks for taking the time to put that together Wilbert. I anxiously await your analogue capture findings.

btw, any particular reason Peach didn't make it into the first round?

[telemike]

Please add Deen to the test too :)

[WyldeF1r3]

I've never seen "From Hell" but if the best an unfiltered clip can achieve out of 100.00 on the SSIM scale is a 79.xx is this really such a great source to use for a test which simply measures the destructiveness of a filter on an original "perfect" image?

I realize that quite possibly the reason for this is the cropping, but still, shouldn't the control clip give a more assuring value than it does? Since the 2nd clip is in refrence to the first clip why would an unfiltered encode produce such a great SSIM gap in output?

I guess I just fail to understand how a filtered encode can look more like the original (be closer to ideal quality of 1) than an unmodified encode.

[Lefungus]

I wonder what method is best ?

Compare original source to the filtered and compressed source
or original source, filtered, to the filtered and compressed source

[Wilbert]

I've never seen "From Hell" but if the best an unfiltered clip can
The part of the movie that I used has dark frames, and very detailed frames, but has not much high motion scenes.

achieve out of 100.00 on the SSIM scale is a 79.xx is this really such a great source to use for a test which simply measures the destructiveness of a filter on an original "perfect" image?

It measures the result of the filters on the compression. See below ...

I realize that quite possibly the reason for this is the cropping
I don't think so. I think it's the encoder and the used bitrate. Using a higher bitrate will result in a higher SSIM value. But I will do some tests on this. Btw, I didn't resize them.

I guess I just fail to understand how a filtered encode can look more like the original (be closer to ideal quality of 1) than an unmodified encode.
That's the whole reason people apply filters on a clean source (specially if the encoding has a low bitrate, or is mpeg): to make it more compressible. The idea is that you have less compression artefacts if you apply soft filtering on a clean source, as opposed to no filtering.

@Lefungus

Compare original source to the filtered and compressed source

That's what I did.

original source, filtered, to the filtered and compressed source

Sorry, I don't understand what you want to compare here.

[WyldeF1r3]

Originally posted by Wilbert It measures the result of the filters on the compression.
I'm sorry, I don't understand what you mean by this. I thought Lefungus said:
Originally posted by Lefungus it is meant to compute a quality metric, based on perceived visual distortion.
which doesn't sound like the same thing to me. Of course I could be just gorssly misunderstanding the whole principal of this...

Originally posted by Wilbert That's the whole reason people apply filters on a clean source (specially if the encoding has a low bitrate, or is mpeg): to make it more compressible. The idea is that you have less compression artefacts if you apply soft filtering on a clean source, as opposed to no filtering.

Obviously, but SSIM is not a measurement tool for compression, neither is PSNR or MSE or anything else, you have your byte sizes for that. If SSIM is supposed to be a way at trying to calculate a possible subjective opinion on quality comparing a clip to the original, how can something that is different from the original look more like the original than something that was completely left alone? (why does c3d and temporalsoften have a higher SSIM than an unfiltered clip if the whole basis for measurement is the difference between the original and the secondary, why it is a smaller size is a statistic not in question.) Unless I'm misunderstanding something about the way SSIM is supposed to work and what a 100.00 source is. If you have a purposely flawed clip and "improve" its quality through filtering, it will have a less than 100.00 because it doesn't look identical to the source, yet subjectively it will "look better," am I wrong?

[Manao]

original source, filtered, to the filtered and compressed source I think Lefungus means compare the source after filtering to the compressed result. But by doing so, you'll only measure the ability of a filter to give an input less modified by the compressor. It is interesting in order to understand interaction between filter and compressor, but not if you goal is to obtain a compressed result the closest to the original.

It could be interesting to try different bitrates ( 800 kbit/sec and 1400 kbit/sec for example ), to see the influence of the compression ratio on the performance of the different codecs. And I would add a test with TemporalSoften and Convolution3D with stronger settings, just in case there performance is improved ( it should of course depend of the bitrate used, but it would be a good basis in understanding overfiltering )

Anyway, a big thank for your test.

@WyldeF1r3 :

You can only obtain 100 if you compare the clip to itself.

The 79.36 score is obtained when you measure the SSIM between the original unfiltered clip and the compressed ( @1000kbit/sec ) unfiltered clip. So it is normal to obtain such a "bad" score , since compressing is lossy.

The fact that a compressed filtered clip have a higher SSIM score ( compared to the original unfiltered clip ) means that ( filtering + compressing ) keeps a better quality that justs ( compressing ) ( assessing that SSIM is a good measurement of the quality of a processing, of course )

[DDogg]

It might be worth posting the below from the original SSim thread as a reference because the value is just a derived indicator. What I mean is 79.36 is just a reference point, its not like it means 79 percent of original or anything like that. Actually, if I remember, 79.36 isn't bad at all. Anyway, Wilbert, this is a valuble reference and thanks a lot for taking the time to do and share it with us.

In version 0.22, the global score is scaled from 0 to 100 instead of 0 to 1. it's just a simple conversion: newValue = 100 * pow( oldValue, 8)
With this, different values between two clips may be more visible.

example on a small xvid sample:

high bitrate clip: 82
low bitrate clip: 66

If you prefer it non-scaled, just add "scaled=false" as parameter.
For the same sample, it'll give:

high bitrate: 0.975
low bitrate: 0.949

[MfA]

WyldeF1r3, compression is about throwing stuff away ... the compressor doesnt always make the best decision on what to throw away, so sometimes if you do it yourself before the compressor gets its hands on it you can improve compression by making it concentrate on information it can compress well.

[ronnylov]

This is a very interesting tool to help us compare different conversion methods against each other!

I'm thinking of use it to compare my mpeg2 compression methods. If my source is 704x576 resolution huffyuv and my destination 352x576 mpeg2, do I need to resize the original to the same resolution as the encoded video before input it into the SSIM or can I compare clips with different resolution? Or is it better to enlarge the mpeg2 clip to the source resolution and then compare them in SSIM? The lower resolution itself make the image less detailed and the pixels more visible on my TV which degrades the visual quality but on the other hand the smaller resolution makes it easier for the encoder to compress which improves the quality at lower bitrates.

I can also use this tool to compare different encoders, like CCE, TMPGEncPlus, Canopus Procoder and Mainconcept encoder.

[ominte]

WyldeF1r3 is right I believe. SSIM is designed to measure the level of differnces against the original clip. It is not an overall quality indicator for your eyes. Let me explain this point further. The objective of noise filtering is to make the source:-

1.easier to compress
or
2. cleaner and with less noise.

For this test it appears Wilbert is trying to use SSIM to measure the different quality of the clips but this will not work for clips that have noise filtering because when you noise filter you are implying that you wish to change the look of the original and don't want to retain the qualities of the original. So when you perform an SSIM test your results will only reflect the differnt strengths used by each filter and hence their closeness to the original. From Wilbert's results we can easily see this point. Take a look at Temporal Smoother, Convolution3d and Fluxsmooth. They are ranked in Wilbert's quality scale in the following order:-

1.Temporal Soften
2.Convolution3d
3.Fluxsmooth

Now take a look at the order of their strength (from my own testing):-

(Weakest at the top)
1.Temporal Soften
2.Convolution3d
3.Fluxsmooth

So you see the SSIM results are not measuring quality but are in fact measuring the difference between the origninal clip - which is not a measure of quality for denoisers because that is not your objective. If you want to measure quality so far the only real method is to use your eyes. Wilbert's results may indicate that Temporal Soften has a higher "quality" than Fluxsmooth however if you view the two encodes you may find that your eyes find the Fluxsmooth encoding more pleaseing on your eyes.

If you want to test this point further Wilbert perform an encode with Fluxsmooth(1,1) and I am almost certain you will find its "quality" equals that of Temporal Soften. This of course makes no sense because how can using a weaker noise filter result in better quality when it removes less noise! It doesn't have better quality but instead is closer to the original.

The best (although probabaly the slowest) method to use for testing the quality of noise filters would be to losslessly encode each noise filtered clip and then compare this clip to a noise filtered encode performed by XviD. From the results you obtain you could determine which filter allows XviD to retain the most "quality" when compared to the noise filtered clip.

PS. Wilbert I still wish to congratulate you on performing a thorough and time consuming test :).

[Manao]

@ronnylov : I think you should extend the compressed clip to the size of the original one, or even both to the resolution at which you playback. ( I would use the first one, though )

@ominte : you want to separe the filtering from the encoding. I see it as a whole. The optimal strongness of the softening filter depends of the bitrate used. The fact that no filtering at all gives better SSIM results prooves the useness of such a filter. Your paradox ( less filtering -> more noise but higher SSIM ) is not a paradox for me. I see a softening filter only as a mean to help the codec taking the good decisions. I don't use a softening filter to reduce noise nor to improve compressibility, but to improve the looking of the encoding. In this case, overfiltering still improves compressibility, but harm the looking.
Wilbert's results may indicate that Temporal Soften has a higher "quality" than Fluxsmooth however if you view the two encodes you may find that your eyes find the Fluxsmooth encoding more pleaseing on your eyes. That has to be checked. We hope the SSIM is correlated to goodlooking. You don't think so. I don't know which one is right ( I still didn't test SSIM in fact ).
The best (although probabaly the slowest) method to use for testing the quality of noise filters would be to losslessly encode each noise filtered clip and then compare this clip to a noise filtered encode performed by XviD. From the results you obtain you could determine which filter allows XviD to retain the most "quality" when compared to the noise filtered clip. There I disagree, you separe filtering from encoding.

[ominte]

I see a softening filter only as a mean to help the codec taking the good decisions. I don't use a softening filter to reduce noise nor to improve compressibility
Just want to point out that you are in fact contradicting yourself in these sentences. Fist you state you filter to help the codec make good decisions. Then secondly you state that you don't filter to reduce noise or improve compressibilty yet the first term is a measures of a codecs ability to make good decisions (compressibility) and the other term, "reduce noise", allows a codec to make better decisions when compressing frames.

We hope the SSIM is correlated to goodlooking.
SSIM does not measure quality as such but more so the differnce between two encodes. Ask yourself this question - If SSIM is measuring "quality" then why does it need two clips to do so? The reason it needs two clips is because it is measuring the diffence between the two and it assumes one of the clips you provide it with (the source) correlates with good quality. However when we denoise we are trying to achieve an encode that is cleaner than the source. Therefore the source is no longer an accurate benchmark for good quality. This is why I proposed an alternative method of testing which benchmarks the new cleaner source against the encode (which is still also denoised).

...I disagree, you separe filtering from encoding.
Firstly I assume by "separe" you actually mean seperate. Now, my proposed test method doesn't seperate filtering from encoding. The encoded clip is still filtered however the source is also filtered. This allows us to compare the difference between the cleaner source (explained above) and the encode and allows us to assess the results and determine which noise filter allows us to maintain the a closer result to the filtered clip (which is ultimately the desired result).

[WyldeF1r3]

Is it useless then to use SSIM to measure the difference between a losslessly encoded filtered clip and the original? Or will that actually produce some kind of useful result.

[Manao]

OK, I've been too vague, I'll try to be more acurate.
We hope the SSIM is correlated to goodlooking. I meant that SSIM ( between the original clip, and the compressed one, filtered or not ) is a measure of the quality of the encode, which takes into account artifacts disturbing to the eye ( In that, it's a better distance than PSNR ). Thus, we can say that if we make two encodes, with different encoding settings, comparing their SSIM from the original clip, the "best" clip will be the one with the higher SSIM.
However when we denoise we are trying to achieve an encode that is cleaner than the source That's the point where I disagree with you. I'll try to explain. I see the encoding process as a unique filter. My goal is to obtain the highest "quality" ( ie SSIM ) between the source and the result, in order to be as close as possible from the original clip, without artifacts (nor overblurring). It happens that a little denoising helps to gain some SSIM (because of the blurring / denoising, without which the codec creates artifacts more "visible"). It depends of the bitrate used, of the strongness of the denoising filter, of which kind of denoiser is used, but they are all parameters of one theorical filter, the encoding process.
Firstly I assume by "separe" you actually mean seperate. Yes, some french words escaped :rolleyes:

@WyldeF1r3 : it is not useless only if you compare different filters, in order to see which one is the weakest. But doing only one measure is useless.

[MfA]

Often times people will use denoisers as smoothers on an already good source because they like the look of a movie after encoding better with than without the denoising at the same bitrate, for those kinds of use the SSIM test is perfectly valid.

[Wilbert]

Let me answer some of the points, as far as MfA and Manao haven't answered it.

btw, any particular reason Peach didn't make it into the first round?
I added Peach after seeing your request, but it couldn't get it much better than 76.89. I will add it with the next update.

Please add Deen to the test too
Maybe I will do that, but I don't promise anything.

I think Lefungus means compare the source after filtering to the compressed result. But by doing so, you'll only measure the ability of a filter to give an input less modified by the compressor. It is interesting in order to understand interaction between filter and compressor, but not if you goal is to obtain a compressed result the closest to the original.
This is very true. That could also be interesting, but it is not the purpose of this test.

It could be interesting to try different bitrates ( 800 kbit/sec and 1400 kbit/sec for example ), to see the influence of the compression ratio on the performance of the different codecs. And I would add a test with TemporalSoften and Convolution3D with stronger settings, just in case there performance is improved ( it should of course depend of the bitrate used, but it would be a good basis in understanding overfiltering )
My idea was (and still is) to do both points here.

WyldeF1r3 is right I believe. SSIM is designed to measure the level of differnces against the original clip. It is not an overall quality indicator for your eyes. Let me explain this point further. The objective of noise filtering is to make the source:-

1.easier to compress
or
2. cleaner and with less noise.

For this test it appears Wilbert is trying to use SSIM to measure the different quality of the clips but this will not work for clips that have noise filtering because when you noise filter you are implying that you wish to change the look of the original and don't want to retain the qualities of the original.
MfA and Manao gave comments on this (yes its not an overall quality, and yes its measures relative to the source). Let me just add that the source is dvd, and it doesn't contain noise. So, we don't want to change the look of the original. I think that we agree that the source has the highest quality.

However I also want to do tests with an analog capture. Of course, here is the situation totally different. The source is noisy and can't be used as a reference. In this case I will apply a superfilter and reencode with huffyuv, to gain the 'best' quality.

Take a look at Temporal Smoother, Convolution3d and Fluxsmooth. They are ranked in Wilbert's quality scale in the following order:-

1.Temporal Soften
2.Convolution3d
3.Fluxsmooth

Now take a look at the order of their strength (from my own testing):-

(Weakest at the top)
1.Temporal Soften
2.Convolution3d
3.Fluxsmooth

So you see the SSIM results are not measuring quality but are in fact measuring the difference between the origninal clip - which is not a measure of quality for denoisers because that is not your objective.

If we agree that the source has the highest quality, then we are measuring quality.

If you want to measure quality so far the only real method is to use your eyes. Wilbert's results may indicate that Temporal Soften has a higher "quality" than Fluxsmooth however if you view the two encodes you may find that your eyes find the Fluxsmooth encoding more pleaseing on your eyes.
Of course, the real question is if the SSIM values reflect this. I think it's nice to set up a blind test, to see whether this is true.

If you want to test this point further Wilbert perform an encode with Fluxsmooth(1,1) and I am almost certain you will find its "quality" equals that of Temporal Soften.
This is a very interesting point. And I don't think it is true in general. Sure, it will be closer to the unprocessed encoding, but not necessary above it (at least I think). I will try some light values for Fluxsmooth and report back.

[Lefungus]

Originally posted by Wilbert
Of course, the real question is if the SSIM values reflect this. I think it's nice to set up a blind test, to see whether this is true.


Actually, this blind test has already been done, on the vqeg test. See the original paper.
My own experience on it was that it seems to do its job. A simple test is somewhat easy, do a comparison, check the ssim value for each frame, and look at low quality frames ssim-wise. Most of the time, you'll find it ugly too.
I think as long as we keep in mind it's far from a perfect solution, and be cautious in conclusions, it'll be ok.

A blind test would be nice though, but hard to set up.

To return to the denoiser test, i agree with wilbert, and his analog/DVD methodology. Then we'll have a nice overview, that will call more subjective tests later to check it :) (ie: with our eyes)

[Wilbert]

@ominte,

Tried Fluxsmooth(1,1), and indeed you are correct. It comes on the third place.

@all,

I updated the first post. I made tests at 1000, 750 and 600 kbps [the latter is not complete yet]. But I'm very disappointed that only a few smoothers produce higher SSIM values than the source (also at low bitrates).

[Wilbert]

I completed the tests with the dvd-rip. The main conclusion is given in the last graph, which I also attached to this post. The best plugins are ranked as follows (using the default settings):

1) TemporalSoften(2,3,3,6,2)
2) Convolution3d(preset="moviehq")
3) unprocessed clip
4) NoMoSmooth()

Another conclusion is that you can get more quality from a plugin (have a look at Fluxsmooth(1,1), for example) by lowering the settings. In other words, most default settings or presets are set to high for encoding dvd rips.

It also shows that there is no need to smooth clean clips to gain compressibility, just encode at a lower bitrate instead. If you do want to filter, use TemporalSoften or Convolution3d with the settings above.

Looking at the clips themselved, I recommend not to go below SSIM values of 80, to get good quality.

If there are no requests for additional data or plots, I would like to proceed with the analog caps tests ...

[DDogg]

Wilbert, out of curiosity, in all of this work did you do a lossless_dump.avi and then compare it back to source? If so what was the SSIM value?

[Kurosu]

The conclusion I draw is that MPEG-1/2/4 compression schemes are more sensitive to dynamic/temporal noise (P/B-frames are indeed usually more than 95% of a filesize), as all the filters having a temporal part are in the top of those charts. But then, SSIM doesn't cover the temporal masking which is at the basis of B-frames: in fact, SSIM comparisons would probably yield that compression with (was: without) B-frames is often worse than without. ANd PSNR already shows little improvements (if any) compared to the visual improvements.

Filters like QMF have some success because they spacially smooth (reduce details/high frequencies/whatever...) more on high action scenes.

[ronnylov]

Originally posted by Kurosu
But then, SSIM doesn't cover the temporal masking which is at the basis of B-frames: in fact, SSIM comparisons would probably yield that compression without B-frames is often worse than without. ANd PSNR already shows little improvements (if any) compared to the visual improvements.

Compression without B-frames is often worse than without?
I don't understand. Do you mean compression with B-frames is often worse than without or do you mean compression with B-frames are often better than without?

Anyway, I have made a SSIM comparison with different mpeg2 encoders and when plotting the ssim graphs it was obvious that B-frames had lower quality than P-frames and I-frames, but that's not strange because B-frames requires much less bitrate. If the B-frames are removed there is less bitrate available to P-frames and I-frames and the overall quality may be lower. I haven't completed my tests yet so I don't know if anything will improve without B-frames. But I will try it and compare the results with my subjective quality impression by watching the video at playback.

How does temporal masking work? Does it mean that the eye can't see the lower quality of the B-frames at normal playback speed?

[Kurosu]

Originally posted by ronnylov
Compression without B-frames is often worse than without?
I don't understand.
Corrected. Besides, what I said isn't a rule of thumbs. In fact, the "often worse" isn't true. It's in fact often better on average. But the B-frames by themselves have a lower PSNR (and probably SSIM value). Yet, this isn't visible at playback speed because of temporal masking (due to retina persistence, I bet).

I think a valid criterion for video compression would need to account for this problem. But that's even more complex to modelize. Maybe MfA or Lefungus have encountered during their reading of VQPEG results some other methods. (was SSIM introduced in quality measure for still (JPEG-like) images?)

How does temporal masking work? Does it mean that the eye can't see the lower quality of the B-frames at normal playback speed?
I think so (I don't know if ffmpeg's option uses the same basis). Don't see in that word something magic either, I used it because it was the most meaningful for me.

What I aim at is saying that temporal cleaning probably has a better result, because spacial filtering mostly affects intrablocks (I-frames and some parts of P-frames - ie without motion compensation), and that the most bits are spent on interblocks. If we follow that direction, then this brings the matter of temporal masking, hence visual quality, which isn't only based on still images comparison. For that kind of reason, people may prefer or not RealVideo 9 over XviD and vice versa.

[Wilbert]

Does it mean that the eye can't see the lower quality of the B-frames at normal playback speed?
Can someone explain the reason of this?

btw, all the encodings didn't have b-frames.

@DDogg,

I encoded the vob to huffyuv, the resulted SSIM value (after converting the huffyuv to YV12) is: 95.93

[JuanC]

@Wilbert:

When subjectively comparing denoisers I usually compare the original and the results of the tested denoisers. My choice of a filter, based on the quality I perceive, is the filter that reduces as much as possible the noise and keeps as much as possible the fine details, and then I take in account the speed. I would sacrifice a little quality for speed. I don't know how I could do something like this, comparing the quality of the denoisers, without a “noisy” clip.

My suggestion: I hope this is possible: Having an original clean clip, then using some mechanism (noise generators? getting noise from other noisy clips?) to mix or overlay some noise to the original clip and then use the resulting noisy clip (encoded to Huffyuv???) to test the denoisers. And this way the results of the filters could be compared (using SSIM) to the original clean clip. I wouldn’t be surprised if applying the filters on a noisy clip; PeachSmoother (very effective and fairly fast) gets to the top of the list.

Hope this makes any sense. :J

[Wilbert]

Can someone explain the reason of this?
Never mind. b-frames contain the moving scenes/parts of the clip. The quality of those "can't be seen" at normal playback.

I don't know how I could do something like this, comparing the quality of the denoisers, without a “noisy” clip.
If you read the whole thread you will see that not everyone smooth to denoise their clip. Personally, I never smooth when having a clean source. (But then, I don't encode to svcd, or at very low bitrates :)) Therefore I also proposed a second test, using an analog capture.

Your suggestion is good, I didn't think of that! There are indeed filters which add noise to a clip (blockbuster or addgrain). But I have some other ideas :)

[ronnylov]

Regarding B-frames I did a test on mpeg2 with Mainconcept encoder 1.4 and the ssim average value was slightly higher with B-frames than without (I tested a home made camcorder video at 8 mbit/s). But there was a lot of motion in this clip (shaky camera). I guess the B-frames in mpeg2 are there for a reason. The situation with divx/xvid and b-frames may be different.

With B-frames (IBBPBBPBBPBB): SSIM average = 82
Without B-frames (IPPPPP): SSIM average = 81

Adding or removing P-frames did not change the value much.

[Kurosu]

Originally posted by Wilbert
Never mind. b-frames contain the moving scenes/parts of the clip. The quality of those "can't be seen" at normal playback.
Ah! That was what you meant. I thought you were familiar with the MPEG standard.
intrablocks are 'new' block, coded as is.
interblocks are 'old' blocks, to which a reference in past/previous frames was found - and only the difference is encoded.
I-frames are frames encoded a la JPEG
P-frames are frames partially predicted by motion estimation (16x16 block search with vectors by 'steps' of at worst 16,16 (I think the famous halfpel and quaterpel are 8,8 and 4,4)
B-frames (between 2 P-frames most often) roughly interpolates the motion vectors, and deduce a new image

The difference between image build by motion estimation and the actual image is a difference (or error) images. B-frame error images have less information than P-frames and can be quantized higher without as much degradation as if it was done on a P-frame. This maybe a bold statement when refering to PSNR, but people in the XviD forum would point you to a threshold inside XviD that tells when frames are encoded as B-frames. I don't know if it's related to temporal considerations. But in any case, motion does hide the lower quality of B-frames. Put too many B-frames between P-frames, and not only the PSNR, but probably the visual quality will be degraded, because artifacts may remain on video enough for the eye to notice.

[trbarry]

I don't know about the effect of seeing b-frame errors. But b-frame errors do not propagate to any other frames, simply because nothing else depends upon them. So the quality effect of any b-frame error is limited to a single frame. This isn't true for errors in I or P frames since other frames may reference them. For instance a file or transmission error in an I-frame can screw up a whole GOP. And over quantization of an I-frame may be visible for the entire GOP.

- Tom

[JuanC]

I have read, again, the whole thread. I think there are very interesting posts. In particular I find the ones from WyldeF1r3 and ominte very interesting.

You are dealing with two different things in your approach to the comparison: denoising (filtering) and compression (encoding: b or not b-frames... which have had most of the attention in the latest posts).

Ideally we should separate the two things, and, if comparing denoisers (the tittle of your first post talks about that) every clip involved in the process should be using the same compression scheme (huffyuv???) And so, the reference clip (Clean), the second refference clip (with noise added to it), and the results of using each denoiser to filter the added noise in that noisy clip should be encoded using the very same codec, using a technique as lossless as possible (Huffyuv?).

Again, I hope this makes sense. :J

[ronnylov]

How about following method to add noise?

Take a clean DVD and put it in your standalone player (which should be macrovision free). Hook it up to your VCR. Play a chapter on the standalone DVD player and record it to VHS on the VCR. Hook up the VCR to your analogue capturing board and capture the VHS tape.

This would be the noisy clip.

Rip the same chapter directly from your DVD disc in a DVD-ROM unit. This would be the clean clip.

Now you can compare the clean clip with different denoisers applied to the noisy clip. I think a problem may be aligning the pictures pixel by pixel because of resizing made in the capture process but it may be possible to test different denoisers from a VHS capture.

You could also make a comparison with an analogue capture direct from the dvd player. This would be a clean analogue capture to compare with the ripped chapter after applying denoisers.

You could also use a capturing board with built in TV-tuner and tune in the RF-signal from the VCR antenna output connected to the antenna input of the TV-tuner on your capture board. This would be something similar to live broadcast on a cable network.

[Wilbert]

Take a clean DVD and put it in your standalone player (which should be macrovision free). Hook it up to your VCR. Play a chapter on the standalone DVD player and record it to VHS on the VCR. Hook up the VCR to your analogue capturing board and capture the VHS tape.

This would be the noisy clip.
No, it's not a noise clip. The noise in the analog capture comes from the cable lying under the street, not because you are recording on VCR.

Ideally we should separate the two things, and, if comparing denoisers (the tittle of your first post talks about that) every clip involved in the process should be using the same compression scheme (huffyuv???) And so, the reference clip (Clean), the second refference clip (with noise added to it), and the results of using each denoiser to filter the added noise in that noisy clip should be encoded using the very same codec, using a technique as lossless as possible (Huffyuv?).

1) I don't think people encode their dvd-rips / analog captures with huffyuv. Of course, the results differ from codec to codec. I think the results are representative for XviD or DivX.

2) I'm not sure if the noise added with blockbuster/addgrain is comparable to noise found in analog captures. This is my problem with the proposal of JuanC.

Here is my proposal for the second test:

We apply a super-denoiser to the capture and encode it to huffyuv. This will be the clean reference clip (cleaner than applying any other denoiser, but we have to agree on that). Then we do the same test as the first one, with stronger settings.

[YY1020]

How about 2DCleansYUY2??

[JuanC]

Originally posted by Wilbert
...
1) I don't think people encode their dvd-rips / analog captures with huffyuv. Of course, the results differ from codec to codec. I think the results are representative for XviD or DivX.

2) I'm not sure if the noise added with blockbuster/addgrain is comparable to noise found in analog captures. This is my problem with the proposal of JuanC.

... 1) My suggestion is to separate the encoding aspect from the filtering one. That's why I though of Huffyuv, as it is a codec as lossless as it can be. Here's how I think it could be: If you have an original clip from say a DVD, you encode it as the original, w/o filtering, then you add the noise and encode it as the refference noisy clip, and then you encode each resulting "denoised" version of the clip and compare them to the clean encoded one.

2) How about this: use the noise generators, and besides that combine (mergeluma & mergechroma) some analog captured clip which contains only noise. I can think of getting together several of those small "black" portions of video between commercials and TV shows....

I appreciate your great work and the time you spend on it. :J

[ronnylov]

Originally posted by Wilbert
No, it's not a noise clip. The noise in the analog capture comes from the cable lying under the street, not because you are recording on VCR.

OK, I thought there was some noise added from the VHS tape itself but perhaps not. But if I use the RF modulator on my VCR to convert to antenna signal and then use a very long antenna cable or an attenuator to make a weak signal and connect it to my TV-tuner card in my PC and capture the demodulated signal I guess there will be some noise. I can skip the VCR recording part if I don't want to test VHS cleaning filters. Or is this a silly thing to do if I want to test filtering methods to restore as much of the original picture as possible? I guess it won't be the same as real TV capturing no matter what I do, because I don't have any access to the original source used by the TV-station. The ssim plugin compares the original with the end result but if I don't have the original before the noise was added then it's impossible to test the cleaning filters used when trying to restore the original picture.

EDIT: I just remembered that some of the channels I can grab directly to mpeg2 with my DVB-C digital capture card are also sent in analogue form on my cable network. So if I can capture with my DVB-C simultaneously as with my ATI All-In-Wonder Radeon (and perhaps simultaneously record it to VHS also) then I could compare the clips, using the DVB capture as reference. The DVB capture is still not the original signal because it is mpeg2 encoded to a DVB stream but it is still better than my analogue captures. But I'm not sure if I can capture with my analogue card simultanesously as with my DVB-C card because they are both in the same computer. But I could record it simultaneously with my VCR at least and capture it from the VCR later! Perhaps it's time to move my DVB-C card to my other PC.

[mikeson]

@Wilbert: Can you please update your results with new released sh0dan's MipSmooth 1.0?

[Wilbert]

OK, I thought there was some noise added from the VHS tape itself but perhaps not.
I don't say there is no noise added at all here (I don't know how much noise is added by the VHS), but it's not the main source. Maybe you can capture both (thus capture from TV, and record on the VCR at the same time and capture later), and see if there is much difference?

The ssim plugin compares the original with the end result but if I don't have the original before the noise was added then it's impossible to test the cleaning filters used when trying to restore the original picture.
I would say it's complicating since we don't have the original source. But, just wait and see ...

Can you please update your results with new released sh0dan's MipSmooth 1.0?
I will do that!

[Manao]

On the test I did, the SSIM value obtained with Mipsmooth was lower ( 72.5 instead of 74 with Convolution3D, look here (http://forum.doom9.org/showthread.php?s=&threadid=63062) ). I used : MipSmooth(preset="MovieHQ",spatial = 2,spatial_chroma = 3,scalefactor = 0.9)However I made a visual inspection, and for me, the mipsmooth clip looks better. The scenes were mipsmooth got the worst score ( compared to C3D ) were "complicated scenes" ( for example, a forest ) in which I could not find visual differences. Some scenes did seem slightly smoother ( but very slightly, and while watching frame by frame ), but I think what prevailed was less artifacts in the case of Mipsmooth. I deblocked both clip when making the comparison, but the C3D's one was still blocky ( and with more mosquito noise ). For this particular test, I may be biased, since I prefer a clip without artifacts. But I think that Mipsmooth is great, so thanks a lot sh0dan ! As for SSIM, I still like the way it behaves ( and it is still the best measurement we have ), but I think it doesn't like smoothing at all.

Does this give more weight to the fact that in Wilbert's test, C3D and TemporalSoften get higher SSIM value than the unfiltered clip ? I think so.

[digitalman]

I agree about Mipsmooth. Even though it has a lower rating, it does look so much better to the eye, with less blocking and less mosquito noise, even on clean sources. It may be slightly slower, but so worth it!!!!

[Dams]

This filter is aiming at encoding which low birate ? 500, 600 ?
Is 900 is low ? :)

[Manao]

Dams, it's depend on the resolution you'll use and of the noise of the input source. I used it on a clean source @ 700 Kbit/sec with a 640x272 clip, and it gave me better results than anything else. Now, I'm trying it on a noisy film ( Holy Grail ), with a higher resolution ( 640x352 ), at a higher bitrate ( 980 kbit/sec ). I'll tell you what are the results. But I think that the bitrate in the second case is 'lower' than in the first one, because of the increased resolution, and of the noise.

[Dams]

Okay, I understand.

The author of this post doesn't edit it at this time to add Mipsmooth 1.0 ?