Filter for eliminating DCT blocks

[SansGrip]

After some testing with my noise-producing filters (see
this thread (http://forum.doom9.org/showthread.php?s=&threadid=37003)), in particular using GaussianNoise in combination with Layer to add noise to a clip, I noticed a couple of problems. First the overall luminosity of the clip was increased (obviously as a side-effect of using Layer's "add" mode), and second it appeared to cause a very slight shift towards green. I'm not sure what might be responsible for this.

It occurred to me that in this specific application (adding noise to a clip), and given that Gaussian noise with mean=0 is both negative and positive, it makes more sense, would be more efficient, and would produce better results if one applied those values directly to a clip rather than layering them together later. So I wrote another filter called, funnily enough, AddNoise.

The reason one would want to add noise to a clip, as suggested in this thread's title, is that it can completely remove once and for all those awful DCT blocks (also known as dark blocks) that MPEG encoders enjoy liberally scattering through our otherwise-flawless encodes.

For a much more detailed rationale and usage instructions see the included documentation.

(Note to moderators: I made this a separate thread because it's a separate filter from my Noise_Generators package, and conceptually is more focused on artifact removal than general noise generation. Feel free to merge the two if you think it more appropriate.)

Edit: Removed links to old version.

[OUTPinged_]

mmm


Did a minor check and still cant get how you are getting better result with noise added.


Maybe you can post here a couple of pictures where the improvement is clearly seen?

[SansGrip]

Maybe you can post here a couple of pictures where the improvement is clearly seen?

Here's an example (http://www.jungleweb.net/~sansgrip/AddNoise.png) from American Pie (I happen to have the vobs sitting on my hard drive right now). It's a montage of four grabs of the same P-frame.

Script:

AudioDub(Mpeg2Source("ap.d2v"), WavSource("ap.wav"))
Crop(11, 8, 700, 462)
LanczosResize(352, 174)

opening = Trim(0, 533)
opening = opening.TemporalSoften(3, 8, 30)

movie = Trim(534, 131644)
movie = movie.AddNoise(yv=?, loop=4, mode="round")

credits = Trim(131645, 0)
credits = credits.Greyscale().TemporalSoften(3, 8, 30)

last = opening + movie + credits

AddBorders(0, 33, 0, 33)

LegalClip()
ConvertToRGB()

Settings:

TMPGEnc MPEG-1 352x240 500-2564kbps CQ-100

Details:

You'll want to zoom in about 300%, particularly on the expanse of wall to the right of the guy. You might want to nudge your monitor's brightness/contrast up a little if the DCT blocks aren't clearly visible.

The top-left grab had no noise applied. As you can see, the whole wall is composed entirely of DCT blocks. They're fairly subtle in this frame, but still very noticible especially on a TV.

The top-right grab had yv=1 noise applied. You should be able to see that the DCT blocks are almost completely eliminated by the added noise, with negligible effect to the rest of the frame. Personally, I think the detail elsewhere is actually enhanced, though that's a subjective thing (as poptones said here (http://arstechnica.infopop.net/OpenTopic/page?a=tpc&s=50009562&f=67909965&m=3890938134&r=5470927074), I find the undithered grab to be somewhat "cartoonish" in comparison. Incidentally, he posted some grabs in that thread that illustrate the benefit of dithering a lot more effectively than my example. I'll continue looking for a better one).

The bottom frames had yv=1.5 and yv=3 noise applied. yv=3 is definitely too strong with this source, and yv=1.5 is borderline at best.

Bear in mind that this encode is intended to be viewed on a TV. If I were encoding for monitor viewing with XviD I would likely need less strong settings. I may not need to add noise at all, though those DCT blocks do tend to crop up in dark areas.

[soulfx]

I'm going to check this out. Hehe, through all the trouble I have to remove noise from my captures now I'm going to see about adding noise.

I do know what is meant by "cartoon" looking and DCT blocks as when I get done applying the noise filters I have running on my captures I get a little bit of each. Right now all the filters are working good at keeping the noise down and detail up, but something just doesn't look right about the faces. They are too clean, almost like everyone is wearing streatched masks.

Yeah! I guess I'm the first one to give this filter a download. I'll run some test and try it out.

[SansGrip]

I'm going to check this out. Hehe, through all the trouble I have to remove noise from my captures now I'm going to see about adding noise.

I didn't believe it either until I saw the results. The good news is that adding noise is much quicker than removing it ;) :D.

Right now all the filters are working good at keeping the noise down and detail up, but something just doesn't look right about the faces. They are too clean, almost like everyone is wearing streatched masks.

Exactly. I think those "invisible" details that get smoothed away to aid compressibility aren't so unimportant after all, especially in faces and fine textures. For example, look without zooming at the cushions on the couch and the detail around the edge of the chair in the grabs I provided. In the grab with noise 1.0 I think they are sharper and (for want of a better phrase) more "three-dimensional".

It seems to me that MPEG isn't designed for very smooth sources; this is precisely the reason that cartoons can, at least in my experience, look so awful when run through the encoder. I've not tried it yet, but I have a hunch that adding some noise to a cartoon might improve that significantly.

I'll run some test and try it out.

I'll be very interested to hear your opinion :)

[Blight]

I don't see the point of adding noise prior to encoding as that just raises the bitrate.

If you want to add noise to compensate for frailties in the video-encoding, do it on the decode (i.e. see: ffdshow).

[SansGrip]

I don't see the point of adding noise prior to encoding as that just raises the bitrate.

Of course. That's what it's meant to do :).

If you want to add noise to compensate for frailties in the video-encoding, do it on the decode (i.e. see: ffdshow).

ffdshow is great, and if Panasonic would include it in my DVD player's firmware I'd stop adding noise during pre-processing ;).

[stickboy]

Originally posted by SansGrip
ffdshow is great, and if Panasonic would include it in my DVD player's firmware I'd stop adding noise during pre-processing ;).What if you use crappy cables? :D

[Xenoproctologist]

Hmm...the next step here would be to create an inverted unsharp mask and use it to weight the noise so it's only added to areas with flat shading.

[SansGrip]

What if you use crappy cables? :D

That's a good idea, but I think constantly switching back and forth depending on whether I'm watching a DVD or VCD would be bad for the connectors :D ;).

[SansGrip]

Hmm...the next step here would be to create an inverted unsharp mask and use it to weight the noise so it's only added to areas with flat shading.

Funnily enough I was just reading up on spatial frequency, because it occurred to me that the problem is worst in areas of low spatial frequency (i.e. flat areas with little detail).

I've not come across any comprehensible (to me anyway) explanation of it, so I was just about to ask in General Discussion :).

[Xenoproctologist]

Hmm...I just had a thought...

What if the noise were only added to the pixels on the edge of the macroblocks? Would this be enough for mpeg4 codecs to latch on to, while having the added benefits of (1) consuming less bandwidth than full-frame noise and (2) having the noise be smoothed away during deblocking?

[SansGrip]

What if the noise were only added to the pixels on the edge of the macroblocks?

If someone (maybe me -- I'll think about it ;)) could come up with an effective algorithm for detecting just DCT blocks, then this would probably help with re-encoding MPEG sources.

However bear in mind that almost all the DCT blocks you see in the grabs I posted above were generated by the second encode (that is, from MPEG-2 to MPEG-1). Since those DCT blocks don't exist yet it would be hard to blend them :).

[SansGrip]

I was thinking about the spatial frequency thing and I remembered an experimental filter a wrote quite some time ago to do "detail detection" on a block by block basis. It wasn't very successful but I decided to take another look at the code and see if I could resurrect it.

After some playing I spotted what was wrong with it and, to my surprise, it actually works really quite well. Here (http://www.jungleweb.net/~sansgrip/AddNoiseShow.png) is a shot of it in "show" mode displaying where the new version will add noise to the same frame I used to illustrate the problem in my post above.

You can see that it thinks there's quite a lot of areas where there's insufficient detail (this is of course configurable; I used detail_threshold=12), particularly that wall which we know gets totally splattered with DCT blocks when there's no noise added.

I've done some frame-by-frame comparisons and it seems where the filter detects insufficient noise is almost always exactly where one gets DCT blocks occurring in the final encode, including dark areas (lair of the evil dark blocks).

Anyway, play around with it (and look at the source if you like). I consider this to be rather a good candidate for a compromise between the yes-add-noise and what-are-you-nuts camps ;).

Edit: Removes links to old version.

[MaTTeR]

SansGrip,

Looking at your screenshot, it seems like the filter is definitely displaying the areas where noise would need to be added based on my experience. So to me this looks very promising and I'm anxious to do some testing in about 12hrs or so. I've been fighting the DCT blocks on stable backgrounds for sometime now and I'm glad to see your effort in the resolution. Thx!

[Xenoproctologist]

Originally posted by SansGrip
What if the noise were only added to the pixels on the edge of the macroblocks?

If someone (maybe me -- I'll think about it ;)) could come up with an effective algorithm for detecting just DCT blocks, then this would probably help with re-encoding MPEG sources.
Pre-existing macroblocks are irrelevant, as what you're trying to do here is prevent macroblocking in the current encode. Macroblock size and offset would have to be passed to the filter as parameters.

Also, the current spatial detail detection could result in some thresholding artifacts in very clean sources. You might want to consider fading the edges of the noise out.

[Didée]

Originally posted by Xenoproctologist
You might want to consider fading the edges of the noise out.
This is true for almost every filter that processes only a part of the frame!

Anyway, what about this:
IMHO, the blocking in flat areas is caused by the high quant coefficients the codec decides to use *because* of the little detail in the respective area.
So, adding noise seems to be mainly needed when encoding to mpeg-1 - when encoding to XviD, the usage of a custom quantizer matrix with less agressive coefficients for the low frequencies should fight this blocking problem as well.

?

[SansGrip]

it seems like the filter is definitely displaying the areas where noise would need to be added based on my experience

It does seem that way, which is surprising since the algorithm is so simple. I wasn't expecting it to work so well. But it needs testing on a variety of sources before I'll get too excited ;).

I'm anxious to do some testing in about 12hrs or so.

I'll be very interested to hear your results.

I've been fighting the DCT blocks on stable backgrounds for sometime now

What version of MPEG do you encode to?

[SansGrip]

Macroblock size and offset would have to be passed to the filter as parameters.

Well AFAIK the offset should always be 0,0 and the DCT blocks always 8,8 for MPEG. But someone might like to correct me on this. I think it's still an interesting idea that I'm going to play with some.

Also, the current spatial detail detection could result in some thresholding artifacts in very clean sources. You might want to consider fading the edges of the noise out.

I was thinking the same thing after writing the "show" code and before I'd finished the actual noising part, but with reasonable means and variances (i.e. those most people would need to use) it's very difficult to spot even when you look real close. I noticed it a couple of times over the course of an hour's testing, but it was only where I already knew (from "show") that a block of noise would appear in an area.

Of course if you jack up the variance to something unrealistic (say, >30) then it starts getting noticible, but I would say that's enough of a noise difference that feathering wouldn't really help.

[SansGrip]

IMHO, the blocking in flat areas is caused by the high quant coefficients the codec decides to use *because* of the little detail in the respective area.

I've not done any testing, but my gut says this is exactly the cause.

So, adding noise seems to be mainly needed when encoding to mpeg-1

And that's what I encode to almost exclusively, hence the filter ;).

when encoding to XviD, the usage of a custom quantizer matrix with less agressive coefficients for the low frequencies should fight this blocking problem as well.

There's a definite chance that would work too. While the docs I've read on MPEG encoding say that it compresses more in high frequency areas, I'd say based on my experience (certainly with MPEG-1 at all bitrates and with MPEG-2 at lower bitrates) the default matrix is too aggressive on very low frequency areas.

I've only done a couple of XviD encodes (generally when I've captured something to test my capture card) and got weird results so I've not really played with it much, but this now has me wondering about the Quantize Matrix section of the TMPGEnc settings... :)

Edit: It seems TMPGEnc's default matrix is far too aggressive with low frequency areas. The "MPEG standard" matrix is significantly better at reducing DCT blocks (except in I-frames, which seems to be unfixable), and there are other non-standard matrices that perform even better. It still can use a little dithering, but nowhere near as much as with the default matrix.

[MaTTeR]

Originally posted by SansGrip
at version of MPEG do you encode to?

90% of the time I'm using MPEG-4/XviD and the other 10% I'm using TMPEG/MPEG-1. Watching the MPEG-1 encodes on my standalone player, well I never see any DCT blocks at all, not even on the stable backgrounds such as walls. However, for MPEG-1 I'm also using the excellent KVCD/XVCD templates which IMHO give excellent results but that's another topic.

[SansGrip]

Watching the MPEG-1 encodes on my standalone player, well I never see any DCT blocks at all, not even on the stable backgrounds such as walls.

This could also be due to contrast/brightness/sharpness settings on your TV as well as the ambient light levels when you watch it. Is your TV calibrated?

However, for MPEG-1 I'm also using the excellent KVCD/XVCD templates which IMHO give excellent results but that's another topic.

Hmmm, I've not heard of those. Where did you find them? Edit: Never mind. www.kvcd.net ;).

[MaTTeR]

Originally posted by SansGrip
Is your TV calibrated?

Yes, I calibrated it about a month ago when it was first bought. In terms of brightness/contrast, it's somewhat close to my monitor. The only difference I can think of is that my standalone is using component outputs which hides the blocks somewhat. I can take the same video, play it from HTPC via S-Video/GeForce2 and the blocks can be seen easily.

I'm going to start a two pass rip on a pretty clean DVD source using AddNoise. This source is pretty sharp and cleaner than most DVD's but does have some smoke/fog scenes which loves to look blocky after encoding. I'd like to use the LegalClip filter along with AddNoise2 with a conservative variable. Would this script be a nice starting point?

LoadPlugin("C:\Program Files\AviSynth2\plugins\MPEG2DEC3.dll")
mpeg2source("D:\DIvX RIPs\Romeo is Bleeding\romeo.d2v",cpu=1,moderate_h=45,moderate_v=65,lumoff=-2,iDCT=2).crop(8,4,704,464).GrapeSmoother(10).UnFilter(3,3).AddNoise(yv=1,show=false,fast=false,block_size=16,loop=4).BicubicResize(624,336,0.333,0.333).LegalClip()

Hmm...maybe a bit too much filtering.

[SansGrip]

In terms of brightness/contrast, it's somewhat close to my monitor.

Then I would suggest your monitor is too bright or your TV is too dark ;). They shouldn't really be the same. The monitor should be darker.

The only difference I can think of is that my standalone is using component outputs which hides the blocks somewhat.

Wow, wish I had component in/out on my stuff :).

...your script...

It looks ok, except I would say that you might not need lumoff=-2 and UnFilter if you use AddNoise. I've always thought the lumoff/UnFilter combination to be something of a hack. It's a clever hack that seems to work, but why? No-one's really pinned that down yet. Also it'll change your levels slightly.

As for AddNoise, I'd say try a short clip with various settings before deciding what you want to use. For example with the TMPGEnc default matrix I would use 0.75 with American Pie, but with the new matrix I just made (and also with the MPEG standard one) I'd use 0.3 instead. (You don't need yv=1 because 1 is the default variance for all components.)

Also try with mode="round" and mode="floor". The former will add slightly more noise because it rounds to the nearest integer instead of towards zero.

I always use loop=5, since it seems to work just as well as generating new noise for each frame, at least for this purpose.

Finally, fiddle with the detail_threshold with show=true to make sure it's in line with your thinking on what already has enough detail. 12 seems a reasonable default but I only tested it with a couple of sources.

Apart from that, there's really nothing to tweak ;) :D.

Oh, and WRT LegalClip, you can include it if you like. It all depends on what XviD does with out-of-range values.

[MaTTeR]

Thanks for the input but right now it seems to be a bit of a struggle to find good values. Guess it's going to take awhile for me to get some conclusive results.

I haven't figured out what's causing it but my speed has dropped from 62FPS using YV12 and no filtering to a whopping 10FPS with AddNoise(yv=0.5,fast=true),GrapeSmoother(8) & LegalClip() in YUY2 mode of course. I'm thinking LegalClip couldn't cause that much of a speed penalty or am I wrong? It must be AddNoise which it made it drop so much.

So far I played around with detail_threshold and it's resulted in the file being way oversized. Lots of variables to play with here:D

Edit- Yep, my monitor is darker but only slightly. Since I just recently switched over to component cables, I can honestly say they are well worth the $80 I paid for them. The colors are so clean and the details are superb when viewing a clean source such as a SuperBit DVD.

Edit2- I've taken Unfilter and lumoff=-2 out of the script for now and doing no preprocessing with MarcFD's filter. In regards to LegalClip, I have no idea how XviD handles this but I'm using it now to see the difference if any.

[SansGrip]

I haven't figured out what's causing it but my speed has dropped from 62FPS using YV12 and no filtering to a whopping 10FPS with AddNoise(yv=0.5,fast=true),GrapeSmoother(8) & LegalClip() in YUY2 mode of course.

The culprit (or one of them) is AddNoise. You'll definitely want to use the loop parameter to pre-generate and loop over frames of noise instead of generating new noise for each frame. Making Gaussian noise is very time-consuming and the code is unoptimized.

Benchmark the script without AddNoise and then with AddNoise(..., loop=x, ...). You shouldn't see a big difference.

I'll be interested to hear the result.

So far I played around with detail_threshold and it's resulted in the file being way oversized. Lots of variables to play with here:D

Yep, and there might be more in the future ;). I was thinking of adding weighting to the noise based on the detail level, and the option of only noising every nth frame. I'm also going to investigate Xenoproctologist's suggestion of noising only at DCT block boundaries.

As far as finding good values for a particular clip, I would suggest first turning on show mode and setting detail_threshold to something that looks right to you (in other words make sure it's highlighting only areas you think need more noise). Then turn off show mode and set the variances to, say, 1.0. This might be too much, so decrease it by say 0.2 each time until you start seeing unacceptable DCT blocks in low frequency areas. This should give you an idea of the minimum amount of noise necessary for a particular clip.

Edit: I forgot to mention there's plenty of other things to try out, such as setting it to noise the luma very gently and the chroma stronger (i.e. set yv to something like 0.25 and uv and vv to something in the range of 0.75-3). This doesn't seem to work too well with MPEG-1, but MPEG-4 is a different beast.

Another thing you can try is smoothing before or after adding noise. I'm guessing the latter would tend to negate the benefit of dithering in the first place, but it's something I've not got around to testing yet. Yet another possibility is noising the whole frame (detail_threshold=0) and then smoothing. I don't know what the result will be, but you never know :).

You can also set loop=1 which will produce a single frame of noise that is added to every frame. This is in effect spatial noise as opposed to regular spatiotemporal noise. At high settings it looks rather like you're watching it through a very speckled window, but at low settings it might reduce the DCT blocks without being so noticible to the eye. I don't know, I've not tried it yet. Let me know ;).

[MaTTeR]

Originally posted by SansGrip
The culprit (or one of them) is AddNoise. You'll definitely want to use the [i]loop parameter to pre-generate and loop over frames of noise instead of generating new noise for each frame.

Well the loop parameter definitely fixed the speed issue. I lost about 4hrs of downtime last night due to avisynth acting strangely but finally got it sorted out this morning with the help of some coffee:)

I've tested alot of different settings on this clean source DVD and I'm still not satisifed with the results that the UnFilter(x,x) and Lumoff=-x givves me. So far I've had to find a middle of the road setting that doesnt cause the codec to eat excess bitrate, in doing so the DCT blocks aren't completely disappearing but I did notice an improvement. Here's what I'm currently using and will continue to tweak-
LoadPlugin("C:\Program Files\AviSynth2\plugins\MPEG2DEC3.dll")
mpeg2source("D:\DIvX RIPs\Romeo is Bleeding\test\romeo.d2v",iDCT=2)
crop(8,4,704,464)
GrapeSmoother(8)
AddNoise(yv=0.25,vv=0.65,fast=true,loop=5,detail_threshold=4)
BicubicResize(624,336,0.333,0.333)
LegalClip()

I like where I have the detail_threshold set at but I'm thinking I need to possibly play with the yv,vv and block_size parameters. Have you seen how much varying the block_size affects the encode? I'm thinking of playing around with it and trying the floor/round modes as well.

[SansGrip]

So far I've had to find a middle of the road setting that doesnt cause the codec to eat excess bitrate, in doing so the DCT blocks aren't completely disappearing but I did notice an improvement.

Of course there's always going to be a path to be struck between bitrate and DCT blocks, especially using this filter. I'm beginning to think that AddNoise achieves much better results with MPEG-1 than with MPEG-4.

The question now is, what exactly is the lumoff/unfilter combo doing to the clip? I think I shall make it my mission today to find out ;).

I like where I have the detail_threshold set at but I'm thinking I need to possibly play with the yv,vv and block_size parameters.

I think the key to it is finding the right variances for the clip. 0.25 in floor mode isn't very much, so I'm not surprised you're still seeing some blocks. Look at it this way:

With a variance of 0.25, 68% of the values generated will be within -0.25 and 0.25. 95% will be within -0.5 and 0.5, and 99% will be within -0.75 and 0.75. The closest-to-zero number that produces a change in the pixel is, obviously, -1.0 or 1.0, which is four standard deviations away from zero when variance=0.25. The chance of getting a -1.0 or 1.0 is very small, and something like 99.9999% of the numbers you're generating will result in a zero, or no change to the pixel.

If you were to use mode=round, though, the closest-to-zero number that affects the pixel is 0.5. Since 0.5 is only 2 standard deviations away from the mean of zero, it's much more likely to occur. Indeed, about 5% of the values you generate should be <= -0.5 or >= 0.5.

As you can see, when using small variances the rounding mode can make a big difference :).

You could also try a higher luma variance with loop=1 and see what you think of the effect. Or try just greyscale noise, with yv=whatever, uv=0, vv=0.

Have you seen how much varying the block_size affects the encode?

To be honest I've never tried. I'd be interested to hear your conclusions.

[iago]

The question now is, what exactly is the lumoff/unfilter combo doing to the clip? I think I shall make it my mission today to find out ;).

:D LOL!

Good luck, pal! I'm sure it "does" something "good", though I -like everyone else- don't know what the real magic is! ;)

Sorry for shifting off-topic, but I just couldn't resist! ;)

kind regards,
iago

[MaTTeR]

The only reasoning I've come up with for the nice results of the Unfilter/Lumoff combo is that UnFilter is doing the majority of the work. With a setting of UnFilter(4,4) for example, maybe it slightly sharpens the existing noise forcing the encoder to add a few more bits to those areas which essentially is what were also trying to do with AddNoise(xx). The only problem with my theory is that I think UnFilter is more designed for sharpening object edges instead of so called mosquito noise:)

Whatever the reasoning, for MPEG-4 it's the best solution I've used up until this point but more testing continues.

[iago]

@MaTTeR

When I first encountered the magic UnFilter does together with lumoff=-2 in one of my test encodes, I also thought just like you. That is, I concluded that the key point is sharpening with UnFilter(+,+). But then, when I discovered that UnFilter with (-,-) parameters is also as effective as (+,+) parameters against black-blocking when used with lumoff=-2, this reasoning, which sounded quite reasonable first, simply collapsed! ;)

best regards,
iago

[SansGrip]

Whatever the reasoning, for MPEG-4 it's the best solution I've used up until this point but more testing continues.

Does it get rid of all DCT blocks, or just those in dark areas? I tried UnFilter(5, 5) with an MPEG-1 clip and it did work somewhat against DCT blocks in "non-dark" areas, but not as effectively as AddNoise (for me anyway).

It also produced, as it's supposed to, a general sharpening that seemed to encourage mosquito noise. Perhaps the ideal solution might be to implement lumoff=-a and an unsharp mask only in the areas lacking detail...?

[SansGrip]

But then, when I discovered that UnFilter with (-,-) parameters is also as effective as (+,+) parameters against black-blocking when used with lumoff=-2, this reasoning, which sounded quite reasonable first, simply collapsed! ;)

That is weird. I can see lumoff=-2 helping in that, perhaps, superblack compression artifacts might be shifted out of the range where they get noticed by the encoder, but I have no idea why smoothing would help if the problem really is a lack of detail.

I'm still pretty confused by this whole thing. By its nature MPEG should compress low-frequency areas much less than high-frequency ones, so why it's only low-frequency areas that exhibit DCT blocks is anyone's guess.

According to this, smoothing the low-frequency areas (thus reducing the frequency even more) should improve the blocking since the encoder should compress those areas even less. But test results do not bear this out. If anything, smoothing low-frequency areas makes the problem worse, while adding noise (increasing the frequency) makes them go away.

One possibility is that higher frequency areas get allocated significantly more bits and thus, even with the higher levels of compression, do not suffer from this problem.

Any MPEG experts out there?? ;)

[MaTTeR]

Originally posted by SansGrip
[i]Perhaps the ideal solution might be to implement lumoff=-a and an unsharp mask only in the areas lacking detail...? You read my mind man*evil grin*:D

The detection algo your using in AddNoise seems to pick out the problematic parts such as walls, doors and in general....stable backgrounds. Those areas are where we usually see the blocks floating around that distract us from what the main object of the frame is suppose to be, say for instance a persons face. Combining your detection scheme with an UnSharp Masking filter might be somewhat of a silver bullet, at least it certainly sounds good when I think about it;) How difficult would such a beast be to code? I'm not a coder so I'm just curious.

Edit- iago is right though, even using (-x,-x) with UnFilter still reduces the blocking problem:confused:

[SansGrip]

The detection algo your using in AddNoise seems to pick out the problematic parts such as walls, doors and in general....stable backgrounds.

Pretty cool considering all it does is count unique luma values ;).

Combining your detection scheme with an UnSharp Masking filter might be somewhat of a silver bullet, at least it certainly sounds good when I think about it;)

I was thinking the same thing. There shouldn't be any of the problems normally associated with sharpening such as mosquito noise and ringing, since those are high-frequency issues.

How difficult would such a beast be to code? I'm not a coder so I'm just curious.

I have a vague idea how unsharp mask works (first you produce a blurred version of the image then use that to make the sharpened version) but I've not looked into it in any depth. It would be slower than AddNoise with loop=x though.

I will ask Dr Google right now and try to get something implemented tonight. I think I'll start by adding it as an alternative "mode" (though I've already used that parameter name, dammit ;)) for AddNoise. If it does turn out to be the "silver bullet" then maybe I'll make it a separate filter.

Edit: After looking at UnFilter more closely it seems it's not an unsharp mask after all, but (as the name suggests, now I think about it) a way of compensating for common filtering techniques that might have been applied to the source.

Unfortunately I don't believe I can use Tom's code directly in my filter because his is released under the GPL and mine is not (I might be wrong here though). For the moment I'll work on adding an unsharp mask for y'all to play with :).

iago is right though, even using (-x,-x) with UnFilter still reduces the blocking problem:confused:

Over the whole frame, or just in dark areas?

[MaTTeR]

Originally posted by SansGrip
It would be slower than AddNoise with loop=x though. Well if that's the price we have to pay then so be it :-)

I will ask Dr Google right now and try to get something implemented tonight.That would be fantastic, I'll be sure to monitor the thread. In regards to the name, well you could always name it "Blockbuster". lol

Over the whole frame, or just in dark areas? Sorry I missed the first time you asked this. Well since I mainly see the blocks only in dark areas anyways then that's where I see them disappear after the UnFilter combo. Maybe I only see them in darker to mid-tone areas because I'm encoding at higher bitrates than others do, not sure:rolleyes:

I'm not so sure that UnSharp masking would need to be used in this filter your writing though. I'm thinking that even a "generic" type of sharpening might even work. I mean after all, the parts that were mainly interested in sharpening have very little detail which is why the encoder somewhat ignores them. It seems a more general sharpening might be faster than UnSharp but I'm only speaking from my old Photoshop(2D) experience. What are your thoughts on this?

Edit-
Yep that's what I was thinking about UnFilter, I think "msharpen" might be the UnSharp masking filter. Tom's filter seems to focus more on the edge detection of frames itself. I think it's specifically to get rid of the nasty "EE" effects from poorly filtered sources such as the crappy "Born on the Fourth of July" DVD. Great movie ruined by EE:devil:

[SansGrip]

In regards to the name, well you could always name it "Blockbuster". lol

hehehe I like it ;).

Well since I mainly see the blocks only in dark areas anyways then that's where I see them disappear after the UnFilter combo.

I would say that MPEG-4 has pretty much solved the DCT block issue except in darker areas. That's why I'm getting better results with AddNoise than you guys, since I'm using MPEG-1 with puts them all over the place (as does MPEG-2 to a certain extent).

It seems a more general sharpening might be faster than UnSharp but I'm only speaking from my old Photoshop(2D) experience. What are your thoughts on this?

I think you're referring to something like the built-in Sharpen filter, which IIRC uses a simple 3x3 matrix. I guess the speed difference would depend on how the blurring was implemented with an unsharp mask. If a 3x3 blur was used the speed difference wouldn't be that much, but if implemented as a Gaussian blur it would be quite a lot slower. Either that or I'm talking nonsense ;). I'll get back to you when I've done some actual research...

Edit: Stop editing in response to the edit I wrote while you were writing, while I'm writing! :D

I don't know if MSharpen uses an unsharp mask. It definitely is edge-constrained though.

[MaTTeR]

Hmm...I'm having second thoughts about the initial idea:D

Personally I'd much rather blur the areas that the detection algo has selected rather than sharpen them for obvious bitrate savings. This is where UnFilter has come in to use I guess because we can either sharpen or blur depending on the source material.

In regards to Gaussian blurring with UM, would we really need that process? I guess my thinking is that even a 3x3 average when blurring or sharpening might suffice for what were trying to accomplish. In my thinking, the most difficult part is successful and that's the detection algo that you implemented.

BTW- Do you have Blockbuster chain reantal stores in Ontario? They are on every street here in the states. That name would be real fitting for this type filter.

Edit- (yet again):D
I mispoke earlier, Donald's "msharpen" filter is not using UM. Have a look here (http://forum.doom9.org/showthread.php?s=&threadid=32757&highlight=unsharp) for more info.

[trbarry]

Tom's filter seems to focus more on the edge detection of frames itself. I think it's specifically to get rid of the nasty "EE" effects from poorly filtered sources such as the crappy "Born on the Fourth of July" DVD. Great movie ruined by EE

UnFilter just approximates an inverse of a simple 3 tap sharpen or soften filter. It doesn't look for edges or try to guess what convolution was previously used. It's just optimistic, figuring the original encoders were lazy and the user is smart enough to set the strength. That is, if it causes artifacts then back it off a bit. ;)

And I still have no idea why it helps the blocking problem. It's probably just a serendipitous bug that cancels one in Xvid. But it makes me afraid to ever change it. It will be interesting to see if the YV12 version still helps. :devil:

- Tom

[Kaiousama]

Thanks Great filter Sansgrip ! :D
Two considerations:
1) it's slow (but this is a secondary problem ;)
2) I'm testing it on anime (btw argento soma :P) and if i've well understood the filter apply the noise to the green areas leaving unchanged the non green areas according to the detail_threshold bcause in that area (the non green ones) the detail is higher than that value.... looking at the attached picture you'll see that the non-noised (non green) areas in this way are the main edges' ones, and irony of the fate are the main parts that in anime are affected by dct artifacts (because the main edges in the scene are like a wavefront when the motion starts), request: can you insert an option in the filter to exactly invert the assegnation of green and non-green (that will be noised & non-noised) areas, so in that way the noise will be added only to the edges&near-edges area (giving a better DCTproblems reduction & better quality in the parts that are not affected by DCTartifacts)? Many thanks! Bye

[edit] i've tryied to attach an image but probably it needs a bit to be authenticated ^_^

[SansGrip]

1) it's slow (but this is a secondary problem ;)

Well, it's very slow when generating new noise for each frame. But with the loop option it shouldn't be too bad. It's all still unoptimized of course.

if i've well understood the filter apply the noise to the green areas leaving unchanged the non green areas according to the detail_threshold

That's right. In show mode areas with insufficient detail (that will have "detail" added) are highlighted in green.

the main parts that in anime are affected by dct artifacts (because the main edges in the scene are like a wavefront when the motion starts)

That's interesting, I didn't know this. To be honest I'm surprised that you get DCT blocks in high-detail (high-frequency) areas, but while I've never tried it myself I know anime encoding is a different beast entirely from live action :).

can you insert an option in the filter to exactly invert the assegnation of green and non-green (that will be noised & non-noised) areas

It will be in the next release, which will also include 3x3 sharpening and blurring. I've written the code, I just need to tidy it up :).

[Kaiousama]

That's right. In show mode areas with insufficient detail (that will have "detail" added) are highlighted in green.
is it possible to make the noised areas( now higlighted in green) highlighted in an alpha blended colour (the same green for instance); so we can see what's under the green to better decide if that part has to be noised or if it's better to reduce the noise threseold?

That's interesting, I didn't know this. To be honest I'm surprised that you get DCT blocks in high-detail (high-frequency) areas, but while I've never tried it myself I know anime encoding is a different beast entirely from live action :).
It's difficult to say with words (mainly for a -bad english speaking- italian ^_^) but the issues in anime are mainly two (in my opinion): the artifacts (blocks) created in an overall fast scene movement (in the sliding or zooming ones) and the artifact created near the edges in movement (slow or fast, it doesn't matter); this second one is the most eyes' sensed problem, i think it's more important in anime than movies because anime has definite and massive edges for each thing, while in reality movies the edges are less focused by eyes (because in proportion to the rest of the picture are only a little and trascurable part)

It will be in the next release, which will also include 3x3 sharpening and blurring. I've written the code, I just need to tidy it up :).
faboulous, amazing, wow!!
i'm thinking: can you code a parameter to decide what kind of noise strenght and blurring effect add separately to the -over the noise threseold- parts and other values to the -under the noise threseold- parts ?

You are doing a great work... Thanks:D

p.s. a little curiosity: why only 3x3 blur?isn't there a way to implement variable diameter blur (it would be very useful to have a high diameter blur in the colour uniform areas and a little one in the detailed parts)?

[SansGrip]

is it possible to make the noised areas( now higlighted in green) highlighted in an alpha blended colour

I already fixed that yesterday. Affected blocks are now outlined in grey and the contents are unaltered.

It's difficult to say with words (mainly for a -bad english speaking- italian ^_^)

Don't worry, your English is molto bene -- definitely much better than my Italian which I "studied" for three weeks ten years ago, both for the pure love of learning and because the two best-looking girls in the school were also in the class :D.

can you code a parameter to decide what kind of noise strenght and blurring effect add separately to the -over the noise threseold- parts and other values to the -under the noise threseold- parts ?

That would create an awful lot of parameters, at least for method="noise", and I'm not sure what benefit there would be in adding noise to areas already sufficiently detailed. What did you have in mind?

why only 3x3 blur?isn't there a way to implement variable diameter blur (it would be very useful to have a high diameter blur in the colour uniform areas and a little one in the detailed parts)?

Because it's fast and easy ;). I agree, though, that while a 3x3 sharpen is more than strong enough, a 3x3 blur isn't very effective. I'm still doubtful, though, as to the positive impact of reducing the frequency in already low-frequency areas.

I just had a thought that is sort-of connected to this reply but would work better as a separate reply for people who haven't read this far ;). I'll write that in a moment.

P.S. As to why the new release isn't out yet, I took some time to (algorithmically) optimize the sharpen and blur methods and gained several fps in the process. I also did a pretty major structural overhaul to make the code more modular and easier to maintain, and managed to (entirely accidentally) gain another few fps. I still have to add the noise generation code back into the restructured version, but that won't take long. Then I have to update the docs. Then I'll release it :).

[SansGrip]

Before I forget it, here's what popped into my head while I was writing the last reply...

As I said I'm not totally sure of the benefits of blurring low-frequency areas (except for extra compressibility), but I can see the benefit of being able to apply separately different filters to low- and high-frequency parts of the frame.

One thing I don't want AddNoise/Blockbuster to become is a filter that tries to do everything and please everyone. I conceived it as a way to eliminate DCT blocks and want it to stay focussed on that job.

For this reason, might it not be a good idea to write something along the lines of:

DetailMixer(threshold=x, low_filter=UnFilter(...), high_filter=SmoothHiQ(...))

where the output of low_filter is used for low-frequency areas and the output of high_filter is used for high-frequency areas and then blended together?

Just a thought.

[Kaiousama]

can you code a parameter to decide what kind of noise strenght and blurring effect add separately to the -over the noise threseold- parts and other values to the -under the noise threseold- parts ?
That would create an awful lot of parameters, at least for method="noise", and I'm not sure what benefit there would be in adding noise to areas already sufficiently detailed.
i think 2 more options are sufficient : a highFreq_strenght and a lowFreq_strenght, and the entropy doesn't become like in my bedroom :D

What did you have in mind?
Simple... maybe your filter can reduce even the mosquito effect,
think: the mosquito effect borns (and it's more visible) in the detailed part of the movie that are moving, think of it as an expansion of the block concept but inverted.
I mean: in high frequency areas, a movement create a zone of temporaneal low quality, the codec has to analyze by macroblocks and zzap! in this macroblock the innocent watcher see a mosquito that surrounds the movement of the edge.
Now, what if in this high_freq moving zones there is some noise that make them more like a uniform moving zone instead of a -surrounded by high frequency edge- moving zone? (yes, the detail goes a little down but remember, this zone is in movement, so a little noise is hard to see with eyes).
Maybe even this bad effect can be attenuated in one shoot ;) (what do you think? i think the best way is to make some compression tests on difficult movie and anime parts)

P.S. As to why the new release isn't out yet, I took some time to (algorithmically) optimize the sharpen and blur methods and gained several fps in the process. I also did a pretty major structural overhaul to make the code more modular and easier to maintain, and managed to (entirely accidentally) gain another few fps. I still have to add the noise generation code back into the restructured version, but that won't take long. Then I have to update the docs. Then I'll release it .
Don't worry for the release... i think is better to have a big changed release (with documentation) instead of 3-4 little releases in a day made in hurry ;)

I already fixed that yesterday. Affected blocks are now outlined in grey and the contents are unaltered.
your solution is even better than the alphablended one, good idea

Don't worry, your English is molto bene -- definitely much better than my Italian which I "studied" for three weeks ten years ago, both for the pure love of learning and because the two best-looking girls in the school were also in the class
Knowledge & women .... the two real people's drugs :D :D (glad to party the new upcoming release with a good glass of chianti :sly: )

Goodnight

[SansGrip]

i think 2 more options are sufficient : a highFreq_strenght and a lowFreq_strenght, and the entropy doesn't become like in my bedroom :D

That's a possibility. But I'm still not convinced about increasing the frequency of high-frequency areas in an attempt to reduce artifacts :).

maybe your filter can reduce even the mosquito effect,
think

I've not tried it, but I doubt it very much. Mosquito noise is a high-frequency artifact, which is why you only see it around edges etc.. Increasing the frequency of those areas will almost certainly make the problem worse, not better. Smoothing (i.e. reducing the frequency) in those areas might work, though.

Knowledge & women .... the two real people's drugs :D :D

hehehe yes, and now I know a handful of Italian words that I wouldn't otherwise. Didn't get anywhere with the girls, though ;).

[High Speed Dubb]

Sansgrip,
I take it you&rsquo;ve figured out what a variance is. ;)

Here&rsquo;s a speed trick you could use. A normal distribution can be closely approximated as a binomial. That can let you replace a slow normally distributed random number generator with a few draws from a uniform distribution.

In other words, get n uniform[0,1] random numbers (i.e., the usual random function), add them together, subtract (n/2), and multiply by a number chosen to get you the desired variance. The bigger then n, the more closely this will resemble a normal distribution. 4 is probably a reasonable n to use.

[SansGrip]

I take it you&rsquo;ve figured out what a variance is. ;)

heheh yes, it took me a while but I got there. I still have a bunch of image processing-related PDF's lying around that I wish I could understand though ;).

In other words, get n uniform[0,1] random numbers (i.e., the usual random function), add them together, subtract (n/2), and multiply by a number chosen to get you the desired variance. The bigger then n, the more closely this will resemble a normal distribution. 4 is probably a reasonable n to use.

How would one normalize this to zero mean unit variance?

I read about this technique (with a recommended n of 12) on Usenet but whoever was writing said it's a fairly poor way of getting a normal distribution. Probably for this filter it would be more than adequate, though.

The speed of the Gaussian PRNG isn't really an issue any more, since I now pre-generate and loop over the cached noise. I also have a trick for the new version which dramatically reduces memory requirements (I make one-quarter-sized frames of noise, and cycle through the cache for each quarter for each component).

[High Speed Dubb]

Subtracting (n/2) gets you a 0 mean. If you need to adjust to variance = 1.0, try multiplying the output by sqrt(12/n). (Maybe that&rsquo;s where the poster came up with an n of 12?) n=4 was my rating of &ldquo;looks pretty much like a normal,&rdquo; since that&rsquo;s the goal, here.

[Kaiousama]

For this reason, might it not be a good idea to write something along the lines of:
DetailMixer(threshold=x, low_filter=UnFilter(...), high_filter=SmoothHiQ(...))
where the output of low_filter is used for low-frequency areas and the output of high_filter is used for high-frequency areas and then blended together?

mhhh.... this is a great idea, because let everyone apply the better filter combination for each case :eek: (i'm imploring to implement this option, it would be too much useful :p :D)