Help to remove the blocks

[shon3i]

@Morte66, thanks for your answers.

Deband solves the problem, blocking is gone, it is good for my taste.

I will try M4G script, if this give me much better result than, than speed isn't important to me, i am suprised with speed of processing with fft3dgpu on my system comparing to fft3dfilter because is much faster, i have ati x800xt, and i can't imagine why i can't get better quality when remove noise.

Did you have some experience with AQ/Psy options, now elecard have very complex options, i tested something, but i can't find proper settings because have three levels of complexity, and all of that three can be in range -100, 100. I am very confused about that?

[Morte66]

@Morte66, thanks for your answers.

I will try M4G script, if this give me much better result than, than speed isn't important to me, i am suprised with speed of processing with fft3dgpu on my system comparing to fft3dfilter because is much faster

I should be clearer: fft3dgpu may be faster than fft3dfilter on its own, but the whole M4G script is not much faster because fft3d is only a part of it. I guess all that motion vector stuff is eating lots of CPU too.

Did you have some experience with AQ/Psy options

Not a lot. I was happy about blocking in x264 after I started using M4G's matrices (the remaining problems are in the source MPEG2). AQ came along later, when I didn't really have a x264 problem, and when I'd finished my big (500 DVD) encoding project. So I didn't look at it very hard. I might try combinations AQ and no-dct-decimate some time soon.

I've only used Psy in Nero. It was nothing special.

[shon3i]

I should be clearer: fft3dgpu may be faster than fft3dfilter on its own, but the whole M4G script is not much faster because fft3d is only a part of it. I guess all that motion vector stuff is eating lots of CPU too.
Uh, i just tested the script and it is very slow, i think is slower than TimeDenoise(), aslo i didn't suprise with quality because isn't much better, it reduce bloks, but not much. I suppose that my bitrate is too low.

How you work your films, 1CD rip?

[Morte66]

Uh, i just tested the script and it is very slow, i think is slower than TimeDenoise(), aslo i didn't suprise with quality because isn't much better, it reduce bloks, but not much. I suppose that my bitrate is too low.

OK, for me it didn't do much to the blocks directly but it made it easier for DeBand in ffdshow to process them later. But if you're happy using DeBand on its own, there is probably not much need to use the pre-process script.

How you work your films, 1CD rip?

No, I average about double your bitrate. I do crf 22 encodes with no target size, using sharp deblocking and an anti-blocking CQM with no bframe prediction. I normally use: --crf 22 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct none --filter -2,-2 --subme 6 --trellis 1 --analyse all --8x8dct --threads 2 --thread-input --cqmfile "C:\Program Files\MeGUI\extra\M4G-Smooth_V1.cfg" --progress --no-psnr --no-ssim --output "" ""

[elguaxo]

*.mp4 guy, your script is pure magic. Slow as hell, but the results are impressive. I used the same script you posted here, but I see you posted something similar here (http://forum.doom9.org/showthread.php?p=964755#post964755).

What are the differences between both scripts? I only need it for removing mpeg2 blocks from DVDs.

Does this apply to both of your scripts?
For stronger processing use higher values of thSAD and Sigma

Thanks.

[*.mp4 guy]

Does this apply to both of your scripts?
It does, but only for removing noise (and really, really bad blocking), higher values won't remove any more banding.

Otherwise the only difference between the two scripts is the masking, the script you linked two should be a little better (at removing noise, no idea about banding), but I never noticed any obviouse differences, so you should probably use the one here in this thread, which should be a little faster.

[elguaxo]

you should probably use the one here in this thread, which should be a little faster.

:eek: Well, I think I won't touch the script in the other thread.

:thanks:

[Tanma]

You can get close to removing all the mpeg2 blocks from most dvd's with this script, but its slow as hell, although it leaves detail almost completely intact.

source = last

backward_vec2 = source.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = source.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = source.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = source.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
mask = mvmask(kind=1, vectors=forward_vec1).UtoY().spline36resize(source resolution)#its very important that this line matches the resolution you are feeding the script with
smooth = source.degrainmedian(mode=3).fft3dgpu(bw=16, bh=16, bt=3, sigma=4, plane=0)
source2 = maskedmerge(source, smooth, mask)

source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
source3.gradfun2db(1.5)

Which plugins I need to load to open this script?

[elguaxo]

MVTools (http://avisynth.org.ru/mvtools/mvtools.html), fft3dGPU (http://forum.doom9.org/showthread.php?t=89941) (you can replace it with FFT3DFilter (http://avisynth.org.ru/fft3dfilter/fft3dfilter.html)), Masktools (http://manao4.free.fr/masktools-v2.0a31.zip), GradFun2DB (http://avisynth.org/warpenterprises/files/gradfun2db_25_dll_20060315.zip). And if you want to use spline36resize, then AviSynth 2.5.6 or above.

[Didée]

elguaxo: you forgot to list DegrainMedian (http://avisynth.org.ru/degrain/degrainmedian.html). ;)
And MaskTools v2 currently is at v2.0a32 (http://manao4.free.fr/masktools-v2.0a32.zip) - though it wouldn't matter for this script.

Technically, there is an error in mp4guy's script (could be it is intentional in this case ... only mp4guy can tell):

The idx value in the MVDegrain2 call should not be "1", but something else. As it is now, the processed clip "source2" is temporally filtered with 4 compensated neighbors of the not-filtered clip "source". Consequently, about ~80% of the effect of DegrainMedian().FFT3Dgpu() are discarded.
In this similar script (http://forum.doom9.org/showthread.php?p=964755#post964755), mp4guy used idx correctly.

Note that this isn't a intentional deblocking script. It's a denoising script that on the way also removes blocking.

Given that blocking is present, it's also suboptimal that the MV search is done on the raw source. If there is visible blocking, it would be better to first use a non-destructive deblocker (hint: deblock_qed), and start from there. This will improve the motion search. When MV search is done directly on a blocky source, then chances are high that the MV search will "snap in" on the source blocks, which is counterproductive for the goal of blocking removal.

[Terranigma]

Yes, all that's required is to change this line
source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
to
source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)

Or they can opt not using idx at all :p

[elguaxo]

Given that blocking is present, it's also suboptimal that the MV search is done on the raw source. If there is visible blocking, it would be better to first use a non-destructive deblocker (hint: deblock_qed), and start from there. This will improve the motion search

Thanks for the tip!

[*.mp4 guy]

Given that blocking is present, it's also suboptimal that the MV search is done on the raw source. If there is visible blocking, it would be better to first use a non-destructive deblocker (hint: deblock_qed), and start from there. This will improve the motion search. When MV search is done directly on a blocky source, then chances are high that the MV search will "snap in" on the source blocks, which is counterproductive for the goal of blocking removal.

There is no such thing as a non-destructive deblocker; however on sources with very heavy blocking, running the motion search on the deblocked source can be helpful, but will result in detail loss. Also Didée is right, the idx error was unintentional, and should be corrected.

[Didée]

There is no such thing as a non-destructive deblocker;
Taken literally, sure. Splitting hairs can be fun.
If it sounds better, I could rephrase to sth like "a deblocker that does as little change to the source as possible, while still doing a fairly good job in removing as much blocking artefacts as possible".

IMO, the sentence with "non-destructive" was way easier to read & get the point. Which should've been clear, anyway.


running the motion search on the deblocked source can be helpful, but will result in detail loss.
Really? Why should that be? :)

[burfadel]

Have you tried blindpp and deblock? Add blindpp when encoding vob files (mpeg2), but it should be done before resizing the picture. This filter is specifically designed for the 8x8 blocks that mpeg2 suffers with. Deblock is a related filter. Have you tried simple filtering like dctfun4b? with a settings like dctfun4b(5,3) or something?

Also have you tried a different deinterlacer? its amazing how these can affect the encodes. Which one are you using, and what settings are you using for it?

The good thing about media player classic (I'm using the HC 1.0.9.0 version) is that you can play your avisynth scripts through it. Hopefully your processor is fast enough and settings not too complex, so you can play the file in real time (you're NOT encoding it remember). Just load the avisynth file with all the filter settings in it. Inspect the picture through there and see if there's any artifacts or anything wrong with the picture. Remember this way, how and what you're seeing on the screen is exactly the same as whats past so the encoder. Its not on the supported file list, you either have to right hand click, and open with media player classic, or open Media player classic, select Objects of type 'All files *.*' etc, and open it that way.

[Tanma]

Thanks for the tips. I'm encoding Sutepri, it has a helluva lot of grains. When you remove the grains, a lot of blocks appears. I don't know what is worst, maybe the blocks that are visible even from far of the screen.

[*.mp4 guy]

Taken literally, sure. Splitting hairs can be fun.
If it sounds better, I could rephrase to sth like "a deblocker that does as little change to the source as possible, while still doing a fairly good job in removing as much blocking artefacts as possible".

IMO, the sentence with "non-destructive" was way easier to read & get the point. Which should've been clear, anyway.



Really? Why should that be? :)

1- The less a deblocker changes the source, the less blocking it can remove, deblockers work by (basically) bluring, so some detail loss is inevitable if the deblocker actually does anything.

2- motion vectors aren't magic, If you run a motion search on a reduced detail source, then make changes to the source based on that motion search, the changes will reduce the level of detail.

[Manao]

2- motion vectors aren't magic, If you run a motion search on a reduced detail source, then make changes to the source based on that motion search, the changes will reduce the level of detail.Blocks don't move, while objects move, so blocks hurt a lot the motion estimation process. So a slight deblocking actually improves the motion estimation.

[Didée]

@ mp4guy

If you run a motion search on a reduced detail source, then make changes to the source based on that motion search, the changes will reduce the level of detail
Without further backing up, that's not a fact, but only a claim. You're implicating that doing motion search on a source with weakened signal will necessarily worsen the motion vectors. Which doesn't quite align with the observations I've made the last years. Often the motion search is even improved by doing so - see also Manao's comment. (After all, we're not talking about searching on a Gauss(variance=1000) filtered clip.)


motion vectors aren't magic
Anyone said so? If so, surely it wasn't me ...

Moreover, mo'comped filtering is not even a generally-suited method to combat source blocking. It often may improve on the problem, true, I know very well. But *basically*, it's not a valid method for the problem, because:
blocking mostly occurs in areas where the original codec had "problems" with the motion. Chances are that MVTools will also not find valid motion (some areas might be "impossible to compensate"), in which case the whole motion thingy obviously is not suited to improve on the problem of blocking.


The less a deblocker changes the source, the less blocking it can remove, deblockers work by (basically) bluring, so some detail loss is inevitable if the deblocker actually does anything.
Well, I point once more to my old case example:

Deblocker comparison: (Huffyuv (http://home.arcor.de/dhanselmann/_stuff/deblock_5ways_new_huffyuv.avi)) - (MSU v6.0 (http://home.arcor.de/dhanselmann/_stuff/deblock_5ways_new_MSU60.avi))

It's just a random 1-frame case example, but it gives hint which deblocker removes how much blocking or not, and which removes how much detail or not. Look close.


For my taste, I'd rather first tackle the obvious blocking with a non-destructive:p deblocker. After that, mo'comped filtering is definitly suited to clean up the leftover.

[Terranigma]

For my taste, I'd rather first tackle the obvious blocking with a non-destructive:p deblocker. After that, mo'comped filtering is definitly suited to clean up the leftover.


I agree, for mpeg2sources, I use cpu=4, and if there's still quite the evidence of blocking, I use your deblock_qed() script, but I noticed that it doesn't work properly on very grainy sources. It tends to eat away on edges, why is that?

[R3Z]

I agree, for mpeg2sources, I use cpu=4, and if there's still quite the evidence of blocking, I use your deblock_qed() script, but I noticed that it doesn't work properly on very grainy sources. It tends to eat away on edges, why is that?

I find cpu=4 way too strong. Deblock_QED doesnt blur the details anywhere near as much. Then again, i always use blockbuster which helps to mask some blocks as well.

PS: I know blockbuster isnt a deblocker.

[Terranigma]

i always use blockbuster which helps to mask some blocks as well.

PS: I know blockbuster isnt a deblocker.
Of course, it adds noise to try and mask whatever blocks that's still in the source. I prefer to use addgrain or grainfactory instead, because noise is evil. :devil:

[Didée]

Terranigma: sorry my english is not lively enough here, could you use a more clinical phrasing than "eats away on edges" -- or even better, post a before-after screenshot? I'm not sure (enough) to get you right on this ... ther're three propable answers, only one will really fit.

In any case: cpu=4 is stone age. ;)

[Terranigma]

Terranigma: sorry my english is not lively enough here, could you use a more clinical phrasing than "eats away on edges"
Ok, would interpolating fine edges do? :D

[Didée]

Something like "thin lines or structures get 'interrupted'" would've been most descriptive. :)

That's a bug-or-feature of Deblock()'s implementation. If deblock() is called with a rather big value for 'quant' (or the sum of 'quant'+'aOffset'+'bOffset'), then it starts doing this funky kind of thing. This once was mentioned by redfordxx as well (edit: this thread (http://forum.doom9.org/showthread.php?t=105383)), by the time he was fiddling with various deblocking methods.

Once more, note that deblock_qed doesn't do any "new" kind of deblocking - it just uses the output of Deblock() as starting point, then modifies the results in the inner 6x6 cell of each block, to keep detail.

Solution is, obviously, to not go too high with quant1, aOff1 and bOff1. I mostly use those ranges:

quantX=20..24, aOffX=2..4, bOffX=4..8

If a source needs way more than that, pipe it direct-stream into your trashbin.

[Terranigma]

Ok, thanks Didée. By the way, what do you think about me using deblock_qed() instead of degrainmedian for *.mp4 guy's script? Better compensation for the blocks present in the stream?

[*.mp4 guy]

@ mp4guy
Without further backing up, that's not a fact, but only a claim. You're implicating that doing motion search on a source with weakened signal will necessarily worsen the motion vectors. Which doesn't quite align with the observations I've made the last years. Often the motion search is even improved by doing so - see also Manao's comment. (After all, we're not talking about searching on a Gauss(variance=1000) filtered clip.)

I've done tests, it does result in detail loss, wheather the loss is significant depends upon how much detail was actually in the source, and the filter being used.

Also, I realise that mo comped denoising will run into problems in the same places as any codec would; thats why fft3d is run on areas with bad block match metrics.

Blocks don't move, while objects move, so blocks hurt a lot the motion estimation process. So a slight deblocking actually improves the motion estimation.
Perhaps on sources with pronounced blocking, but thats not even what this script is for. Its for 1- heavy noise removal, 2-while maintaining or improving gradients, and 3- removing minor blocking (usually associated with banding) that may be enhanced when reencoding while 4- losing as little detail as possible. Basically its ideal for noisy mpeg2 compressed video, because thats what I made it for. I recomended its use on very similar video as what it was designed for, in another thread I recomended it for again, primarily noise removal of an excessively noisy source.

[Tanma]

Some scenes can just be trimmed to use with strong deblocker settings... I'm doing it with some mpeg2 sources that have terrible dark scenes. But that requires more work.

[Tanma]

MVTools (http://avisynth.org.ru/mvtools/mvtools.html), fft3dGPU (http://forum.doom9.org/showthread.php?t=89941) (you can replace it with FFT3DFilter (http://avisynth.org.ru/fft3dfilter/fft3dfilter.html)), Masktools (http://manao4.free.fr/masktools-v2.0a31.zip), GradFun2DB (http://avisynth.org/warpenterprises/files/gradfun2db_25_dll_20060315.zip). And if you want to use spline36resize, then AviSynth 2.5.6 or above.

I see now why you told to change to FFT3DFilter. FFT3dGPU doen't work with MT().

mp4 guy's function is really nice, and can work well in any source with blocks.

[SilaSurfer]

You can get close to removing all the mpeg2 blocks from most dvd's with this script, but its slow as hell, although it leaves detail almost completely intact.

source = last

backward_vec2 = source.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = source.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = source.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = source.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
mask = mvmask(kind=1, vectors=forward_vec1).UtoY().spline36resize(source resolution)#its very important that this line matches the resolution you are feeding the script with
smooth = source.degrainmedian(mode=3).fft3dgpu(bw=16, bh=16, bt=3, sigma=4, plane=0)
source2 = maskedmerge(source, smooth, mask)

source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)
source3.gradfun2db(1.5)

Hello Mp4 Guy.

Great script. I was just wondering could this script which uses Mvtools be changed to work with Mvtools2. Thank you in advance