Need Suggestions for VERY GRAINY source

[Zep]

I have a 1280 x 720p source. it is VERY GRAINY to the point of be annoying and my enjoyment during viewing is adversely affected. :(

What is even worse and makes this grain even more noticeable is the fact that the grain DANCES around frame to frame so on as an example walls and sky your eye is drawn to this as if thousands of insects are buzzing about lol So I end up seeing that and not what I should be paying attention to.

I have tried DeGrainMedian(limitY=8,limitUV=15,mode=0) and it does not even come close to helping this. it gets rid of some grain but doe not fix the dancing.


I have tried

vbw1=MVAnalyse(isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=8,overlap=2,idx=1,sharp=1)
vfw1=MVAnalyse(isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=8,overlap=2,idx=1,sharp=1)
vbw2=MVAnalyse(isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=8,overlap=2,idx=1,sharp=1)
vfw2=MVAnalyse(isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=8,overlap=2,idx=1,sharp=1)
MVDegrain2(last,vbw1,vfw1,vbw2,vfw2,thSAD=100,idx=1)

Same result basically. However I have a feeling someone who knows MVDegrain2 very well could help me with parms to better make this work since I never used MVDegrain2 until to today.


I have tried HQDN3D with interesting results. It gets the grain semi ok but same problem in that any grain left over is dancing
so your eye is drawn to it.


I have tried FFT3DFilter and same as all of the above and it tend to blur a little more. Though using this after some of the above with a light setting helps.


I have also tried a temporal smoother after all the above and it helps some but starts to blur/lose detail.


ok you can get a 9 meg clip from this link which first shows a WALL with grain then the sky with mega grain.

http://www.megaupload.com/?d=Z8EUNJY4

any suggestions?

Thanks!

[buzzqw]

file not avaiable

BHH

[DeathAngelBR]

I have also tried a temporal smoother after all the above and it helps some but starts to blur/lose detail.

Maybe a sharpener after the denoise filter chain can fix that.

[mikeytown2]

MVDegrain2(last,vbw1,vfw1,vbw2,vfw2,thSAD=100,idx=1)

If you increase this 10x to oh
MVDegrain2(last,vbw1,vfw1,vbw2,vfw2,thSAD=1000,idx=1)
it actually works fairly well. Add these and you got a nice looking video
FFT3dGPU()
LimitedSharpenFaster()


In terms of detail loss, the grain is what gives that clip its perceived detail.

[Spuds]

Hard to say exactly what to do with that since the vid does not have much detail to use as an anchor. I ran mc_spuds with various settings on the clip, pick what you think looks best to your eye if any.

Here is the wall:
http://img81.imageshack.us/img81/6376/walloneaj0.th.jpg (http://img81.imageshack.us/my.php?image=walloneaj0.jpg)


Here is the sky:
http://img66.imageshack.us/img66/5757/skyoneur4.th.png (http://img66.imageshack.us/my.php?image=skyoneur4.png)

[R3Z]

How about posting a non cropped original source ?

The source you posted has no detail apart from the noise, so any recomendations made on the source you posted might not work.

[Zep]

file not avaiable

BHH

that is strange I just Downloaded the file from the link fine.


hmmm...

[Zep]

MVDegrain2(last,vbw1,vfw1,vbw2,vfw2,thSAD=100,idx=1)

If you increase this 10x to oh
MVDegrain2(last,vbw1,vfw1,vbw2,vfw2,thSAD=1000,idx=1)
it actually works fairly well. Add these and you got a nice looking video
FFT3dGPU()
LimitedSharpenFaster()


In terms of detail loss, the grain is what gives that clip its perceived detail.


I will give that a go thanks!

[Zep]

Hard to say exactly what to do with that since the vid does not have much detail to use as an anchor. I ran mc_spuds with various settings on the clip, pick what you think looks best to your eye if any.



ahhh very good results with some of those settings.

The in between settings work very well. Leave a little grain
and still keeps all the detail.


thanks

[Zep]

How about posting a non cropped original source ?

The source you posted has no detail apart from the noise, so any recomendations made on the source you posted might not work.


I will make one and put it up tomorrow along with
a clip I did that came out very well in my first dozen or so
tests with all the above suggestions. In fact I will un crop
the original clip so you all get an better idea of the detailed stuff
as well as able to compare to what I feel is the best filtered
clip thus far.

thanks!

[Sagekilla]

This looks like a clip from 300, because the grain looks very similar to the one in it. But that's just what I think.

[Dark Shikari]

This looks like a clip from 300, because the grain looks very similar to the one in it. But that's just what I think.If it is, I would recommend a combination of HQdn3D and RemoveGrain(19), as thats what I used.

[Sagekilla]

If it is, I would recommend a combination of HQdn3D and RemoveGrain(19), as thats what I used.

Yeah, I'm quite certian now that it is.. Those clouds look like the ones from 300. Also, could you post a sample screenshot of before and after? I'm curious as to how well that works, since I plan on re-encoding 300 (Despite the fact that I managed to get it to about 1.5 GB at full DVD resolution somehow without any external filters..) Here's what I mean:

http://img.photobucket.com/albums/v621/Sagekilla/300_clip.png

Looks almost identical to the types of clouds up there, and the noise too. It's kind of hard to tell in the picture I have here because this was post-encode, not the source file. Even if it's not 300, the grain type looks very similar to 300, and that combination you used might work just as well.

[Dark Shikari]

Yeah, I'm quite certian now that it is.. Those clouds look like the ones from 300. Also, could you post a sample screenshot of before and after? I'm curious as to how well that works, since I plan on re-encoding 300 (Despite the fact that I managed to get it to about 1.5 GB at full DVD resolution somehow without any external filters..) Here's what I mean:

http://img.photobucket.com/albums/v621/Sagekilla/300_clip.png

Looks almost identical to the types of clouds up there, and the noise too. It's kind of hard to tell in the picture I have here because this was post-encode, not the source file. Even if it's not 300, the grain type looks very similar to 300, and that combination you used might work just as well.I got 300 at 720p to 700MB without serious quality issues (slight background blocking, detail loss, but nothing atrocious) so you should be able to shrink it well beyond that at standard definition.

Basically the issue is that any noise reduction that actually gets rid of the grain is going to cut the detail enough to make it very compressible.

[Zep]

This looks like a clip from 300, because the grain looks very similar to the one in it. But that's just what I think.

you are correct! great eyes! :)

I have been ripping the BluRay over and over trying to get rid of the damn grain which almost
ruins the movie for me ( I love the movie enough to try and fix it for my own personal tastes)

I'm uploading an original clip and my best effort at grain removal now. note the size difference as
well. The original clip 98 megs and the grain removal 73 megs. I compressed the clips single
constant quality of Q=1 in xvid and no bframes and everything turned up to max for these clips.
It shows very well the huge difference grain removal has on this movie.

anyway,

I thought about turning up the grain removal even more and trying a sharpen filter after
but I'm afraid of losing too much detail so for now I'm trying to find a grain removal amount
that I can stand and that does not hurt the detail much. I think this latest round it damn good
as you will see after you DL both clips and compare.

here are the clips

Original (http://www.megaupload.com/?d=UALDTUIM)

With grain removal (http://www.megaupload.com/?d=HQFBODID)

Thanks

[Dark Shikari]

you are correct! great eyes! :)


I have been ripping the BluRay over and over trying to get rid
of the damn grain which ruins the movie for me (well sorta since
I love the movie enough to try and fix it for my own personal tastes)


I'm uploading an original clip and my best effort at grain removal now
and will post link in a few minutes once both are all up.I got 300 to under 700 megabytes and completely denoised, so you should be able to do it too.

As I said, HQDN3D, RemoveGrain(19), and fft3dfilter are your friends.

[Sagekilla]

you are correct! great eyes! :)

I have been ripping the BluRay over and over trying to get rid of the damn grain which almost
ruins the movie for me ( I love the movie enough to try and fix it for my own personal tastes)

I'm uploading an original clip and my best effort at grain removal now. note the size difference as
well. The original clip 98 megs and the grain removal 73 megs. I compressed the clips single
constant quality of Q=1 in xvid and no bframes and everything turned up to max for these clips.
It shows very well the huge difference grain removal has on this movie.

anyway,

I thought about turning up the grain removal even more and trying a sharpen filter after
but I'm afraid of losing too much detail so for now I'm trying to find a grain removal amount
that I can stand and that does not hurt the detail much. I think this latest round it damn good
as you will see after you DL both clips and compare.

here are the clips

Original (http://www.megaupload.com/?d=UALDTUIM)

With grain removal (http://www.megaupload.com/?d=HQFBODID)

Thanks

Indeed, if you can manage a 25% reduction in file sizes perhaps I can get the same (If not more, since I'm not using matrixes that suck up extra bitrate) reduction in bitrate using Dark Shikari's suggestions.


@Dark Shikari: What settings did you use for HQdn3D to get it to such low bitrates?

[Dark Shikari]

Indeed, if you can manage a 25% reduction in file sizes perhaps I can get the same (If not more, since I'm not using matrixes that suck up extra bitrate) reduction in bitrate using Dark Shikari's suggestions.


@Dark Shikari: What settings did you use for HQdn3D to get it to such low bitrates?
I think I used (I no longer have the script):

src=last
removegrain(19)
removegrain(19)
DeGrainMedian(mode=0)
fft3dgpu(precision=2)
hqdn3d(0,0,3,6)
fft3dfilter(sigma=0,sharpen=1)
last.mergechroma(src)
Lanczos4Resize(1280,720)

[Sagekilla]

I think I used (I no longer have the script):

src=last
removegrain(19)
removegrain(19)
DeGrainMedian(mode=0)
fft3dgpu(precision=2)
hqdn3d(0,0,3,6)
fft3dfilter(sigma=0,sharpen=1)
last.mergechroma(src)
Lanczos4Resize(1280,720)

Hm, wish I could do the same with my source but seeing as it's SD-only I'd lose too much sharpness. The following seems to do the job well though, cutting the needed bitrate by over 66%

hqdn3d()
RemoveGrain(19)

[Zep]

I got 300 to under 700 megabytes and completely denoised, so you should be able to do it too.

As I said, HQDN3D, RemoveGrain(19), and fft3dfilter are your friends.

there is is NO WAY in hell you can get 300 to 700 megs at 1280 x 720p even with max settings in x264. Even at 4.4 gigs I tell the difference in quality VS direct from the bluray.

I cringe at what the quality must be on your encode :)

I tried HQDN3D, RemoveGrain(19), and fft3dfilter.

none were good on their own but maybe all 3 together would be ok but I would expect loss of detail. Note if you are doing the low rez DVD then the grain will not as bad and you will need different methods and more light weight filters could do the job odds are. IMHO of course :)


so did you look at the clips? what did you think?

[Dark Shikari]

there is is NO WAY in hell you can get 300 to 700 megs at 1280 x 720p even with max settings in x264. Even at 4.4 gigs I tell the difference in quality VS direct from the bluray.Obviously you can tell the difference; you can't get visually lossless at any smaller than the original in most cases anyways.

The quality is quite good overall though if you're not a quality snob. You'd be quite surprised how good x264 is at low bitrates, especially when you've denoised it heavily (thus also reducing the bit cost for highest detail areas).

[Zep]

I think I used (I no longer have the script):

src=last
removegrain(19)
removegrain(19)
DeGrainMedian(mode=0)
fft3dgpu(precision=2)
hqdn3d(0,0,3,6)
fft3dfilter(sigma=0,sharpen=1)
last.mergechroma(src)
Lanczos4Resize(1280,720)

the fact you had to Lanczos4Resize(1280,720) means you smoothed out the encode IMHO too much and lost detail else no need for that filter call. Which is why I decided not to go that route.

I would love to see a clip of that of the same scenes in my clips. :P

[Zep]

Obviously you can tell the difference; you can't get visually lossless at any smaller than the original in most cases anyways.

The quality is quite good overall though if you're not a quality snob. You'd be quite surprised how good x264 is at low bitrates, especially when you've denoised it heavily (thus also reducing the bit cost for highest detail areas).

yeah but I want it really good. what is the point of spending all this time tweaking the grain removal only to have lower quality elsewhere?

My goal was to get rid of a good amount of grain and keep it looking as good as the bluray as much as possible on 1 DVD-R.

[Dark Shikari]

yeah but I want it really good. what is the point of spending all this time tweaking the grain removal only to have lower quality elsewhere?

My goal was to get rid of a good amount of grain and keep it looking as good as the bluray as much as possible on 1 DVD-R.300 has enough grain that if you remove the grain, you will lose loads and loads of detail. Its pretty much unavoidable.

[Sagekilla]

300 has enough grain that if you remove the grain, you will lose loads and loads of detail. Its pretty much unavoidable.

Yeah, RemoveGrain(19) alone gave the grain a jittery look so I had to keep the HQdn3D before that so it wouldn't have a flicker. A simple LimitedSharpenFaster added on helped retain some "detail" so it didn't look horrendous though.


Wish I had a blu-ray source so I could encode to 720p though.

[Zep]

300 has enough grain that if you remove the grain, you will lose loads and loads of detail. Its pretty much unavoidable.

which is why if you look at my clips you will see I did not remove all the grain.

Comparing to the Bluray 95% of detail is still there after my grain removal and is the best
balance I could get. i.e. get rid of just enough grain and grain dancing to not annoy me
while watching and thus keep most of detail intact to not annoy me on that front as well :D

[Zep]

300 has enough grain that if you remove the grain, you will lose loads and loads of detail. Its pretty much unavoidable.

BTW

interestingly enough it is not the grain that annoys me. It
is the DANCING grain. So I feel you can get rid of a lot of
dancing grain and keep detail if you use filters that look for
grain dance and only Degrain those areas while leaving the
rest alone. Most of the grain for instance is NOT on actors
faces and real live stuff. It appears they added in forced grain
to all the CGI created stuff like the cool skies.


I have been experimenting with a edge/detail finding filters than then mask those areas
which I then use so that the grain removal does NOT remove from the masked areas.
In fact I'm doing it right now and the results are very impressive. I have been able
to turn up grain removal while keeping even more detail than before. This
is how I plan to do the final encode for sure. The down side is SUPER SLOW :(
It will take days to encode the final.

[Sagekilla]

which is why if you look at my clips you will see I did not remove all the grain.

Comparing to the Bluray 95% of detail is still there after my grain removal and is the best
balance I could get. i.e. get rid of just enough grain and grain dancing to not annoy me
while watching and thus keep most of detail intact to not annoy me on that front as well :D

I wish I had a Blu-ray source to work off of like you do, I just went with HQdn3D() after a resize since RemoveGrain(19) decimated WAY too much detail and softened the image a lot.

[Pookie]

Didee's suggestions from previous threads:


source=last
denoised=fft3dfilter(sigma=8,sigma2=8,plane=4,degrid=1)
backward_vec2 = MVAnalyse(denoised,isb = true, delta = 2, pel = 2, blksize=16, overlap=8, sharp=1, idx = 1)
backward_vec1 = MVAnalyse(denoised,isb = true, delta = 1, pel = 2, blksize=16, overlap=8, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(denoised,isb = false, delta = 1, pel = 2, blksize=16, overlap=8, sharp=1, idx = 1)
forward_vec2 = MVAnalyse(denoised,isb = false, delta = 2, pel = 2, blksize=16, overlap=8, sharp=1, idx = 1)
MVDegrain2(source,backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=800,idx=2)

[Didée]

Similar, but not exactly like that. The prefilter should be able to cut down the grain almost completely. Some loss of detail is nothing to worry about at this stage. Then, thSAD should not be increased, but instead decreased from the default 400.
Using a prefilter together with such a high thSAD is guaranteed to introduce artifacts in areas where MC fails.

Alltogether, that suggestion was just an "easy" version of a general methodology. A script allowing for full control over all aspects is ten times as long as that, and quite a bit slower as well.

[Pookie]

"Using a prefilter together with such a high thSAD is guaranteed to introduce artifacts in areas where MC fails"

Indeed. You can't tell from the cropped example source file what thSAD values to use because it is only a small segment of wall and sky. It would be nice to see something of detail to determine if the degraining is going to be too strong.

[Sagekilla]

Similar, but not exactly like that. The prefilter should be able to cut down the grain almost completely. Some loss of detail is nothing to worry about at this stage. Then, thSAD should not be increased, but instead decreased from the default 400.
Using a prefilter together with such a high thSAD is guaranteed to introduce artifacts in areas where MC fails.

Alltogether, that suggestion was just an "easy" version of a general methodology. A script allowing for full control over all aspects is ten times as long as that, and quite a bit slower as well.

I can see plenty of places where the MC fails and 300 becomes a soupy mess.. Namely the snow scene with the wolf, where the snow just accumulates and becomes a soup of artifacts.

Can you recommend a version that'll work without those problems?

[Didée]

@ Sagekilla:
In other words, you mean if the holy grail of noise removal already has been discovered?
I fear it has not, yet.


@ Zep: Some more thoughts on "dancing" grain.

This sort of "dancing" usually isn't a property of the original grain, at least for the most part. Grain in itself usually is a high-frequency distortion only. The "dancing" effect is introduced by lossy DCT-based compressors, where in the lossy compression process some error is introduced into the low-frequency parts, caused mainly by the hi-frequency parts.

Something to try: the following script will remove the low-frequency flicker, leaving the high-frequencies intact.

#LoadPlugins: MedianBlur.dll
# RemoveGrain.dll
# mt_masktools.dll
# FluxSmooth.dll

AviSource("K:\300_original.avi")

o = last
f = o.MinBlur(1,2).MinBlur(2,2).RemoveGrain(11,-1)
f.FluxSmoothT(7).mt_AddDiff(mt_MakeDiff(o,f,U=2,V=2),U=4,V=4)

# eventually, limit the maximum pixel change to +/- 2 :
# mt_LutXY(o,last,"x 2 + y < x 2 + x 2 - y > x 2 - y ? ?",U=2,V=2)

# to compare:
stackvertical(o,last)
#interleave(o,last)

return(last)

#------------------------------------------
# Taken from MCBob.avs:
function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200

RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

In result, there will be almost no smoothing, and the grain basically is fully preserved. It's just the flicker, or "dancing" effect, that will be removed.
As a side-effect, there might occur some slight toning-down of shadings when there is motion. One can definetly see it in single-frame comparisons by flipping between original and processed. But one will hardly note that during watching the movie.

[Zep]

"Using a prefilter together with such a high thSAD is guaranteed to introduce artifacts in areas where MC fails"

Indeed. You can't tell from the cropped example source file what thSAD values to use because it is only a small segment of wall and sky. It would be nice to see something of detail to determine if the degraining is going to be too strong.


download the original clip it is in a link above

[Zep]

@ Sagekilla:
In other words, you mean if the holy grail of noise removal already has been discovered?
I fear it has not, yet.


@ Zep: Some more thoughts on "dancing" grain.

This sort of "dancing" usually isn't a property of the original grain, at least for the most part. Grain in itself usually is a high-frequency distortion only. The "dancing" effect is introduced by lossy DCT-based compressors, where in the lossy compression process some error is introduced into the low-frequency parts, caused mainly by the hi-frequency parts.

Something to try: the following script will remove the low-frequency flicker, leaving the high-frequencies intact.

#LoadPlugins: MedianBlur.dll
# RemoveGrain.dll
# mt_masktools.dll
# FluxSmooth.dll

AviSource("K:\300_original.avi")

o = last
f = o.MinBlur(1,2).MinBlur(2,2).RemoveGrain(11,-1)
f.FluxSmoothT(7).mt_AddDiff(mt_MakeDiff(o,f,U=1,V=1),U=4,V=4)

# eventually, limit the maximum pixel change to +/- 2 :
# mt_LutXY(last,o,"x 2 + y < x 2 + x 2 - y > x 2 - x ? ?",U=2,V=2)

# to compare:
stackvertical(o,last)
#interleave(o,last)

return(last)

#------------------------------------------
# Taken from MCBob.avs:
function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200

RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

In result, there will be almost no smoothing, and the grain basically is fully preserved. It's just the flicker, or "dancing" effect, that will be removed.
As a side-effect, there might occur some slight toning-down of shadings when there is motion. One can definetly see it in single-frame comparisons by flipping between original and processed. But one will hardly note that during watching the movie.


ok I guess I finally found all the same versions of the plugins you are using.
My result was not that good. for me my removal is much better. not only is more grain
removed and less dancing grain with what does remain but more detail in mine is retained.

however mine runs slower than the above by a fair bit.


thanks for the ideas though :)

[Sagekilla]

Thanks for the script Didee! It worked much better than any of the other methods before (for me anyway.) I didn't get much of a bitrate reduction, 100-150 kbps at best, but at least the grain isn't jumping around like it used to, and I still get retain some of the "detail," even if some of it is fake from the grain. It may not be the Holy Grail of removing noise/grain, but for 300 it works perfectly.

Edit: I tried it out on my copy of "Knocked Up" as well, and it seemed to do a very nice job of stabilizing the grain so the backgrounds look static for the most part.

[Zep]

Thanks for the script Didee! It worked much better than any of the other methods before (for me anyway.) I didn't get much of a bitrate reduction, 100-150 kbps at best,


because it is not really doing much. At least for me it only reduces the amount of grain by a very small amount and what is left is dancing around almost as much.

if you want to see a fast and really good grain removal use ffdshow noise removal and turn on Denoise3d but only use the HQ setting and max it out. Very fast and very good.


Now mix that with your avi synth script and tweak from there.


I think you see better results because you are not coming from the Bluray but are using the lower rez version which has far less grain and most of these plugs are geared to low rez DVD. Removegrain just can't cut it on high rez Bluray source and this is where removegrainHD is better and why it was written.

[Terranigma]

because it is not really doing much.
Did you remove the stackvertical line, and uncomment interleave/mt_LutXY and added selectevery(2,1)?

[Didée]

Zep, it seems you were expecting the wrong thing from that script...

My result was not that good.
because it is not really doing much. At least for me it only reduces the amount of grain by a very small amount
That was exactly the aim of the exercise. The grain is hardly touched at all. Removed are only the bad effects that the grain forced the original source encoder to do.

...and what is left is dancing around almost as much.
Can't agree with that. To my eyes it looks noticeable different.
If I literally go one step back - so that I hardly can see the hi-freq grain at all anymore - then I still see lots of flickering in the "original", while the lowpass-processed one looks pretty solid.


for me my removal is much better. not only is more grain removed and less dancing grain with what does remain...
no doubt about that. Since my script didnt't try at all to remove any grain, your script surely did remove more.

Again: That script was in no way meant for grain removal. It was also not meant as script to use for final encoding (although one can do so, if the result happens to be sufficient and pleasant.)
It was only showing that one of the major annoyances of very grainy sources, namely the low-frequency flicker - can be tackled with rather simple means.
One step further, something in this direction often is very useful for subsequent motioncompensated denoising. If the flicker is left in, it eventually will also disturb the ME engine (making the vectors follow the flicker, causing spatial shifts where in fact there should be none), which will lower the benefit one can get from MC-NR. When the flicker is taken out before the motion search, chances are better to get more clean vectors.

...but more detail in mine is retained.
Is it so? Or is it perhaps wishful thinking?

Random example: Frame 180.

The overall difference caused by ...

left: my script, right: your 'best.so.far' clip
http://img503.imageshack.us/img503/4121/lowpassdifferencevx4.th.png (http://img503.imageshack.us/my.php?image=lowpassdifferencevx4.png) - http://img207.imageshack.us/img207/7982/zepdifferencehl7.th.png (http://img207.imageshack.us/my.php?image=zepdifferencehl7.png)

Or: flip forth-back between these three: my script - original - your 'best.so.far'
http://img511.imageshack.us/img511/23/lowpassprocessedyn8.th.png (http://img511.imageshack.us/my.php?image=lowpassprocessedyn8.png) - http://img511.imageshack.us/img511/9413/originalut6.th.png (http://img511.imageshack.us/my.php?image=originalut6.png) - http://img228.imageshack.us/img228/799/zepsbestsofarej7.th.png (http://img228.imageshack.us/my.php?image=zepsbestsofarej7.png)

Okay, we didn't see either your latest script or its results, but from what you said it was basically similar to what you showed with 'best.so.far", plus masking of certain areas, allowing for even stronger settings. Now, I doubt that the kinds of negative impacts, as shown above, can be rescued by only a little masking...
Get this right: the result of your 'best.so.far' is not bad at all, it's quite good. But there *is* smearing / shifting, and there *is* detail loss. Hence I can't follow your finding that your script "keeps more detail", when mine doesn't loose any to start with.

We surely all are interested in seeing the latest script you came up with. However, after years of toying around with very grainy sources in all kinds of even-so-sophisticated ways, I can tell: grain removal on strong-grained sources won't ever be perfect. It's always a "you win here, you lose there" game. Winning without losing is out of reach.

---

BTW, what BlureRay is this that we're juggling around with 720p?

[Terranigma]

One step further, something in this direction often is very useful for subsequent motioncompensated denoising. If the flicker is left in, it eventually will also disturb the ME engine (making the vectors follow the flicker, causing spatial shifts where in fact there should be none), which will lower the benefit one can get from MC-NR. When the flicker is taken out before the motion search, chances are better to get more clean vectors.

Didée, are you saying it'd be more beneficial to use this idea you came up with, to remove low-frequencies, will work better if it's used before mo-comp. denoising?

[Didée]

this idea you came up with The idea in itself is not exactly new - Fizick once made a basic script with multi-freq filtering, and IIRC that was before his first implementation of FFT3DFilter. It's been a while since then. :D

are you saying it'd be more beneficial to use this idea you came up with, to remove low-frequencies, will work better if it's used before mo-comp. denoising? It's still the basic method of using a pre-filter before doing the motion search. This can be done in several different ways, and this here was just one of them that can be used. E.g., if one is using a pure spatial prefilter, the effect is the opposite: it will take out the hi-freq's, but leave the flicker mostly intact, therefore still irritating the ME engine.
Fact is, with strong grain there is so much uncertainty at the pixel level that there is hardly any "this is the right way to do". There are plenty of different possible points to break into the circle of catch-22 ... but there's no "correct" one.

[Terranigma]

Ah I see, I understand now. Thanks for clarifying. :)

[Sagekilla]

Personally, I prefer Didee's script over most plugins I've tried for removing grain since it doesn't blur the image at all, and when it comes to the dancing grain like in 300 it stabilizes it so it doesn't look horrendous after encoding. Really, my biggest beef is how soft the image becomes after most denoise/grain filters so Didee's script is really what I've been looking for.

I really wasn't expecting too much in the bitrate reduction department after I read his explanation, but just the sheer increase in video quality (to my eyes) from not having dancing grain is more then enough.

[Pookie]

Another one for the cookbook.:D Thanks (again), Didée.

[Zep]

Another one for the cookbook.:D Thanks (again), Didée.

ok all I finally did the final encode which took 3 days to crunch :D
Please download and take a look I am curious as to all your thoughts!

The clips (http://www.megaupload.com/?d=90HDTJL0)

I learned A LOT from this thread. Nothing worked straight up to the level I wanted and the best by far was to use a little of everything I learned in this thread and test test test then tweak and test some more lol I hope you all check out the final result clip I think you will be very surprised at just how awesome it came out and IMHO trounces the original in all aspects.

Some Notes and things I learned about grain.

File Size. Grain kills the ability to compress more than I thought. I used x.264 and to reach a Q=16 average the original needed a 9000 bitrate and mine without grain only needed a 4000 bitrate and mine IMHO looks better and has in some areas more detail. (get to that in a minute)

the codec it self plays a huge part on detail loss even with massive bitrate. In this case I learned x.264 blows out detail no matter how much bitrate you give it because of the inloop filters so if you have the bitrate turn them off. If you do not then yeah leave them on as they help a ton with blocking etc....

grain and dancing grain (dancing grain is what really annoys me) really is only noticeable to the point of distraction on walls, and sky and wheat fields and skin (not face as much like say a women's smooth tummy, i.e. not so detailed skin areas). Where there is lots of detail it gets lost in that detail so that made it easy to find the level of detail and mask then only remove A LOT of grain from the low level detail areas, some from middle detail areas and almost none from high detail areas so we can keep as much detail a possible since the filtering will always get rid of some. Down side is SLOW SLOW SLOW.

Where you place filters in the chain is so important when it comes to what I feel is re-mastering type of filtering. an example is my huge filter chain washed out the colors some and lightened up everything so I had to add more filters to adjust that and when you look at the original you even see that they had the same problem and I was able to make it look even better than the original in that area also like on the shields where the highlights were so bright and huge it made the them look fake but my tweaks added back the bronze color and toned down the contrast so it looked real and the color more rich and vibrant. The thing is if I place the filters at the end of chain it just sorta gave the whole frame a boost and it looked fake since some things shouldn't get the boost. However, if I used it at the start of chain then let all the removal and detail enhancement at it, they knocked it back down to not fake levels and they did it selectively. A bronze shield that needed a touch of more yellow-bronze got it but a silver sword did not else it would have turned purple or blue and not stayed silver looking. very interesting results here.


Anyway, if you watch the clips note the wheat behind the bad guy while he says "the hot gates" huge difference in dancing pixels between the clips. A ton to none. After that the sky is next place that looks so much better. Even the kings son's face looks better because in this example the dancing grain was drawing your eye away from detail. This is what I meant above by how some detail is even better with the grain removed. Another is the pull back shot with wheat all around. The original has so much dancing grain it makes it hard to see each stalk since the grain is jumping on and off them and since each grain was as large or larger that the stalk width it hid them but with it gone you can see each stalk and the fine lines. The wheat has more detail now and so does the mountain range in the distance or perhaps I should say it has more perceived detail. :P


Anyway I am very pleased with the results and the only thing I may do differently is to run it again with more bitrate :)

[Sagekilla]

because it is not really doing much. At least for me it only reduces the amount of grain by a very small amount and what is left is dancing around almost as much.

if you want to see a fast and really good grain removal use ffdshow noise removal and turn on Denoise3d but only use the HQ setting and max it out. Very fast and very good.


Now mix that with your avi synth script and tweak from there.


I think you see better results because you are not coming from the Bluray but are using the lower rez version which has far less grain and most of these plugs are geared to low rez DVD. Removegrain just can't cut it on high rez Bluray source and this is where removegrainHD is better and why it was written.

The point for me is not to touch the grain in 300 since it's very difficult to remove grain and retain sharpness. I refuse to degrain if it means my image is becoming smoother, and I've already encoded the video and quite like how the results have come out.

Edit: To clarify, none of the degrain scripts I've tried have offered a decent way to get rid of the dancing grain (Which is my goal really) while maintaining sharpness, and Didee's script is what really does this for me. Plus, I don't use ffdshow at all so I can't do postprocessing that way.

[Zep]

The point for me is not to touch the grain in 300 since it's very difficult to remove grain and retain sharpness. I refuse to degrain if it means my image is becoming smoother, and I've already encoded the video and quite like how the results have come out.

Edit: To clarify, none of the degrain scripts I've tried have offered a decent way to get rid of the dancing grain (Which is my goal really) while maintaining sharpness, and Didee's script is what really does this for me. Plus, I don't use ffdshow at all so I can't do postprocessing that way.


you say all that without looking at my final clip (for now anyway odds are I will run more testing next weekend) or reading my last post I take it.

My clip has almost no grain and almost no loss of detail and in some areas more detail and I do it at half the bitrate of the original. If I had to choose which one to watch I would take mine by miles and I bet most would and that is right now. I could easily make it even better if I wanted to spend more time and use the same bitrate as the original. (I would get the Q average in x.264 down to 12 I bet and could turn off the in loop filters hahaha)

The bit rate issue alone is such a huge plus you could really go to town with more filtering like mega masking for selective removal as well as selective sharpening. Furthermore, IIRC you said you do not have the Bluray. if you did you would see that Didee's script doesn't work on it that well. Dancing grain was almost as bad and there was loss of detail on top of that. My guess is the same reason removegrainHD was made. With higher rez source you need go beyond just 9 pixel box to catch the grain. Anyway, that script may work on low rez stuff but it didn't do much on direct from the Bluray source. Trust me I wish it did as it would have saved me a lot of time.

Anyway, download my latest clip and compare as an example the start of the clip the kings face VS the original. The detail is still there. (read my post above about MASKING and you will know why it is still there)

[StifflerStealth]

Zep, could you post your final AVS script that you used to make that final build? I would love to learn off of it. I have other movies that I own that are grainy and it ticks me off each time I watch them. They are as bad as 300, especially the first two Harry Potter movies. It's like they cone back to Video Tape days. :(

And I too have learned a lot about grain in this thread. The tips posted here are invaluable.

[Sagekilla]

you say all that without looking at my final clip (for now anyway odds are I will run more testing next weekend) or reading my last post I take it.

My clip has almost no grain and almost no loss of detail and in some areas more detail and I do it at half the bitrate of the original. If I had to choose which one to watch I would take mine by miles and I bet most would and that is right now. I could easily make it even better if I wanted to spend more time and use the same bitrate as the original. (I would get the Q average in x.264 down to 12 I bet and could turn off the in loop filters hahaha)

The bit rate issue alone is such a huge plus you could really go to town with more filtering like mega masking for selective removal as well as selective sharpening. Furthermore, IIRC you said you do not have the Bluray. if you did you would see that Didee's script doesn't work on it that well. Dancing grain was almost as bad and there was loss of detail on top of that. My guess is the same reason removegrainHD was made. With higher rez source you need go beyond just 9 pixel box to catch the grain. Anyway, that script may work on low rez stuff but it didn't do much on direct from the Bluray source. Trust me I wish it did as it would have saved me a lot of time.

Anyway, download my latest clip and compare as an example the start of the clip the kings face VS the original. The detail is still there. (read my post above about MASKING and you will know why it is still there)

Yeah but there's exactly that: You have HD vs my SD. I don't know how well my source would hold up to your script. I'd love to test your script on my SD source but I haven't found a script (or didn't look hard enough, will later when I'm not so busy)

[Didée]

Please download and take a look I am curious as to all your thoughts!
In a nutshell:

- Grain removed, at the price of:

- faces often times look soapy
- some detail is emphasised, some detail is lost or blurred
- there is edge ringing and mosquito noise / texture echoes caused by FFT3DFilter

Anyway I am very pleased with the results and the only thing I may do differently is to run it again with more bitrate :) For what you have there, the bitrate is fully sufficient. More bitrate won't help for the downsides introduced by your filterchain.

It's by all means a respectable try, but "remastering" (your words) needs a bit more than this.