Which script for removing grain?

[Carpo]

I have been looking at removing grain from Aliens Bluray, as using crf19 gives me a 35gb file, i was advised to de grain it, but after looking at a few scripts I'm unsure on which one to use.

Can someone advise me on which one(s) to look at :thanks:

[mastrboy]

Screenshots / Sample could be useful.

Without looking at it, your safest bet might be a simple MDegrain[1,2,3]:

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)


If you are encoding it at 1080p, a larger blocksize could give a speed improvement: http://avisynth.org.ru/mvtools/mvtools2.html

[Didée]

I haven't seen a HD version of Aliens yet. But if it has the same grain as the original DVD release (not the "digitally remastered" one), then MDegrain is not suited to perform good degraining. Aliens' grain is way too strong for that. If it's the original grain, then you'd need preprocessing like in TemporalDegrain or MCTemporalDenoise. But it'll be no fun with a 1080p source.

Recommendation: "post a sample of the source" :)

[Carpo]

Backing up the drives atm, but as soon as i have that done i will post a few screenshots :)

[Didée]

Take your time. The bad news is that screenshots help next to nothing in deciding on a denoising strategy. Video samples are needed.

[Carpo]

what tools would i need to splice up an .m2ts file, smallest one on the bluray is 2gb ;)

[Didée]

Ha, good question. Me, I would simply use DGIndexNV to extract scenes ... but of course you need to have registered it (not free, it costs a small fee to neuron2).

There are some AVC cutting tools out there - I remember e.g. H264TS_cutter - but I've _no_ experience with them. (Don't even have a BR drive, and my TV reception still is SD, too. If I work with HD, it's mostly "problem samples" provided by other users.)

[Carpo]

http://www.mediafire.com/?pq4t634bjk8yn6b - seems its mainly in the darker scenes, hopefully you can see it here

[Didée]

Yep, thank you. Nice grain there, a pity you want to reduce/remove it. But in and on itself, the grain is not as severe as I had thought. I'll look at it later, right now I've to leave, reallife is calling.

[Carpo]

OK, thanks, it's not so much i want to get rid of it, its just using -crf19 will give me a 35GB .mkv file, which is 1GB bigger than the actual bluray, some have said to remove the grain as this is more than likely causing the encoder to jump up to try and keep all the details. I have been looking at a crf of around 23/24, but atm I'm just trying to cover all the bases.

Have only really just got into backing up HD content, so I'm rather cluless atm :p

[mastrboy]

If the purpose is not removing the grain, calming it might work for an increase in compression.
I can't get your sample a the momement: "Invalid or Deleted File. "

i have had alot of good results with TTempSmooth(F) for increasing the compression on various clips, but i never deal with 1080P content so i don't know if it's the right filter to use.
Adjust maxr for radius (larger=more stable, but slower)

http://forum.doom9.org/showthread.php?t=77856

Adjusting your x264 options could also be useful, lower trellis and AQ, and set deblock higher to "remove" the grain.

[Carpo]

You are correct, its gone, strange!!! I am re uploading it again, will post link when its done :)

[Carpo]

http://www.mediafire.com/?akglcjfkgdwhwiz - its there again

[7ekno]

I would use MCTemporalDenoise(settings="low", gpu=true,enhance=true), if that is still too grainy for your liking you could move to MCTemporalDenoise(gpu=true,enhance=true) ....

7ek

[Carpo]

Thanks 7ekno, will give your suggestion a try, see who it goes :)

[Carpo]

have not been able to test "MCTemporalDenoise(settings="low", gpu=true,enhance=true)" as it errors out - RemoveGrain: Invalid Mode 20

I grabed all the files it said it needed

[Didée]

This version (http://home.arcor.de/kassandro/RemoveGrain/RemoveGrain.rar) of RemoveGrain package. Use the ...SSE2.dll variants of RemoveGrain.dll and Repair.dll.

[Carpo]

Yes, they worked, going to be damn slow if i do choose to remove/reduce grain/noise, normally on an encode i can get 3-4fps using some of the methods listed here im getting .90fps, took 10 mins to do a 24 second clip :(

[Didée]

Look at post#3 once more. That's what I meant with "it'll be no fun with a 1080p source". :)

[Carpo]

I knew that, just didn't think it would be that much non fun ;)

[Didée]

Any script using Motion Compensation will be inherently slow when the source is 1080p. Achieving a noticable compression gain without using motion compensation, and still do no harm to picture quality, that's ... difficult.

The following script achieved 30% compression gain with a simple x264 --crf 19 --preset slow --tune film :

DGsource("01149.dgi")
changefps(last,last,true)
o = last
m2 = o.removegrain(11,0).medianblur(2,0,0).removegrain(11,20)
m2.fluxsmootht(5).mt_adddiff(mt_makediff(o,m2,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - 0.751 * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)
return(last)
There is some slight loss in moving low-contrast areas, but not too serious. Speed impact on encoding is rather low.
This could be improved by adding motion compensation, but then the script immediately will be 2 or 3 times slower.

The strategy is to separate spatial high-frequencies from low-frequencies, reduce temporal instability on the LF part, and reduce the amplitude of th HF's by 25%. I.e. the grain generally is kept, but appears weaker and "finer" now.

[Carpo]

so is that best to put that before or after resize? to be honest i don't really need 1080p atm as laptop is only 720p (1280x800) and main PC is on an old LCD that can do at its max native resolution 1360x768, yet on the Mac i can do 1920x1200, so i could go 720p, which would speed things up a lot, but looking towards the future when tv's can do higher (if they don't already) then again in the future 720 and 1080p will look crap :)

Then again i suppose in the future PC's will be fast enough for me to re-rip them if so required.

What do you think 720p or 1080p ?

[Lyle_JP]

so is that best to put that before or after resize? to be honest i don't really need 1080p atm as laptop is only 720p (1280x800) and main PC is on an old LCD that can do at its max native resolution 1360x768, yet on the Mac i can do 1920x1200, so i could go 720p, which would speed things up a lot, but looking towards the future when tv's can do higher (if they don't already) then again in the future 720 and 1080p will look crap :)

Then again i suppose in the future PC's will be fast enough for me to re-rip them if so required.

What do you think 720p or 1080p ?

Wait, you're resizing to 1280x720p, and the encodes you're getting at crf 19 are larger than the 1080p blu-ray source? Something doesn't sound right there. Are you doing any heavy sharpening on your resize?

[Carpo]

no Lyle, atm i am leaving them full size 1080p, i was just wondering if i should stay at 1080p even though the laptop and pc (LCD TV) have screen sizes of 1280x800 and 1360x768, or if i should resize them to 720p

[Lyle_JP]

no Lyle, atm i am leaving them full size 1080p, i was just wondering if i should stay at 1080p even though the laptop and pc (LCD TV) have screen sizes of 1280x800 and 1360x768, or if i should resize them to 720p

I guess that all depends on your plans for the video file long term. I always make my encodes with my playback devices in mind, so I would go with 720p. Really, If I were going to keep it 1080p I would do no re-encoding at all; I would just move the video and audio streams I wanted into an .mkv file and call it a day. Disk space is (relatively) cheap.

OTOH, if you're not de-graining for space considerations but merely because you find the grain aesthetically unpleasant, then apply your denoising script after resize (it'll be faster).

[Carpo]

Storage is only cheap if you have the cash for it, i'm unemployed atm, so i have to make do with what i have, long term plan is get a stand alone bluray player, so i guess i could go with 720p for now, the reason i was removing the grain is that when i tried to do aliens before it was giving me a 35GB file, people suggested the grain maybe what was causing it, i will do it at -crf19 720p and see what happens :)

[Didée]

people suggested the grain maybe what was causing it
Not "maybe", but "definetly". I made some small tests with your sample, and a full-fledged denoising (MDegrain3 with pre-NR for motion search) came out 1/3 the size of a direct encoding without filtering. 66% smaller is a drastical bitrate reduction. However, with all grain removed it's not the same movie anymore. (Though, at 720p this can be masked again, more or less, by adding grain during playback via ffdshow.)

[Carpo]

I meant this was suggested in another thread as one of a few possible issues, i stopped using ffdshow after i started using lav splitter+lav audio+lavcuvid+madvr, i will give it some thought after a nights sleep :)

[Carpo]

so does this look right?

LoadPlugin("C:\Program Files (x86)\MeGUI\tools\dgindexnv\DGDecodeNV.dll")
DGSource("F:\DISC_RIPS\aliens.dgi",fieldop=0)
#deinterlace

changefps(last,last,true)
o = last
m2 = o.removegrain(11,0).medianblur(2,0,0).removegrain(11,20)
m2.fluxsmootht(5).mt_adddiff(mt_makediff(o,m2,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - 0.751 * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)
return(last)

crop( 0, 22, 0, -22)
Spline36Resize(1280,688) # Spline36 (Neutral)
#denoise

never was too sure where to put filters, before or after resize?

[kypec]

never was too sure where to put filters, before or after resize?
I'd definitely put the filters always after crop and in this case also after resize. The smaller is the resolution for heavy temporal and/or spatial processing the faster it can run.

[Carpo]

Thanks, its running now :)

[SilaSurfer]

I would also go after resize to speed everything up. I use dfttest right now but I will also have a look at Didée's script above.

Update:

Didée could you present a sample of the above script with motion compensation please? :rolleyes: ;)

[Didée]

Didée could you present a sample of the above script with motion compensation please? :rolleyes: ;)

Sure. But I've a problem to make a decision ...

Once you start to add motion compensation, you're immediately faced with the caleidoscope of possibilities.

Means: one can take the script literally as-it-is, and support it with motion compensation:

KEEP = "0.75" # how much HiFreq-grain to keep. 0.0=keep nothing, 1.0=keep all. !! String value !!

DGsource("01149.dgi")
o = last
m2 = o.medianblur(2,0,0).removegrain(11)
lop = m2.fluxsmootht(5)
sup = lop.msuper(pel=1)
sup2 = m2.msuper(pel=1,levels=1)
bv1 = sup.manalyse(isb=true,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
fv1 = sup.manalyse(isb=false,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
bc1 = m2.mcompensate(sup2,bv1)
fc1 = m2.mcompensate(sup2,fv1)

interleave(fc1,m2,bc1).fluxsmootht(5).selectevery(3,1)
\.mt_adddiff(mt_makediff(o,m2,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)



But oh my ... this is inefficient! The major effort lies in the motion search. Then a relatively costly explicite compensation is done, only to use a very "particular" filter named fluxsmooth, which "reacts" only to one very certain temporal characteristc. Hence, once we took the initial effort of doing a motion search, it would be more efficient to simply use MDegrain instead of FluxSmooth.

KEEP = "0.75" # how much HiFreq-grain to keep. 0.0=keep nothing, 1.0=keep all. !! String value !!

DGsource("01149.dgi")
o = last
m2 = o.medianblur(2,0,0).removegrain(11)
lop = m2.fluxsmootht(5)
sup = lop.msuper(pel=1)
sup2 = m2.msuper(pel=1,levels=1)
bv1 = sup.manalyse(isb=true, delta=1,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
fv1 = sup.manalyse(isb=false,delta=1,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
bv2 = sup.manalyse(isb=true, delta=2,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
fv2 = sup.manalyse(isb=false,delta=2,blksize=16,overlap=8,truemotion=false,search=5,dct=5)

m2.mdegrain2(sup2,bv1,fv1,bv2,fv2,thSAD=320)
\.mt_adddiff(mt_makediff(o,m2,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)



Well. Next point. The spatial part is laid out for 1920'ish FullHD. You guys wanna encode at 720p? This has to be taken into consideration. A spatial median filter is a nice tool to "cut grain down", but spatial medians have only few "notches" that you can tune in. A radius=2 (5x5) median on 1080p is one thing, but is another thing at 720p, yet another thing on SD resolution.

For the grain characteristic of this particular source a radius=2 median is a bit too rough at 720p, and radius=1 is a bit too small. Pity!

The possibilities are

a) "I don't care, and Didée is babbling way too much anyway. I just use the previous script. Hehe!"


b) using a combination of 2 different medians:

KEEP = "0.75" # how much HiFreq-grain to keep. 0.0=keep nothing, 1.0=keep all. !! String value !!

DGsource("01149.dgi")
o = spline36resize(1280,720)
m2 = o.medianblur(2,0,0)
m2 = o.removegrain(4).mergeluma(m2,0.5).mt_lutxy(m2,"x y < x 1 + x y > x 1 - x ? ?",U=2,V=2).removegrain(11)
lop = m2.fluxsmootht(5)
sup = lop.msuper(pel=1)
sup2 = m2.msuper(pel=1,levels=1)
bv1 = sup.manalyse(isb=true, delta=1,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
fv1 = sup.manalyse(isb=false,delta=1,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
bv2 = sup.manalyse(isb=true, delta=2,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
fv2 = sup.manalyse(isb=false,delta=2,blksize=16,overlap=8,truemotion=false,search=5,dct=5)

m2.mdegrain2(sup2,bv1,fv1,bv2,fv2,thSAD=320)
\.mt_adddiff(mt_makediff(o,m2,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)



c) make the spatial part on a smaller resolution, thereby including the lowpassing that otherwise is done via RemoveGrain(11) anyway

KEEP = "0.75" # how much HiFreq-grain to keep. 0.0=keep nothing, 1.0=keep all. !! String value !!

DGsource("01149.dgi")
o = spline36resize(1280,720)
m2 = spline16resize(640,360).removegrain(4).bicubicresize(1280,720,1,0)
lop = m2.fluxsmootht(5)
sup = lop.msuper(pel=1)
sup2 = m2.msuper(pel=1,levels=1)
bv1 = sup.manalyse(isb=true, delta=1,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
fv1 = sup.manalyse(isb=false,delta=1,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
bv2 = sup.manalyse(isb=true, delta=2,blksize=16,overlap=8,truemotion=false,search=5,dct=5)
fv2 = sup.manalyse(isb=false,delta=2,blksize=16,overlap=8,truemotion=false,search=5,dct=5)

m2.mdegrain2(sup2,bv1,fv1,bv2,fv2,thSAD=320)
\.mt_adddiff(mt_makediff(o,m2,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)



Those are just a few possibilities out of the caleidoscope. And that's without even considering the fact that we also could throw a little enhancement into the mix, instead of simply reducing things in the source. This would open - you guess it - the next half-dozen of script variants that could be used ... :D



What was the question .... ah, yes: you wanted a sample encoding. Which one? One of these, or one of the twenty-five other possibilities?


Oh nooooo, I just can't decide. Seems you need to try for yourself. ;)

[Carpo]

okay now you have fried my brain and i have a hard day tomorrow, i think i will stick with what you gave me first time around, and encoding to 720p its a lot faster :p

[SilaSurfer]

Hehe! Ask for one example and you get four, but I did not expect anything less from you Didée. ;). Now I need to take the day off from work tomorow to test this out.:p Thanks

[Didée]

Uuhh, I just realized there's a mistake in the scripts: need to use a 2nd super clip, actually the fluxsmooth-preprocessing is getting re-used, which isn't menat to be. Oh, and a mistake with the downscaling dimensions.

Where did I have my brain. I'll correct it in a minute.

Edit: Done.

[Carpo]

So if i wanted to apply this over all encodes, which you suggest to use? Or would it be an idea not to use this in all encodes?

Lets for argument sake say i wanted 1080p what would i need to change in the script(s) to achieve this.

Sorry if these seem like newbie questions, when it comes to this i am a newb :p but best to get it right before i start encoding :)

edit: ANyone know why avisynth might be complaining that it doesnt know what "KEEP" means, guessing its to do with

m2.mdegrain2(sup2,bv1,fv1,bv2,fv2,thSAD=320)
\.mt_adddiff(mt_makediff(o,m2,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + /

* 128 + ?",U=2,V=2),U=3,V=3)

I believe i have all the dll required installed, then again im probably wrong :p

[Gavino]

ANyone know why avisynth might be complaining that it doesnt know what "KEEP" means
Did you include this line?
KEEP = "0.75" # how much HiFreq-grain to keep. 0.0=keep nothing, 1.0=keep all. !! String value !!

[Didée]

Look at the very-first lines of the posted scripts. They all start with >> KEEP = "0.75" <<. Now ask yourself why that line is missing in your script. :)

Which script to use? Can't tell you that. If at all, probably either one of the last two.
(But most probably, I'd fiddle around a little more.)

The other possibility is to do nothing at all, and use CRF21 instead of that holy-moly CRF19. x264 does a good job too.

I can tell you how to juggle with pixels. But i cannot tell what kind of picture is more pleasing to YOUR eyes. That's your pigeon. :)

----

In any case,
o = spline36resize(1280,688, 0,21,-0,-21)
and respectively
m2 = o.spline16resize(640,344).removegrain(4).bicubicresize(1280,688),1,0)
because it's simply better without the black mattes from 1.85 academy format.

[Carpo]

i clicked select all - ggggrrrrrr, stupid wordpad and stupid me

edit: Okay last stupid question from me, and I'm guessing this is just a case of personal preference, but what would i need to do to know whether or not i need to remove grain, or as i say is it a case of looking at the film and going, nah, don't like that grain ;)

After reading a the other thread i made about crf, you might be right about -crf 21, a lot of people seem to use that for 1080p