AddGrainC (now with Chroma noise & YUY2, RGB)

[foxyshadis]

As an experiment I made a version of Tom Barry's AddGrain with chroma noise, no more need for scripting workarounds.

AddGrain(var,hcorr,vcorr,uvar,seed)
AddGrainC(var,uvar,hcorr,vcorr,seed)

var, uvar:
luma and chroma variance.
chroma does nothing in rgb mode.

hcorr,vcorr:
horizontal & veritcal correlation.

seed:
int, to keep the same grain on every frame.

Unlike the original, these are all named and optional and default to 0.0 except var, which is 1.0.

It may be faster, it may also be SSE2 only (haven't run it on the Athlon yet). Thought it might be useful to others, I'll finish up the readme if there's any demand.

Download v1.7.1 here (http://ldesoras.free.fr/src/avs/AddGrainC-1.7.1.7z).

One name is meant to be backward compatible, one is meant to make more sense, I don't know which I prefer so I just left both. :p

Changelist:
1.0 2003/06/18 Tom Barry Initial Release
1.1 2006/06/01 Foxyshadis Chroma grain + constant seed
1.2 2006/06/06 Foxyshadis Supports YUY2, RGB. Fix cache mess.
1.3 2006/06/10 Foxyshadis Crashfix, noisegen optimization
1.4 2006/08/11 Foxyshadis Constant replaces seed, seed repeatable
1.5 2010/05/07 Foxyshadis Limit the initial seed generation to fix memory issues.
1.5.1 2010/05/13 Firesledge The source code compiles on Visual C++ versions older than 2008
1.5.2 2011/10/26 Firesledge Removed the SSE2 requirement.
1.5.3 2011/10/26 Firesledge Fixed coloring and bluring in RGB24 mode.
1.5.4 2011/10/27 Firesledge Fixed bad pixels on the last line in YV12 mode when constant=true,
fixed potential problems with frame width > 4096 pixels
and fixed several other minor things.
1.6.0 2011/10/28 LaTo INV. Added SSE2 code (50% faster than MMX).
1.6.1 2011/10/29 LaTo INV. Automatic switch to MMX if SSE2 is not supported by the CPU.
1.7.0 2012/08/16 Firesledge Supports Y8, YV16, YV24 and YV411 colorspaces.
1.7.1 2013/11/25 Firesledge 64-bit version.

[Backwoods]

404 on the link good buddy, over.

[Didée]

Ah, foxyshadis is entering plugin land ... /me rubs hands ;)

Nice extension to AddGrain. Less juggling-around with separating & merging color planes.

BTW: It seems there's a long standing issue with AddGrain. Somehow, if during processing a AddGrain'ed clip is duplicated, used otherwise, and later on combined again with the original AddGrain'ed clip, then the noise pattern is no more the same. Seems like the filter starts with a different seed for processing the child frame, instead of using the seed from the mother frame.
At times, this is highly unpropitious. The usual workaround is to not apply AddGrain directly to a clip, but to apply it to a neutral grey clip instead, and subtracting this one from the clip you want to process. This way mother and child keep having the same pattern.
Sorry for I've no example at hand right now (and can't make one, since I'm already sleeping :p )

From the sourcecode I can tell nothing (no clue about how these things are handled). Perhaps someone in the knowings can see where it's going wrong.

[BigDid]

...(and can't make one, since I'm already sleeping :p )...
:) You can type and sleep, wonderful :D

To stay on the subject: same 404 error when trying to download :(

Did

[foxyshadis]

Heh, I forgot that I always put stuff in the /random folder... ^^;; Sorry.

I've been doing plugins for a while, it's just that they usually don't pan out so there's no point in releasing. :p

What I suspect is happening there is the cache gets blown, so that even though both paths lead back to the same addgrain'd clip, it gets done twice - but I might be entirely wrong. I'll take a look.

[Backwoods]

http://foxyshadis.slightlydark.com/random/AddGrainC.zip

[Mug Funky]

hey, cool. i just made a scripty workaround the other day.

would it also be possible to keep the same seed for each frame? this could be useful for avoiding banding when using addgrain for dither (perhaps it could even be added into gradfun2db?)

[Chainmax]

Sounds like it would make a great addition to BlockTerminator :). One thing though: I assume the chroma noise won't look like rainbows or color bands, what would it look like?

[foxyshadis]

It will look like chroma noise. :p The output should be identical to this:

function AddChromaGrain(clip c, int "grain") {
grainu=c.utoy().addgrain(grain,0,0)
grainv=c.vtoy().addgrain(grain,0,0)
return ytouv(grainu,grainv).mergeluma(c)
}

I need to organize my invented script functions better, they're all over several hard drives...

Here, I'll put up a screenshot.
http://foxyshadis.slightlydark.com/random/funky_cn.jpg
Go easy on chroma noise; heavy luma noise can look artsy, but heavy chroma noise just looks like a horrible capture. (That was uvariance 100, btw.)

I was thinking about having a single starting seed, but a constant seed actually sounds useful. Btw, I made modifications of blockbuster as well, so that its arguments are more useful. I have other ideas for it, when I remember to pick it back up.

Edit: Okay, I updated, fixing a bug with the luma/chroma noise interaction (maybe I only broke it today :p), and adding a seed parameter for Mug. Use anything >= 0 to enable.

Actually it appears to be using std. dev., not variance, the variable names lied. =p So maybe I'll change var and uvar to dev and udev.

Edit 2 jun: updated readme in archive as well.

[Fizick]

Thanks for update!
IMHO, first parameter of AddGrain is variance, or squared deviation (sigma*sigma), and is not standard deviation.

[Fizick]

And feature request: YUY2 support.

[foxyshadis]

I'll look into it, I've only used YV12 before but I have a Plan that might let me use the existing mmx writeover code. I'm sure there's some function I could use in RemoveGrain, but ouch that code hurts my brain, or Tritical's filters; if I can't figure it out then Interleaved2Planar/Planar2Interleaved or straight C will work.

Didée, IanB explained the problem as coming from playing fast and loose with the cache, and offered a quick workaround I'll insert.

[Fizick]

Probaly it may be implemented as interleaved, not planar.
At last with equal luma and chroma noise it is trivial :)

[foxyshadis]

And feature request: YUY2 support.
Done.

RGB24/32 as well, but luma only. (How would you make gaussian noise across 3 color channels?)

Somehow, if during processing a AddGrain'ed clip is duplicated, used otherwise, and later on combined again with the original AddGrain'ed clip, then the noise pattern is no more the same.
Fixed. Or at least covered up, I'm not quite sure which.

Enjoy. It was a lot of fun to do, really, even if I had to recode much of it. Hope it isn't horrifying to you real plugin devs. Threw comments and a new readme in. Now I just have to learn SSE and figure out how to parallelize the inner update loop. ;D

[Fizick]

thanks, i will try use it.
I often used AddGrain for denoise filter development.

[Fizick]

I tried it.
"Evaluate: System exception. Illegal instruction."

I have AthlonXP.
Need in SSE2?
But when I compile your source myself (VCToolkit2003), DLL works fine for me.
Wrong source? :)

[foxyshadis]

No, I just compiled it with SSE2 for testing and didn't turn it off. I uploaded the normal SSE one.

Maybe I'll do what Kassandro does. ;) (But seriously, thanks for reminding me to turn that option off for a final release, it doesn't change the speed anyway.)

[edit] It seems to have some bug that causes crashes on reload with mid-to-large res. I'll fix soon.

[foxyshadis]

[edit] It seems to have some bug that causes crashes on reload with mid-to-large res. I'll fix soon.
Fixed. I also partly optimized the random generation, which you'd only notice if you use it many times in a script, like me. >.>

[Chainmax]

So, to use AddGrain(x,0,0) on luma and AddGrain(y,0,0) on chroma, what would be the corresponding AddGrainC call?

[foxyshadis]

AddGrainC(x,y)

That's it.

[Chainmax]

I see, thanks :) http://smilies.vidahost.com/otn/wink/thumb.gif.

[zambelli]

Does anybody know how the size and strength and of the grain correlate to compression quantization? In other words, what size and strength of grain needs to be used in order for it to be preserved by H.264, VC-1 or MPEG-2 quantization?

[Sagittaire]

Does anybody know how the size and strength and of the grain correlate to compression quantization? In other words, what size and strength of grain needs to be used in order for it to be preserved by H.264, VC-1 or MPEG-2 quantization?

Well not simple because for example grain retention is not the same between AVC implementation. Anyway for all codec big grain size with low temporal complexity is alway more simple to compress. Small grain size with high temporal complexity is really hard to compress for all codec and particulary for AVC. In this case the better way to compress is degrain/regrain pre-process.

[Manao]

Zambelli : i advise you to make a gradient of noise (stacking grey videos with noises of different variances), and encode that in intra at different quantizer. You'll easily see the quantizer at which a noise of variance X disappear (or the variance needed for a noise to remain at quantizer Y). Do remember that in AVC, quantizers are exponential, so you don't need to test standard deviation 1,2,3,4,5,6....,n but rather 1.12^n (1.12^6 = 2 is the key factor behind AVC's quantization).

[McCauley]

Hi foxyshadis,

thanks for this nice little PlugIn.
Would it be possible to add the option to add a grain pattern that is constant for a specified amount of frames and then changes to the next pattern for the specified length and so on?
I basically want to use it for something like GrainOptimizer does.

Regards and thanks in advance
McCauley

[Adub]

Just use trim.

Ex.

source=avisource("example.avi)
a=source.trim(0.500).addgrainc(blah)
b=source.trim(501,1000).addgrainc(blah)
c=.....
a+b+c+....

[Didée]

More likely the aim is to change the noise pattern after only a few frames, ie. have like 2, 3, 4 or 5 frames with the same noise pattern, then switch to the next pattern, repeat, a.s.o.

x = 4 # set the no. of frames with same noise pattern

clp = last
grey = BlankClip(clp,color_yuv=$808080)
noise = grey.AddGrainC(...)
noise = noise.changefps(framerate(clp)*x).trim(0,framecount(clp))

clp.mt_makediff(noise,U=3,V=3) # if it's only luma noise, use "U=2,V=2"

edit: "clip" => "clp", silly me.

[McCauley]

Thanks Didée, that's what i need for my purposes! As always you had a few simple lines that did the trick!
But since this should'nt be closely as fast as the same thing in AddgrainC, i would be happy too see it implemented directly in AddgrainC.

Regards
McCauley

[foxyshadis]

Test it, that shouldn't be noticeably slower unless you use timecodec. Since my computer was stolen last weekend, I won't be updating anything for a while, but the source is in the package.

[Leak]

Since my computer was stolen last weekend, [...]
o_O;;;

(Thank god my machine + radiator is so bulky they'd probably leave it behind; or should I fill it with concrete just to be sure?)

[Blue_MiSfit]

I'm wondering...

Why would one add chroma grain? I know it can be beneficial in certain cases or for artistic reasons to add luma grain, but chroma?

Just curious ;)


Since my computer was stolen last weekend

NO! That's terrible :( I hope you didn't loose any vital data. Best of luck getting things back in order!

~MiSfit

[pitch.fr]

just FIY, this plugin creates mad memory leaks above 2 threads in MT via ffdshow.

problem doesn't exist in v1.0 and v1.1...they strictly obey SetMemoryMax()

too bad you said noisegen was improved in v1.3, but 1.5 Gb of memory leak on 4 threads is a tad too much :D

GrainFactory() works fine w/ any version

[swaaye]

Can someone post AddGrainC v1.4 somewhere? The link to foxyshadis site is down. Mad googling has only gotten me v1.3.

[J_Darnley]

It might not remain there but sure: http://users.telenet.be/darnley/avisynth/addgrainc_1.4.zip

[swaaye]

It might not remain there but sure: http://users.telenet.be/darnley/avisynth/addgrainc_1.4.zip

Thanks much

I put it up on megaupload too for anyone else in case your link dies
http://www.megaupload.com/?d=90EXJZGZ

[foxyshadis]

I finally reinstalled a build environment and built a fix (http://www.megaupload.com/?d=9ZWS9HDF) for a memory bug that mainly rears its head with MT. Thanks to leeperry for bugging me to fix it a while back.

[leeperry]

yay, too awesome! I've been stuck w/ 1.1 since forever...what are the improvements exactly in the newer versions please? Could they make a noticeable difference in GrainFactory3() :confused:

1.2 2006/06/06 Foxyshadis Supports YUY2, RGB. Fix cache mess.
1.3 2006/06/10 Foxyshadis Crashfix, noisegen optimization
1.4 2006/08/11 Foxyshadis Constant replaces seed, seed repeatable

what does "noisegen optimization" mean exactly? :thanks: a million!

[foxyshadis]

I believe that was from combining the two chroma noise planes into one, faster & less memory w/ no real difference. Wish I'd recognized the other major memory hog at the time.

I was also informed that this version won't run on Win2k, thanks to using VC 2010. Intentionally, since 2000 is out of support; I bet IanB's going to be hatin'. Damn it, Microsoft. Apparently there might be a way to patch it out, in the meantime cretindesalpes sent me a working fix (http://ldesoras.free.fr/src/AddGrainC-1.5.1.7z) built with 2005.

[leeperry]

oh, so it's mostly about speed improvements making compromises on PQ? coz somehow I prefer 1.1 over 1.4 in GrainF3() I think...but maybe it's placebo, I'll try again several times.

[foxyshadis]

The grain's the grain, I've never messed with how it gets created or combined with the frame. The only thing that I've ever changed is how it randomizes the per-frame, in order to support constant grain and defined grain even with random access, and enabling chroma (which is identical to luma in YV12) and RGB/YUY2 (which won't affect YV12 at all).

If you do prefer it, that might be because in 1.1, if a frame gets flushed out of cache it'll have a whole new randomized grainy image. I don't know how GrainF3 munges the image, but if it uses that fact, it will look different. It might be worth enlarging the possible grain size though.

The only thing pre-1.3 calculation would be good for is having different strength in the U and V planes... I can't think of any good reason why you would.

[leeperry]

it looks to me that the 3 grain layers in GrainF3 w/ 1.4 are not random anymore...they all seem to use the same pattern(except for the chunk size). w/ 1.1 I do get my "Grain EE 3D pop effect" feeling I've spent so much time working on...making halo-based EE is so yesterday :D

[leeperry]

..which makes me think that "seed repeatable" implies that provided I would find a "best" combination, I might be able to modify GrainF3 to always use the very same seeds/patterns....interesting!

[leeperry]

BTW:
seed (-1)
Specifies a repeatable grain sequence. Set to at least 0 to use.

constant (false)
Specifies a constant grain pattern on every frame.
at least 0? so this will set the grain sequence? what's the max? I'm not sure how to take advantage of the "repeatable seed" feature of 1.4 :o

[foxyshadis]

The max is probably either 2 billion or 4 billion, I forget. Any number will work about as well as any other. Every one will give you a different sequence of grain, but it'll always be identical for the same source parameters. Different colorspace or size will change everything. Give it a shot, set seed to something and keep reloading the script on different frames.

[leeperry]

ok, interesting...this could do exactly what I want.

EDIT: and it does! many :thanks:

[TurboPascal7]

Looks like a bug here. Bug appears with uvar > 0 and constant=true.
Example:
http://img529.imageshack.us/img529/3412/41349393.th.png (http://img529.imageshack.us/img529/3412/41349393.png)http://img832.imageshack.us/img832/2852/97947380.th.png (http://img832.imageshack.us/img832/2852/97947380.png)
Right down corner. Seems to work incorrect on any resolution on yv12.

[SSH4]

hmm, i have some bug

[AlekseiV]

hmm, i have some bugdrink lots of orange juice

[Reino]

...it may also be SSE2 only (haven't run it on the Athlon yet).I have and your filter doesn't work an my AMD Athlon XP 3200+.
For MCTemporalDenoise your filter is required, but here it comes down with this error:Evaluate: System exception - Illegal Instruction
(GradFun2DBmod.v1.5.avsi, line 291)GradFun2DBmod.v1.5.avsi:290 grain = custom=="empty" ? dither.addgrainC(str,strC,0,0)
291 \ : Eval("dither." + custom)
292 diff = custom=="empty" ? blankclip(dither,color_yuv=$808080).addgrainC(str,strC,0,0)
293 \ : Eval("blankclip(dither,color_yuv=$808080)." + custom)Removing the AddGrainC calls makes MCTemporalDenoise work, so I guess a SSE cpu, as mentioned in the readme, is not enough and your filter is actually SSE2 only at the moment.
If you find some time, could you please have a look?

[cretindesalpes]

I have and your filter doesn't work an my AMD Athlon XP 3200+.

Does this version (http://ldesoras.free.fr/src/avs/old/AddGrainC-1.5.3.7z) work better?

EDIT: Link now points to v1.5.3 instead of 1.5.2, I also fixed a bug occurring with RGB24 colorspace, causing unwanted coloration.