Saving grain for later use

[Audionut]

Is there any way (perhaps with masks and such) to create a copy of the grain and perhaps fine details of video, and save that information in to a small file, with the ability to add that information back to the video at playback?

It's only an idea, I would not be able to attempt it myself, nor have I really any idea how to do it.

But I know you guy's use masks and such for denoising/sharpening/etc. Perhaps is there was a way to save that information into say a smaller than 1mb file, then use ffdshow's avisynth to insert it back in at playback.

Would be great for x264!!

[Guest]

My company's silicon can do something like this (STMicroelectronics' 7109 MPEG/AVC/VC1 decoder). It doesn't save the grain, but it regenerates random grain and adds it to the display frames. It is very effective at eliminating that plastic look. There's no need to save the actual grain because it is random.

I also noticed Sony's big LCOS TVs have a slightly grainy display panel surface that achieves the same effect!

[FredThompson]

ffdshow can add some grain during playback. It IS a good way to reduce excess smoothness.

Hmm...I wonder if harshly removing grain and using a slight application of MSU's cartoon filter would give a semi-CGI appearance. That would be kind of funny, emulating CGI with live action.

[Mug Funky]

do you mean something similar to wavpack's dualstream feature, where a lossy track is stored, and a residual is also stored losslessly so they can be combined later?

easiest way would be to run your favourite denoiser, then subtract the result from the original.

then encode both versions (maybe the noisy one with a different codec?) and subtract them again on playback using an avisynth script.

i can't see it being terribly useful though - random noise doesn't compress very well at all, and will tend to swim around just like it would if it were left in the original (perhaps it'll swim in different directions which could be an improvement). this means you'll have twice (or more) the filesize to do what an addgrain on playback would do.

as neuron alluded to, newer encoding technologies can add grain on playback. i think h.264 (though no current software implementations that i know of) will store grain paramerically and reproduce it on playback. it wont be the same grain, but will have the same properties and look the same. quite clever actually, considering how much grain is removed by h.264 even at relatively low quantisers (~17-20).

[foxyshadis]

Use this prior to denoising or encoding and save it:

grain=default(grain,10)
gthresh=default(gthresh,1.0)
gmask=fft3dfilter(sigma=gthresh*10,sigma2=gthresh*4,sigma3=gthresh,sigma4=0,plane=4) \
.mt_lutxy(last,expr="x y - abs",chroma="process") \
.BilinearResize(m(4,width/4),m(4,height/4)).mt_lut(expr="x 2 ^",chroma="process")

and on playback:

mt_merge(last,addgrainc(grain*10,grain*2),gmask.bilinearresize(width,height),chroma="process")


This is an extremely avisynth-specific implementation of the general idea: form a mask of noisy areas from the main picture, shrink it to a blobby mess, then add random noise back to the picture based on the (enlarged) mask. It'll definitely compress extremely well. I believe that's exactly what AVC FGS does. You'd have to hack ffdshow to support playback merging of an external stream this way, probably so much that it'd be easier to add FGS to lavc instead.

A smart enough grain filter could probably determine the best areas to apply it back to without an external reference, which would be an interesting area of research someday. Probably too mathematical for me though!

[Boulder]

A grain factory created by Didée, needs MaskTools v1.5.x and AddGrain:

function GrainFactory(clip last,int "grain1_strength",int "grain2_strength",int "grain1_sharpness",int "grain2_sharpness")
{

grain1_strength=default(grain1_strength,5)
grain2_strength=default(grain2_strength,2)
grain1_sharpness=default(grain1_sharpness,75)
grain2_sharpness=default(grain2_sharpness,75)

#grain1_strength = 7 # [ 0 - ???] strength of grain / for mid-tone areas
grain1_size = 1.15 # [0.5 - 4.0] size of grain / for mid-tone areas

#grain2_strength = 0 # [ 0 - ???] strength of grain / for dark & bright areas
grain2_size = 0.8 # [0.5 - 4.0] size of grain / for dark & bright areas

#grain1_sharpness = 75 # [ 0 - 100] very soft (0) to very sharp(100)
#grain2_sharpness = 75 # [ 0 - 100] grain definition

grain1_texture = 0 # [ 0 - 100] strength of "material texture" appearance
grain2_texture = 0 # [ 0 - 100] (in fact: 'bump' effect)

ontop_grain = 0 # [ 0 - ???] additional grain to put on top of prev. generated grain

temp_avg = 20 # [ 0 - 100] percentage of noise's temporal averaging

brightness_bias = 0.96 # [0.5 - 2.0] bias for dark|bright --> midtone fading

#-----------------------------------------------------------------------------------
o = last
ox = o.width
oy = o.height
sx1 = m4(ox/grain1_size)
sy1 = m4(oy/grain1_size)
sx1a = m4((ox+sx1)/2.0)
sy1a = m4((oy+sy1)/2.0)
sx2 = m4(ox/grain2_size)
sy2 = m4(oy/grain2_size)
sx2a = m4((ox+sx2)/2.0)
sy2a = m4((oy+sy2)/2.0)
b1 = grain1_sharpness/(-50.0) + 1.0
b2 = grain2_sharpness/(-50.0) + 1.0
b1a = b1/2.0
b2a = b2/2.0
c1 = (1.0-b1)/2.0
c2 = (1.0-b2)/2.0
c1a = c1/2.0
c2a = c2/2.0
t1a = string(grain1_texture)
t1b = string(100-grain1_texture)
t2a = string(grain2_texture)
t2b = string(100-grain2_texture)
tmpavg = temp_avg/100.0

#-----------------------------------------------------------------------------------

ADD_DIFF = "x y - 128 +"
BUMPKERNEL1 = T1a + " 0 0 0 " + T1b + " 0 0 0 0"
BUMPKERNEL2 = T2a + " 0 0 0 " + T2b + " 0 0 0 0"
CENTER2MAX_BIAS = "x 120 - abs "+string(brightness_bias)+" ^ 2.5 *"

#-----------------------------------------------------------------------------------

grainlayer1 = blankclip(o, width=sx1, height=sy1, color=$808080) .AddGrain(grain1_strength, 0, 0)

grainlayer1 = (grain1_texture==0)
\ ? grainlayer1
\ : yv12lutxy(grainlayer1,grainlayer1.DEdgeMask(0,255,0,255, BUMPKERNEL1), ADD_DIFF)

grainlayer1 = (grain1_size == 1.0) ? grainlayer1
\ : (grain1_size < 0.75) ? grainlayer1.bicubicresize(sx1a,sy1a, b1a, c1a).bicubicresize(ox,oy, b1a,c1a)
\ : grainlayer1.bicubicresize(ox,oy, b1,c1)

#-----------------------------------------------------------------------------------

grainlayer2 = blankclip(o, width=sx2, height=sy2, color=$808080) .AddGrain(grain2_strength, 0, 0)

grainlayer2 = (grain2_texture==0)
\ ? grainlayer2
\ : yv12lutxy(grainlayer2,grainlayer2.DEdgeMask(0,255,0,255, BUMPKERNEL1), ADD_DIFF)

grainlayer2 = (grain2_size == 1.0) ? grainlayer2
\ : (grain2_size < 0.75) ? grainlayer2.bicubicresize(sx2a,sy2a, b2a, c2a).bicubicresize(ox,oy, b2a,c2a)
\ : grainlayer2.bicubicresize(ox,oy, b2,c2)

#-----------------------------------------------------------------------------------

grainlayer = MaskedMerge(grainlayer1, grainlayer2, o.yv12lut(CENTER2MAX_BIAS), U=1,V=1)

grainlayer = (temp_avg==0) ? grainlayer
\ : grainlayer.mergeluma(grainlayer.temporalsoften(1,255,255,255,2), tmpavg)
grainlayer = (ontop_grain==0) ? grainlayer
\ : grainlayer.AddGrain(ontop_grain,0,0)

#-----------------------------------------------------------------------------------

result = yv12lutxy(o, grainlayer, ADD_DIFF, U=2,V=2)

return result
}
#-----------------------------------------------------------------------------------
function m4(float x) {return( x<16?16:int(round(x/4.0)*4)) }
#===================================================================================

It would be nice if someone could modify it for MaskTools v2, I recently tried it but I'm not sure if it worked just like this older version.

The function can be used to create a very nice film grain effect and add fake details to the image.

[Audionut]

I've never really liked any of the addgrain implementations of ffdshow for one reason or the other.

X264 removes grain at lower bitrates.

I was thinking that if it was possible to take the "original" grain from the video and add it back at playback, that would be better than adding artificial grain.

[Guest]

Compression is achieved by throwing away grain and small detail (among other things). Keeping it is equivalent to using a higher bitrate. As they say, you can't have your cake and eat it too.

[Alain2]

My company's silicon can do something like this (STMicroelectronics' 7109 MPEG/AVC/VC1 decoder). It doesn't save the grain, but it regenerates random grain and adds it to the display frames. It is very effective at eliminating that plastic look. There's no need to save the actual grain because it is random.

I also noticed Sony's big LCOS TVs have a slightly grainy display panel surface that achieves the same effect!
Can someone explain to me why it is bad to remove noise (not details)? In the real world, my eyes aren't seeing random noise when I look at someone, so why should we expect noise on films?

You all say that x264 is removing too much noise and gives a plastic look, but to me it's getting closer to reality... When I filter the noise but keep details, and then encode with --filter -3:-1 or so to limit the blur from inloop, this preserves quite well the textures, and I don't have the "plastic" impression that I had with the rv10 codec for instance..

[MrTroy]

Can someone explain to me why it is bad to remove noise (not details)?Another example of 'you can't have one without the other'. Removing noise removes details. Too much detail removing creates a plastic-y look. Adding grain masks this bad effect.

[Alain2]

Yes, but you can greatly preserve details with motion compensation tools, and remove mainly noise ; I do plenty of comparison between my source and my filtered clip, I hardly see what detail is altered, only the noise ; I think the noise is a way to compensate the small pixel resolution we have, that makes all small movements obvious and unnatural on a noiseless pictures, but as we increase the resolution of the clips, I think it will be less and less needed to do this..

[Soulhunter]

There's no need to save the actual grain because it is random.
Wrong! Filmgrain isnt random... While noise has mostly fixed characteristics, the characteristics of filmgrain depend on many things like brightness, processing or the film material... To quote myself: "Basically film grain is nothing more than a lot silver particle clusters... Most motion-picture film material is made of 3 layers (one for each color -> RGB) and they contain dye + silver salts. In the dark parts of the image the salts get transformed into metallic silver (which blocks the light) when the film gets exposed, this is also the reason grain is usually much rougher in dark areas than in bright ones! Why are some flicks more grainy than others? Well, you can get celluloid in different granularity grades! Why? Celluloid with low granularity has a lower photo-sensitivity (lower ISO speed) and celluloid with high granularity has a higher photo-sensitivity (higher ISO speed) Means: For celluloid with low granularity you need much more illumination than youd need for celluloid with a high granularity! The film format makes a difference as well, Super8 film for example looks usually more grainy than 35mm film (8mm = smaller than 35mm = needs more upsizing = grain becomes more visible). And, last but not least there is also postprocessing/filtering which can be used to remove or enhance the grain! And what is noise? Noise could be anything, depends on the context... For video its basically everything that distorts the signal, and its mostly a unwanted by-product (analog interferences, compression artefacts etc.)" Another thing to consider is the color... Each layer of the film contains a different amount of grain, the blue layer for example is usually much more grainy than the red/green one! So, to reproduce grain accurately youd need to analyze the luma and the underlying chroma of each pixel and add different amounts and sizes of grain depending on this!

Anyway, here (http://soulhunter.chronocrossdev.com/data/gnoise_r4.zip) is another filter to add noise via avisynth... ^^

[Guest]

Well, I oversimplified. Yes, the grain that is recreated is parameterized from local image content. What I meant was it is not necessary to save the exact grain from the source.

[FredThompson]

@soulhunter,

You're vague, again. Would it be too much to ask for you to be specific in your posts?
A little attention to detail and some background information never hurt anyone. http://home.mindspring.com/~fredthompson/Smiley-LOL.gif

[Soulhunter]

You're vague, again. Would it be too much to ask for you to be specific in your posts?

What do you mean? Why grain in dark areas looks rougher? The dark areas are dark coz silver transforms and blocks the light, so this area becomes less transparent, therefore the grain becomes more visible... The bright areas stay more transparent (let more light through) so you see less grain! Why grain is usually more visible in blue areas? Coz the grain in the blue emulsion has usually a higher diameter (up to 1/3 higher iirc) than in the red / green one! However, all this things can vary a lot depending on the exact purpose the film material was made / developed / thought / tweaked for -> in-door shooting, outdoor shooting, low light condition, etc... Or you meant that you want some more general info about film / photography? Some random links: 1 (http://64.78.42.182/sweethaven/MiscTech/Photog01/default.asp?unNum=2&lesNum=2), 2 (http://en.wikipedia.org/wiki/Photographic_film), 3 (http://en.wikipedia.org/wiki/Film_stock), 4 (http://www.diyphotography.net/techniques/something-about-analog-film), 5 (http://www.imx.nl/photosite/technical/Filmbasics/filmbasics.html). But theres much much more out there, google is your friend! ;)


Bye

[FredThompson]

Sorry, I still don't understand what you're trying to explain. Can you give more details?

It's obvious you're trying to be helpful. I appreciate that. It's still a little...to...vague...

[Guest]

Fred, please knock it off. A rule 11 strike is imminent. Thank you.

http://forum.doom9.org/forum-rules.htm

If you have any questions, please follow up in PM, not here.

[FredThompson]

Geez, lighten up. The smiley should make the first reply obvious. He's got thick enough skin to recognize the backhanded comment and the humor.

edit: That's supposed to read "compliment" - should be obvious ...

[Guest]

Geez, lighten up. The smiley should make the first reply obvious. He's got thick enough skin to recognize the backhanded comment and the humor. I asked you to take discussion to PM. Therefore, struck for rule 16.

[bond]

as neuron alluded to, newer encoding technologies can add grain on playback. i think h.264 (though no current software implementations that i know of) will store grain paramerically and reproduce it on playback. it wont be the same grain, but will have the same properties and look the same. quite clever actually, considering how much grain is removed by h.264 even at relatively low quantisers (~17-20).are you sure that it wont be the same grain?

actually i think if someone would be willing to write an avs plugin that outputs the grain info into a file aku will add support for the avc filmgrain feature to x264

[Soulhunter]

actually i think if someone would be willing to write an avs plugin that outputs the grain info into a file aku will add support for the avc filmgrain feature to x264

Hmm, remembers me this... (http://forum.doom9.org/showthread.php?p=674223#post674223) But yeah, a configurable avs plugin which produces a info file about the grain characteristics would probably do a much better job! Coz this way you can also reproduce the grain that was lost by compression, and not just the grain that was lost by pre-processing. Btw, if someone needs samples with a lot film grain, I could upload some minutes of this flick here... (http://img61.imageshack.us/img61/8262/dr40um.png) ^^


Bye

[Kopernikus]

are you sure that it wont be the same grain?

it is not the same. The film grain feature of H.264 gives you the possibility to model grain by a few parameters. The actual decoded grain is random. A decoder implementation doesn't need to use the models described in the spec, it can use what it likes.

[akupenguin]

actually i think if someone would be willing to write an avs plugin that outputs the grain info into a file aku will add support for the avc filmgrain feature to x264
Sure. The tricky part (aside from determining the correct parameters) is if you want to compare unfiltered source against filtered + compressed x264 output. See, the grain parameters are stored as SEI packets interleaved into the video stream. So in order to add FGM to an existing h.264 file, you need to demux, modify the stream, and remux. Which isn't theoretically hard, but it's not something I want to add to x264cli. I might start another frontend for such tasks (e.g. a cavlc<->cabac converter too).
The alternative would be to compare against the 1st pass, then generate FGM, and add it to the 2nd pass. Which is better than nothing, but would lose a bit of precision from using the wrong QPs (and might also be affected by some of the other x264 options disabled in fast 1st pass).

[bond]

Sure. The tricky part (aside from determining the correct parameters) is if you want to compare unfiltered source against filtered + compressed x264 output. See, the grain parameters are stored as SEI packets interleaved into the video stream. So in order to add FGM to an existing h.264 file, you need to demux, modify the stream, and remux. Which isn't theoretically hard, but it's not something I want to add to x264cli. I might start another frontend for such tasks (e.g. a cavlc<->cabac converter too).
The alternative would be to compare against the 1st pass, then generate FGM, and add it to the 2nd pass. Which is better than nothing, but would lose a bit of precision from using the wrong QPs (and might also be affected by some of the other x264 options disabled in fast 1st pass).hm i thought about something like an avs plugin thats used in the first pass writing the grain info to a file. after that plugin a "degrainer" plugin (whatever denoise someone might prefer) gets used in the script outputting what gets fed into x264
the grain plugin is obviously only needed in the first pass, the degrainer in both

there would be two problems i think:
1) difference between the grain stored by the grain plugin and the grain removed by the degrainer
2) difference between the degrained source and the x264 output

i dunno if the difference if 1) is in any ways important but i think 1) could be fixed by a smart plugin combining the grain-to-file and degrain function (eg writing the info it removes by degraining into a file)
i think 2) can propably never be solved

[Soulhunter]

Hmm, I thought of a plugin or tool which saves the grain characteristics to a separate info file by comparing the non-denoised source with the denoised and compressed result! This file could be muxed into the mp4 as private track or so... Then we'd need a decoder side function which is able to read/understand this info (maybe in ffdshow) and tries to reproduce the grain... Decoders that are not aware of this stuff should simply ignore the private track and decode the video normally, no?


Bye

[foxyshadis]

SEI packets are basically bitstream-level private streams. So it'd make more sense to put it there instead of muxing it into the container, but of course that needs a tool to mux it into the bitstream.

It'd make for an interesting external tool! The big question is what sort of format would the file have to be in? How do you measure and describe the characteristics of noise?

[Soulhunter]

SEI packets are basically bitstream-level private streams. So it'd make more sense to put it there instead of muxing it into the container, but of course that needs a tool to mux it into the bitstream...
But, can you pack whatever you want into it, or is this defined in the specs? Coz then your possibility are sorta limited again... while some new developed info file format would give you much more freedom, huh?


Bye

[akupenguin]

H.264 defines a few grain models and a syntax to store them in SEI packets. Of course you get more freedom from ignoring the standard, but then the same could be said of ignoring H.264 for the compression part too, and writing a new codec from scatch. But freedom is useful only if you think you can design a better codec.

[Soulhunter]

H.264 defines a few grain models and a syntax to store them in SEI packets. Of course you get more freedom from ignoring the standard, but then the same could be said of ignoring H.264 for the compression part too, and writing a new codec from scatch. But freedom is useful only if you think you can design a better codec.
Uhm, but replacing H.264 with some new codec would result in a file you cant decode with a H.264 decoder anymore, while adding some txt as private track should be perfectly legal (spec and decoding wise). Normal decoders should simply ignore the the private track and decode the video normally, as they do now... But I see what you mean, why re-inventing the wheel, eh? Well, it was just a suggestion for the case that the info the SEI packets are able to store is too insufficient/limited to reproduce film grain. But as you (as dev) know much more about H.264 than me, I simply gonna trust ya expertise... ^^


Bye

[bond]

what should be the advantage of storing the grain info somewhere instead of storing it the way avc defines it?

[Soulhunter]

Well, you could use some self developed info file format which would allow you to store info you maybe cant store via SEI packets, dunno... Guess to reproduce film grain properly you have to pack all kinds of info into it, like: amount, opacity, diameter, sharpness etc... and this for each frame!

Ok, to reproduce grain perfectly storing the info global per frame wouldnt be enough... Sharpness should be constant over a frame in most cases, but amount / opacity / diameter can change over a frame, so you have to grade em somehow ie. by the levels and color ratios... But then you'd need to analyze the video again while playback (levels and color ratios) and this would be fu*king slow, heh? So using only frame global params is the way to go, no? Leaves the question if you can store info like this -> FrameNumber#, Amount#, Opacity#, Diameter#, Sharpness# <- via SEI packets? Dunno, lets wait for response from some dev, chances are good I underestimate the h.264 specs... ^^


Bye

[Manao]

The AVC standard provides two standardized ways to store the grain information.

First one : you generate random 16x16 DCT coefficients, then you apply a filter much like DCTFilter on these coefficients, to "shape" the noise. Then you perform an iDCT and add the resulting noise to the picture. The parameters of the noise are roughly the parameters of the DCTFilter-like filter

Second one : you generate an AR noise. Noting N(x,y) the noise at the point (x,y), you have the following formula :

N(x,y) = A*gaussian-noise + B*N(x,y-1) + C*N(x-1,y) + D*N(x-1,y-1) + E*N(x+1,y-1) +F*N(x-2,y) + G*N(x,y-2)

A,B,C,D,E,F and G are the parameters of the noise.

In both cases, parameters can be mathematically identified once the grain is isolated. Isolating the noise is another story...

[foxyshadis]

Maybe this should be moved to the AVC forum anyway.

What about using --nr? nr works in the transform domain, and if one of the noise profiles is based on DCT transform, perhaps that could map over somehow?