Grain/noise retention outside the encoder

[Morte66]

I orignally meant to post this as a tangent in the "x264 MUCH better quality than Xvid?" thread, but it seems better to start a new thread about it.

If it's an absolutely essential component of the director's vision, xvid can often keep the grain more perfectly modeled than even the best x264 settings+post-added grain, as morte found. Most people probably aren't the kind of hardcore film buffs like him, but enough are that it's important to remember them.

Mostly I don't like to reencode the noise from DVD (which I think is mainly MPEG2 crud from the godawful DVD encodes rather than actual film grain). I don't normally like grain, I generally consider it a flaw of the chemical filmmaking process which can be removed in software. So I'm usually trying to make a "cleaned up" encode with denoise and sharpen etc. Faithfulness to the DVD is the last thing I want, and faithfulness to the negative is no biggie. Sometimes I add some borderline-subliminal noise on playback to create texture where there was no real detail in the source.

But once in a while I hit some material where the noise on the DVD is a good thing and I want to keep it. And I know there are people who'd like to retain all of the noise/grain, all of the time. Xvid seems to handle that well and relatively easily, provided you don't mind really big encodes (sometimes 200% the size of the source). I'm thinking that x264 with deazone tweaking and a CQM might come close at usefully lower bitrate, but for my occasional usage it's not worth getting into the "tweak and tweak and tweak again" cycle. Xvid is easier.

The really interesting option is to denoise and use a noise generator (e.g. ffdshow) on playback. But there are a couple of problems at the moment. Firstly you can't get the noise quite right, e.g. on "The Shield" it packs the noise too close, it needs a "stretch" parameter. Second, it's a pain. Every time you start playing something -- as I might do half a dozen times in an evening -- you have to pop up that ffdshow dialogue and pull sliders around. Third, noise is sometimes different in certain scenes, and there's no way I'm halting playback on "Boomtown" every time a flashback starts/ends.

I would love to see an add-on system that did noise reduction up front, characterised the noise (probably per-GOP/scene), and fed parameters to a noise "regenerator" in the decoder or a post-processor. It should be something that just works once configured, with no need to fiddle with sliders unless you want to. If you could dial the noise regenerator defaults down to "low" or "none" then the same encode might serve people who like to retain noise and people who like to lose it.

I know there's a facility for some sort of grain retention in the h264 specs, but I've never seen any examples I don't know how well it works in practice. At the moment, it's just an intriguing concept for me. It might be pie in the sky.

So I wonder does anyone have any practical experience they'd like to share? Any samples? Any thoughts?

[Soulhunter]

Actually youd just need to calculate/save 2 params per gop/scene... Grain amount [depends on the ISO speed of the film and the luminance/color] and grain size [how huge the grain clusters are... depends on the ISO speed of the film and how much it was upsized...]. While playback you scale the visibility/opacity of the generated grain depending on the luminance [per pixel, not global]. For the Y channel if it needs to be fast, or for each RGB channel if speed doesnt matter... Unfortunately both methods should be too slow for realtime post-processing, at least atm!


Bye

[shon3i]

Film Grain modeling, like Aku says earlier, maybe isn't magic bulet for transparency, but IMHO, encodings will be looking much better especialy 1CD rips.

Adding noise on playback is remind me, on time when i playing audio casstess on non Dolby unit, that is horrible, like now playing AVC video without ffdshow(noise on)

[Morte66]

Addition to my last post:

For the Y channel if it needs to be fast, or for each RGB channel if speed doesnt matter...

I've been thinking about the RGB issue...

Any still photograhper who does digital postprocessing will tell you that in practice you get most of your noise in the blue channel, some in red, not much in green. Furthermore, most chromogenic (dye-based) colour film is made of three layers sensitive to cyan/magenta/yellow light (complementary colours to RGB). These are not always the same materials, they may have entirely different grain characteristics. So it might be necessary to process R/G/B separately at all stages of the chain.

But I don't know how much this matters in practice.

Unfortunately both methods should be too slow for realtime post-processing, at least atm!

I wonder if it's a suitable task for GPU acceleration.

[R3Z]

I happen to really love film grain and can accept noise in most areas, as long as its there because of the camera technology.

I really would like to get x264 to have some sort of option to tell ffdshow to add grain during playback and how much.

How does On2VP7 compare to x264 in grain retention ? They seem to be beating their own drum in regards to their technology and how much better than h264 it is.

Sorry if i kinda strayed here !

[Sharktooth]

i remember vp6/vp7 had something abuout grain regeneration during playback but i dont think it's based on the source grain (i can be wrong though).

[lexor]

In relation to Soulhunter's example, that second picture can be mimicked by Perlin noise (with the right parameters). Is there an AviSynth or FFDshow plugin for Perlin noise? (my search skills yield - "No") Would be nice though.

[Sharktooth]

well, the picture shows one kind of grain... there are at least 6 different kinds...

[lexor]

well, the picture shows one kind of grain... there are at least 6 different kinds...
From what I understand the point of hunter's post was that Gaussian noise isn't one of those 6 types, so having a Perlin noise generator may not give us everything, but at least it'll give us something.

[Sharktooth]

yep, but dont forget noise and film grain are 2 completely different things... ;)

[lexor]

not if one is made to look like the other.

[Sharktooth]

ehr... noise cant look like grain and vice versa.

[CruNcher]

Kodak (who else) has some very good Digital Film Grain Models of their stock available for professional usage

[Mug Funky]

@ lexor: i wrote one for avisynth, but it's teh suxxor :)

@ sharktooth: grain and noise are both wideband signals... with the right filtering you should be able to make one look like the other. sort of like how outdoor ambience can often be faked with filtered pink noise (like waterfalls, wind, rain, etc).

[Sharktooth]

uhm, faking doesnt mean it will be the same.
digital film grain models are way more complex than simple noise.
as i said, noise and grain look completely different and are 2 completely different things.

[lexor]

uhm, faking doesnt mean it will be the same.
digital film grain models are way more complex than simple noise.
as i said, noise and grain look completely different and are 2 completely different things.

Maybe, but we aren't really competing against proper grain either. Most of my encodes are for HD episodes and their "grain" is no where near real (or really good digital) grain. Just look at pathetic graining in 24/Star Trek Enterprise/House, not only can noise algorithms (especially dynamic systems based ones) produce equivalent, we can actually get a better one going.

It is true that we aren't going to magically get a really good film grain algorithm, but we aren't competing with really good grain algorithms in "pro" space either. And it's not just TV shows, even movies that are shot in digital are starting to drop the ball and moving from "aesthetically pleasing" grain (heard that phrase used by directors and such) to just noise. Only the biggest blockbasters are keeping up the visual quality nowadays.

[HeadBangeR77]

Could someone tell me, without going too deep into physical characteristics of noise and grain, which is which in this crappy sample o'mine?

http://forum.doom9.org/showthread.php?p=945832#post945832
http://forum.doom9.org/showpost.php?p=947397&postcount=85

If you looked at that thread, *.mp4 guy has managed to keep like 75% of grain and noise using x264, what I would name a miracle, if he wasn't the mp4 guy ;) - target bitrate 3500 kbps. 2-pass encodes help much + Qpel in both passes while using XviD.

[Soulhunter]

Have a look at this! (http://img246.imageshack.us/img246/5830/dr4dz4.png) Its from "The Devils Rejects" [PAL/R2 DVD]. Very grainy flick!

But it offers a detail grade you wont find on much DVDs (http://img257.imageshack.us/img257/2231/dr2rf9.png), huh?

Try to keep this via h.264... >.>


Bye

[bond]

i remember vp6/vp7 had something abuout grain regeneration during playback but i dont think it's based on the source grain (i can be wrong though).those simply add artificial grain to the video iirc

[Morte66]

I wonder whether grain/noise would have to be actually generated during playback. Perhaps you could have a library of 30 second noise patterns that just get looped and mixed in.

[akupenguin]

That only works on purely additive noise. If your grain depends on the content of the video, then you can only precompute a string of random numbers, not the grain modelling part.

[*.mp4 guy]

I cobled together a function that does a pretty good job simulating the grain found in the Matrix (first movie), its slow, though. It should be possible to make other types of noise aswell by changing the convolutions applied to the grain.

function filmgrain(clip c, int "str"){
str = default(str, 4)
X = str
c
grain = addgrain(x,0,0)
graindiff = mt_makediff(last, grain, u=3, v=3)
grainc = graindiff.unsharp(vary=1.5, varc=1.5, strength=1).unsharp(vary=0.75, varc=0.75, strength=0.5).gaussianblur(vary=0.3, varc=0.3).unsharp(vary=0.75, varc=0.75, strength=1)
blank = mt_makediff(last, addgrain(last, round(x/3), 0, 0))

grainld = maskedmerge(grainc, blank, last)
mt_adddiff(last, grainld)
return(last)}

[R3Z]

I cobled together a function that does a pretty good job simulating the grain found in the Matrix (first movie), its slow, though. It should be possible to make other types of noise aswell by changing the convolutions applied to the grain.

function filmgrain(clip c, int "str"){
str = default(str, 4)
X = str
c
grain = addgrain(x,0,0)
graindiff = mt_makediff(last, grain, u=3, v=3)
grainc = graindiff.unsharp(vary=1.5, varc=1.5, strength=1).unsharp(vary=0.75, varc=0.75, strength=0.5).gaussianblur(vary=0.3, varc=0.3).unsharp(vary=0.75, varc=0.75, strength=1)
blank = mt_makediff(last, addgrain(last, round(x/3), 0, 0))

grainld = maskedmerge(grainc, blank, last)
mt_adddiff(last, grainld)
return(last)}

Thanks for that mp4guy, i was just wondering what unsharp reffers to ? I can only find the plugin tunsharp.

Cheers,

jase

[*.mp4 guy]

unsharp is part of the (very usefull) variable blur plugin, which is also used for bluring in the script.

[R3Z]

unsharp is part of the (very usefull) variable blur plugin, which is also used for bluring in the script.

Thanks for that ! I am sorry but i cant get the function to work.

I tried making filmgrain into an avsi and calling it as FilmGrain() in my avs, but what seems to happen is it just uses up all the available memory when i try to preview it in mplayerc.exe and when opening it in megui, it says says the avs might be corrupt.

These are my plugins and their versions;


AddGrain.dll (0.1.0)
AddGrainC.dll
ColorMatrix.dll
colors_rgb.avsi
Convolution3DYV12.dll
corrector.dll
Decomb.dll
degrainmedian.dll
DGDecode.dll
DirectShowSource.dll
EEDI2.dll
filmgrain.avs
FluxSmooth.dll
LeakKernelDeint.dll
MaskTools.dll
MCBob_v03b.avsi
medianblur.dll
mt_masktools.dll (2.0.30.0)
mvbob.avs
mvbob.avsi
mvtools.dll (1.6.2.0)
NicAudio.dll
ReduceFlicker.dll
ReduceFlickerSSE2.dll
ReduceFlickerSSE3.dll
RemoveGrain.dll
RemoveGrainS.dll
RemoveGrainSSE2.dll
RemoveGrainSSE3.dll
Repair.dll
RepairS.dll
RepairSSE2.dll
RepairSSE3.dll
SangNom.dll
seesaw.avsi
SimpleResize.dll
SmoothDeinterlacer.dll
SSE2Tools.dll
SSE3Tools.dll
SSETools.dll
SSEToolsS.dll
SSIM.dll
TCPDeliver.dll
TDeint.dll
test.txt
TIVTC.dll
TMonitor.dll
TomsMoComp.dll
UnDot.dll
VagueDenoiser.dll
VariableBlur.dll (0.4.0)


I am trying not to be a noobie about this :confused:

[*.mp4 guy]

Make sure you have the latest version of both masktools 2 and masktools 1.5, the script only uses 200MB of memory on a HD clip I'm testing so memory shoudn't be a problem unless you are using it on a huge resolution video.

[R3Z]

Make sure you have the latest version of both masktools 2 and masktools 1.5, the script only uses 200MB of memory on a HD clip I'm testing so memory shoudn't be a problem unless you are using it on a huge resolution video.

I re-downloaded the variableblur plugin and replaced the existing one which now works fine :confused: Strange yeah !

This functions is awesome, i give you two thumbs up mate. This along with the noise factory from didee are great for adding the grain back to mimic grain and some small details.

[Didée]

There's an obvious (mini-) optimization, getting rid of those 2 instances of mt_makediff() :

[...]
c
greyclip = blankclip(color_yuv=$808080)
graindiff = greyclip.addgrain(x,0,0)
grainc = graindiff.unsharp( ......
# blank = greyclip.addgrain(last, round(x/3), 0, 0)) # wrong semantic ... copy/paste error ;-)
blank = greyclip.addgrain(round(x/3), 0, 0))
[...]

Not a big deal, since most cycles are spend in the blur/sharpen functions ... but the change is easy, so why not do it. ;)

[*.mp4 guy]

Thanks Didée, I'm working on a potential speedup for the gaussian operations, I think I can replace them wih a dct based nyquist limiting function that will run faster, but I'm not sure, there is a possibility it might run slower (or not look as good).

[*.mp4 guy]

Well I wrote the function to do the lowpass, but at low precision (when its faster) it looks worse, and at high precision it doesn't look different enough to justify the speed hit. I'll look into fft3dgpu, but I don't think I can make the function any faster and keep the quality good enough to bother with.

@Didée I can't get you're suggestion to work, addgrain complains about bad input parameters (sorry to paraphrase, but I don't remember exactly what the error was) I'm using avisynth 2.5.7.

[Didée]

My fault, too much hurry. Corrected.

[Soulhunter]

@ *.mp4 guy

How about some LUT magic to lower the grain opacity towards white, and rise it towards black? And/or a option to set the grain dots diameter [to adjust it to the resolution of the source] like in this noise factory script Didée wrote some time ago... ^^


Bye

[*.mp4 guy]

It does lower the grain opacity as brighness increases, grain also gets smaller as brighness increases, I could add a levels function and some variables to make it more tweakable though. Grain size could be made adjustable by turning the gaussian radii into variables, but right now all of my attention is going into the dct based resizer I'm working on, so it will have to wait.

[Soulhunter]

It does lower the grain opacity as brighness increases, grain also gets smaller as brighness increases...

It does? *Reads script again* >.>


Bye

[anton_foy]

Great script *.mp4 Guy!

How can I get the noise also colourized like AddgrainC(3,3,0) with the FilmGrain func.?

[*.mp4 guy]

Its probably possible using tweak on the noise pattern before it is aplied to the picture, but I don't know what the specifics would entail.

[Sagekilla]

Neat little idea I had would be if it were possible for x264, during the encoding process, to set some flags on a per-frame basis that could be used to dynamically change how the grain/noise is generated for each and every frame. To be more specific.. It would probably require some grain/noise models to analyze the frame and pick the "best" choice, as well as tweak a few of those presets on the fly for a best fit. Of course, you'd probably need a special decoder that handles this.. It would take the little overhead from those noise/grain flags (Maybe a few bytes just to say something like "strength=32, contrast=12" etc) and convert it to noise to be added at decode time.

Just an idea.. I don't know if anyone would actually implement it or anything, but that would be pretty cool to see. I'd imagine it would increase encode times too since it'd be analyzing the frame and comparing it to a noise/grain model, tweaking it, etc for a best fit.

[Sharktooth]

there are film grain modeling algos defined in the standards and they should work pretty well. obviously it takes time to implement them in both the encoder and the decoder and not all codecs support them.