Log in

View Full Version : Posterization of flat areas with FFT3DGpu

denissio
16th December 2007, 16:37
Hi!
First of all I would like to thank the authors of FFT3DGpu (and FFT3DFilter) for this great denoiser. But I've got some problems..
I'm now going to rip a DVD with 'Stranger than Fiction' movie inside and it has a lot of noise in under-lighted scenes, mostly in flat areas. My usual script for filtering the majority of not very noisy DVDs looks like this:

Undot()
Temporalsoften(2,3,3,mode=2,scenechange=6)
mergechroma(blur(1.3))
FluxSmoothST(5,7)

But in this case I felt that this was not enough, and decided to browse the Doom9 forum :)
Finally, I've chosen FFT3DGpu. The results were awesome, and I was about to start the encode, but .. I've noticed strange effect in flat areas - is it posterization?! The transition between colors was not gradual, it became jerky..

Here are some screenshots..
First, we can clearly see that the pictire is not too overfiltered:

frame 56140 original:
http://img218.imageshack.us/img218/7581/stf5614000ia2.th.png (http://img218.imageshack.us/my.php?image=stf5614000ia2.png)

frame 56140 filtered:
http://img229.imageshack.us/img229/1989/stf56140qi5.th.png (http://img229.imageshack.us/my.php?image=stf56140qi5.png)

frame 74162 original:
http://img217.imageshack.us/img217/4162/stf7416200ng0.th.png (http://img217.imageshack.us/my.php?image=stf7416200ng0.png)

frame 74162 filtered:
http://img231.imageshack.us/img231/857/stf74162xw0.th.png (http://img231.imageshack.us/my.php?image=stf74162xw0.png)

Now we've reached the 'problematic' scene. The first frame in the scene is filtered correctly:

frame 136334 original:
http://img220.imageshack.us/img220/9199/stf13633400cu0.th.png (http://img220.imageshack.us/my.php?image=stf13633400cu0.png)

frame 136334 filtered:
http://img229.imageshack.us/img229/6985/stf136334pr5.th.png (http://img229.imageshack.us/my.php?image=stf136334pr5.png)

The next frame gets posterized (Maybe problem with the source? I dunno): >.<

frame 136335 original:
http://img220.imageshack.us/img220/8336/stf13633500xb7.th.png (http://img220.imageshack.us/my.php?image=stf13633500xb7.png)

frame 136335 filtered:
http://img229.imageshack.us/img229/3135/stf136335bm9.th.png (http://img229.imageshack.us/my.php?image=stf136335bm9.png)

The effect gets even stronger here:

frame 136468 original:
http://img231.imageshack.us/img231/4338/stf13646800zo1.th.png (http://img231.imageshack.us/my.php?image=stf13646800zo1.png)

frame 136468 filtered:
http://img216.imageshack.us/img216/6267/stf136468hj4.th.png (http://img216.imageshack.us/my.php?image=stf136468hj4.png)

And here:

frame 136773 original:
http://img229.imageshack.us/img229/1669/stf13673300ch7.th.png (http://img229.imageshack.us/my.php?image=stf13673300ch7.png)

frame 136773 filtered:
http://img216.imageshack.us/img216/4053/stf136733yc1.th.png (http://img216.imageshack.us/my.php?image=stf136733yc1.png)

frame 136773 filtered (I've stressed the effect):
http://img176.imageshack.us/img176/3623/stf136733explmu0.th.png (http://img176.imageshack.us/my.php?image=stf136733explmu0.png)


PS. My script looks like the following:

# PLUGINS
Loadplugin("D:\Program Files\AviSynth 2.5\plugins\FFT3dGPU.dll")
LoadPlugin("D:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("D:\PROGRA~1\GORDIA~1\AviSynthPlugins\UnDot.dll")
LoadPlugin("D:\PROGRA~1\GORDIA~1\AviSynthPlugins\Convolution3d.dll")
LoadPlugin("D:\PROGRA~1\GORDIA~1\AviSynthPlugins\FluxSmooth.dll")
Loadplugin("D:\Program Files\AviSynth 2.5\plugins\VagueDenoiser.dll")
#
# VIDEO SOURCE
Video=mpeg2source("M:\VIDEO_ED\stranger_than_fiction\stf---1.d2v")
#
# NOISE REDUCTION
Video=UnDot(Video)
Video=FFT3dGPU(Video,sigma=2)
#Video=Vaguedenoiser(Video,0,3,6,0,False,False,1,True,0.5,75)
#Video=Temporalsoften(Video,2,3,3,mode=2,scenechange=6)
#
# FINISH
Return(Video)


So, any ideas? Problem with the source or with the filter? Anyway, I'd like to hear an advice about denoising this movie.
Sagekilla
16th December 2007, 21:33
Does the video look this way when you are opening up the file using VirtualDubMod?

If not, does it look that way after you've encoded it using [insert favorite codec here]?
denissio
16th December 2007, 22:25
Does the video look this way when you are opening up the file using VirtualDubMod?
Yes. The original frames are affected only by Undot(), and filtered ones - with Undot() and FFT3dGPU(sigma=2). Undot doesn't make any visual difference, it's just for better compressibility...

If not, does it look that way after you've encoded it using [insert favorite codec here]?
Of course not, I would have mentioned if I used some codec...
Terranigma
16th December 2007, 22:28
Could you try the slower, more accurate version, fft3dfilter? That flat rainbow spot could be the fault of your gfx card (which fft3dgpu relies on.)
Sagekilla
16th December 2007, 22:34
Yes. The original frames are affected only by Undot(), and filtered ones - with Undot() and FFT3dGPU(sigma=2). Undot doesn't make any visual difference, it's just for better compressibility...


Of course not, I would have mentioned if I used some codec...

Just checking, I wanna make sure all the bases are covered! I honestly don't know otherwise what may be causing this, since I'm not too well versed with denoising/graining. You can try lowering sigma slightly on FFT3DGPU to see if it affects things any, perhaps leave a little bit of grain in so as to mask the potential banding.


Edit: Like Terranigma said, the problem could be your video card as well. I myself cannot use FFT3dGPU on my 7900 GT because, while fast, my video card introduces a ton of artifacts, much worse and noticeable then what you're getting (Think of getting pure black frames as output, video that's been chopped up and rearranged, etc)
denissio
16th December 2007, 22:34
Could you try the slower, more accurate version, fft3dfilter? That flat rainbow spot could be the fault of your gfx card (which fft3dgpu relies on.)

Alas, the result is the same.
denissio
16th December 2007, 22:37
Just checking, I wanna make sure all the bases are covered! I honestly don't know otherwise what may be causing this, since I'm not too well versed with denoising/graining. You can try lowering sigma slightly on FFT3DGPU to see if it affects things any, perhaps leave a little bit of grain in so as to mask the potential banding.

Ok, thanks, I'll try to play with the tweaks a little, but I'm not sure this would help ...
Sagekilla
16th December 2007, 22:42
Ok, thanks, I'll try to play with the tweaks a little, but I'm not sure this would help ...

Well with lower sigma you're degraining less, which means more grain will be left in to maybe hide those ugly posterization artifacts. You can also try tweaking block width/height/overlap (bw, bh, ow, oh) to see if that changes anything.
denissio
16th December 2007, 23:30
Hmm, lowering sigma didn't help much, however setting the 'bt' to 0 resulted in lesser banding.. yet I got slightly visible grid and ringing (e.g. near hair). IMHO, this banding is a result of some temporal denoising, implemented in FFT3D's Wiener filter.
I'd like to hear some piece of advice from more experienced users than me:)
Fizick
16th December 2007, 23:36
what versions do you use?
Short vob cut (about 10 frames including problem ones) would be useful.
May be reason is that I do not see any dark regions on my old CRT monitor :)
banding is quite poossible. the question is: how big.
denissio
17th December 2007, 18:28
what versions do you use?
Short vob cut (about 10 frames including problem ones) would be useful.
May be reason is that I do not see any dark regions on my old CRT monitor :)
banding is quite poossible. the question is: how big.

Well..
fft3dgpu.dll is version 0.8.2.0, avisynth 2.57, virtualdub 1.6.12, DGIndex 1.4.3
The huffyuv cut (20 frames, 4.4 MB) is here: http://dump.ru/files/n/n825084981/
Fizick
17th December 2007, 18:51
Is huffyuf the original video format?
If not, please provide VOB cut.
denissio
17th December 2007, 20:20
I've got problems with making the vob cut.. please tell what software can perform it? I've tried TMPGEnc, Jummfa DVDCutter and Merger, ChopperXP, VobBlanker2010.. but couldn't get a needed cut. Besides, HuffYUV is quite lossless, maybe you'll examine it?
Leak
17th December 2007, 20:43
I've got problems with making the vob cut.. please tell what software can perform it?
DGIndex.

np: Yello - Daily Disco (1980-1985: The New Mix In One Go)
denissio
17th December 2007, 20:53
Thank you, Leak.
Here it is: http://dump.ru/files/n/n2314233/
Fizick
17th December 2007, 22:25
Ok, I got it! :)
Now question. What is the most problem frame (with artifactes of FFt3dgpu)? I will look tomorrow.
denissio
17th December 2007, 22:32
Fizick, imho, frame #17 or #21 :)
tsp
18th December 2007, 22:49
denissio: try this: FFT3dGPU(Video,sigma=2,sigma3=1,sigma2=0.1,sigma4=1)
by using such a low value for sigma2 the small amount of medium sized grain prevents posterization while the large amount of small grain is removed by the high sigma value.

Sagekilla: have you considered better cooling (liquid N2) :)
denissio
18th December 2007, 23:01
denissio: try this: FFT3dGPU(Video,sigma=2,sigma3=1,sigma2=0.1,sigma4=1)
by using such a low value for sigma2 the small amount of medium sized grain prevents posterization while the large amount of small grain is removed by the high sigma value.

Sagekilla: have you considered better cooling (liquid N2) :)

Thank you, it removed most of the banding. ;)