How to denoise rendered image sequence

[pancserzso]

I am trying to filter image sequences rendered by a visualization program. The problem is that the program was not created with animation in mind, and it created an image sequence full of artifacts / temporal noise.

You can see an example of such source here (http://storage.z-e-r-o.in/zsolt/2013/10/source.mp4).

Some time ago I was trying to solve the same problem, and I got some help from Didée in this thread (http://forum.doom9.org/showthread.php?t=161954) however this time some of the samples are worse in quality and the original script didn't really work on them.

Here is the code which I tried to use for filtering:
LoadPlugin( "e:\apps\utils\Video tools\AVSplugins\RemoveGrainSSE3.dll" )
LoadPlugin( "e:\apps\utils\Video tools\AVSplugins\mt_masktools-26.dll" )
LoadPlugin( "e:\apps\utils\Video tools\AVSplugins\EEDI2.dll" )

o=last
ox=o.width
oy=o.height

# round/soften hard corner pixels
o.pointresize(ox*3,oy*3).removegrain(4)
mt_lutxy(last,last.removegrain(11,-1),"x x y - 2.51 * +",U=2,V=2).bilinearresize(ox,oy)
o3=last

bubb0 = o3.eedi2(maxd=8).bicubicresize(ox,oy,0,0,0,0.5).turnleft().eedi2(maxd=8).turnright().bicubicresize(ox,oy,0,0,-0.5,0)
return(bubb0)

And here is the filtered result I got: filteredYV12.mp4 (http://storage.z-e-r-o.in/zsolt/2013/10/filteredYV12.mp4)

I think my problem is that the code which I'm trying to use is not really strong enough for my source.

Is there any way to create a filter which removes this strong temporal flickering / noise? It's not a big problem if the end result is a bit blurry.

[raffriff42]

The images are very aliased - is this how the original images are? (crop + 2x blowup)
https://www.dropbox.com/s/gdo15fmcjvxkfk4/t%3D168979-fr330.png?raw=1

Can you turn on antialiasing in your render engine? If not, can you render at 4x final size? If so, you can downsample your images with smoothing in Avisynth, or in an image batch processor.

[pancserzso]

The images are very aliased - is this how the original images are? (crop + 2x blowup)
https://dl.dropboxusercontent.com/u/108089426/Screenshots/t%3D168979-fr330.png

Can you turn on antialiasing in your render engine? If not, can you render at 4x final size? If so, you can downsample your images with smoothing in Avisynth, or in an image batch processor.

I know, this is the problem. The object in 3D is initially very far from the origin, and the rendering program was not written with this in mind. So when the object is transparent and is far from the origin, these artifacts appear.

The problem with re-rendering is that even in 1.5x resolution it would take me at least 7 days non-stop rendering, and I don't have enough time for that. Also, since its a GPU based program, I might not be able to render some objects, as it'd need >2 GB of GPU memory, and my card only has 2.

Pretty much my only choice is to do filtering in Avisynth.

I've tried a few spatio-temporal denoiser, and so far FFT3DFilter was able to get it really smooth, although very blurry.

I was thinking about using the FFT3D as an input to MVTools and somehow combine it with the original source, but I don't know how to do it. What do you think?

[raffriff42]

I'm looking at the script you posted - I like it. Much better than my naive GaussianBlur>Sharpen. The remaining temporal noise is helped by increasing the frame rate from 24 to 60fps with Interframe (http://forum.doom9.org/showthread.php?t=160226). This not only adds more temporal smoothing, but the eye is less sensitive to noise at the higher frame rate. Brightening this particular source seems to help as well.

Allowing a small amount of twinkling/shimmering is OK, IMHO- to the uneducated eye it says "sharpness." ;) The added blur of Interframe may allow you to back off some of the smoothing elsewhere.LoadPlugin( "e:\apps\utils\Video tools\AVSplugins\RemoveGrainSSE3.dll" )
LoadPlugin( "e:\apps\utils\Video tools\AVSplugins\mt_masktools-26.dll" )
LoadPlugin( "e:\apps\utils\Video tools\AVSplugins\EEDI2.dll" )
LoadPlugin(pathBase + "mvtools-2.5.11.9-svp\mvtools2.dll")
Import(pathBase + "Interframe\InterFrame2.avsi")
LoadPlugin(pathBase + "SmoothAdjust-2,00\avs-2.5.x\x86\SmoothAdjust.dll")

SmoothLevels(0, 1.0, 160, 0, 255)

o=last
ox=o.width
oy=o.height

# round/soften hard corner pixels
o.pointresize(ox*3,oy*3).removegrain(4)
mt_lutxy(last,last.removegrain(11,-1),"x x y - 2.51 * +",U=2,V=2).bilinearresize(ox,oy)
o3=last

bubb0=o3.eedi2(maxd=8).bicubicresize(ox,oy,0,0,0,0.5)
\ .turnleft().eedi2(maxd=8)
\ .turnright().bicubicresize(ox,oy,0,0,-0.5,0)

return bubb0.InterFrame(NewNum=60, NewDen=1)

EDIT - one more thing: return bubb0.InterFrame(NewNum=60, NewDen=1).Sharpen(0.4)

[Mystery Keeper]

Do you really need to denoise it? The flickering you see doesn't seem like artifacts to me, but rather like usual 3D->2D projection result. I don't see z-order glitch either, so program has probably sorted the faces too. Though I can not be sure, since the objects look quite small. I too suggest using multisampling or supersampling->downsampling in the rendering program.

[pancserzso]

Thanks for the suggestions, but there is nothing I can do with the source at the moment. I am not sure if I need to do denoising or something else, but I would like to make it visually pleasing, without bright points / flickering in brightness.

Here are some filtering I have been trying:

DeGrainMedian(limitY=10, limitUV=13, mode=1, interlaced=false, norow=false)
DeGrainMedian(limitY=10, limitUV=13, mode=1, interlaced=false, norow=false)
=> result (http://storage.z-e-r-o.in/zsolt/2013/10/process3.mp4) not too bad, handles the extreme noisy part quite well (2-7 sec)

FFT3DFilter(sigma=3, plane=4, bw=48, bh=48, bt=4, ow=16, oh=16, sharpen=0)
=> result (http://storage.z-e-r-o.in/zsolt/2013/10/process4.mp4)

tr = 24
FFT3DFilter(sigma=12,bw=tr,bh=tr,ow=tr/2,oh=tr/2,bt=5,plane=0)

=> result (http://storage.z-e-r-o.in/zsolt/2013/10/process7.mp4)

FFT3DFilter( sigma=6, plane=0, bt=5, bw=32, bh=32, ow=16, oh=16 )
=> result (http://storage.z-e-r-o.in/zsolt/2013/10/process8.mp4) my favourite so far, but may be a bit too blurry

FluxSmoothST(temporal_threshold=7, spatial_threshold=7)

=> result (http://storage.z-e-r-o.in/zsolt/2013/10/process5.mp4)

o = last
pre = o.fft3dfilter(sigma=8,sigma2=12,sigma3=10,sigma4=5,bw=24,bh=24,ow=12,oh=12,bt=5).FluxSmoothT(4)

sup1 = o.MSuper(hpad=16, vpad=16, pel=2, levels=1,sharp=2)
sup2 = pre.MSuper(hpad=16, vpad=16, pel=2, levels=0,sharp=0)
bv3 = MAnalyse(sup2,delta=3,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=true, dct=5)
bv2 = MAnalyse(sup2,delta=2,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=true, dct=5)
bv1 = MAnalyse(sup2,delta=1,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=true, dct=5)
fv1 = MAnalyse(sup2,delta=1,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=false,dct=5)
fv2 = MAnalyse(sup2,delta=2,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=false,dct=5)
fv3 = MAnalyse(sup2,delta=3,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=false,dct=5)

o.MDegrain3(sup1,bv1,fv1,bv2,fv2,bv3,fv3,thSAD=360)
=> result (http://storage.z-e-r-o.in/zsolt/2013/10/process6.mp4) really nice looking result at the beginning (0-1 sec) and the end (7-16 sec), but really bad in the middle (2-7 sec)

[cretindesalpes]

My attempt, similar to the last try of your previous post :
d = 2
a1 = RemoveGrain (4).RemoveGrain (12)
a = a1.TemporalSoften (4, 25, 25, 15, 2)
sup_a = a.MSuper ()
vec = sup_a.MAnalyse (multi=true, delta=d, blksize=8, overlap=4, search=5, searchparam=4, DCT=5)
sup = MSuper (levels=1)
MDegrainN (sup, vec, d, thSAD=800)
Quite effective, but a bit blurry at the end.
=> result (http://www.mediafire.com/download/51ldm51ojq00q0a/pancserzso-result.mkv)

[pancserzso]

cretindesalpes, thanks a lot! It is almost perfect!! Only a bit too strong / blurry for my taste, because as you can see the most visible fruits in the animation are at the 'end' of their single path.

raffriff42, thanks, but I cannot modify the frame rate nor the brightness, I have to use the clip as it is.

I run the filters on the final clip, which is composed both from flickering fruits and smooth ones.

The challenge here is how to keep the details in the 'big' good examples while filtering the flickering ones out! Cretindesalpes' solution is the best so far, but it's a bit loosing too much details.

Cretindesalpes, would it be possible to either post sharpen or use a bit less filtering?

Here are the results from the final composition:
source-final (http://storage.z-e-r-o.in/zsolt/2013/10/source-final.mp4)

cretindesalpes (http://storage.z-e-r-o.in/zsolt/2013/10/process-cretindesalpes.mp4)

dgm573 (http://storage.z-e-r-o.in/zsolt/2013/10/process-dgm573.mp4)
DeGrainMedian(limitY=5, limitUV=7, mode=3, interlaced=false, norow=false)
DeGrainMedian(limitY=5, limitUV=7, mode=3, interlaced=false, norow=false)

dgm703 (http://storage.z-e-r-o.in/zsolt/2013/10/process-dgm703.mp4)
DeGrainMedian(limitY=7, limitUV=0, mode=3, interlaced=false, norow=false)
DeGrainMedian(limitY=7, limitUV=0, mode=3, interlaced=false, norow=false)

Didée old (http://storage.z-e-r-o.in/zsolt/2013/10/process-didee.mp4)

fft3d-3-48-16 (http://storage.z-e-r-o.in/zsolt/2013/10/process-fft3d-3-48-16.mp4)
FFT3DFilter(sigma=3, plane=4, bt=4, bw=48, bh=48, ow=16, oh=16)


fft3d-6-32-16 (http://storage.z-e-r-o.in/zsolt/2013/10/process-fft3d-6-32-16.mp4)
FFT3DFilter(sigma=6, plane=0, bt=5, bw=32, bh=32, ow=16, oh=16)

flux (http://storage.z-e-r-o.in/zsolt/2013/10/process-flux.mp4)
FluxSmoothST(temporal_threshold=7, spatial_threshold=7)

mdg3 (http://storage.z-e-r-o.in/zsolt/2013/10/process-mdg3.mp4)
o = last
pre = o.fft3dfilter(sigma=8,sigma2=12,sigma3=10,sigma4=5,bw=24,bh=24,ow=12,oh=12,bt=5).FluxSmoothT(4)

sup1 = o.MSuper(hpad=16, vpad=16, pel=2, levels=1,sharp=2)
sup2 = pre.MSuper(hpad=16, vpad=16, pel=2, levels=0,sharp=0)
bv3 = MAnalyse(sup2,delta=3,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=true, dct=5)
bv2 = MAnalyse(sup2,delta=2,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=true, dct=5)
bv1 = MAnalyse(sup2,delta=1,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=true, dct=5)
fv1 = MAnalyse(sup2,delta=1,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=false,dct=5)
fv2 = MAnalyse(sup2,delta=2,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=false,dct=5)
fv3 = MAnalyse(sup2,delta=3,truemotion=false,global=true,blksize=16,overlap=16/2,search=5,searchparam=4,isb=false,dct=5)

o.MDegrain3(sup1,bv1,fv1,bv2,fv2,bv3,fv3,thSAD=360)

[pandy]

MLAA?

http://visual-computing.intel-research.net/publications/papers/2009/mlaa/mlaa.pdf
http://iryoku.com/aacourse/downloads/06-MLAA-on-PS3.pdf

However im not a AVS script guru...