Adaptive noise removal

[Dust Signs]

Hello,

I am currently trying to re-encode the episodes from the UK Blu-ray of "The Prisoner", mainly because the noise in some scenes is so unbearably bad. While MDegrain is doing a really great job in most scenes (while retaining most of the details in the not so noisy parts), there are a few scenes which are so noisy that even MDegrain3 cannot restore anything remotely acceptable. Here are some (unfiltered) examples from episode 3 (last episode on the first Blu-ray):
http://imagizer.imageshack.us/v2/280x200q90/674/tFbD18.png (http://imageshack.com/f/iqtFbD18p)
http://imagizer.imageshack.us/v2/280x200q90/912/GWmfQ3.png (http://imageshack.com/f/pcGWmfQ3p)
http://imagizer.imageshack.us/v2/280x200q90/673/sYQah8.png (http://imageshack.com/f/ipsYQah8p)
I am now looking for a way to adaptively filter scenes like that (I planned on using FFT3D), but without degrading the quality in the other scenes (which MDegrain can handle). Is there such a way without manually looking for the time codes and adding corresponding conditions to the script?
Any hints (including suggestions for improving my current setup) are welcome. Here is the script that I am currently using (I am using StaxRip for editing the script and previewing the results):

LoadPlugin("C:\StaxRip\Applications\AviSynth plugins\ffms2\ffms2.dll")
SetMemoryMax(2047)
SetMTMode(5,4)
FFVideoSource("D:\Blu-rays\The Prisoner\The Prisoner E03.mkv", cachefile="D:\Blu-rays\The Prisoner\The Prisoner E03.ffindex")
AssumeFPS(25.000)
Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()
SetMTMode(2)
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\mvtools2.dll")
super = MSuper(pel=4, sharp=2)
backward_vec3 = MAnalyse(super, isb = true, delta = 3, overlap=4)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
forward_vec3 = MAnalyse(super, isb = false, delta = 3, overlap=4)
MDegrain3(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400)
return(last)


Thanks in advance
Best regards
Dust Signs

PS.: In case you wondered how the "good" scenes look like, here is an example from the same episode: http://imagizer.imageshack.us/v2/280x200q90/537/mPNIay.png (http://imageshack.com/f/exmPNIayp)

[feisty2]

use an edge mask
apply mdegrain on "edge" part
use dfttest on flat (background) part

[feisty2]

xxxsource
edge=mdegrainn (xxx)
flat=dfttest (xxx)
mask=flat.mt_edge ("prewitt")
mt_merge (flat,edge,mask,luma=true)

[Dust Signs]

dfttest makes AviSynth crash on my system. I tried several different versions of libfftw. Is there a different filter which does something similar?

Best regards
Dust Signs

[feisty2]

fft3dfilter or tnlmeans
fft3dfilter is not kinda quality wise
tnlmeans is very slow

[Groucho2004]

dfttest makes AviSynth crash on my system.
You're probably running out of memory, especially considering this:
SetMemoryMax(2047)
Reduce the number of threads and/or run dfttest single-threaded if you are using Avisynth MT.
Check memory usage with Task manager or better this tool (http://forum.doom9.org/showthread.php?t=165528).

[Dust Signs]

You're probably running out of memory, especially considering this:

Reduce the number of threads and/or run dfttest single-threaded if you are using Avisynth MT.
Check memory usage with Task manager or better this tool (http://forum.doom9.org/showthread.php?t=165528).

No, I'm not running out of memory; even with only one thread, I get an access violation, followed by a crash.

Best regards
Dust Signs

[Groucho2004]

No, I'm not running out of memory; even with only one thread, I get an access violation, followed by a crash.

Best regards
Dust Signs
Odd. I never had dfttest crash.


Clever signature, by the way. :)
The number you dialed is imaginary. Please turn your phone by 90° and try again

[StainlessS]

I get an access violation, followed by a crash.

Maybe because you ran out of memory, you only have 2GIG user memory under 32 bit. Do as Groucho suggests and reduce it, any more than 512MB is purely wastful (IanB suggests exactly the same, 512MB is maximum default).

[Dust Signs]

fft3dfilter or tnlmeans
fft3dfilter is not kinda quality wise
tnlmeans is very slow

You are right, fft3dfilter is really not good in this context; I'll try tnlmeans as soon as I can find a link which is not dead.

Best regards
Dust Signs

[Dust Signs]

Maybe because you ran out of memory, you only have 2GIG user memory under 32 bit. Do as Groucho suggests and reduce it, any more than 512MB is purely wastful (IanB suggests exactly the same, 512MB is maximum default).

I have 32 GB of main memory, so this should not be a problem.
EDIT: Reducing the argument of SetMemoryMax to 512 still produces a crash

Best regards
Dust Signs

[Groucho2004]

I'll try tnlmeans as soon as I can find a link which is not dead.
Try this one (http://forum.doom9.org/showthread.php?t=168090).

[Groucho2004]

I have 32 GB of main memory, so this should not be a problem.
The user address space for 32 Bit applications is 2 GB. It does not matter how much memory you have installed. The usable address space can be increased to 4 GB on a 64 Bit OS but only if the program using Avisynth is "large address aware".

[Dust Signs]

The user address space for 32 Bit applications is 2 GB. It does not matter how much memory you have installed. The usable address space can be increased to 4 GB on a 64 Bit OS but only if the program using Avisynth is "large address aware".

I am aware of that. But I don't see how this would be related to the crash. I have been using this template (SetMemoryMax) with AviSynthMT for years and never had any problems with it. I think it is highly unlikely that it is a problem now with dfttest; it is much more likely that the reason for the access violations are completely different.

Best regards
Dust Signs

[Groucho2004]

I am aware of that. But I don't see how this would be related to the crash.
I did not write that it is related to your crash. I was simply commenting on your statement:
"I have 32 GB of main memory, so this should not be a problem."

Also, if you are indeed aware of that, why would you enable avisynth to use up all your available address space? Your basically asking for trouble by doing so.

[Dust Signs]

Try this one (http://forum.doom9.org/showthread.php?t=168090).

Thank you. Using it in the "script" feisty2 suggested does however not really improve the quality compared to MDegrain by itself.

Best regards
Dust Signs

[Dust Signs]

I did not write that it is related to your crash. I was simply commenting on your statement:
"I have 32 GB of main memory, so this should not be a problem."

By mentioning the amount of main memory I just meant that there should be more than enough for AviSynth (out of the 32 GB total, there are about 28 GB free most of the time). Since the maximum amount of memory AviSynth as a 32-bit process can use (address) is 2 GB, I chose 2047 MB in the first place.

Best regards
Dust Signs

[feisty2]

increase "a" and "h" values in tnlmeans

[Groucho2004]

Since the maximum amount of memory AviSynth as a 32-bit process can use (address) is 2 GB, I chose 2047 MB in the first place.
I don't know what kind of encoding chain staxrip uses but if you allocate the full 2 GB for avisynth there won't be anything left for the encoder (which runs in the same process unless the frames are piped through stdout to the encoder).

[Dust Signs]

Using the example from http://forum.doom9.org/showthread.php?p=859522#post859522, I increased h to 2.2 and a to 1000, but the quality is still pretty bad. In addition, the previously "good" looking still also look quite blurry.

Best regards
Dust Signs

[Dust Signs]

I don't know what kind of encoding chain staxrip uses but if you allocate the full 2 GB for avisynth there won't be anything left for the encoder (which runs in the same process unless the frames are piped through stdout to the encoder).

As far as I can tell, staxrip launches x264 with the AviSynth script as input file as well as my additional command line parameters.
Now that I think about it, it is really odd that this works at all. Maybe AviSynth does not allocate the full 2 GB after all.

Best regards
Dust Signs

[Groucho2004]

Now that I think about it, it is really odd that this works at all. Maybe AviSynth does not allocate the full 2 GB after all.
If you specify 2G with SetMemoryMax() it means that avisynth will use up to that value for its frame cache.

[Dust Signs]

I'll reduce it in my future scripts then. Thank you.

Best regards
Dust Signs

[feisty2]

to improve quality, set bx=by=0 and multiscale false, increase ax ay values til its so slow that u cant stand

[feisty2]

if still no good, could play around "curve" stuff,
example can be found in "dither" document

[feisty2]

and also add mcompensate along with an az value greater than 0 if u still think dat ain't good enough

[Dust Signs]

The quality got a bit better, but the results in the "good" scenes are still not what they were before. Here is the script that I am currently using:

LoadPlugin("C:\StaxRip\Applications\AviSynth plugins\ffms2\ffms2.dll")
SetMemoryMax(512)
FFVideoSource("H:\test.mkv", cachefile="H:\test.ffindex")
AssumeFPS(25.000)
Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\mvtools2.dll")
original = last
super = MSuper(pel=4, sharp=2)
backward_vec3 = MAnalyse(super, isb = true, delta = 3, overlap=4)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
forward_vec3 = MAnalyse(super, isb = false, delta = 3, overlap=4)
MDegrain3(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400)
edge=MDegrain3(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400)
flat=TNLMeans(original,h=2.2,a=1000)
mask=flat.mt_edge("prewitt")
mt_merge(flat,edge,mask,luma=true)


Any suggestions? Speed is not really an issue. It is only important for me to keep the "good" frames good, which was why I was asking for an adaptive filter of some sort. Don't get me wrong - the results are quite impressive in the "bad" frames.

Best regards
Dust Signs

[feisty2]

remove mdegrain3 between fvec3 and edge=xx
you used mdegrain3 twice

[feisty2]

xxxsource
c=last
super=msuper(xxx)
bv3=manalyze (xxx)
bv2
bv1
fv1
fv2
fv3
c.mdegrain3 (xxx)
edge=last
bc3=c.mcompensate (super, bv3,xxx)
bc2
bc1
fc1
fc2
fc3
interleave (bc3,bc2,bc1,c,fc1,fc2,fc3)
tnlmeans (ax=16,ay=16,az=3,bx=0,by=0,multiscale=false,sse=true)
selectevery (7,3)
flat=last
mask=xxxx.removegrain (20)
mt_merge

[feisty2]

fill in the rest urself basing on what you want, im on cellfone now, typing is a pain in the ass

[Dust Signs]

Oops, I must have duplicated that line by accident. I'll try your suggestion. Thank you.

Best regards
Dust Signs