dfttest - 2D/3D frequency domain denoiser.

[nonsens112]

Originally Posted by Great Dragon

It's a great filter you've made, tritical. Thanks.
And I have one question: can I disable completely a temporal filtering to use only spatial features?

tbsize=1 makes filter spatial.

[Dogway]

Im wondering whether this is the spiritual sucesor of tnlmeans. is it?
I use tnlmeans because its wonderful results on anime, although its deadly slow and blurs a little bit on undifined areas. Its also abandoned and not supported anymore.

Is there a setting on dfttest aimed at animation? or similar to tnlmeans(ax=3,ay=3,az=6,sx=2,sy=2,bx=1,by=1,h=0.6,a=1000.0,sse=false)

EDIT:

[tritical]

dfttest isn't related to tnlmeans in any way other than being a denoiser. dfttest works in the frequency domain. tnlmeans is basically bilateral filtering with no distance weighting and similarity weighting based on local neighborhood similarity (difference) instead of individual pixel difference. The only way to find equivalent settings (if they even exist) is to test I'm afraid.

[Dogway]

Thanks for the answer. I added the adjective "spiritual" only because its the next denoiser you made after tnlmeans. I still havent fully tested dfttest but sigmas and tbsize, but results doesnt seem much animation oriented at defaults settings as tnlmeans thats why Im trying to speed up this one with MT+ms=true (doesnt seem to work) or find out a good dfttest setting for this kind of sources. But its good to know that unlike tnlmeans sse=false, dfttest doesnt have any specific setting which would undoubtedly work better in animation.

[foxyshadis]

I think the noise power spectrum analysis might be a little off, it smooths the image way out. For instance, I get this on one image:

     0.013  18650.967   1413.832    255.401    143.324     45.611      0.009 
 43671.273    846.859    561.533    310.418     87.381     18.431      7.488 
  4325.101    438.551    167.673     90.019     45.118     18.157     23.700 
   134.528    189.763      4.576     86.177     57.927     23.606     24.473 
    13.794      7.955      4.505     34.447     15.854      4.820      5.515 
     2.716      2.340      6.929      0.477      0.114      0.087      0.327 
    11.135      3.224      0.270      0.179      0.501      0.372      0.417 
     2.716      3.802      0.464      1.590      0.274      0.150      0.327 
    13.794     26.689     67.291     47.565      6.188      7.503      5.515 
   134.528      1.487     60.844    123.280     64.566     35.531     24.473 
  4325.101   3644.524    187.824     80.187    130.267     33.799     23.700 
 43671.273  40664.180   1224.066     10.090     92.101     19.694      7.488 


avg noise power = 2006.033321

The text file is just "0,0,486,368", and attached is the image that I used to create it. The script is:

ImageReader("C:\Temp\grafix\4454627606_fb119cdbfb_o.jpg",end=0).crop(0,0,0,-1).converttoyv12(matrix="pc.709")
dfttest(tbsize=1,nfile="nclip.txt")

On the other hand, when I crop the image to just that 12x12 square and use "0,0,0,0", I get this:

     0.001     21.195     74.256     26.044      4.195      1.394      0.660 
     2.550     17.687     66.975     59.390      7.629      1.961      0.575 
    33.436     24.428     33.947     27.089      2.309      0.416      0.342 
    51.260     28.261     10.396      8.658      2.295      0.064      0.096 
    11.952      6.892      0.965      3.310      1.639      0.018      0.067 
     7.062      3.432      0.490      3.285      1.215      0.011      0.127 
     6.039      3.123      0.284      2.904      1.121      0.030      0.101 
     7.062      3.752      0.355      3.274      1.124      0.198      0.127 
    11.952      7.017      1.482      6.035      2.040      0.227      0.067 
    51.260     36.053      4.138      5.044      4.195      0.013      0.096 
    33.436     15.871      3.712     13.901      8.128      0.281      0.342 
     2.550     11.079     28.088     11.395      5.175      1.396      0.575 


avg noise power = 10.156793

Which seems much more reasonable (it's rather chunky noise).

Edit: Further investigation reveals that the numbers keep increasing as the sbsize goes up.

[tritical]

I think you have x and y switched in nclip.txt. The ordering is:

frame_number,plane,y,x

If I use:

0,0,368,486

instead of

0,0,486,368

I get the same result as you show for the 12x12 cropped image. As far as dependence on sbsize/sosize/tbsize/tosize, the values are normalized based non-coherent power gain, and should be independent of window size (of course that only applies to ideal cases, such as different windows containing only white noise, etc...). Overlapping isn't considered when estimating the noise spectrum, but the estimated values are adjusted accordingly before use. If you continue to see any weirdness please let me know .

[foxyshadis]

Thanks. Sorry for the bad call; given that I've been staring at the code in question all day, I have no excuse. Also, I made a patch which may or may not be of any use to you: It allows specifying nframe, nplane, nx, ny as arguments if you only want one set, without having to make a file. Also included is an experimental nclip parameter, where instead of specifying a location, it uses a naive random algorithm to find a suitable location in the blank area of an edge map. It's pretty half-baked right now. My original plan was to use tdtrans to find the farthest locations from any edge, but it wasn't really any better than just choosing a location at random.

For noisy edge maps, one of my work in progress is:

dn=dfttest(tbsize=1,sigma=40,sigma2=40)
ed1=dn.scalededge(1)
ed2=dn.scalededge(2)
ed4=dn.scalededge(4)
ed=average(ed1,.4,ed2,.4,ed4,.2).mt_binarize(64,chroma="process").removegrain(2)

function scalededge(clip c, int "scale") {
c
sm=Spline16Resize(m(2,width/scale),m(2,height/scale))
ved=sm.mt_convolution(horizontal="1 2 1",vertical="1 0 -1",chroma="process")
hed=sm.mt_convolution(vertical="1 2 1",horizontal="1 0 -1",chroma="process")
return mt_lutxy(hed,ved,expr="x y +",chroma="process").Spline16Resize(width,height)
}

dfttest(tbsize=1,nframe=0,nclip=ed)

There are probably more elegant ways of getting a good mask, but I'm a bit sleepy now.

[tritical]

Definitely good ideas. For some reason, trying automatic placement of the window for noise estimation never even crossed my mind. I'll try to get your changes added.

[dansrfe]

Is there any way to speed up dfttest() at default settings. I'm really happy with the results of this filter, but It's just way to slow to use in practice. I MT'd it but I still get about the same speed maybe a bit more.

[tritical]

dfttest is internally multithreaded so I wouldn't expect much if any speed increase from using MT. By default dfttest uses threads = # of processors... in some cases it might be faster with slightly more threads, but I've never tested that. First option to make it faster is decrease tbsize to 3 or 1 (default is 5). Second option is to decrease the amount of spatial overlap and increase the spatial window size. By default it uses a spatial window size of 12 with 9 overlap (75%). For the same window size less overlap will be faster. For the same overlap percentage, larger window size will be faster.

[dansrfe]

Thanks tcritical! Also can you post the documentation for dfttest so I can read up on all the options. I can't find the documentation anywhere.
Also do you have any plans to revise and upgrade dfttest in the future?

[Didée]

Originally Posted by dansrfe

I can't find the documentation anywhere.

Huh? The documentation is included in dfttestv16.zip from tritical's page.

[dansrfe]

*Oop's*, sorry about that. I had the dll downloaded a while ago and must have forgotten to save the documentation. I didn't look at the one place it had to be

Thanks Didée!

[tritical]

Posted v1.7. Changes:

+ added nstring/sstring/ssx/ssy/sst parameters and functionality
+ allow space as delimiter in input files
- fixed missing emms in sse routine for f0beta != (1.0 or 0.5) and ftype=0

Explanations for new parameters (from v1.8 readme):

   nstring -

       Same functionality as 'nfile', but allows entering window locations directly in
       the script instead of creating a separate file. The list of frame/plane/ypos/xpos
       quadruples is stored as a string with each quadruple separated by a space.
       Example:

          If you use an nfile that looks like:

              a=4.0
              35,0,45,68
              28,0,23,87

          You can use the following nstring and get the same result:

             nstring="a:4.0 35,0,45,68 28,0,23,87"

       The one restriction is that the oversubtraction factor (a:x.x) must be the first
       entry in the string (as opposed to nfiles where the a=x.x can be placed anywhere).
       If it is not supplied, then the same default oversubtraction factor is used as
       is used for the nfile option.

       default:  ""


   sstring/ssx/ssy/sst -

       Used to specify functions of sigma based on frequency. If you want sigma to vary
       based on frequency, then use 'sstring' instead of the 'sigma' parameter. sstring
       allows you to enter values of sigma for different normalized [0.0,1.0] frequency
       locations. Values for locations between the ones you explicitly specify are computed
       via linear interpolation. The frequency range, which is dependent on sbsize/tbsize,
       is normalized to [0.0,1.0] with 0.0 being the lowest frequency and 1.0 being the
       highest frequency. You MUST specify sigma values for those end point locations
       (0.0 and 1.0)! You can specify as many other locations as you wish, and they don't
       have to be in any particular order. Each frequency/sigma pair is given as "f.f:s.s".
       The list of frequency/sigma pairs is saved as a string, with each pair separated by
       a space.

       For example, if you want a linear ramp of sigma from 1.0 for the lowest frequency
       to 10.0 for the highest frequency use:

            sstring = "0.0:1.0 1.0:10.0"

             "0.0:1.0"  =>  this means sigma=1.0 at frequency 0.0

             "1.0:10.0"  => this means sigma=10.0 at frequency 1.0

             Sigma values for frequencies between 0.0 and 1.0 will be computed via
             linear interpolation.

       Or if you want a band-stop filter that passes low and high frequencies (filters
       middle frequencies) use something like:

            sstring = "0.0:0.0 0.15:10.0 0.85:10.0 1.0:0.0"

       To help visualize the process, the resulting filter spectrum is output to
       "filter_spectrum-date_string.txt" using the same format as the "noise_spectrum.txt"
       file that is output by the nfile/nstring options. The format of this file is compatible
       with 'sfile' input.

       There are two methods for computing sigma values for a given frequency bin based on
       sstring. The first computes the normalized frequency location of each dimension 
       (horizontal,vertical,temporal), interpolates sigma for each of those dimensions, 
       and then multiples the individual sigmas to obtain the final sigma value. So that
       everything scales correctly, all sigma values entered in sstring are first raised to
       the 1/#_dimensions power before perform performing linear interpolation and multiplying.
       The second method (based on fft3dfilter's system) works by computing a single location
       from the seperate dimension locations (x,y,z) as:

           new = sqrt((x*x+y*y+z*z)/3.0)

       sigma is then interpolated to this location. By default the first system is used.
       To use the second system simply put a '$' sign at the beginning of sstring as shown
       below:

            sstring = "$ 0.0:1.0 1.0:10.0"


        ---------------- ssx/ssy/sst explanation -------------------------------

       sstring breaks the 1D (sbsize=1), 2D (for tbsize=1), or 3D (for sbsize>1 and tbsize>1) 
       frequency spectrum into chunks by normalizing each dimension to [0.0,1.0]... i.e. the
       frequency range [0.0,0.25] is a cube covering the first 1/4 of each dimension. This works
       fine if you want to treat all dimensions the same in terms of how sigma should vary.
       However, if you wanted to ramp sigma based only on temporal frequency or horizontal
       frequency, this is too limited. This is where ssx/ssy/sst come in!
       
       ssx/ssy/sst allow you to specify sigma as a function of horizontal (ssx), vertical (ssy),
       and temporal (sst) frequency only. The syntax is exactly the same as that of sstring. To
       get the final sigma value for a frequency location, the three separate values (one for
       each dimension) are computed and then multiplied together. As with sstring the sigma values
       are first raised to the 1/#_dimensions power before performing linear interpolation and
       multiplying. If you don't specify all three strings, then a flat function equal to the
       'sigma' parameter is used for the missing dimensions. For dimensions of size one (the
       spatial dimenions if sbsize=1 or the temporal dimension for tbsize=1) the corresponding
       string is ignored.

       For example:

            ssx="0.0:1.0 1.0:10.0",ssy="0.0:1.0 1.0:10.0",sst="0.0:1.0 1.0:10.0"

       will give the same result as

            sstring="0.0:1.0 1.0:10.0"

       Or if you want to ramp sigma based on temporal frequency:

            sigma=10.0,sst="0.0:1.0 1.0:10.0"

            This will use 10.0 for the horizontal/vertical dimensions, and ramp
            sigma from 1.0 to 10.0 in the temporal dimension.

       If 'sstring' is specified, it takes precedence over ssx/ssy/sst. Again, the
       "filter_spectrum-date_string.txt" output file is helpful in visualizing the result.

       default:  ""


   dither -

       Controls whether dithering is performed when converting from float to unsigned char
       for output. Internally dfttest works on floating point values. For output the
       result must be quantized back to unsigned char values. Prior to v1.8 this was always
       done by simply rounding. Possible settings:

           0 -     no dithering (same as v1.7 and prior)
           1 -     Floyd-Steinberg dithering
           2-100 - Floyd-Steinberg dithering with increasing amounts of uniform random
                   noise added prior to the dithering process

       Obviously dither=0 is the fastest, and dither=1 is slightly faster than dither>=2
       due to not having to generate a random number for every pixel. However, this part
       doesn't take much time compared to the actual filtering operation. dither=1 should
       combat any banding introduced by dfttest's quantization, but probably wont help
       banding in the source. At least, not anymore than the filtering itself. dither>=2 can
       combat banding in the source that is left over after filtering.

       default:  0

Needless to say 'nstring' came from foxyshadis' idea for not having to create a separate file. The sstring/ssx/ssy/sst I had been meaning to do for a while since it is a frequently requested feature.

For reference, the sigma4/sigma3/sigma2/sigma system of fft3dfilter converts to - sstring="$ 0.0:sigma4 0.1768:sigma3 0.3536:sigma2 1.0000:sigma". Note the '$', which makes dfttest use the second sstring method for determining sigma values.

[tritical]

Posted v1.8, changes:

+ added dither parameter and functionality
+ attach date string to filter_spectrum.txt and noise_spectrum.txt output
+ changed sstring handling and added option to function like fft3dfilter

See post above for more info on parameters.

[foxyshadis]

Wow, nice. Thanks!

AddGrain pre-generates a field of grain and quickly applies it to the image each frame, starting from a random offset, rather than coming up with a new random number each pixel. Since dither isn't nearly as visible, you could probably get away with pre-generating as few as 128-256 values to repeatedly add prior to floyd-steinberg.

[Terranigma]

Thanks a lot for the update. After fiddling around with it, i've really come to love this new version, even more so than fft3dfilter in every conceivable way.

And oh yea, thanks for adding in my request.

[Lorax2161]

Thanks for your hard work on this. It just keeps getting better and better.

Question: might it be in your future plans to include a "show" parameter which would display either the noise identified for denoising without the video--or which would paint the pixels that have been identified as noise to be removed a different color?

Of course I presume this would make it slower, but it would only be for testing/analysis... not to be left on during an encode.

In any event, I really do appreciate your contributions.

[Stephen R. Savage]

Lorax, that feature would not need to be implemented in dfttest. You can already create the difference map between two clips with the internal Subtract() filter or the mt_makediff() function from the MaskTools plugin.

[tormento]

Sob. If only it could be a little faster on HD contents.

Tritical: do you plan to update the x64 version too?