TempLinearApproximate plugin

[Mystery Keeper]

Here's what seems to be stable release of TempLinearApproximate plugin rewrite for VapourSynth.
This plugin is meant for motion-compensated temporal denoising.

TempLinearApproximate r3 - Windows binaries. (https://bitbucket.org/mystery_keeper/templinearapproximate-vapoursynth/downloads/templinearapproximate-r3.7z)
Sources. (https://bitbucket.org/mystery_keeper/templinearapproximate-vapoursynth/src)

Usage:
tla.TempLinearApproximate(
clip:clip
radius:int:opt
planes:int[]:opt:empty
gamma:int:opt
)

radius: (default 5) The size of neighborhood of the current frame. Approximation is made by (radius * 2 + 1) points, unless close to beginning or end - it is trimmed in these cases.

planes: The array of planes you want to process. If not specified - all planes are processed. Empty array is allowed for testing purposes.

gamma: Boolean. True by default. Indicates if luma values are converted from gamma corrected to linear light before processing and back after. Never applied to chroma planes. Setting this argument to False gives a speed up.

Motion compensated denoising script. (https://bitbucket.org/mystery_keeper/templinearapproximate-vapoursynth/raw/master/MCDenoise.py)
Usage example:
import sys
sys.path.append('E:\\vapoursynth-plugins\\py\\')
import MCDenoise as mcd

tlamcArguments = {
'radius': 5,
'search': 5,
'pelsearch': 2,
'subpelinterp': 2,
'searchparam': 5,
'truemotion': True,
'chromamotion': True,
'thsad': 600,
'thscd1': 200,
'thscd2': 150,
'refine': True,
'_lambda': 2000,
}

denoised = mcd.TempLinearApproximateMC(input=source, **tlamcArguments)
Note the "refine" argument. It makes the function ignore the "blocksize" and "overlap" parameters, start motion estimation with blocksize=32 and overlap=16 and gradually refine down to blocksize=4 and overlap=2. Very accurate, but very slow.

CHANGELOG:

r3: Support for single precision float input.
Processing in linear light. On by default.
Motion compensated script modified into standalone function. "gamma" argument added.

r2:
Fallback to averaging when current frame is the middle in the range (it is not the middle only when close to the beginning or the end). Speedup.
License change to MIT.
Fallback to C90. Compilable with MSVC.

[handaimaoh]

As there are no motion compensation means in VapourSynth at the moment, it uses AviSynth MVTools2 plugin and thus is only usable in x32 VapourSynth.

You are just using MAnalyse, MSuper and MCompensate right now, correct? I noticed that making an MVTools replacement was on the list of Vapoursynth issues. If no one else has started I can try to whip up a quick port of those functions. Now, I haven't yet looked at the code so it could be far less "quick", though, as it's likely my eyes will begin bleeding reading the source...

[Mystery Keeper]

You are just using MVAnalyse, MVSuper and MVCompensate right now, correct? I noticed that making an MVTools replacement was on the list of Vapoursynth issues. If no one else has started I can try to whip up a quick port of those functions. Now, I haven't yet looked at the code so it could be far less "quick", though, as it's likely my eyes will begin bleeding reading the source...Community will worship you if you do. Though I think it would be better if Myrsloik ported it personally. MVTools relies on AviSynth specifics, and VapourSynth port could make use of VapourSynth capabilities. Like frame properties. I hope he'll at least supervise it if you undertake the port.

[handaimaoh]

Community will worship you if you do. Though I think it would be better if Myrsloik ported it personally. MVTools relies on AviSynth specifics, and VapourSynth port could make use of VapourSynth capabilities. Like frame properties. I hope he'll at least supervise it if you undertake the port.

I'll see what I can do. :) Having just started looking at the code I can see this will be interesting... :scared: :eek: :D

[handaimaoh]

Also looking at the code, what "Avisynth specifics" are you referring to? I'm not seeing what you are talking about. It's mostly using a bunch of its own internal functions. The most I'm seeing it use of the Avisynth API is the getReadPtr/getWritePtr, getPitch, etc. functions.

[Mystery Keeper]

Also looking at the code, what "Avisynth specifics" are you referring to? I'm not seeing what you are talking about. It's mostly using a bunch of its own internal functions. The most I'm seeing it use of the Avisynth API is the getReadPtr/getWritePtr, getPitch, etc. functions.
Some data is stored in audio stream or something like that. I'm not sure.

[handaimaoh]

Some data is stored in audio stream or something like that. I'm not sure.

Ok, I see that in the documentation. It uses the "num_audio_samples" as a field to push data through. Just requires using the propSet* functions to do the same thing. So we would just need to decide on what to call the frame properties that are needed to be pushed out. It's quite simple.

[Myrsloik]

Some data is stored in audio stream or something like that. I'm not sure.

A few words on "porting" MVTools and motion estimation in general.

I won't.

When I get to it (sometime early next year I guess), I will start writing something new. It will be inspired by mvtools and the goal is to have equivalent useful functions.

The only vapoursynth features mvtools would benefit from is the possibility to attach data to frames, that would remove the insanity of passing a data pointer in an int for the audio properties. That and the general parallelism.

Don't expect wonders from running it in parallel though, it already breaks things down into many small and fairly balanced steps as separate functions so even the avisynth version scales in an acceptable way.

Feel free to shitport the existing lump of mvtools code but I won't waste my time doing that.

Now I let you go on with the thread's actual topic....

[handaimaoh]

Feel free to shitport the existing lump of mvtools code but I won't waste my time doing that.

Well if you're going to make something different to replace it I won't bother. My eyes have had enough bleeding for one day reading its code even for 10 minutes or so. I'll just continue working on other stuff instead.

I'll bow out for now too.

[Mystery Keeper]

When I get to it (sometime early next year I guess), I will start writing something new. It will be inspired by mvtools and the goal is to have equivalent useful functions.That's exactly why I said it is better be done by you =)

[mastrboy]

Feel free to shitport the existing lump of mvtools code but I won't waste my time doing that.

Even if the code is a lump of unreadable/messy code from a programming perspective, for video encoding isn't it the result that matter? and not how that result is achieved?

I've yet to find a better temporal denoiser/degrainer which can produce the exact result I'm looking for combined with masktools2 for any source yet. (Of course not at default settings...)

[handaimaoh]

Even if the code is a lump of unreadable/messy code from a programming perspective, for video encoding isn't it the result that matter? and not how that result is achieved?

I've yet to find a better temporal denoiser/degrainer which can produce the exact result I'm looking for combined with masktools2 for any source yet. (Of course not at default settings...)

Sure, if you don't care about the fact that no one else can pick up development on it if the original author stops.

[Mystery Keeper]

Even if the code is a lump of unreadable/messy code from a programming perspective, for video encoding isn't it the result that matter? and not how that result is achieved?

I've yet to find a better temporal denoiser/degrainer which can produce the exact result I'm looking for combined with masktools2 for any source yet. (Of course not at default settings...)
AviSynth is in such a poor state because its authors have sacrificed maintainability to speed. There were no options at the time. No intrinsics and multicore CPUs. Now we don't need to do such thing and can keep the code neat and maintainable.

Also, motion compensated denoising is great, but no way universal. When two moving scenes get blended (and that happens a lot in movies and even more in music video) - you can expect motion compensation to create all kinds of distortion.

[handaimaoh]

AviSynth is in such a poor state because its authors have sacrificed maintainability to speed. There were no options at the time. No intrinsics and multicore CPUs. Now we don't need to do such thing and can keep the code neat and maintainable.

Compiler intrinsics for MSVC have been around since at least Visual Studio 2003. Now there have been numerous code generation bugs over the years, but Avisynth's lack of their usage is not because they are something recent. Lack of their usage is likely because the core devs never moved passed Visual Studio 6 and what you said about both the core devs and plugin writers sacrificing any maintainability for writing the most "optimized" code. Even x264's ASM is far easier to follow than the inline asm macro mess that makes up the vast majority of asm in Avisynth plugins.

[Mystery Keeper]

Question!
Since my filter calculates linear approximation, should I use gamma correction? Should I make it default? Should I only apply it to Y-plane? Or make an option to specify the planes?

[Mystery Keeper]

r2 is out with speedup, license change and source compilable with MSVC.

[mawen1250]

It seems the 64bit binary is an old one?

[Mystery Keeper]

It seems the 64bit binary is an old one?Oops. You're right. Fixed.

[Mystery Keeper]

r3 is out with processing in linear light, single precision float input support and MC script changed into standalone function.

[TurboPascal7]

http://pastebin.com/1KqGiCQR
I didn't actually run this but it should probably work. Just made your python less bad.

[Mystery Keeper]

Thank you. Updated the archive and the script link.

[Are_]

At least with gcc you need to add "#include <stddef.h>" to "src/processplane.h" for it to compile.

If you don't do that it complains non stop with "src/processplane.h:11:2: error: unknown type name ‘ptrdiff_t’".

[Mystery Keeper]

Weird. I did test compilation with GCC, and it doesn't complain. I'm using MinGW Builds.

[Are_]

gcc-4.9.2 http://paste.kde.org/po3ffxcgc/ytyhyo/raw

And I'm not a coder so maybe I misunderstood, but doesn't this (http://pubs.opengroup.org/onlinepubs/009695399/basedefs/stddef.h.html) mean you should use that include if you plan to use ptrdiff_t? Why is working for you without it? :|

[Mystery Keeper]

Well, it seems that my version of GCC (4.8.1) has stddef.h included in stdint.h.

[Are_]

I checked my stdint.h from mingw cross compile package and indeed stddef.h is right there included, but not in the header from glibc the system uses to build stuff. Maybe it's a windows thingy.

[Mystery Keeper]

Looks like FFmpeg had the same issue. Well, fixed.

[metyo]

Hi
I'd like to try the denoise func of TLA MC script, but I get the error when I'd like to use the lib
QWidget::insertAction: Attempt to insert null action
vsedit: symbol lookup error: /usr/lib/vapoursynth/libtemplinearapproximate.so: undefined symbol: gcToLinear


Achlinux 64 and 32 bit as well, compiled from the AUR git (with success)

At the end, my script was simplified to
import vapoursynth as vs
core = vs.get_core ()
ret = core.ffms2.Source(source='*****OUT.dv')
ret=core.tla.TempLinearApproximate(clip=ret,radius=2)
ret.set_output ()

vspipe gives the same :
vspipe: symbol lookup error: /usr/lib/vapoursynth/libtemplinearapproximate.so: undefined symbol: gcToLinear


Any help ?
thx++
metyo

[Are_]

Probably misscompiled, did you gcc -shared -fPIC ${CFLAGS} ${LDFLAGS} -o libtemplinearapproximate.so $(pkg-config --cflags vapoursynth) src/processplane.c src/main.c ? Note "-shared -fPIC"

[sl1pkn07]

yes (is my pkgbuild from [AUR] (https://aur.archlinux.org/packages/va/vapoursynth-plugin-templinearapproximate-git/PKGBUILD))

EDIT: mmm possible missing src/processplane.c in gcc line?

EDIT2: try now @metyo

[metyo]

Thank you sl1pkn07 and Are_, now it works perfectly, and the TLA MC script is ok

thank you again

metyo

[Mystery Keeper]

Updated the motion compensated denoising script (https://bitbucket.org/mystery_keeper/templinearapproximate-vapoursynth/raw/master/MCDenoise.py). Now supports float point input and DCT for 32x32 blocks (DCT gets changed to the closest supported mode).

[asarian]

Why, thank you for this tool! It's awesome! :) And very fast too (relatively, naturally). I was looking at a 90 hours job, with AviSynth and MCTD, and now, in VapourSynth and TempLinearApproximateMC, I'm looking at an eta of ~11 hours! :)

[asarian]

For future releases, btw, would it be possible to add strength-presets, like 'very low', 'low', 'medium', etc? Kinda like in MCTD and QTGMC. Or add some more-or-less equivalent values to the documentation. Thx.

[Mystery Keeper]

The algorithm is straightforward. There's no "strenght". But you can lower the temporal radius.

[asarian]

The algorithm is straightforward. There's no "strenght". But you can lower the temporal radius.

Shouldn't the 'thsad', 'thscd1', 'thscd2' arguments also matter?! I pulled some comparison table off of MCTD:


### +-------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
### | SETTINGS | VERY LOW | LOW | MEDIUM | HIGH | VERY HIGH |
### |-------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------|
### | radius | 1 | 2 | 3 | 2 | 3 |

(...)

### | thSAD | 200 | 300 | 400 | 500 | 600 |
### | thSAD2 | 200 | 300 | 400 | 500 | 600 |
### | thSCD1 | 200 | 300 | 400 | 500 | 600 |
### | thSCD2 | 90 | 100 | 100 | 130 | 130 |
### |-------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------|
### | truemotion | false | false | false | false | false |
### | MVglobal | true | true | true | true | true |
### | pel | 1 | 2 | 2 | 2 | 2 |
### | pelsearch | 1 | 2 | 2 | 2 | 2 |
### | search | 2 | 2 | 2 | 2 | 2 |
### | searchparam | 2 | 2 | 2 | 2 | 2 |

[Mystery Keeper]

Those arguments affect the accuracy of motion compensation. More strict settings give you better accuracy, but let some areas remain unfiltered. Looser settings ensure better coverage, but let in false vectors, resulting in more blurry image. They shouldn't affect the speed much or maybe at all.

[Mystery Keeper]

Updated the motion compensated denoising script. It requires updated versions of mvtools and/or mvsf now. Removed DCT workaround, since all DCT are now supported on all block sizes by mvtools. Added support for using DFTTest instead of TLA and arguments for it. TLA is still default. Arguments 'tmode', 'tbsize' and 'tosize' are fixed and not exposed.

[xekon]

Thank you so much for this plugin, it seems fast! i am in the process of moving from avisynth to vapoursynth, replacing mctd and can't seem to get as much of the mosquito noise filtered as i did in avisynth.

Any help is appreciated, I took a small sample of my project and included both the avisynth and vapoursynth scripts as well as comparison screenshots of frame 216.

removed dead link.

My original goal is a denoised upscale similar to one i found on youtube, this is how close i got using avisynth (mine saves a bit more detail and has little less blurring, but does not get quite as much of the noise):
http://screenshotcomparison.com/comparison/192795

And then here a comparison between my vapoursynth and avisynth scripts, I just cant seem to get as much of noise filtered when using vapoursynth:
http://screenshotcomparison.com/comparison/192804

I know everyone has their own things to do, but if you get free time to help I would really appreciate it. I have tried upping the thsad and thscd1 values to 900, but it didn't seem to change much, I don't have a very good understanding of the script and its parameters.

EDIT: I may have just needed a good nights sleep, this morning it occurred to me to use this filter as part of the prefiter for V-bm3d, so far I really like the output, however I have to wonder if I could have gotten similar output with less functions(faster). I will have to see how fast this encodes compared to my avisynth version(however this one does seem a bit cleaner than my avisynth one)

comparison: http://screenshotcomparison.com/comparison/192841

vapoursynth script: http://pastebin.com/dpA2eA8Y

[bin.n2f]

TempLinearApproximateMC has ghosting artifacts to prevent this ... i used QTGMC to increase frame rate from 24 to 48

:thanks: for this tool

[KingLir]

How can I build this on macOS ? I am not a coder, but I can follow instructions :)

[sl1pkn07]

for linux i use this


gcc -c -fPIC ${CFLAGS} ${CPPFLAGS} -I. $(pkg-config --cflags vapoursynth) -o main.o src/main.c
gcc -c -fPIC ${CFLAGS} ${CPPFLAGS} -I. $(pkg-config --cflags vapoursynth) -o processplane.o src/processplane.c
gcc -shared -fPIC ${LDFLAGS} -o lib${_plug}.so main.o processplane.o

[KingLir]

for linux i use this


Thank you! Using this I found the needed steps on macOS:

A. Add the missing header files in src/vapoursynth
VapourSynth.h , VSHelper.h , VSScript.h

B. Change both main.c vapoursynth includes to use "quotes" instead of <angled>

C. Run this:


gcc -c -fPIC ${CFLAGS} ${CPPFLAGS} -I. $(pkg-config --cflags vapoursynth) -o main.o src/main.c
gcc -c -fPIC ${CFLAGS} ${CPPFLAGS} -I. $(pkg-config --cflags vapoursynth) -o processplane.o src/processplane.c
g++ -dynamiclib -undefined suppress -flat_namespace *.o -o libtla.dylib


Using this info, if anyone can do a commit for building script for macOS (and Linux) - please do.

[Mystery Keeper]

TempLinearApproximateMC has ghosting artifacts to prevent this ... i used QTGMC to increase frame rate from 24 to 48

:thanks: for this tool
The only reason it could produce ghosting would be bad motion estimation settings.

[KingLir]

I am getting strange motion artifact (like missing frames - I don't know the right term) with the following script and R36. When I change TLA-MC line with any other denoiser - I don't get the issue. What should I change in TLA parameters ?

clip = core.ffms2.Source(source=filename)
clip = core.vivtc.VFM(clip=clip, order=1) # IVTC part 1
clip = core.vivtc.VDecimate(clip=clip) # IVTC part 2
clip = mcd.TempLinearApproximateMC(clip=clip)
clip.set_output()

Original clip:
https://mega.nz/#!ggxF0bxA!IksjCkXKATd1VOS3ps4NrgV91p1k6YdQIRYcyvIyowY

After TLA-MC:
https://mega.nz/#!YlpT0QaD!4-ZbMsOcVQN67QFrJ1LKe6BObIMSesp8I7SZ1BnvmLk

[KingLir]

Anyone ?

[Mystery Keeper]

I am getting strange motion artifact (like missing frames - I don't know the right term) with the following script and R36. When I change TLA-MC line with any other denoiser - I don't get the issue. What should I change in TLA parameters ?Try thsad=400, thscd1=200, thscd2=150. Remove _lambda argument.

[KingLir]

Try thsad=400, thscd1=200, thscd2=150. Remove _lambda argument.

Thanks! Using your answer and the documentary I ended up using the following.The output seems good to me, but what are the shortcomings of these ? I aim for "best" image quality and OK with it being very slow.

clip = mcd.TempLinearApproximateMC(clip=clip, radius=3, thsad=800, thscd1=800, thscd2=150, _lambda = None)

[Mystery Keeper]

Thanks! Using your answer and the documentary I ended up using the following.The output seems good to me, but what are the shortcomings of these ? I aim for "best" image quality and OK with it being very slow.

clip = mcd.TempLinearApproximateMC(clip=clip, radius=3, thsad=800, thscd1=800, thscd2=150, _lambda = None)This method relies entirely on the reliability of motion estimation. And motion estimation on noisy source is not quite reliable. Depending on your settings you may get either washed out details - usually thin lines, or some noise unfiltered.

[KingLir]

This method relies entirely on the reliability of motion estimation. And motion estimation on noisy source is not quite reliable. Depending on your settings you may get either washed out details - usually thin lines, or some noise unfiltered.

Thanks for the explanation. So just to make sure I got it right, if I use strict and slow settings like showing in my last code line above (is it indeed the recommended "strict and slow" settings ?) - I should worry only for some noise unfiltered or it still can produce "washed out details - usually thin lines" ?