MVTools, Depan, DepanEstimate for VapourSynth

[jackoneill]

v7 is here (https://github.com/dubhater/vapoursynth-mvtools/releases/tag/v7), mostly with fixes to bugs introduced in v6.



* Use SATD functions optimised for SSE4.1, AVX, XOP, and AVX2, copied from x264. Affects Analyse and Recalculate used with dct=5..10.
* Add sanity check for thscd1 parameter. It will no longer accept values larger than 8*8*255. That was always the maximum value, it just wasn't checked.
* Fix bug in Analyse with isse=True and 16 bits per sample input (introduced in v6).
* Fix Recalculate (completely broken in v6).
* Fix pink tint in Degrain and Compensate when overlap is used and isse=True (introduced in v6).
* Store the ml parameter in Mask as float, as it's supposed to be, instead of integer. Fix by HolyWu. (introduced who knows when, possibly right at the beginning).

[buchanan]

Thank you.
Encoding with 16bit fed into QTGMC stops after ~3000 frames / 1 min. Same script with 8 bit into TGMC runs fine until the end

[jackoneill]

Thank you.
Encoding with 16bit fed into QTGMC stops after ~3000 frames / 1 min. Same script with 8 bit into TGMC runs fine until the end

32 or 64 bit? Does v6 work better?

Run it in VirtualDub to get a somewhat useful crash report.

[zerowalker]

Here is two shots from Avisynth and Vapoursynth.

http://imgur.com/fvlPQ7q,SITTfOS#0

The image difference is minimal but it's there (you will have to download and switch between them to see it).

[feisty2]

what's "mv.Finest"?
it's new, no avs mvtools got this
and... I couldn't find a readme for it

[jackoneill]

Here is two shots from Avisynth and Vapoursynth.

http://imgur.com/fvlPQ7q,SITTfOS#0

The image difference is minimal but it's there (you will have to download and switch between them to see it).

Is that with overlap and v6? Because at least some of the differences there could be caused by a bug in v6 (the slight brightening of the dark object with the Logitech logo on it). It's fixed in v7. Other than that, the differences in degraining must be due to the small differences in the output of Super. They're so insignificant, I don't feel like doing anything about it.


what's "mv.Finest"?
it's new, no avs mvtools got this
and... I couldn't find a readme for it

Finest is a helper used by some of the Flow* filters. It does a thing to the super clip. It's not new. In the Avisynth plugins it's not public, that's all.

[Are_]

Here is two shots from Avisynth and Vapoursynth.

http://imgur.com/fvlPQ7q,SITTfOS#0

The image difference is minimal but it's there (you will have to download and switch between them to see it).

Second image looks like it has some kind of dithering mesh applied to it, how did you produce it? Because I'm not having that with vapoursynth.

[Reel.Deel]

Finest is a helper used by some of the Flow* filters. It does a thing to the super clip. It's not new. In the Avisynth plugins it's not public, that's all.
MVTools from SVP (http://www.svp-team.com/wiki/Plugins:_MVTools2#The_interface) made it available.
This function is commented out in latest MVTools 2.5 versions and it's obviously a mistake cause it can helps a lot with caching the big frame for hpel (pel=2) and qpel (pel=4) modes.
So MVFinest is only used on the Flow* filters, and can help caching on large frames when pel=2 or 4 is used?

I tried the following but I get this error in VDub x64: An out-of-bounds memory access (access violation) occurred in module 'libmvtools'......reading address 000000000E2F76C4.
If I comment out the mv.Finest call it works fine. I might be doind something wrong but if I'm not then here's the crash report: http://privatepaste.com/9bf3b6e1f9
import vapoursynth as vs
core = vs.get_core()
src = core.lsmas.LWLibavSource(r'M:\1080p BD test.mkv')
super = core.mv.Super(src)
super = core.mv.Finest(super)
mvbw = core.mv.Analyse(super, isb=True)
mvfw = core.mv.Analyse(super)
out = core.mv.FlowFPS(clip=src, super=super, mvbw=mvbw, mvfw=mvfw, num=50, den=1)
out.set_output()

[jackoneill]

MVTools from SVP (http://www.svp-team.com/wiki/Plugins:_MVTools2#The_interface) made it available.

So MVFinest is only used on the Flow* filters, and can help caching on large frames when pel=2 or 4 is used?

I tried the following but I get this error in VDub x64: An out-of-bounds memory access (access violation) occurred in module 'libmvtools'......reading address 000000000E2F76C4.
If I comment out the mv.Finest call it works fine. I might be doind something wrong but if I'm not then here's the crash report: http://privatepaste.com/9bf3b6e1f9
import vapoursynth as vs
core = vs.get_core()
src = core.lsmas.LWLibavSource(r'M:\1080p BD test.mkv')
super = core.mv.Super(src)
super = core.mv.Finest(super)
mvbw = core.mv.Analyse(super, isb=True)
mvfw = core.mv.Analyse(super)
out = core.mv.FlowFPS(clip=src, super=super, mvbw=mvbw, mvfw=mvfw, num=50, den=1)
out.set_output()


Ah, so it's public in the SVP fork.

The Flow* filters invoke it internally when they need it, so you don't have to do anything. I don't know what additional use the SVP folks have for it (maybe in their secret filters?), but here it's already used as much as it can be. I made it public because why not.

It crashes because Analyse doesn't expect to receive the output of Finest.

[Reel.Deel]

@HolyWu and jackoneill
Thanks for explaining, I understand now. http://www.cheesebuerger.de/images/midi/boese/d074.gif

[MonoS]

The plugin crash using QTGMC

Here the crash info
VirtualDub crash report -- build 35491 (release-AMD64)
--------------------------------------

Disassembly:
6efb07e0: 008b81100100 add [rbx+11081], cl
6efb07e6: 00448b81 add [rbx+rcx*4-7fh], al
6efb07ea: 1c01 sbb al, 01h
6efb07ec: 0000 add [rax], al
6efb07ee: 39d0 cmp eax, edx
6efb07f0: 0f8efa010000 jle 16efb09f0
6efb07f6: 4139c0 cmp eax, eax
6efb07f9: 0f8e61020000 jle 16efb0a60
6efb07ff: 8981f8000000 mov [rcx+f8], eax
6efb0805: 8b8108010000 mov eax, [rcx+108]
6efb080b: 398114010000 cmp [rcx+114], eax
6efb0811: 8b9120010000 mov edx, [rcx+120]
6efb0817: 0f4d8114010000 cmovge eax, [rcx+114]
6efb081e: 39d0 cmp eax, edx
6efb0820: 0f4cc2 cmovl eax, edx
6efb0823: 8981fc000000 mov [rcx+fc], eax
6efb0829: 80b98800000000 cmp byte ptr [rcx+88], 00h
6efb0830: 741a jz 16efb084c
6efb0832: 488b81f4000000 mov rax, [rcx+f4]
6efb0839: 488981e8000000 mov [rcx+e8], rax
6efb0840: 8b81fc000000 mov eax, [rcx+fc]
6efb0846: 8981f0000000 mov [rcx+f0], eax
6efb084c: 448b8928020000 mov r9d, [rcx+228]
6efb0853: 8b8124020000 mov eax, [rcx+224]
6efb0859: 448b81f0000000 mov r8d, [rcx+f0]
6efb0860: 410fafc1 imul eax, ecx
6efb0864: 41d1f8 sar eax, 1
6efb0867: 4501c8 add eax, ecx
6efb086a: 99 cdq
6efb086b: 41f7f8 idiv eax, eax <-- FAULT
6efb086e: 410fafc1 imul eax, ecx
6efb0872: 99 cdq
6efb0873: 41f7f8 idiv eax, eax
6efb0876: 898124020000 mov [rcx+224], eax
6efb087c: 5b pop ebx
6efb087d: 5e pop esi
6efb087e: 5f pop edi
6efb087f: 5d pop ebp
6efb0880: 415c pop esp
6efb0882: c3 ret
6efb0883: 8ba9f0010000 mov ebp, [rcx+1f0]
6efb0889: 448d4dff lea r9d, [rbp-01h]
6efb088d: 39fd cmp ebp, edi
6efb088f: 440f4fcf cmovg ecx, edi
6efb0893: e964feffff jmp 16efb06fc
6efb0898: 0f db 0fh
6efb0899: 1f db 1fh
6efb089a: 8400 test [rax], al
6efb089c: 0000 add [rax], al
6efb089e: 0000 add [rax], al
6efb08a0: 448b89ec010000 mov r9d, [rcx+1ec]
6efb08a7: 418d59ff lea ebx, [r9-01h]
6efb08ab: 4139e9 cmp ecx, ebp
6efb08ae: 4189c1 mov ecx, eax
6efb08b1: 0f4fdd cmovg ebx, ebp
6efb08b4: 39c7 cmp edi, eax
6efb08b6: 0f8c40feffff jl 16efb06fc
6efb08bc: ebc5 jmp 16efb0883
6efb08be: 6690 nop
6efb08c0: 8bb1f0010000 mov esi, [rcx+1f0]
6efb08c6: 448d56ff lea r10d, [rsi-01h]
6efb08ca: 4439ce cmp esi, ecx
6efb08cd: 450f4fd1 cmovg edx, ecx
6efb08d1: e9c3fdffff jmp 16efb0699
6efb08d6: 662e db 2eh
6efb08d8: 0f db 0fh
6efb08d9: 1f db 1fh
6efb08da: 8400 test [rax], al
6efb08dc: 0000 add [rax], al
6efb08de: 0000 add [rax], al

Built on Althena on Sun Oct 27 16:00:02 2013 using compiler version 1400

Windows 6.1 (Windows 7 x64 build 7601) [Service Pack 1]
Memory status: virtual free 8387123M/8388608M, commit limit 8056M, physical total 4029M

RAX = 2737100
RBX = 310
RCX = 68ae1a0
RDX = 0
RSI = 7b9ebf0
RDI = 47
RBP = 440
R8 = 0
R9 = 64640
R10 = 440
R11 = 3
R12 = 44bd7e
R13 = 64640
R14 = 14
R15 = fff37380
RSP = 75bf660
RIP = 6efb086b
EFLAGS = 00010247


Crash reason: Integer Divide-by-Zero

Crash context:
An integer division by zero occurred in module 'libmvtools'.

Pointer dumps:

RAX 02737100: 6170b690 00000000 6170b6b0 00000000 6170b6d0 00000000 54143f73 88000035
RCX 068ae1a0: 00000059 00000047 00000010 00000010 000018af 00000002 00000001 00000001
RSI 07b9ebf0: 00000008 00000000 0061a6aa 00000000 00000000 0061e57e 00000000 00000000
RSP 075bf660: 068ae1a0 00000000 00000030 00000000 00000000 00000000 20368b0c 00000000
075bf680: 00000019 00000000 6efb810b 00000000 00000010 00000000 003d0000 00000000
075bf6a0: 00000008 00000000 00000008 00000000 003d0288 00000000 77253448 00000000
075bf6c0: 075bf700 00000000 00000008 00000004 075bf6e0 00000000 000bf017 000befe5
R12 0044bd7a: 0000004b 00000000 b221ffff 0002982b 03fe0000 bdc00001 00000044 00010000

Thread call stack:
6efb086b: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+2eb8b]
6efb810b: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+3642b]
77253448: ntdll!RtlAllocateHeap [77200000+53360+e8]
77253448: ntdll!RtlAllocateHeap [77200000+53360+e8]
6efb8e9d: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+371bd]
6ef82a21: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+d41]
6efa77b4: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+25ad4]
6ef87c84: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+5fa4]
7fed8ff0349: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+5009]
7fed90644d7: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+79197]
7fed8fff19b: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+13e5b]
7fed8ffbd83: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+10a43]
7fed8ffd4bc: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+1217c]
7fed9072769: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+87429]
7fed9063e75: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+78b35]
7fed9037aeb: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+4c7ab]
7fed8fffae8: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+147a8]
7fed903b5c0: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+50280]
7fed9069156: VapourSynth!getVapourSynthAPI [7fed8fc0000+2b340+7de16]
770f59ed: kernel32!BaseThreadInitThunk [770e0000+159e0+d]
7722c541: ntdll!RtlUserThreadStart [77200000+2c520+21]

-- End of report


This is the script
import vapoursynth as vs
import havsfunc as has

core = vs.get_core()

src = core.d2v.Source("D:/JDownloader/Film/VTS_01_1.d2v", nocrop=True).fmtc.bitdepth(bits=16)

deint = has.QTGMC(src, Preset="Very Slow", SourceMatch=3, MatchPreset2="Slow", Lossless=2, NoisePreset="Slow", TFF=False, EZDenoise=1.5).fmtc.nativetostack16()

deint.set_output()
[As a note, the dvd i'm trying to convert is a dvd the author itself sent me, hy neuron2 :D]

I get the same error without nativetostack16, i was forced to use it cause VD don't support 16bit input

Using v6 crash at startup, instead with v7 i can also jump between frames in the preview and can encode about 1k frames

[jackoneill]

The plugin crash using QTGMC

Here the crash info


Thanks, that's helpful. The thing is, the disassembly doesn't match my own copy of libmvtools.dll from vapoursynth-mvtools-v7-win64.7z (and looks totally wrong, no wonder it crashes). Can you verify the md5sum of the library, please? It should be 6231b53114e98b71f751bfa26468b7ee. It's probably the same, but let's make sure.

[MonoS]

Thanks, that's helpful. The thing is, the disassembly doesn't match my own copy of libmvtools.dll from vapoursynth-mvtools-v7-win64.7z (and looks totally wrong, no wonder it crashes). Can you verify the md5sum of the library, please? It should be 6231b53114e98b71f751bfa26468b7ee. It's probably the same, but let's make sure.

Yes the hash is the same, the only thing that come up in my mind is that some instruction must have rewritten some code during the execution.

Is this even possible?? Are we sore that mvtools and vs mark all the code page as not writable??

[jackoneill]

Yes the hash is the same, the only thing that come up in my mind is that some instruction must have rewritten some code during the execution.

Is this even possible?? Are we sore that mvtools and vs mark all the code page as not writable??

Looking more closely, it's only VirtualDub's translation of the machine code into instructions that doesn't match my own disassembly. The machine code itself (second column) is correct. VirtualDub must have a bug there.

I found the cause of the crash, though. This binary should work: http://ulozto.net/xJMBQnV5/vapoursynth-mvtools-v7-div0-win64-7z

buchanan: Perhaps you had the same problem?

[MonoS]

Looking more closely, it's only VirtualDub's translation of the machine code into instructions that doesn't match my own disassembly. The machine code itself (second column) is correct. VirtualDub must have a bug there.

I found the cause of the crash, though. This binary should work: http://ulozto.net/xJMBQnV5/vapoursynth-mvtools-v7-div0-win64-7z

buchanan: Perhaps you had the same problem?

A VD bug, long time no see :)

I'm trying it right now with the same clip as before [a ~30m extra], i'll let you know how it goes

[Overdrive80]

Actually I want port my scripts to vapoursynth but I can´t use MDegrainN from mvtools' mod by cretindesalpes. Could you consider it to implement?

Thanks.

[MonoS]

MVTools continue to crash even using the new dll.
The exact same crash report [yes, i made sure to use the new dll, hash A72268104CBE17049112FDF839BC0B57 ].
Only some register are different, but i think this is due to the windows address randomization
VirtualDub crash report -- build 35491 (release-AMD64)
--------------------------------------

Disassembly:
6efb07e0: 008b81100100 add [rbx+11081], cl
6efb07e6: 00448b81 add [rbx+rcx*4-7fh], al
6efb07ea: 1c01 sbb al, 01h
6efb07ec: 0000 add [rax], al
6efb07ee: 39d0 cmp eax, edx
6efb07f0: 0f8efa010000 jle 16efb09f0
6efb07f6: 4139c0 cmp eax, eax
6efb07f9: 0f8e61020000 jle 16efb0a60
6efb07ff: 8981f8000000 mov [rcx+f8], eax
6efb0805: 8b8108010000 mov eax, [rcx+108]
6efb080b: 398114010000 cmp [rcx+114], eax
6efb0811: 8b9120010000 mov edx, [rcx+120]
6efb0817: 0f4d8114010000 cmovge eax, [rcx+114]
6efb081e: 39d0 cmp eax, edx
6efb0820: 0f4cc2 cmovl eax, edx
6efb0823: 8981fc000000 mov [rcx+fc], eax
6efb0829: 80b98800000000 cmp byte ptr [rcx+88], 00h
6efb0830: 741a jz 16efb084c
6efb0832: 488b81f4000000 mov rax, [rcx+f4]
6efb0839: 488981e8000000 mov [rcx+e8], rax
6efb0840: 8b81fc000000 mov eax, [rcx+fc]
6efb0846: 8981f0000000 mov [rcx+f0], eax
6efb084c: 448b8928020000 mov r9d, [rcx+228]
6efb0853: 8b8124020000 mov eax, [rcx+224]
6efb0859: 448b81f0000000 mov r8d, [rcx+f0]
6efb0860: 410fafc1 imul eax, ecx
6efb0864: 41d1f8 sar eax, 1
6efb0867: 4501c8 add eax, ecx
6efb086a: 99 cdq
6efb086b: 41f7f8 idiv eax, eax <-- FAULT
6efb086e: 410fafc1 imul eax, ecx
6efb0872: 99 cdq
6efb0873: 41f7f8 idiv eax, eax
6efb0876: 898124020000 mov [rcx+224], eax
6efb087c: 5b pop ebx
6efb087d: 5e pop esi
6efb087e: 5f pop edi
6efb087f: 5d pop ebp
6efb0880: 415c pop esp
6efb0882: c3 ret
6efb0883: 8ba9f0010000 mov ebp, [rcx+1f0]
6efb0889: 448d4dff lea r9d, [rbp-01h]
6efb088d: 39fd cmp ebp, edi
6efb088f: 440f4fcf cmovg ecx, edi
6efb0893: e964feffff jmp 16efb06fc
6efb0898: 0f db 0fh
6efb0899: 1f db 1fh
6efb089a: 8400 test [rax], al
6efb089c: 0000 add [rax], al
6efb089e: 0000 add [rax], al
6efb08a0: 448b89ec010000 mov r9d, [rcx+1ec]
6efb08a7: 418d59ff lea ebx, [r9-01h]
6efb08ab: 4139e9 cmp ecx, ebp
6efb08ae: 4189c1 mov ecx, eax
6efb08b1: 0f4fdd cmovg ebx, ebp
6efb08b4: 39c7 cmp edi, eax
6efb08b6: 0f8c40feffff jl 16efb06fc
6efb08bc: ebc5 jmp 16efb0883
6efb08be: 6690 nop
6efb08c0: 8bb1f0010000 mov esi, [rcx+1f0]
6efb08c6: 448d56ff lea r10d, [rsi-01h]
6efb08ca: 4439ce cmp esi, ecx
6efb08cd: 450f4fd1 cmovg edx, ecx
6efb08d1: e9c3fdffff jmp 16efb0699
6efb08d6: 662e db 2eh
6efb08d8: 0f db 0fh
6efb08d9: 1f db 1fh
6efb08da: 8400 test [rax], al
6efb08dc: 0000 add [rax], al
6efb08de: 0000 add [rax], al

Built on Althena on Sun Oct 27 16:00:02 2013 using compiler version 1400

Windows 6.1 (Windows 7 x64 build 7601) [Service Pack 1]
Memory status: virtual free 8387182M/8388608M, commit limit 8056M, physical total 4029M

RAX = 2737100
RBX = 310
RCX = 787ae00
RDX = 0
RSI = 7b5b280
RDI = 47
RBP = 440
R8 = 0
R9 = 64640
R10 = 440
R11 = 3
R12 = 44bd7e
R13 = 64640
R14 = 14
R15 = fff37380
RSP = 77bf660
RIP = 6efb086b
EFLAGS = 00010247


Crash reason: Integer Divide-by-Zero

Crash context:
An integer division by zero occurred in module 'libmvtools'.

Pointer dumps:

RCX 0787ae00: 00000059 00000047 00000010 00000010 000018af 00000002 00000001 00000001
RSI 07b5b280: 00000008 00000000 0061a6aa 00000000 00000000 0061e57e 00000000 00000000
RSP 077bf660: 0787ae00 00000000 00000030 00000000 00000000 00000000 38fe8b0c 00000000
077bf680: 00000019 00000000 6efb810b 00000000 00000010 00000000 00590000 00000000
077bf6a0: 00000008 00000000 00000008 00000000 00590288 00000000 778a3448 00000000
077bf6c0: 077bf700 00000000 00000008 00000004 077bf6e0 00000000 000bf017 000befe5

Thread call stack:
6efb086b: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+2eb8b]
6efb810b: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+3642b]
778a3448: ntdll!RtlAllocateHeap [77850000+53360+e8]
778a3448: ntdll!RtlAllocateHeap [77850000+53360+e8]
6efb8ea0: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+371c0]
6ef82a21: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+d41]
6efa77b4: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+25ad4]
6ef87c84: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+5fa4]
7fedf950349: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+5009]
7fedf9c44d7: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+79197]
7fedf95f19b: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+13e5b]
7fedf95bd83: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+10a43]
7fedf95d4bc: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+1217c]
7fedf9d2769: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+87429]
7fedf9c3e75: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+78b35]
7fedf997aeb: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+4c7ab]
7fedf95fae8: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+147a8]
7fedf99b5c0: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+50280]
7fedf9c9156: VapourSynth!getVapourSynthAPI [7fedf920000+2b340+7de16]
776459ed: kernel32!BaseThreadInitThunk [77630000+159e0+d]
7787c541: ntdll!RtlUserThreadStart [77850000+2c520+21]

-- End of report

[jackoneill]

MVTools continue to crash even using the new dll.
The exact same crash report [yes, i made sure to use the new dll, hash A72268104CBE17049112FDF839BC0B57 ].
Only some register are different, but i think this is due to the windows address randomization

That makes no sense... What about this one? Can you show the crash report from this one? http://ulozto.net/xVhfoGVd/vapoursynth-mvtools-v7-div0-win64-7z

[MonoS]

Of course, here the report
VirtualDub crash report -- build 35491 (release-AMD64)
--------------------------------------

Disassembly:
6efb07e0: 008b81100100 add [rbx+11081], cl
6efb07e6: 00448b81 add [rbx+rcx*4-7fh], al
6efb07ea: 1c01 sbb al, 01h
6efb07ec: 0000 add [rax], al
6efb07ee: 39d0 cmp eax, edx
6efb07f0: 0f8efa010000 jle 16efb09f0
6efb07f6: 4139c0 cmp eax, eax
6efb07f9: 0f8e61020000 jle 16efb0a60
6efb07ff: 8981f8000000 mov [rcx+f8], eax
6efb0805: 8b8108010000 mov eax, [rcx+108]
6efb080b: 398114010000 cmp [rcx+114], eax
6efb0811: 8b9120010000 mov edx, [rcx+120]
6efb0817: 0f4d8114010000 cmovge eax, [rcx+114]
6efb081e: 39d0 cmp eax, edx
6efb0820: 0f4cc2 cmovl eax, edx
6efb0823: 8981fc000000 mov [rcx+fc], eax
6efb0829: 80b98800000000 cmp byte ptr [rcx+88], 00h
6efb0830: 741a jz 16efb084c
6efb0832: 488b81f4000000 mov rax, [rcx+f4]
6efb0839: 488981e8000000 mov [rcx+e8], rax
6efb0840: 8b81fc000000 mov eax, [rcx+fc]
6efb0846: 8981f0000000 mov [rcx+f0], eax
6efb084c: 448b8924020000 mov r9d, [rcx+224]
6efb0853: 8b8128020000 mov eax, [rcx+228]
6efb0859: 448b81f0000000 mov r8d, [rcx+f0]
6efb0860: 410fafc1 imul eax, ecx
6efb0864: 41d1f8 sar eax, 1
6efb0867: 4501c8 add eax, ecx
6efb086a: 99 cdq
6efb086b: 41f7f8 idiv eax, eax <-- FAULT
6efb086e: 410fafc1 imul eax, ecx
6efb0872: 99 cdq
6efb0873: 41f7f8 idiv eax, eax
6efb0876: 898128020000 mov [rcx+228], eax
6efb087c: 5b pop ebx
6efb087d: 5e pop esi
6efb087e: 5f pop edi
6efb087f: 5d pop ebp
6efb0880: 415c pop esp
6efb0882: c3 ret
6efb0883: 8ba9f0010000 mov ebp, [rcx+1f0]
6efb0889: 448d4dff lea r9d, [rbp-01h]
6efb088d: 39fd cmp ebp, edi
6efb088f: 440f4fcf cmovg ecx, edi
6efb0893: e964feffff jmp 16efb06fc
6efb0898: 0f db 0fh
6efb0899: 1f db 1fh
6efb089a: 8400 test [rax], al
6efb089c: 0000 add [rax], al
6efb089e: 0000 add [rax], al
6efb08a0: 448b89ec010000 mov r9d, [rcx+1ec]
6efb08a7: 418d59ff lea ebx, [r9-01h]
6efb08ab: 4139e9 cmp ecx, ebp
6efb08ae: 4189c1 mov ecx, eax
6efb08b1: 0f4fdd cmovg ebx, ebp
6efb08b4: 39c7 cmp edi, eax
6efb08b6: 0f8c40feffff jl 16efb06fc
6efb08bc: ebc5 jmp 16efb0883
6efb08be: 6690 nop
6efb08c0: 8bb1f0010000 mov esi, [rcx+1f0]
6efb08c6: 448d56ff lea r10d, [rsi-01h]
6efb08ca: 4439ce cmp esi, ecx
6efb08cd: 450f4fd1 cmovg edx, ecx
6efb08d1: e9c3fdffff jmp 16efb0699
6efb08d6: 662e db 2eh
6efb08d8: 0f db 0fh
6efb08d9: 1f db 1fh
6efb08da: 8400 test [rax], al
6efb08dc: 0000 add [rax], al
6efb08de: 0000 add [rax], al

Built on Althena on Sun Oct 27 16:00:02 2013 using compiler version 1400

Windows 6.1 (Windows 7 x64 build 7601) [Service Pack 1]
Memory status: virtual free 8387149M/8388608M, commit limit 8056M, physical total 4029M

RAX = 2737100
RBX = 310
RCX = 7de6530
RDX = 0
RSI = 5094c470
RDI = 47
RBP = 440
R8 = 0
R9 = 64640
R10 = 440
R11 = 3
R12 = 44bd7e
R13 = 64640
R14 = 14
R15 = fff37380
RSP = 75af660
RIP = 6efb086b
EFLAGS = 00010247


Crash reason: Integer Divide-by-Zero

Crash context:
An integer division by zero occurred in module 'libmvtools'.

Pointer dumps:

RAX 02737100: 0008ffff 0009ffff 0007ffff 000affff 000cffff 000affff 000cffff 000e000d
RCX 07de6530: 00000059 00000047 00000010 00000010 000018af 00000002 00000001 00000001
RSI 5094c470: 00000008 00000000 0061a6aa 00000000 00000000 0061e57e 00000000 00000000
RSP 075af660: 07de6530 00000000 00000030 00000000 00000000 00000000 2c97cb0c 00000000
075af680: 00000019 00000000 6efb810b 00000000 00000010 00000000 004b0000 00000000
075af6a0: 00000008 00000000 00000008 00000000 004b0288 00000000 778a3448 00000000
075af6c0: 075af700 00000000 00000008 00000004 075af6e0 00000000 000bf017 000befe5
R12 0044bd7a: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Thread call stack:
6efb086b: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+2eb8b]
6efb810b: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+3642b]
778a3448: ntdll!RtlAllocateHeap [77850000+53360+e8]
778a3448: ntdll!RtlAllocateHeap [77850000+53360+e8]
6efb8ea0: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+371c0]
6ef82a21: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+d41]
6efa77b4: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+25ad4]
6ef87c84: libmvtools!VapourSynthPluginInit [6ef80000+1ce0+5fa4]
7feddf90349: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+5009]
7fede0044d7: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+79197]
7feddf9f19b: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+13e5b]
7feddf9bd83: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+10a43]
7feddf9d4bc: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+1217c]
7fede012769: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+87429]
7fede003e75: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+78b35]
7feddfd7aeb: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+4c7ab]
7feddf9fae8: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+147a8]
7feddfdb5c0: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+50280]
7fede009156: VapourSynth!getVapourSynthAPI [7feddf60000+2b340+7de16]
776459ed: kernel32!BaseThreadInitThunk [77630000+159e0+d]
7787c541: ntdll!RtlUserThreadStart [77850000+2c520+21]

-- End of report


Edit: If you are on the vs/avs IRC i can come here

[foxyshadis]

I spent a while trying to get my head around why
nLambda = nLambda*LSAD/(LSAD + (predictor.sad>>1))*LSAD/(LSAD + (predictor.sad>>1));
is crashing, given that LSAD is 64640, and the only thing I can come up with is that predictor.sad>>1 is so huge that it overflows and negates LSAD. I think it's more likely that memory corruption or an uninitialized value is screwing up the execution, like maybe RecalculateMVs and PseudoEPZSearch didn't find anything, but I can't get it to happen at all. The annoying part of the line is that nLambda is usually 0 here anyway, so it's a meaningless calculation unless truemotion is enabled. (Virtualdub really does need to work on its 64-bit disassembly, r8d and eax aren't even close.)

MonoS, do you have a specific script and input that someone could test on?

[MonoS]

Ok taking a closer look to the simple x264/x265 launcher logs i noticed that with the new version with the tentative fix the plugin crash at frame 1756/1757, instead with the standard version it crashed at frame 981.

I uploaded a portion of the video i'm trying to deinterlace, i'm sorry but is a bit NSFW [i tried with a BlankClip but as i expected, no errors], i included the d2v, remember to change the directory of the m2v into it: https://www.mediafire.com/?cb8h4u3y3u8xds5

The script i used is this one:
src = core.d2v.Source("test.d2v", nocrop=True).fmtc.bitdepth(bits=16)

deint = has.QTGMC(src, Preset="Very Slow", SourceMatch=3, MatchPreset2="Slow", Lossless=2, NoisePreset="Slow", TFF=False, EZDenoise=1.5)#.fmtc.nativetostack16()

deint.set_output()

It's pretty slow [2.5fps on my toaster with an i3 3225] and i didn't tried using faster settings because i want quality [and being this setting quite extreme they should touch about all the sides of QTGMC].

For completion sake i used this settings using simple x264 launcher: VS@64bit, 8bit x264@64bit, preset veryslow, tune film, crf 17, --ref 9 --bframes 9 --b-adapt 2 --aq-mode 3 --sar 16:11 --colormatrix smpte170m --input-range tv --range tv

Trying to reproduce the crash using VSEditor but it didn't give me any problem on those frames, maybe this could help

[jackoneill]

Okay, really fixed this time: http://ulozto.net/xEmVsWye/vapoursynth-mvtools-v7-div0-win64-7z

There were two bugs that could have caused a division by zero at that location:
* the first one was an uninitialised variable.
* the second one, the real cause of this crash, was an overflow due to larger SADs with 16 bit input.

[MonoS]

It went past the troublesome frame, now i'll let the encode goes and hope that it will not crash, thanks a lot jackoneill :D

EDIT: 45000 frames later and the encode is still going, i think we can consider the bug closed

[buchanan]

Hi Jackoneill

I didn't have much time in the last days to do a proper bug report, but will try this latest version to see if it fixes the bug I mentioned previously ;-)

[zerowalker]

Here is SMDegrain with the latest MVTools v7.

http://imgur.com/5mcJoPQ,VCsV6u1,D0dGf4x,RUulFdh

[MonoS]

Ok, found another bug in mvtools.

Using the 16bit codepath, function FlowInter create interesting artifact [i did not encoded this, so i don't know if they are even more interesting in motion :D] http://abload.de/img/yagi.vpy-5448viucx.png

The script used is this one
deint2 = has.QTGMC(p2,Preset="Very Slow", SourceMatch=3, MatchPreset2="Slow", Lossless=2, NoisePreset="Slow", TFF=True).std.SelectEvery(5,[0,1,3,4])

dsup = core.mv.Super(deint2)
bf = core.mv.Analyse(dsup,isb=False, delta=1, overlap=4, truemotion=True)
bb = core.mv.Analyse(dsup,isb=True, delta=1, overlap=4, truemotion=True)
fin2 = core.mv.FlowInter (deint2, dsup, bb, bf, time=50).std.SelectEvery(2,0)

fin2.set_output()

p2 are interleaced credits, i wanted to use cretindesalpes ivtc_txt60mc even if i know is totally useless in this situation [also using 16bit is useless, but who cares].

Using 8bit codepath remove the artifacts

[Are_]

Now that you mention it I saw the same kind of artifacts on a 10bit clip using mv.BlockFPS and I can confirm they dissapear when you convert the video to 8bit prior to mvtools filtering.

EDIT: meh, maybe it's just working as expected, lowering blksize fixes it at 10bit for me.

[MonoS]

Now that you mention it I saw the same kind of artifacts on a 10bit clip using mv.BlockFPS and I can confirm they dissapear when you convert the video to 8bit prior to mvtools filtering.

EDIT: meh, maybe it's just working as expected, lowering blksize fixes it at 10bit for me.

Did the same and seems to work, i'm encoding to see if it's fixed for sure for the whole clip

[pinterf]

I was trying to port a 8mm film restoring script to VS, and (among other things) got stuck at RemoveDirtMC that uses MFlow.
Any plans to port this function?

[feisty2]

Just take mv.compensate instead, pixel compensate (flow) ain't no good to denoising

[jackoneill]

I was trying to port a 8mm film restoring script to VS, and (among other things) got stuck at RemoveDirtMC that uses MFlow.
Any plans to port this function?

Maybe, later? I don't know.

[chainik_svp]

The result finest clip can't be used as normal super clip because they are totally different. Finest filter is automatically invoked by Flow* filters internally when pel>1. It's really not meant to be called by users.

Sadly it's not working as expected in mt environment, at least in original MVTools ;)
trust me cause I know :D

[jackoneill]

Sadly it's not working as expected in mt environment, at least in original MVTools ;)
trust me cause I know :D

What isn't working as expected?

[chainik_svp]

Internal invokation of MFinest.

AFAIR MFinest result was not actually cached...

[zerowalker]

Anyone looked at the images i did on the update?
As far as i can tell the "lightning" problem is still there, as both versions seems to have the same brightness?

[jackoneill]

Anyone looked at the images i did on the update?
As far as i can tell the "lightning" problem is still there, as both versions seems to have the same brightness?

I assume the second image is the original, unfiltered. What exactly are the other three? (For future reference, you can label images with core.text.Text() (http://www.vapoursynth.com/doc/functions/text.html)).

[pinterf]

Just take mv.compensate instead, pixel compensate (flow) ain't no good to denoising

Thank you, tried mv.Compensate in RemoveDirtMC, but it gave blocky artifacts like this (at the armband):
Original / Avisynth MFlow / VapourSynth mv.Compensate

http://i.imgur.com/8RR95Nlt.jpg (http://imgur.com/8RR95Nl)

RemoveDirtMC is used for removing bigger chunks of dirt. Later on mv.Degrain2 is used for noise reduction.

[feisty2]

Overlap=0 is nuts to block based compensate, make it half the value of blksize

[zerowalker]

I assume the second image is the original, unfiltered. What exactly are the other three? (For future reference, you can label images with core.text.Text() (http://www.vapoursynth.com/doc/functions/text.html)).

Oh thought the filenames would show, even confused myself now.

I think the first and fourth are Vapoursynth (first should be v7).
Second is unfiltered, and third is Avisynth.

At least that's how i the files should have been ordered:S

Thanks for telling me else i would have continued to rely on filename which doesn't even show there;P

[pinterf]

Overlap=0 is nuts to block based compensate, make it half the value of blksize
Yes, it's unusual. Tried both 0 and 4 for Overlap with blocksize=8 but in some cases 0 gave better results for specific scenes (fast moving string-like objects did not disappear), could not find out why.

[kaefert]

is there something like SVP or Interframe for Vapoursynth?

I mean some library that allows to calculate extra frames between other frames based on movement analysis?

If I understood it correctly MVTools is only the "basis" for such a tool since it provides motion analysis functions, but it does not contain a function to calculate extra frames based on those analysis, is that correct?

[foxyshadis]

SVPFlow is based on the same idea as MVFlow, which is an extension of MVCompensate from blocks to single pixels. It certainly is possible, it just won't be as high quality as SVP, as they've done a bang-up job of enhancing the quality as much as possible. See a rather old thread (https://forum.doom9.org/showthread.php?t=130332) for an idea of how to get MVFlowFPS working. I'm a little too tipsy to convert to VS right now.

[kaefert]

okey, thanks for the reply foxyshadis!
did I understand you correctly:
Is the recommendation for getting the best possible quality of Motion Interpolation to stick to using Avisynth 32bit + SVP?

[foxyshadis]

okey, thanks for the reply foxyshadis!
did I understand you correctly:
Is the recommendation for getting the best possible quality of Motion Interpolation to stick to using Avisynth 32bit + SVP?

Yes, SVP's gpu-accelerated quality is much nicer than either mvtools or SVP's cpu mode. They've put a lot of professional effort into it.

You can use SVP in VSynth through the avs wrapper. Right now that means limited to 8-bit, but kolak and SEt are pushing them to include stacked 16-bit as well.

[kaefert]

oh. okey. so does that mean that if I'm using a a Server that has a nice octacore Xeon CPU, but virtually no GPU "Matrox G200e (Server Engines)" that I have no chance to get this quality?
I guess I would need nvidia or ati to be able to use gpu-accelerated version, correct?

UPDATE: okey so I've found this compatibility list for which GPUs work:
http://www.svp-team.com/wiki/GPU_Compatibility
But I could not yet find an answer to the question if I can't simply generate the "same" quality of motion interpolated in-between frames by using SVP with the gpu=0 parameter set. (slower of course, but since I want to encode the result instead of displaying it live (like the SVP-team seems to intend) slower is annoying but not a deal breaker for me)

[jackoneill]

Here is v8 (https://github.com/dubhater/vapoursynth-mvtools/releases/tag/v8).


* Fix occasional division by zero in Analyse with 16 bit input


Not much new in this one.

[feisty2]

uh... really don't wanna be a dick here, but gotta ask about "vmulti, mdegrainN" stuff again, those are planned to be implemented someday or like, never gonna happen?
I'm like, ditching avisynth and moving everything to vaporsynth and mvtools, it's the only major problem I got now
plz, gimme an answer, functions added by Firesledge, will they ever appear in vsmvtools?

[jackoneill]

uh... really don't wanna be a dick here, but gotta ask about "vmulti, mdegrainN" stuff again, those are planned to be implemented someday or like, never gonna happen?
I'm like, ditching avisynth and moving everything to vaporsynth and mvtools, it's the only major problem I got now
plz, gimme an answer, functions added by Firesledge, will they ever appear in vsmvtools?

At the moment, I have no plans to touch MVTools except to fix bugs.

[captainadamo]

Results for Degrain test:

8 threads:
VapourSynth Windows = 32.35 fps (100% cpu)
VapourSynth Linux = 37.50 fps (100% cpu)
AviSynth firesledge = 8.32 fps (35% cpu)
1 thread:
VapourSynth Windows = 5.32 fps (12% cpu)
VapourSynth Linux = 5.50 fps (12% cpu)
AviSynth Vanilla = 4.42 fps (12% cpu)
AviSynth SVP = 6.05 fps (12% cpu)
VapourSynth script:
import vapoursynth as vs
core = vs.get_core() # threads=1
v = core.lsmas.LWLibavSource(r'720x480 YUV420P8 mpeg2.mkv')
super = core.mv.Super(src)
mvbw3 = core.mv.Analyse(super, isb=True, delta=3, overlap=4)
mvbw2 = core.mv.Analyse(super, isb=True, delta=2, overlap=4)
mvbw = core.mv.Analyse(super, isb=True, delta=1, overlap=4)
mvfw = core.mv.Analyse(super, isb=False, delta=1, overlap=4)
mvfw2 = core.mv.Analyse(super, isb=False, delta=2, overlap=4)
mvfw3 = core.mv.Analyse(super, isb=False, delta=3, overlap=4)
v = core.mv.Degrain3(clip=src, super=super, mvbw=mvbw, mvfw=mvfw, mvbw2=mvbw2, mvfw2=mvfw2, mvbw3=mvbw3, mvfw3=mvfw3)
v.set_output()
AviSynth script:
LWLibavVideoSource("720x480 YUV420P8 mpeg2.mkv")
super = MSuper(last)
mvbw3 = MAnalyse(super, isb=True, delta=3, overlap=4)
mvbw2 = MAnalyse(super, isb=True, delta=2, overlap=4)
mvbw = MAnalyse(super, isb=True, delta=1, overlap=4)
mvfw = MAnalyse(super, isb=False, delta=1, overlap=4)
mvfw2 = MAnalyse(super, isb=False, delta=2, overlap=4)
mvfw3 = MAnalyse(super, isb=False, delta=3, overlap=4)
MDeGrain3(last, super, mvbw, mvfw, mvbw2, mvfw2, mvbw3, mvfw3)

This was done more for fun, but with this test using v8 of MVTools and replacing only the 4x4 and 8x8 Sad_C with NEON intrinsics, I was able to get around 10.5 fps from a 720x480 raw source on an iPad Air 2 with 3 threads. Nothing blazing, but not bad for being on a tablet.

[jackoneill]

This was done more for fun, but with this test using v8 of MVTools and replacing only the 4x4 and 8x8 Sad_C with NEON intrinsics, I was able to get around 10.5 fps from a 720x480 raw source on an iPad Air 2 with 3 threads. Nothing blazing, but not bad for being on a tablet.

So 10.5 fps after. How was it before?