Problem with a hand crafted script

[MonoS]

Just some hours ago i updated my denoise function to allow the recalculate parameter.

The script seems to works just fine, but if i use it with the vectors creation of my script the entire script crash.

The script used to reproduce the issues is this one

import vapoursynth as vs
import MFunc as mon

core = vs.get_core()

file = "c:/Script/address.vpy"

src = core.std.BlankClip(width=32, height=32, format=vs.YUV420P8, length=10, fpsnum=25)

den = mon.Denoise2(src, 200, prefix = file, recalculate=8, fast=True)

den.set_output()


For creating the vector file, simply use a cmd like that

vspipe "c:/Script/address.vpy" "c:/Script/address.vpy.vec" -p


I'm quite sure is not a bug neither of the recalculate part nor in the vector creation part because
1)without 'create' the script works fine
2)an encoded version of a real film with and without vectors, but without recalculate, gave me the SAME file [the same MD5]

Debugging a bit the script [my way to debug a python script is putting line like a = x/0 where x is the instance of the breakpoint all over the code in a bisect way] it seems to crash when reaching the final iteration inside the
while blksize > recalculate:
loop, before the first call to Recalculate

This would suggest some problem when reaching the blksize=8 iteration, but lowering the recalculate parameter don't solve the problem

You can find MFunc here https://github.com/MonoS/MonoS-VS-Func/blob/master/MFunc.py

Thanks for the attention

[jackoneill]

It sounds like a bug in Recalculate. What are overlap and blksize when it crashes?

[MonoS]

It crash with the pairs: 8,2;8,4;16,4;16,8

Changing the code from a loop to

if blksize > recalculate:
blksize = int(blksize / 2)
if fast:
overlap = int(overlap / 4)
else:
overlap = int(overlap / 2)

bvec1 = core.mv.Recalculate(superRep, bvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
bvec2 = core.mv.Recalculate(superRep, bvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
fvec1 = core.mv.Recalculate(superRep, fvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
fvec2 = core.mv.Recalculate(superRep, fvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)

if blksize > recalculate:
blksize = int(blksize / 2)
if fast:
overlap = int(overlap / 4)
else:
overlap = int(overlap / 2)

bvec1 = core.mv.Recalculate(superRep, bvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
bvec2 = core.mv.Recalculate(superRep, bvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
fvec1 = core.mv.Recalculate(superRep, fvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
fvec2 = core.mv.Recalculate(superRep, fvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)

if blksize > recalculate:
blksize = int(blksize / 2)
if fast:
overlap = int(overlap / 4)
else:
overlap = int(overlap / 2)

bvec1 = core.mv.Recalculate(superRep, bvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
bvec2 = core.mv.Recalculate(superRep, bvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
fvec1 = core.mv.Recalculate(superRep, fvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
fvec2 = core.mv.Recalculate(superRep, fvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)


The code seems to crash right before the first recalculate in the second if statement with blksize and overlap of 16,8 or 18,4

[jackoneill]

Does it crash if you use Analyse to create the initial vector clip?

Also, remove all lines that aren't necessary to reproduce the crash. (This goes for any bug you encounter, in any software.)

[feisty2]

Recalculate works without any problem here, I used (blksize/overlap) 16/8 8/4 4/2 in Placebo.py and everything is just nice

[MonoS]

Following your suggestion i reworked mon.Denoise2 to the bare minimum

import os.path

def Denoise2(src, denoise=400, blur=None, lsb=True, truemotion=True, chroma=True, fast=False, blksize=None, prefix=None, recalculate=None, thSAD=None):
core = vs.get_core()

src16 = mon.Up16(src, lsb)

if fast:
if blksize is None:
blksize = 32

overlap = int(blksize/4)
else:
if blksize is None:
blksize = 8

overlap = int(blksize/2)

if recalculate is None:
recalculate = blksize

if thSAD is None:
thSAD = int(denoise * 1.25)

super = core.mv.Super(src16, chroma=chroma)

superRep = super

bvec1 = mon.ReadVecs(0, prefix, 4)
bvec2 = mon.ReadVecs(1, prefix, 4)
fvec1 = mon.ReadVecs(2, prefix, 4)
fvec2 = mon.ReadVecs(3, prefix, 4)

while blksize > recalculate:
blksize = int(blksize / 2)
if fast:
overlap = int(overlap / 4)
else:
overlap = int(overlap / 2)

bvec1 = core.mv.Recalculate(superRep, bvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
bvec2 = core.mv.Recalculate(superRep, bvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
fvec1 = core.mv.Recalculate(superRep, fvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
fvec2 = core.mv.Recalculate(superRep, fvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)

fin = core.mv.Degrain2(src16, super, bvec1,fvec1,bvec2,fvec2, denoise, plane = 4 if chroma else 0)

return fin


The crash still occurs

I didn't understand what you mean by "Does it crash if you use Analyse to create the initial vector clip?"
If i understood correctly, without using the vector file no it don't crash, it only crash in conjunction between vector file and recalculate

Also i first opened the script into VirtualDub, because usually it's crash reports are "very useful" and the crash originated from VD not from any of the script

[jackoneill]

If i understood correctly, without using the vector file no it don't crash, it only crash in conjunction between vector file and recalculate

Then maybe don't do that. (Sure, Recalculate should fail more gracefully when you give it a vector clip it doesn't like.)

[MonoS]

Then maybe don't do that. (Sure, Recalculate should fail more gracefully when you give it a vector clip it doesn't like.)

the problem is exactly that, the vector file is too much useful [saved me days of time encoding]

Also, why recalculate should fail if the vector file is an exact copy of the vector produced by analysis [as i said doing an encode in x264 with and without vector file gave me the same file] and why VD don't report the crash in recalculate but in the vapoursynth dll??

It's not much of a problem not using it but i'd like to understand what's happening :(

[feisty2]

solution: write the "recalculated vectors" as an external file, that's what I been doing

[feisty2]

an exact copy of the vector produced by analysis

not the "exact" copy, it comes with additional resolution tags, the exact copy should have "unknown dimensions", maybe remove those 2 tags and try again

[MonoS]

solution: write the "recalculated vectors" as an external file, that's what I been doing

This is not feasible because the dimension of the vector file will become too big [talking about >100GB file] and i can't spare that much space :(

not the "exact" copy, it comes with additional resolution tags, the exact copy should have "unknown dimensions", maybe remove those 2 tags and try again

YEAH, you're right O_O, i'll try that

[MonoS]

@feisty: Yes, you were right, making the vector as fixed dimensions recreated the problem
Now i should find a way to remove the dimensions tags

[feisty2]

now I get what's all that "varying dimensions" stuff on vector frames for,
it's for "recalculate"
the width of vector frames grows as you recalculate them, cuz smaller blocks = more blocks = more vectors, obviously
you don't have to care about the growing width at all if the width is already "variable"

as for dimension tag removal, guess Myrsloik wrote something about stripping frame property on the ModifyFrame page

[jackoneill]

Ah, hmm. VapourSynth filters aren't allowed to return frames with dimensions that don't match those declared in the video info. (Recalculate should reject vector clips that have known dimensions.)

I don't think you can create clips with unknown dimensions using the Python module. It's not a matter of removing frame properties. The information is stored in the VSVideoInfo struct associated with the video node.

[Myrsloik]

Ah, hmm. VapourSynth filters aren't allowed to return frames with dimensions that don't match those declared in the video info. (Recalculate should reject vector clips that have known dimensions.)

I don't think you can create clips with unknown dimensions using the Python module. It's not a matter of removing frame properties. The information is stored in the VSVideoInfo struct associated with the video node.

If you want an unknown dimension clip you can do something like this:

Splice([BlankClip(width=200), BlankClip(width=100)], mismatch=True)

Not the most obvious way but there's rarely a good reason to do this...

[MonoS]

If you want an unknown dimension clip you can do something like this:

Splice([BlankClip(width=200), BlankClip(width=100)], mismatch=True)

Not the most obvious way but there's rarely a good reason to do this...

Thanks Myrsloik, this seems to solve the problem :D


if blksize > recalculate:
bvec1 = core.std.Splice([core.std.BlankClip(bvec1, width=1, length=1), bvec1], mismatch=True).std.Trim(1)
bvec2 = core.std.Splice([core.std.BlankClip(bvec2, width=1, length=1), bvec2], mismatch=True).std.Trim(1)
fvec1 = core.std.Splice([core.std.BlankClip(fvec1, width=1, length=1), fvec1], mismatch=True).std.Trim(1)
fvec2 = core.std.Splice([core.std.BlankClip(fvec2, width=1, length=1), fvec2], mismatch=True).std.Trim(1)


Now i'll try to make some real encode and see if i get the same result with and without the vectors file