MVTools, Depan, DepanEstimate for VapourSynth

[captainadamo]

Around 6.5fps. The NEON for the 4x4 SAD only improved things consistently up to about 6.8 fps. The big gain was the 8x8 SAD in NEON which gave it the bump to 10.5 fps.

Just to note this was only when optimizing based off of profiling that single test script. And even with the NEON, those two SAD functions still basically swamp everything else in the performance counters so I don't know how much more I can squeeze out.

[feisty2]

mv.compensate seems like, well.. busted

import vapoursynth as vs
core = vs.get_core()
clp = core.raws.Source("Y.rgb", 736, 960, src_fmt="Y8")
clp = core.fmtc.stack16tonative(clp)
super = core.mv.Super (clp)
bv = core.mv.Analyse (super, isb=True,delta=1)
bcmp = core.mv.Compensate (clp, super, bv, thsad=400, thscd1=248, thscd2=130)
bcmp.set_output ()

bv.set_output () works okay
bcmp.set_output () = instant crash

[jackoneill]

mv.compensate seems like, well.. busted

import vapoursynth as vs
core = vs.get_core()
clp = core.raws.Source("Y.rgb", 736, 960, src_fmt="Y8")
clp = core.fmtc.stack16tonative(clp)
super = core.mv.Super (clp)
bv = core.mv.Analyse (super, isb=True,delta=1)
bcmp = core.mv.Compensate (clp, super, bv, thsad=400, thscd1=248, thscd2=130)
bcmp.set_output ()

bv.set_output () works okay
bcmp.set_output () = instant crash

Fixed. Thanks for the report.

Here is v9 (https://github.com/dubhater/vapoursynth-mvtools/releases/tag/v9).

* Fix crash in Compensate with Gray input

[feisty2]

any way to encode the vector clip to an actual vid and load it later, like whenever I need it?
tried to encode it directly and vspipe moaned about "cannot output clips with varying dimensions"
I need vectors encoded so I can do motion estimation just once and use it to motion compensate several times

[MonoS]

any way to encode the vector clip to an actual vid and load it later, like whenever I need it?
tried to encode it directly and vspipe moaned about "cannot output clips with varying dimensions"
I need vectors encoded so I can do motion estimation just once and use it to motion compensate several times

I'm also interested in this, and thus got interested as you asked.

Save the image data using imwri is impossible, the plugin create image as you visualize them, and due to impossibility to use vspipe to request all frames this is a manual process, also save 100 frames in png require about 30MB.

The ye olde mvtools for avs had an interesting outfile parameters, but seems lacking an infile parameter, thus making it useless, also it was removed.

Then i took a look at the code and tried to see how the analyse function saved the image.
It did so creating a frame 1xnVector and then writing/reading directly into it.
Replacing this frame for a simple array and then writing this array to a file in a binary format [and maybe applying some compression if needed].

If you don't want to modify mvtools code [for reason i can understand], creating an extra plugin [i dunno if it can be done also as a script] that read the image and binarize that into a file and vice versa, it should be a trivial task i think.

if you are not capable [iirc you said that you are not so good with c] and jackoneill don't want to make this i can spare some time.

EDIT1: I should go to splice cause almost 1AM but i got really interested in this problem.
I sketched something in my head and i think i only need how to access sequentially all the frames of a video without requesting it directly [so not something like imwri, but like lsmash] and maybe have also a progress bar of some sort [you know, you don't want to stare at a spinning circle for hours or more]
Probably something like getFrameAsync() that call a writing function with fmSerial threading mode.

EDIT1.1: Reading vspipe source i thought about modifying it to create a new output mode that do exactly that, and then create a plugin to read that binary file and recreate the vec data and i also have some idea on how to make this.
Now sleep!

[feisty2]

awesome! well, I failed to figure out a way to do it by scripting and, I suck at C/C++, so counting on u to save the world bro!

[jackoneill]

You can do this in Python:


import vapoursynth as vs

c = vs.get_core()

src = c.ffms2.Source("asdf.mp4")

superclip = c.mv.Super(src)
analysis = c.mv.Analyse(superclip, delta=1, isb=True)

f = open("/tmp/vectors.bin", "wb")

for i in range(analysis.num_frames):
if (i+1) % 100 == 0:
print(i+1, "/", analysis.num_frames)

frame = analysis.get_frame(i)

view = frame.get_read_array(0)

for y in range(len(view)):
# there is only one row (this loop runs once)
f.write(view[y])

f.close()

It's about 82 KiB/frame.

[MonoS]

and for reading?? if all the frames have different width there should be a frameLen in the file, also i don't see any function for creating varying dimension clip for then writing to it

[jackoneill]

and for reading?? if all the frames have different width there should be a frameLen in the file, also i don't see any function for creating varying dimension clip for then writing to it

I don't think the width varies in a clip. The output of Analyse has unknown dimensions simply because they are not known until a frame is returned, not because they could change from one frame to the next. Anyway, I'll leave all that as exercises for the reader.

[MonoS]

Then it should be a trivial task to do via scripting.
i'll try to.implemebt as soon as i return home.

@feisty2: i return home at about 15 utc, if you expect to write it yourself let me know so we don't write it twice

[Myrsloik]

I approve of this insanity. My idea for solving the reading (not tested but works in theory):


import vapoursynth as vs

c = vs.get_core()

src = c.ffms2.Source("rule6.avi")

superclip = c.mv.Super(src)
analysis = c.mv.Analyse(superclip, delta=1, isb=True)

vf = open("/stored/vectors.bin")
vector_width = analysis.get_frame(0).width
vector_template_clip = c.std.BlankClip(analysis, height=1, width=vector_width)

def vector_read(n, f):
vf.seek(n * f.width)
vector_data = vf.read(f.width)
fout = f.copy()
view = fout.get_write_array(0)
for x in range(fout.width):
view[0,x] = vector_data[x]
return fout

vector_clip = c.std.ModifyFrame(clip=vector_template_clip, clips=vector_template_clip, selector=vector_read)
vector_clip.set_output()

[feisty2]

@MonoS
gonna try to do it myself, but will let u know if I failed :)

[feisty2]

http://i.imgur.com/HFdBLFx.png
the writing part worked, the reading part popped out an error when I tried to preview it, soon after the error, vaporsynth crashed

[Myrsloik]

line 16 should be:
view = fout.get_write_array(0)

[feisty2]

still error, now it's
"Error getting the frame number 0:
Object is not writable."

[MonoS]

I humbly suggest to add an extra parameter to vector_read for selecting which file to read from.
Something like
vecFile = "/tmp/vectors.bin"
def vector_read(n, f, file):
vf= open("file", "rb")
vf.seek(n * f.width)
# etc...

return fout

vector_clip = core.std.ModifyFrame(clip=vector_template_clip, clips=vector_template_clip, selector=functools.partial(vector_read, file=vecFile)

[feisty2]

yeah, uh, organizing kinda stuff like this can wait I guess, right now, I can't get it to work, I got "Object is not writable" error

[Myrsloik]

I updated my previous post with a fixed script.

It's basically how you write a raw source filter using only python. Actually you probably could replace most of this with chikuzen's raw source...

[feisty2]

so I decide to take the easiest way

simply this works

vw = vec.get_frame(0).width
vecr=core.raws.Source(r"D:\test\vectors.bin", vw, 1, src_fmt="Y8")
vecr.set_output()


edit: replaced "vw" with a log file, no need to create a template anymore now
writing part
vw = vec.get_frame(0).width
log = open(r"D:\test\log.txt", "w").write (repr (vw))
reading part
w=int (open(r"D:\test\log.txt", "r").read ())
vecr=core.raws.Source(r"D:\test\vectors.bin", w, 1, src_fmt="Y8")
vecr.set_output()

[MonoS]

Every time i create the vector file with the same parameters and clip i get a different md5, is this to be expected??

[feisty2]

Every time i create the vector file with the same parameters and clip i get a different md5, is this to be expected??

guess ur source filter is the one to blame here, not mvtools, I remember Myrsloik said something like "Don't expect FFMPEG to be bit exact" or whatever similar.

[Myrsloik]

Every time i create the vector file with the same parameters and clip i get a different md5, is this to be expected??

The answer is mostly not but with a bit of yes. Some formats can have +-1 differences due to rounding I think. Try it with something lossless like utvideo or huffyuv as input and see if it's still different.

[jackoneill]

Every time i create the vector file with the same parameters and clip i get a different md5, is this to be expected??

Compare the contents.

Edit: Nevermind. I think it's just a little unused memory that never gets initialised.

[feisty2]

Do "thsad" and "thscd1/2" have a varying range related to bit depth like "limit" in mv.degrain or they stay in some fixed certain range (always "400" as default whatever input depth is) ?

[jackoneill]

Do "thsad" and "thscd1/2" have a varying range related to bit depth like "limit" in mv.degrain or they stay in some fixed certain range (always "400" as default whatever input depth is) ?

The ranges for thsad and thscdX do not vary with the bit depth. They are scaled internally as needed.

[foxyshadis]

The ranges for thsad and thscdX do not vary with the bit depth. They are scaled internally as needed.

I'm convinced this is the proper mode of operations for all filters, which should take fractional values instead of requiring the user to know what bit depth they're in anytime a filter is called. I'm glad havs does the heavy lifting for the filters that don't scale input. (Although it's going to have to be done all over again for anything that wants 0...1.0 in float. rean has the right idea there.)

[feisty2]

any plan to get mvtools to work on float point clips?

[jackoneill]

any plan to get mvtools to work on float point clips?

No.    

[feisty2]

No.    

okay then, how do I make all this work, like how, should I modify the cpp code to add support for float point clips? a short and simple guide like that "python removegrain19", can I ask for one plz?
I been running from C/CPP like, forever, cuz they are obscure and simply look like machine code to me, and I've always favored stuff easier like python or that "home grown avisynth language"
but that's officially over now, guess it's just, well, time to face the devil finally.

[jackoneill]

okay then, how do I make all this work, like how, should I modify the cpp code to add support for float point clips? a short and simple guide like that "python removegrain19", can I ask for one plz?
I been running from C/CPP like, forever, cuz they are obscure and simply look like machine code to me, and I've always favored stuff easier like python or that "home grown avisynth language"
but that's officially over now, guess it's just, well, time to face the devil finally.

Convert your float clips to integer.

[feisty2]

Convert your float clips to integer.

not doing that unless we got int32 or things even more delicate
and int16 is the most precise int type we got in vaporsynth for now, so, no.

[Bloax]

What kind of sick and twisted things are you doing that require 4294967296 states of precision over 65536 states of precision on monitors that have all of 256 states of precision?

[feisty2]

What kind of sick and twisted things are you doing that require 4294967296 states of precision over 65536 states of precision on monitors that have all of 256 states of precision?

color grading, int16 fails in some extreme cases, gotta take float32/int32 to prevent that

[Bloax]

MVTools is used for color grading? o_O

[feisty2]

MVTools is used for color grading? o_O

no, but once you convert your clip to a lower precision, for whatever purpose, precision will be lost FOR GOOD!

[Bloax]

Well then clearly do the MVTools stuff first in int16 and then convert it back to a more precise format afterwards.
This might be a bit awkward, but then again it's not like we even had more than 8 bits to fuck around with 3-4 years ago.

[feisty2]

Well then clearly do the MVTools stuff first in int16 and then convert it back to a more precise format afterwards.
This might be a bit awkward, but then again it's not like we even had more than 8 bits to fuck around with 3-4 years ago.

not tested but I'm actually worried about precision lost in the "float32-int16-float32" process might lead to some unexpected disasters.

[Bloax]

"Do it first" implies doing input->int16->mvtools->float32->etc, unless that can't be done either for arcane reasons.

[feisty2]

maybe not related to mvtools, but the following shows errors introduced in the "float32-int16-float32" process will actually affect the final result even if it's just 8bits

import vapoursynth as vs
core = vs.get_core()
clp = whatever
clp = core.std.ShufflePlanes(clp, planes=0, colorfamily=vs.GRAY)
clp = core.fmtc.bitdepth(clp, fulls=False, fulld=True, bits=32, flt=True, dmode=1)
clp = core.fmtc.transfer(clp, transs="470bg", transd="linear", fulls=True, fulld=True)
clp1 = core.fmtc.bitdepth(clp, fulls=True, fulld=True, bits=16, flt=False, dmode=1).fmtc.transfer(transs="linear", transd="srgb", fulls=True, fulld=True).fmtc.bitdepth(fulls=True, fulld=True, bits=8, flt=False, dmode=3)
clp2 = core.fmtc.transfer(clp, transs="linear", transd="srgb", fulls=True, fulld=True).fmtc.bitdepth(fulls=True, fulld=True, bits=8, flt=False, dmode=3)
dif = core.std.MakeDiff(clp1,clp2).std.Expr ("x 128 - 100 * 128 +")
dif.set_output ()

[Nevilne]

Not sure if you understand just how much precision 16bit int has, but you can just add [diff of float32->int16 and float32->int16->mvtools] to [float32], to keep non-mvtooled float32 pixels.

[feisty2]

Not sure if you understand just how much precision 16bit int has, but you can just add [diff of float32->int16 and float32->int16->mvtools] to [float32], to keep non-mvtooled float32 pixels.

I'm reading CPP tutorials actually, gonna face this son of a gun sooner or later, so why not now :)
edit: new colorspaces could be defined by filters according to vs doc, so I'm thinking about double or int64 if I'm gonna write filters in cpp

[Myrsloik]

I'm reading CPP tutorials actually, gonna face this son of a gun sooner or later, so why not now :)
edit: new colorspaces could be defined by filters according to vs doc, so I'm thinking about double or int64 if I'm gonna write filters in cpp

Not supported because both are pointless. If floats aren't accurate enough for you then you're much more likely to be using an algorithm not suitable for the task than actually miss the few extra bits a double gives you.

[feisty2]

Not supported because both are pointless. If floats aren't accurate enough for you then you're much more likely to be using an algorithm not suitable for the task than actually miss the few extra bits a double gives you.

okay then, no double or long long, it's just, I was screwed over by rounding errors some time before, and got allergic to it ever since

[foxyshadis]

Not sure if you understand just how much precision 16bit int has, but you can just add [diff of float32->int16 and float32->int16->mvtools] to [float32], to keep non-mvtooled float32 pixels.

Tacking old LSB on top of filtered pixels is roughly equivalent to AddGrain(). Don't do that unless you just need a little randomness, in which case AddGrain will do you better.

Floating point is about working with WTW and BTB, not precision, and you'll generally want to grade it before you filter it so that you don't run into anomalies in areas you can't see, and don't accidentally filter away your important details. If 12- to 16-bit is enough for Camera Raw, 16-bit is enough for video grading.

[cretindesalpes]

maybe not related to mvtools, but the following shows errors introduced in the "float32-int16-float32" process will actually affect the final result even if it's just 8bits
Of course. The “errors” you see are differences between dithering patterns. Clips with extremely small differences will always exhibit these artefacts once dithered, especially with error-diffusion algorithms (*). The only way to accurately check errors is to upconvert the clip of the lowest bitdepth then subtract them and amplify the difference.

(*) For example in 1-D and 1 bit, a value of 0.500 will dither as 10101010… and a value of 0.501 will dither as 11010101… resulting in a 0+1-1+1-1+1-1+1… difference.

[feisty2]

upconvert? as bitdepth upconvert or resolution upconvert?

import vapoursynth as vs
core = vs.get_core()
clp = whatever
clp = core.std.ShufflePlanes(clp, planes=0, colorfamily=vs.GRAY)
clp = core.fmtc.bitdepth(clp, fulls=False, fulld=True, bits=32, flt=True, dmode=1)
clp = core.fmtc.transfer(clp, transs="470bg", transd="linear", fulls=True, fulld=True)
clp1 = core.fmtc.bitdepth(clp, fulls=True, fulld=True, bits=16, flt=False, dmode=1).fmtc.transfer(transs="linear", transd="srgb", fulls=True, fulld=True).fmtc.bitdepth(fulls=True, fulld=True, bits=8, flt=False, dmode=3).fmtc.resample (clp.width*2,clp.height*2,fulls=True, fulld=True)
clp2 = core.fmtc.transfer(clp, transs="linear", transd="srgb", fulls=True, fulld=True).fmtc.bitdepth(fulls=True, fulld=True, bits=8, flt=False, dmode=3).fmtc.resample (clp.width*2,clp.height*2,fulls=True, fulld=True)
dif = core.std.MakeDiff(clp1,clp2).std.Expr ("x 32768 - 100 * 32768 +")
dif.set_output ()

import vapoursynth as vs
core = vs.get_core()
clp = whatever
clp = core.std.ShufflePlanes(clp, planes=0, colorfamily=vs.GRAY)
clp = core.fmtc.bitdepth(clp, fulls=False, fulld=True, bits=32, flt=True, dmode=1)
clp = core.fmtc.transfer(clp, transs="470bg", transd="linear", fulls=True, fulld=True)
clp1 = core.fmtc.bitdepth(clp, fulls=True, fulld=True, bits=16, flt=False, dmode=1).fmtc.transfer(transs="linear", transd="srgb", fulls=True, fulld=True).fmtc.bitdepth(fulls=True, fulld=True, bits=8, flt=False, dmode=3).fmtc.bitdepth(fulls=True, fulld=True, bits=32, flt=True, dmode=1)
clp2 = core.fmtc.transfer(clp, transs="linear", transd="srgb", fulls=True, fulld=True).fmtc.bitdepth(fulls=True, fulld=True, bits=8, flt=False, dmode=3).fmtc.bitdepth(fulls=True, fulld=True, bits=32, flt=True, dmode=1)
dif = core.std.MakeDiff(clp1,clp2).std.Expr ("x 0.5 - 100 * 0.5 +")
dif.set_output ()

anyways I tried both and differences are still there

[cretindesalpes]

bitdepth upconvert. If you’re processing something in float, convert everything to float to compare.

clp = core.std.ShufflePlanes(clp, planes=0, colorfamily=vs.GRAY)
clp = core.fmtc.bitdepth(clp, fulls=False, fulld=True, bits=32, flt=True)
clp = core.fmtc.transfer(clp, transs="470bg", transd="linear", fulls=True, fulld=True)

clp1 = core.fmtc.bitdepth(clp, fulls=True, fulld=True, bits=16, flt=False, dmode=1)
clp1 = core.fmtc.transfer(clp1, transs="linear", transd="srgb", fulls=True, fulld=True)
clp1 = core.fmtc.bitdepth(clp1, fulls=True, fulld=True, bits=32, flt=True)

clp2 = core.fmtc.transfer(clp, transs="linear", transd="srgb", fulls=True, fulld=True)

dif = core.std.MakeDiff(clp1,clp2).std.Expr ("x 0.5 - 100 * 0.5 +")
dif = core.fmtc.bitdepth(dif, fulls=True, fulld=True, bits=8, flt=False)
dif.set_output ()

And the difference will be even tinier if you use dmode 3 instead of dmode 1.

[Reel.Deel]

@feisty2
Not sure what's the obsession with 32-bit float but didn't you say your sources are usually 8-bit heavily compressed Katy Perry (or other pop) music videos? I highly doubt that processing in 32-bit float will magically make the output substantially better. I doubt the highly minute differences will be visible when playing in real-time. Also, do you have an appropriate 10-bit+ panel and workflow? If not then this discussion is moot.

I process my Canon DSLR raw photos in Lightroom (32-bit), output to 16-bit to do post processing in an external HDR software, I export to16-bit again and import into Lightroom for final grading. It would be nice to have a complete 32-bit float workflow but my current workflow is more than adequate to achieve very good results. I don't think 32-bit float will improve the results marginally.

*sorry for any grammar errors, not very efficient on my phone.

[feisty2]

1.Got that obsession from 3dsmax, I was playing with exposures of the objects and noticed 16bits actually produced some bandings when switching the exposure from an extreme to the opposite extreme, and float32 works fine without bandings
2.music videos, well, first, I like music videos, but more importantly, they are short so easy to test filters and demonstrate and they are lossily compressed so kinda like gets you easier to judge how filters work in extreme cases, but I do process other vids, mostly things I shot for fun
3.I got a color corrected monitor but it's not 10bpc, but like I said in 1., you don't really need 10bpc screen when you can already tell that shit happens and visible even at 8bpc screen
4.since we don't have infinite precision here to do all the intermediate steps, rounding errors will accumulate and might cause some serious shit, so gotta pick the data type closest to "infinite precision"
5.I think float32 would be better for linear light

[feisty2]

holy lord, I'm staring at the "mvtools" source code and it's gonna be... painful to add native float point support to it, practically gotta rewrite the whole thing, so I'm thinking of an alternative, we keep int stuff here, but pick uint32 instead of uint16, and vs got no support for Gray32, so convert uint32 to single at the final step, guess gotta remove some asm here cuz they won't work on uint32