Vapoursynth

[Pat357]

Wanted to compare some mvtools versions. Same script as I posted before, FFMS2 1 thread, through vfw, VS r11:
How did you limit FFMS2 to 1 thread ?
I've tried this :
core.std.LoadPlugin(path=r"k:\programs\ffms2-r725-icl\ffms2.dll", threads=1)
but I got the (atm not yet, but will soon be) famous 3 colorscreen.

[lansing]

How did you limit FFMS2 to 1 thread ?


you do it in the core


core = vs.Core(threads=1)

[Myrsloik]

How did you limit FFMS2 to 1 thread ?
I've tried this :
core.std.LoadPlugin(path=r"k:\programs\ffms2-r725-icl\ffms2.dll", threads=1)
but I got the (atm not yet, but will soon be) famous 3 colorscreen.

Hint:
core.ffms2.Source(threads=1)

[Pat357]

for the dither one, you need to set mt=false on all manalyse, msuper, and mdegrain in order to work in vpy
It's not MVtools v2.6.05 that is the problem for vapoursynth, but the included "avstp.dll" is the one that causes problems.
Just rename this small "avptp.dll" file and you can use mvtools without problems. This way mvtools will fall back to single thread.
The "avstp.dll" makes that some filters ((re)written for this) capable of using multi-threading on their own (without using a MT version of Avisynth).
This makes it possible to have MT on standard Avisynth.

Info : avstp thread (http://forum.doom9.org/showthread.php?t=164407)

[Zarxrax]

Theoretical question here.

Over the years, several people tried and failed at creating a graphical Non Linear Editing system using AviSynth as a backend.
Might vapoursynth be better suited for this, and possibly hold some promise in that regard?

[Myrsloik]

Theoretical question here.

Over the years, several people tried and failed at creating a graphical Non Linear Editing system using AviSynth as a backend.
Might vapoursynth be better suited for this, and possibly hold some promise in that regard?

It was already quite possible with avisynth, the problem being that all those projects stayed as vaporware. Unless a skilled, sane person steps up (a rare happening indeed) I predict the same outcome as for avisynth.

So yes, very possible.

(Fun fact the, the toilet paper logo of ffms2 is actually a shameless insult of celluloid (http://forum.doom9.org/showthread.php?t=127187). Celluloid/cellulose, what's the difference?)

[lansing]

(Fun fact the, the toilet paper logo of ffms2 is actually a shameless insult of celluloid (http://forum.doom9.org/showthread.php?t=127187). Celluloid/cellulose, what's the difference?)

thanks for the reading, now i see where the word "vapor" is coming from. :)

http://forum.doom9.org/showthread.php?t=127187&page=3


And can you explain a bit more on how frame leveling multithreading works and how it compares to the current avisynth mt by SEt?

[Chikuzen]

I wrote an avisynth plugin to preview P216/P210/P016/P010 .vpy.

http://forum.doom9.org/showthread.php?t=166038

[kolak]

There is also real v210 preview from vpy via fake AVI (aka avfs) coming :)

update: v210 is shaping very well :)

[Myrsloik]

R12 is here. Changelog in the first page as usual.

The usual bloggy post here (http://www.vapoursynth.com/2012/10/r12-vapoursynth-takes-a-step-in-the-enterprise-direction/).

It would be nice if you redid all benchmarks with this version. It should be a bit faster than the previous ones where it couldn't reach 100% CPU usage.

[kolak]

Fake v210 AVI works great- most (if not all apps) read it fine. Fake AVI can be also loaded to Resolve through ref MOV file (way of passing AVC-I to Resolve if ffms2 can decode it).

[lansing]

benchmarks for r12.

mdegrain2 and mdegrain3 all have significant speed up comparing to r11, cpu usage reaching close to 99%. Mdegrain1 stayed the same.

[Myrsloik]

benchmarks for r12.

mdegrain2 and mdegrain3 all have significant speed up comparing to r11, cpu usage reaching close to 99%. Mdegrain1 stayed the same.

That's because mdegrain1 only has 3 filters and one source in total. Since only one frame can be processed by one filter at a time you become bottlenecked by the slowest filter.
If mdegrain1 speed is what you want at any cost then some tricks with splitting even and odd frame processing to different filter instances can be done with a clever script.

[Pat357]

MPC-HC with Madvr seems to crash when I drop this .vpy on MPC-window :

import vapoursynth as vs
import sys
core = vs.Core(accept_lowercase=True)
core.std.LoadPlugin(r'k:\programs\ffms2-r725-icl\ffms2.dll')
ret = core.ffms2.Source(r"k:\film\10bit\Madoka_v210.mkv")
last = ret
enable_v210=True


The error :

Faulting application name: mpc-hc.exe, version: 1.6.5.6055, time stamp: 0x506a1744
Faulting module name: vsvfw.dll, version: 0.0.0.0, time stamp: 0x506df320
Exception code: 0xc0000005
Fault offset: 0x00002fe0
Faulting process id: 0x10b8
Faulting application start time: 0x01cda31ff99abcae
Faulting application path: K:\programs\MPC-Homecinema-Lite\mpc-hc.exe


If I remove the last line in . vpy(= without "enable_v210=True"), the v210 file plays OK and shows up as P210 in Madvr.
This makes me think that the "v210 format" is the culprit somehow...
Is the VSVFW module not supposed to support v210 input ?
Any idea why this crash happens ?

If I create a fake AVI using PFM/VSFS.DLL, the resulting AVI plays OK and MadVR gives "v210" as format.
(with the line "enable_v210=True" included)

[Myrsloik]

If I remove the last line in . vpy, the v210 file plays OK and shows up as P210 in Madvr.
This makes me think that the "v210 format" is the culprit somehow...
Is the VSVFW module not supposed to support v210 input ?
Any idea why this crash happens ?

What's the resolution of the file?

test_stuff.zip (https://dl.dropbox.com/u/73468194/test_stuff.zip) <- you can also try these two replacement files and see if they help

[Pat357]

What's the resolution of the file?
It's full HD : 1920x1080
test_stuff.zip (https://dl.dropbox.com/u/73468194/test_stuff.zip) <- you can also try these two replacement files and see if they help
Thanks, will test it right away.

Tested the VSFS and VSVFW you've provided : All combinations old-new lead to crash when dropping .vpy on MPC.
As far as I can see, VSFS is OK, it seems VSVFW has problems. (of course, there might be interactions I don't know about)
If I remove the line "enable_v210=True", everything is OK.

Can you reproduce this problem ?

[kolak]

I've noticed the same- graphedit also crashes when vpy with v210 enabled is loaded, but when mounted than it works fine.
In the same time I'm yet to manage to crash any app with fake v210 AVI- this looks stable.

[mastrboy]

Is there any vapoursynth equilant command for histogram(mode="luma") ?
If not, is it planned?

[Myrsloik]

Is there any vapoursynth equilant command for histogram(mode="luma") ?
If not, is it planned?

Dig a few pages back for the same question and answer.

Also, I don't know what's planned. There is enough useful code in plugins for me to spend a whole year porting/improving it to vs standards.

This is where I suggest you "adopt" your favorite filter from avisynth and start coding.

[Myrsloik]

I turned a lot of the internal avisynth functions into a plugin. Here's a link (http://vapoursynth.com/downloads/avisynthfilters.7z) to it.

Note that Bob() (and possibly some other filters) won't work unless you also use the experimental vapoursynth.dll included that fixes several avisynth compat bugs.

I also had to add a tff=true/false argument to Bob() since vs doesn't keep track of that in the same way. This was just a quick delete key exercise so there may be more than one filter that doesn't work right.
Simply do print(core.avs.list_functions()) to see all the included fun.

The experimental vapoursynth.dll is in the archive.

[Pat357]

I turned a lot of the internal avisynth functions into a plugin. Here's a link (http://vapoursynth.com/downloads/avisynthfilters.7z) to it.

Simply do print(core.avs.list_functions()) to see all the included fun.
The experimental vapoursynth.dll is in the archive.
Thanks a lot !
I will create some scripts ; will post as soon as I encounter a problem.
Keep up building this marvelous tool !

Edit :
The docs mention YCOCG as "Color Family Constants", but it seems not to have a format. (can't specify 8bit / 420 sub-sampling).
How can I convert from RGB to YCOCG ? (I suppose this YCOCG is identical to YCgCo with U en V swapped ?)
The reason for my interest in this is the lossless conversion between YCgCo / RGB

[lansing]

I turned a lot of the internal avisynth functions into a plugin. Here's a link (http://vapoursynth.com/downloads/avisynthfilters.7z) to it.


Tested out. All functions work except bob, temporalsoften and spatialsoften, all returns "Avisynth Compat: requested frame xyz not prefetched, using slow method that may deadlock".

And there's no separate functions for assumettf/assumebff to be call

[Myrsloik]

Tested out. All functions work except bob, temporalsoften and spatialsoften, all returns "Avisynth Compat: requested frame xyz not prefetched, using slow method that may deadlock".

And there's no separate functions for assumettf/assumebff to be call

Of course there isn't. I even wrote that. I added the tff=true/false stuff to Bob just to get around it. If there is any other field order dependent function in there that I missed please tell me so I can hack it too.

I'll add those 3 functions to the internal prefect list so you won't get the prefetch warning. And remember, it's just a warning. If it doesn't get stuck completely the final result will always be fine.

[Reel.Deel]

@ Myrsloik
Thanks for the standalone Avisynth filters. I respect your decision on using the 2.5 interface. I think it would of been great if the the extra colorspaces in 2.6 would of been supported also.

On another note someone is currently porting histogram (https://github.com/dubhater/vapoursynth-histogram) and scxvid (https://github.com/dubhater/vapoursynth-scxvid) (never heard of it till today) to native Vapoursynth plugins. I wish I knew how to compile. :(

[Myrsloik]

@ Myrsloik
Thanks for the standalone Avisynth filters. I respect your decision on using the 2.5 interface. I think it would of been great if the the extra colorspaces in 2.6 would of been supported also.

On another note someone is currently porting histogram (https://github.com/dubhater/vapoursynth-histogram) and scxvid (https://github.com/dubhater/vapoursynth-scxvid) (never heard of it till today) to native Vapoursynth plugins. I wish I knew how to compile. :(

Avisynth 2.6 isn't even compatible with itself. I don't like moving targets. If some wants to add 2.6 support they're free to look at the 2.5 compat code and modify it. I'm not against it but I won't do it as I see it as a waste of time.

Maybe histogram works a little bit right now but scxvid definitely doesn't. We're not keeping secret stashes of plugins from you, they're just not ready.

[pbristow]

Myrsloik... :eek:

I'm almost speechless. I stopped paying much attention to the forum for a month or so (too much Real Life(TM) going on...), and I come back to find a revolution taking place! :)

This is an astonishing achievement for such a short space of time, a real lateral-thinking approach to circumventing the problems that have dogged Avisynth development in recent years that doesn't just look promising... it's already delivering!

(Haven't got it working on my own machine yet, mind you, but I will, I will... :o )

I look forward to future developments with great interest.

Bravo! :)

[Fullmetal Encoder]

benchmarks for r12.

mdegrain2 and mdegrain3 all have significant speed up comparing to r11, cpu usage reaching close to 99%. Mdegrain1 stayed the same.

Just to be clear, this is using the original filters? Assuming this, can we then expect a much greater advantage for VapourSynth over AviSynth with those filters specifically re-written to take advantage of VapourSynth's multithreading scheme?

[Myrsloik]

Just to be clear, this is using the original filters? Assuming this, can we then expect a much greater advantage for VapourSynth over AviSynth with those filters specifically re-written to take advantage of VapourSynth's multithreading scheme?

No, this is it. You won't gain any speed at all from porting them. You'll simply have to optimize the code more. Sad truth.
And yes, those benchmarks were made with the original avisynth binaries.

[SassBot]

In case anyone is interested I have a git mirror of the vapoursynth code on Github here (https://github.com/amichaelt/vapoursynth). Thanks for the awesome work, too, Myrsloik. At some point if you or someone else doesn't tackle it I was going to look into see what would be needed to support 2.6 Avisynth plugins.

[lansing]

Just to be clear, this is using the original filters? Assuming this, can we then expect a much greater advantage for VapourSynth over AviSynth with those filters specifically re-written to take advantage of VapourSynth's multithreading scheme?

well you cannot expect any greater speed when your cpu was already running at 99%. If you're looking for more speed, get a better cpu. I'm pretty sure an intel i7 can easily blown away my amd 4 core.

Memory wise, vapoursynth will beat avisynth in any instance. I can recall the many times that i got pissed when trying to encode my bluray video, and avisynth crashed on me because of the "out of memory" thing, and i have 12G RAM on my machine.

[mandarinka]

Avisynth obviously can't use more than 4 GB. It's 32-bit a program.

[Groucho2004]

Avisynth obviously can't use more than 4 GB. It's 32-bit a program.

The address space for a 32 bit process on 32 bit Windows is 2 GB. With some hacking this can be extended to 4GB, but only on 64 bit Windows.

[LoRd_MuldeR]

It doesn't require "hacking", it just needs adding the /LARGEADDRESSAWARE option when compiling the executable.

And of course the code must able to deal with addresses that have the most significant bit set. Just setting the /LARGEADDRESSAWARE flag for arbitrary code might break it!

Because Avisynth is a DLL that gets loaded into the process of some other application, whether /LARGEADDRESSAWARE is set or not totally depends on how that application was compiled.

BTW: On 32-Bit systems you can still get 3 GB virtual address space with /LARGEADDRESSAWARE, but it requires a special Windows boot option.

The full 4 GB are indeed only possible for 32-Bit processes when compiled with /LARGEADDRESSAWARE running and when running on a 64-Bit operating system...

[CarlPig]

Im very newbie to Vapoursynth, so i hope someone can help me with a step by step guide.

1. How to make this avs script to vs script (I have a 2 core cpu with 2 threads):
LoadPlugin("C:\Program Files (x86)\MeGUI\tools\dgindexnv\DGDecodeNV.dll")
DGSource("D:\Bluray Dumps\Fast.Og.Furious.2009\Fast.Og.Furious.2009.dgi",fieldop=0, deinterlace=1)
crop(2, 140, -2, -148)

2. How to open a vs script in VirtualDub

3. How to make this avs script to vs script:
AviSource("D:\Bluray Dumps\Fast.Og.Furious.2009.vdr")
converttoYV12()
hqdn3d(4.0,3.0,6.0,4.5).f3kdb
hqdn3d(4.0,4.0,1.0,1.0).f3kdb
LSFMod(dest_x=1920, dest_y=824)

4. How to send the vs script to x264

[LoRd_MuldeR]

CarlPig, have you already looked at the documentation?
ttp://www.vapoursynth.com/doc/index.html

And at this post, which explains usage with x264?
http://forum.doom9.org/showpost.php?p=1521791&postcount=1

:readguid:

[Groucho2004]

It doesn't require "hacking", it just needs adding the /LARGEADDRESSAWARE option when compiling the executable
Semantics... :)

BTW: On 32-Bit systems you can still get 3 GB virtual address space with /LARGEADDRESSAWARE, but it requires a special Windows boot option.
I choose to ignore that option, never got it to work properly and from what I've read I'm not the only one.

[Chikuzen]

I wrote rawsource plugin for VapourSynth.
vsrawsource-0.1.0.7z (http://www.mediafire.com/download.php?xfr11bbuv0qi3ze)

BTW, I think that each plugin should have its own thread.
However, I'm not clear about where is the right place for VS plugins.
Since vsavsreader had relation very strong against avisynth, there was no problem in particular at the present place.

[Myrsloik]

I wrote rawsource plugin for VapourSynth.
vsrawsource-0.1.0.7z (http://www.mediafire.com/download.php?xfr11bbuv0qi3ze)

BTW, I think that each plugin should have it's own thread.
However, I'm not clear about where is the right place for VS plugins.
Since vsavsreader had relation very strong against avisynth, there was no problem in particular at the present place.

Create all the threads you want, the worst thing that could happen is that a moderator splits or renames the forum name to avisynth and vs I guess.

Once the raw source plugin has been tested and is stable I think I'd like to include it into the vs source tree if you don't mind and I'll add a link to it in the vs documentation in the next release.

[Chikuzen]

Create all the threads you want, the worst thing that could happen is that a moderator splits or renames the forum name to avisynth and vs I guess.

ok, done (http://forum.doom9.org/showthread.php?t=166075).

Once the raw source plugin has been tested and is stable I think I'd like to include it into the vs source tree if you don't mind and I'll add a link to it in the vs documentation in the next release.
no problem

[SassBot]

Hey Myrsloik have you written or do you plan on writing any sort of documentation on porting over Avisynth plugins to Vapoursynth? I understand you are busy and I'm definitely going to dig into the code but at least some basic guidelines for porting over would be helpful.

[SassBot]

So I'm taking a crack at porting ColorMatrix to Vapoursynth and I have a question about how to handle the fact that it requests a frame within the constructor. I see that the vsapi->getFrame says:

do never use inside a filter's getframe function, for external applications using the core as a library or for requesting frames in a filter constructor

So do I use the vsapi->newVideoFrame instead or how should I handle this?

To add the code section I'm dealing with is this:


if (hints)
{
int temp;
child->SetCacheHints(CACHE_RANGE, 1);
hintClip = child;
PVideoFrame pv = child->GetFrame(0, env);
getHint(pv->GetReadPtr(PLANAR_Y), temp);
if (temp == -1)
env->ThrowError("ColorMatrix: no hints detected in stream with hints=true!");
}

[Myrsloik]

Hey Myrsloik have you written or do you plan on writing any sort of documentation on porting over Avisynth plugins to Vapoursynth? I understand you are busy and I'm definitely going to dig into the code but at least some basic guidelines for porting over would be helpful.

You are allowed to use getframe in the constructor if you really have to. But in colormatrix you probably shouldn't. Just wait until the first frame request and initialize it then. Really, you'll have to initialized it any time the colorimetry changes anyway.

You can find a well commented invert filter example in trunk/sdk in the source tree.

[SassBot]

Yeah after I posted and reading more of the ColorMatrix code I noticed that it was basically doing the same check in its getframe function as well anyway. No real point in doing it twice.

[SassBot]

Since I've been unable to find any examples of its use within your example plugins or any of the ported plugins: What is the proper way to call internal functions from within a Vapoursynth plugin? I see there is an invoke function on the VSAPI object but where do I get the VSPlugin pointer from? Is that the object that you get passed in from the VapourSynthPluginInit function?

[Myrsloik]

Since I've been unable to find any examples of its use within your example plugins or any of the ported plugins: What is the proper way to call internal functions from within a Vapoursynth plugin? I see there is an invoke function on the VSAPI object but where do I get the VSPlugin pointer from? Is that the object that you get passed in from the VapourSynthPluginInit function?

I don't think I use that in any simple code. Basically if you want to use a built-in function you use getPluginId to obtain the VSPlugin pointer to the plugin you want. Use com.vapoursynth.std as the identifier if you want to call an internal function.

Now I'm curious, why do you need this in colormatrix?

[SassBot]

Trying to figure out the best way to port over the invoke calls it makes for SeparateFields and Weave for interlaced footage and to replace the Limiter call to a call to Lut. I figured if there are already internal functions for them no need to duplicate.

[Myrsloik]

Trying to figure out the best way to port over the invoke calls it makes for SeparateFields and Weave for interlaced footage and to replace the Limiter call to a call to Lut.
I can accept the separatefields and weave stuff. Seems reasonable.
But I will slap you with a Norwegian salmon if you add on a lut at the end for range limiting.
What you should do is add a pmaxub/pminub at the beginning and end of the already existing pile of inline asm. Using a lut for what is in practice only 2 cpu instructions is absurd.

[SassBot]

Hehe. Well the limiter switch was just throwing out based on what you said to replace the call with in the docs. But, that is a better idea for sure. :-)

[TheFluff]

While you're at it, replace the hinting shit with custom per-frame properties.

[SassBot]

I'll look into that once I get it up and running.