VapourSynth - AUDIO SUPPORT AND NEW API BETA

[Myrsloik]

It's finally happened!

Biggest changes:

Audio support
Cleaner API
More performance


I'll provide a longer post later but things have now reached a state I'm happy with in general. Basically full backwards compatibility is provided with only a few minor caveats:

YCoCg is no longer a separate format so any scripts that reference it will break (now fixed in all known script, simply update your things).
Filters that output multiple clips (old api FFMS2 in alpha mode) will simply have outputs beyond the first discarded.
Compat formats have been removed so applications that rely on them for output are broken.
Avisynth filters now all have a secret argument called compatpack added, set it to True to pack YV16 and RGB24 before passing to the filter.
get_read/write_array() functions have been removed in Python and replaced by a better implementation that can't accidentally cause access violations
Histogram filter no longer bundled in windows installer since it's not a part of the VS source tree



Note that none of the previously compiled plugins for api4 will work so use the provided versions in this release. If you want to do speed comparisons I'd recommend using the linked FFMS2 binary which supports both old and new VS APIs. The alpha plane is now attached to the main frame and has to be extracted with PropToClip().

https://github.com/vapoursynth/vapoursynth/releases/tag/R55-API4-RC2
https://github.com/vapoursynth/vapoursynth/releases/tag/R54-API4-test1 <- get filter binaries from this release

[DJATOM]

Apparently removing COMPAT stuff broke existing script previewers
Failed to convert to RGB:
The VSVideoInfo structure passed by Spline16 is invalid.
Couldn't create preview node for output number 0.

[Myrsloik]

Apparently removing COMPAT stuff broke existing script previewers
Failed to convert to RGB:
The VSVideoInfo structure passed by Spline16 is invalid.
Couldn't create preview node for output number 0.

Anything VFW/AVFS based still works. Also, compat formats needed to die sooner or later

[ChaosKing]

Can't test ffms2 v4 in vdub. I always get this error if click on "run video analysis pass":
An out-of-bounds memory access (access violation) occurred in module 'VirtualDub64'...
...reading address 0000000000000290.

Scrolling through the video can also lead to a crash.

Old ffms2 is ok.

Tested on Win 10 x64, Ryzen 3600 CPU. Vdub2 44282
Video source mpeg2 dvd muxxed to mkv.

EDIT
And it seems that old ffms2 is very slow in this build. The dvd clip went from 220fps to like 6fps after 2000 frames. 1080p avc clip starts with 8fps. Something seems not right.

[Myrsloik]

Can't test ffms2 v4 in vdub. I always get this error if click on "run video analysis pass":
An out-of-bounds memory access (access violation) occurred in module 'VirtualDub64'...
...reading address 0000000000000290.

Scrolling through the video can also lead to a crash.

Old ffms2 is ok.

Tested on Win 10 x64, Ryzen 3600 CPU. Vdub2 44282
Video source mpeg2 dvd muxxed to mkv.

EDIT
And it seems that old ffms2 is very slow in this build. The dvd clip went from 220fps to like 6fps after 2000 frames. 1080p avc clip starts with 8fps. Something seems not right.

Read the initial post: "Old api source filters that used nfMakeLinear (most of them since seeking is slooow) will perform worse until ported to the new API."

I'll see if I can figure out the crash

[Myrsloik]

Can't test ffms2 v4 in vdub. I always get this error if click on "run video analysis pass":
An out-of-bounds memory access (access violation) occurred in module 'VirtualDub64'...
...reading address 0000000000000290.

Scrolling through the video can also lead to a crash.

Old ffms2 is ok.

Tested on Win 10 x64, Ryzen 3600 CPU. Vdub2 44282
Video source mpeg2 dvd muxxed to mkv.


I can't reproduce the crash and I unfortunately don't have dvds muxed to mkv specifically.

[ChaosKing]

I can't reproduce the crash and I unfortunately don't have dvds muxed to mkv specifically.

It happens with other videos as well.

[Myrsloik]

It happens with other videos as well.

Found the bug. The binaries have been sneakily updated.

[ChaosKing]

No more crashes now :D
But there's still something wrong when using v3 api ffms2. It is painfully slow. 190 fps (api v4) vs 6fps (api v3)
I tried multiple ffms2 versions, same low fps. CPU usage seems similar to v4.

EDIT
will perform MUCH WORSE until ported to the new API.
But will it be that slow?

[Myrsloik]

No more crashes now :D
But there's still something wrong when using v3 api ffms2. It is painfully slow. 190 fps (api v4) vs 6fps (api v3)
I tried multiple ffms2 versions, same low fps. CPU usage seems similar to v4.

EDIT

But will it be that slow?

Yes, I suppose that's possible. I got ~1/15 speed with a quick qtgmc test. Basically all the logic to try to make requests linear are gone and the new (and much better) system requires additional filter support.

If relevant source filters aren't updated by the time this branch is stable and tested maybe I'll add some kind of additional workaround.

[ChaosKing]

My results (vdub - run analysis pass)
R54_test1: ffms2 v4 + qtgmc() = 170fps
R54: ffms2 v3 + qtgmc() = 185fps

EDIT:
More reliable results with vspipe (run 3 times each):
R54 ffms2 v3 1080p source - 264.8 fps
R54 ffms2 v3 + qtgmc 480p source - 182.05 fps

R54_test1 ffms2 v4 1080p source - 264.3 fps
R54_test1 ffms2 v4 + qtgmc 480p source - 204.12 fps

Now test1 is faster with qtgmc xD

[ChaosKing]

Very heavy filtering test:
haf.MCTemporalDenoise + knlm.KNLMeansCL + haf.ContraSharpening + haf.FineDehalo + nnedi3_rpow2 + MfTurd + haf.FineDehalo + haf.LSFmod + f3kdb.Deband

R54: 11.3 fps
R54: 12.11 fps (max_cache_size = 15000) (I have 64gb ram)
R54_test1: 12.5 fps

I noticed that test1 does not show the "Script exceeded memory limit. Consider raising cache size." message. So caching is now smarter?

[Myrsloik]

Very heavy filtering test:
haf.MCTemporalDenoise + knlm.KNLMeansCL + haf.ContraSharpening + haf.FineDehalo + nnedi3_rpow2 + MfTurd + haf.FineDehalo + haf.LSFmod + f3kdb.Deband

R54: 11.3 fps
R54: 12.11 fps (max_cache_size = 15000) (I have 64gb ram)
R54_test1: 12.5 fps

I noticed that test1 does not show the "Script exceeded memory limit. Consider raising cache size." message. So caching is now smarter?

Yes, the cache insertion is completely reworked and it wouldn't surprise me if there're only 1/4 as many caches inserted (and in better places) for scripts like qtgmc.

[feisty2]

when will the documentation for API v4 be ready?

[Myrsloik]

when will the documentation for API v4 be ready?

You can find mostly complete notes here: https://github.com/vapoursynth/vapoursynth/blob/doodle1/APIV4%20changes.txt

For plain video filters it's mostly a question of renamed things and you should be able to convert just by looking at the filter samples.

[_Al_]

vsedit would need to be updated, it uses COMPATBGR32 for RGB conversion then using QT's QImage.Format_RGB32 to convert it to QPixmap. In Python (not C ) it uses something along like this:
img = QImage(compatbgr.get_frame(f).get_read_array(0), w, h, stride, QImage.Format_RGB32).mirrored()
pix = QPixmap.fromImage(img).scaled(scale_w, scale_h, **modes)

[Myrsloik]

vsedit would need to be updated, it uses COMPATBGR32 for RGB conversion then using QT's QImage.Format_RGB32 to convert it to QPixmap. In Python (not C ) it uses something along like this:
img = QImage(compatbgr.get_frame(f).get_read_array(0), w, h, stride, QImage.Format_RGB32).mirrored()
pix = QPixmap.fromImage(img).scaled(scale_w, scale_h, **modes)

Fortunately packing planar RGB24 is trivial so applications should get updated quickly and would still work with both. If someone wants to compile a list of all applications that break because of my changes that'd be useful. I already have a good idea about which source plugins need to be updated and I'll get to work on d2vsource tomorrow.

[DJATOM]

Afaik, https://github.com/Endilll/vapoursynth-preview and https://bitbucket.org/mystery_keeper/vapoursynth-editor/src/master/ broken, both relies on Qt and Format_RGB32 (0xffRRGGBB).

[Myrsloik]

Test2 is released. Has fixes for crashes that could occur with certain filters but otherwise no real changes.

[lansing]

Is RGB24 compatible with Virtualdub filter if I'm to load it in the script?

[l33tmeatwad]

If someone wants to compile a list of all applications that break because of my changes that'd be useful.
Yeah, having a master list would be super helpful.

[Myrsloik]

Is RGB24 compatible with Virtualdub filter if I'm to load it in the script?

No, not at the moment. Will work before I make a full release.

[Myrsloik]

Yeah, having a master list would be super helpful.

Here's the list of known things so far. Note that several scripts got updated already and have been removed. I'm currently working on d2vsource and that should be ready sometime this week.

https://github.com/vapoursynth/vapoursynth/issues/718

[Myrsloik]

I'm adding back packed YUY2 and RGB32 support to Avisynth compatibility but I'm not sure how it should be done. Always packing YV16 to YUY2 will be counterproductive since nowadays more plugins support YV16 than YUY2 and it'll error out. Planar RGB to packed has similar problems. So I guess there are 2 options:

1. Inject a secret packing argument into every Avisynth function. Example: core.avs.GBlur(clip, usepacking=True)
2. Try to automatically detect things. Would be equivalent to first invoking a function using planar formats and if that fails pack inputs (if any suitable) to YUY2/RGB32 and retry. No idea how well this would work in practice.

[cretindesalpes]

Nice to see that audio will be supported in VS. But are you really going to support all kind of different sample formats at the plug-in level? Because nowadays we only need float for audio processing (sometimes double for temporary calculations, but this is not needed for streaming between audio modules), and conversion happens only at storage or interface level.

[DJATOM]

Simple (trim, join, etc) filters can live without any processing, isn't it better to provide untouched data?

[tebasuna51]

Nice to see that audio will be supported in VS. But are you really going to support all kind of different sample formats at the plug-in level? Because nowadays we only need float for audio processing (sometimes double for temporary calculations, but this is not needed for streaming between audio modules), and conversion happens only at storage or interface level.

My opinion was in the post https://forum.doom9.org/showthread.php?p=1917206#post1917206

Lossless functions can work with any format (and already work, more or less), but lossy functions only need the 32 bits float format (of course 64 bits float format is better but maybe in the future)

[cretindesalpes]

Any integer data up to 24 bits (even 25 bits IIRC) can be represented in 32-bit floating point data, and converted back and forth losslessly by trivial means. Just make sure not to enable dithering for the float to int conversion.

[Myrsloik]

Nice to see that audio will be supported in VS. But are you really going to support all kind of different sample formats at the plug-in level? Because nowadays we only need float for audio processing (sometimes double for temporary calculations, but this is not needed for streaming between audio modules), and conversion happens only at storage or interface level.

I allow 16-32 bit audio and float. I suspect most plugins will settle on 16,32 and float support or something similar but having the rest available fpr source filters to output is free.

[Myrsloik]

What would you say if I dropped 32 bit builds but in return added back windows 7 (x64) support (Python 3.8 modules in addition to whatever the latest Python is)?

Also, new test build coming soon with avisynth compatibility fully restored.

[Reel.Deel]

What would you say if I dropped 32 bit builds but in return added back windows 7 (x64) support (Python 3.8 modules in addition to whatever the latest Python is)?

Also, new test build coming soon with avisynth compatibility fully restored.

That sounds like a fair trade :D ... I have r52 (x64) installed because of that reason.

[cretindesalpes]

:thanks:

[ChaosKing]

Hmm I think a x64 win7 build is "more useful" than a x86 build.

[Myrsloik]

New version with exciting things:

Python 3.8 and 3.9 are supported now. And 32 bit compatibility remains as well for now I guess.

The portable version requires vs-detect-python.bat to be run once before using to select the correct dlls.

Avisynth compatibility has been restored and will always pass planar formats unless the the avs function is called with the argument compatpack=True.
core.avs.GBlur(clip, compatpack=True)
This seems to be the most reasonable tradeoff in modern times since many filters actually support planar RGB nowadays. May require some scripts to be modified obviously but it's only mildly breaking at most.

Have fun with testing things. The only breaking changes remaining are to overhaul all the memory view stuff in Python. It's currently an access violation deathtrap.

[vxzms]

My friend and I did some simple tests in R55-API4-test3. Here are some of our findings between it and R54-Release:

1. Audio support is wonderful, but I don’t know if it can process lossy audio almost losslessly like ffmpeg, e.g. trim .aac / .m4a audio
2. New API vs scripts reduces a lots of memory (reduced by about 30%), especially based CUDA plugins (reduced by about 80%)
3. Vspipe’s new option --filter-time is interesting; --argus input format also has some changes (byte -> string), user may need attention if vpy decode byte to string
4. Plugins which has been ported to api4 get a small speed increase (about 10%, zimg resize may be higher), but if don’t consider plugins which using nfMakeLinear, api3 plugins also have different degrees of speed reduction (about 5%-15%), e.g. eedi2, znedi3, neo_f3kdb, descale and so on (tcanny and warp similar to api3 speed). If multiple api3 filters are used, it will more obvious, such as 2pass eedi2 / znedi3, they generally become the bottleneck of script speed. These is just part of vs plugin library, the most important is that we are not sure that these filters will be ported to api4 in the future, some developers reduced their activity, so I think if you consider more compatibility with api3.

I don’t know if there are more other people test api4, didn't see much discussion in forum.


* After reminding to correct 2nd point, it may be bm3d.VAggregate reduced so much memory instead of BM3DCUDA.

[Myrsloik]

My friend and I did some simple tests in R55-API4-test3. Here are some of our findings between it and R54-Release:

1. Audio support is wonderful, but I don’t know if it can process lossy audio almost losslessly like ffmpeg, e.g. trim .aac / .m4a audio
2. New API vs scripts reduces a lots of memory (reduced by about 30%), especially based CUDA plugins (reduced by about 80%)
3. Vspipe’s new option --filter-time is interesting; --argus input format also has some changes (byte -> string), user may need attention if vpy decode byte to string
4. Plugins which has been ported to api4 get a small speed increase (about 10%, zimg resize may be higher), but if don’t consider plugins which using nfMakeLinear, api3 plugins also have different degrees of speed reduction (about 5%-15%), e.g. eedi2, znedi3, neo_f3kdb, descale and so on (tcanny and warp similar to api3 speed). If multiple api3 filters are used, it will more obvious, such as 2pass eedi2 / znedi3, they generally become the bottleneck of script speed. These is just part of vs plugin library, the most important is that we are not sure that these filters will be ported to api4 in the future, some developers reduced their activity, so I think if you consider more compatibility with api3.

I don’t know if there are more other people test api4, didn't see much discussion in forum.

1. No, obviously not in the scope of this project.
2. The CUDA drop sounds too big to be correct.
3. The new behavior is based on the type hint in the passed VSMap. If used through vspipe it's always a utf-8 string which turns into the str type in Python. I think strings generally make more sense for what comes from a command line argument.
4. There's no speed difference for api3 and api4 video filters unless they use any flags (99% don't) in which case things may get weird. All speed differences seen are due to general threading changes in the core. At most you'd see a slight memory usage drop when using api4 filter since useless caches can be avoided.

Btw, what system did you test things on?

[vxzms]

1. No, obviously not in the scope of this project.
2. The CUDA drop sounds too big to be correct.
3. The new behavior is based on the type hint in the passed VSMap. If used through vspipe it's always a utf-8 string which turns into the str type in Python. I think strings generally make more sense for what comes from a command line argument.
4. There's no speed difference for api3 and api4 video filters unless they use any flags (99% don't) in which case things may get weird. All speed differences seen are due to general threading changes in the core. At most you'd see a slight memory usage drop when using api4 filter since useless caches can be avoided.

Btw, what system did you test things on?

Thanks your reply, I test on Windows 11 22000.120.

CUDA tested by my friend, mainly (V-)BM3DCUDA. She is also win11.

[Myrsloik]

Does anyone actually use v210 output ever? as in set enable_v210 to true? I'm curious since it isn't the default.

[DJATOM]

Not ever needed to set this option before.

[kedautinh12]

Wow 3900x -> 5950x, waiting for 3090 to full option

[DJATOM]

Wow 3900x -> 5950x, waiting for 3090 to full option

I have bought it for $909 which I think is reasonable.
Maybe I will buy 4090 once it releases... I don't want to overpay for a card, so probably gonna pass on Ampere generation.

[videoh]

You're rich Me too.

[poisondeathray]

Does anyone actually use v210 output ever? as in set enable_v210 to true? I'm curious since it isn't the default.

Yes - when avfs is used for 10bit422 input into programs like retail NLE's

[Myrsloik]

Yes - when avfs is used for 10bit422 input into programs like retail NLE's

Ok, are there any other weird input formats NLEs support?

[poisondeathray]

Ok, are there any other weird input formats NLEs support?

Not really "weird" - v210 is the defacto standard for uncompressed professional video for the last 15-20 years . "p210" is universally not supported by those sorts of programs, including VFX, grading programs


For many NLE's , if uncompressed 8bit 4:2:0 is supported, it's usually supported as "IYUV" , and 8bit 4:2:2 as "UYVY" .

v210 (10bit422) , IYUV (8bit420), UYVY (8bit422) are usually the "magical" pixel format configurations that get passthrough or proper YUV treatment on Windows based NLE's . v210 also on Mac, but "AVI" is not as compatible on most Mac programs

YV12 (for 8bit 4:2:0) and YUY2 (for 8bit 4:2:2 "packed") or YV16 (for 8bit 4:2:2 "planar") as uncompressed imports usually get converted to RGB.



This has been mentioned before several times in other threads - but for the avfs side - MOV container emulation would be more compatible for some programs

[Myrsloik]

I could add alternate output support for IYUV and UYVY fourccs if it really matters.

Patches welcome for MOV support I guess. Personally I'd rather do other things.

[poisondeathray]

I could add alternate output support for IYUV and UYVY fourccs if it really matters.


I would put it on lower priority for your "things to do" list . I'm sure there's more important stuff to do. But if you had to prioritize them UYVY > IYUV

[Myrsloik]

I would put it on lower priority for your "things to do" list . I'm sure there's more important stuff to do. But if you had to prioritize them UYVY > IYUV

All you have to do is say some expensive enterprise NLEs that it'd improve compatibility with and I'll do it soon...

[poisondeathray]

All you have to do is say some expensive enterprise NLEs that it'd improve compatibility with and I'll do it soon...

UYVY will for sure - "Uncompressed video" to a NLE generally means UYVY for 8bit422 and v210 for 10bit422.

(Many do not handle uncompressed 8bit 4:2:0, but for the ones that do, the "magic" key is IYUV)

Most other variants/fourcc's are mishandled and converted to RGB (if they import at all)


(MOV support would increase compatibility too)

[videoh]

The gross performance degradation for API3 source filters with the beta API4 Vapoursynth effectively means you have abandoned backward compatibility for your API. That is a terrible thing. I strongly suggest you find a way to solve that.

[ChaosKing]

But that only applies to plugins which uses nfMakeLinear. That's only a hand full.

[Myrsloik]

The gross performance degradation for API3 source filters with the beta API4 Vapoursynth effectively means you have abandoned backward compatibility for your API. That is a terrible thing. I strongly suggest you find a way to solve that.

Was this not a clear enough statement for you?

https://forum.doom9.org/showpost.php?p=1949996&postcount=2672

[videoh]

You said "probably". That is far from clear. I am trying to convince you to make it a certainty.

[Myrsloik]

UYVY will for sure - "Uncompressed video" to a NLE generally means UYVY for 8bit422 and v210 for 10bit422.

(Many do not handle uncompressed 8bit 4:2:0, but for the ones that do, the "magic" key is IYUV)

Most other variants/fourcc's are mishandled and converted to RGB (if they import at all)


(MOV support would increase compatibility too)

Implemented. Accessed by using vs.set_option('alt_output', x). Where 1 gives you I420, YUY2 or v210 and 2 gives you IYUV or UYVU depending on input. Will be in the next build.

[poisondeathray]

Implemented. Accessed by using vs.set_option('alt_output', x). Where 1 gives you I420, YUY2 or v210 and 2 gives you IYUV or UYVU depending on input. Will be in the next build.

Thanks,

Was UYVU a typo ? UYVY is the desired one

[Myrsloik]

Thanks,

Was UYVU a typo ? UYVY is the desired one

Yes, a typo.

[feisty2]

is there a way to query the cache mode of a specific node in api v4? this is possible in api v3 by reading the "flags" field of the struct returned by getvideoinfo

[Myrsloik]

is there a way to query the cache mode of a specific node in api v4? this is possible in api v3 by reading the "flags" field of the struct returned by getvideoinfo

getNodeDependencies is the closest you get. The cache mode is then determined from the number of consumers and a the request pattern specified.

Why do you need to know it anyway?

[feisty2]

well I never had a use case that requires the knowledge of the cache mode. I'm just cross-checking the API v3 functionalities and the corresponding API v4 ones.

[lansing]

The test installer didn't include the api4 headers in the sdk folder

[Myrsloik]

The test installer didn't include the api4 headers in the sdk folder

Will be fixed in the next build.

[Myrsloik]

It's RC1 time!

Notable changes from test3:

nfMakeLinear source filters now perform well again (FFMS2 was 50% faster than in the R54 API3 release in my very unscientific test)
It's now possible to select additional fourccs for output. Use vs.set_option('alt_output', x). Where 1 gives you I420, YUY2 or v210 and 2 gives you IYUV or UYVY depending on input. Invalid alt_output modes are simply ignored.
In Python you now access frame data with "plane = frame[0]" and then that's a 2d array, get_read/write_array() functions have been removed since they were broken in many subtle ways
The API4 headers are included in the installer
Lots of small bug fixes


Note that non-source filters that use nfMakeLinear (very few) still work poorly and will need to be manually updated.

[videoh]

nfMakeLinear source filters now perform well again Thank you.

Is there a way to install and test this without affecting my API3 install?

[Myrsloik]

Thank you.

Is there a way to install and test this without affecting my API3 install?

Use the portable version if you're fine with vspipe for output. Simply grab the python 3.9.x embedded version (https://www.python.org/ftp/python/3.9.6/python-3.9.6-embed-amd64.zip) and extract the portable VS version on top. Done!

Extracting vsedit in the same dir used to work too.

[_Al_]

Using that latest API4 R 55 portable RC1 version with latest portable python 3.9 on windows 7, running VSPipe.exe I get:
The program can't start because api-ms-win-core-path-|1-1-0.dll is missing from computer. Try reinstalling the program to fix this problem.
What do i have outdated? Older R51 portable is working.
Do I need Visual Studio 2019 for portable setup?

[videoh]

Use the portable version if you're fine with vspipe for output. Simply grab the python 3.9.x embedded version (https://www.python.org/ftp/python/3.9.6/python-3.9.6-embed-amd64.zip) and extract the portable VS version on top. Done! I don't know what this means. Any chance of a step-by-step? Thank you.

[DJATOM]

1) download https://www.python.org/ftp/python/3.9.6/python-3.9.6-embed-amd64.zip
2) extract it to for example VS-R55-API4
3) download https://github.com/vapoursynth/vapoursynth/releases/download/R55-API4-RC1/VapourSynth64-Portable-R55-API4-RC1.7z
4) extract contents of that archive into VS-R55-API4 (confirm overwrite of some files)
5) copy plugins into VS-R55-API4\vapoursynth64\plugins
6) if you want to add some importable scripts, copy them into VS-R55-API4 or edit python39._pth and add extra line with your scripts location (I'm using ./vs-scripts)
7) if you willing to use some preview application (VapourSynth Editor), there's https://github.com/YomikoR/VapourSynth-Editor/releases/download/r19-mod-1/VapourSynth-Editor-r19-mod-1-x86_64.7z which works with APIv4 Vapoursynth (extract contents into VS-R55-API4)

I believe that's enough to get things working.

[poisondeathray]

embedded + portable vapoursynth seems to work for me on simple scripts so far (using similar procedure as DJATOM) , but I have same installed python version as the embedded version and it doesn't work with the installed version - "Failed to initialize VSScript"

Is there anyway I can use installed python version with the portable vapoursynth version?

[poisondeathray]

It's now possible to select additional fourccs for output. Use vs.set_option('alt_output', x). Where 1 gives you I420, YUY2 or v210 and 2

gives you IYUV or UYVY depending on input. Invalid alt_output modes are simply ignored.

Does not seem to work correctly - I tried different combinations like putting it before/after set_output(),

VapourSynth-Editor-r19-mod-1 shows no error, preview ok


import vapoursynth as vs
core = vs.core
clip = core.std.BlankClip(width=1920,height=1080,format=vs.YUV422P8, length=300, color=[16, 128, 128])
#clip = core.std.BlankClip(width=1920,height=1080,format=vs.YUV422P10, length=300, color=[64, 512, 512])
clip = core.std.AssumeFPS(clip, fpsnum=30000, fpsden=1001)
clip = core.text.FrameNum(clip)
#vs.set_option('alt_output', 1)
#vs.set_option('alt_output', 2)
clip.set_output()
#vs.set_option('alt_output', 1)
vs.set_option('alt_output', 2)


bundled vspipe -v says
Core R54
API R4.0
API R3.6

using avfs
"PATH\python-3.9.6-embed-amd64\avfs" yuv422_alt1.vpy gives yv16 according to mediainfo (should be YUY2)
"PATH\python-3.9.6-embed-amd64\avfs" yuv422_alt2.vpy gives yv16 according to mediainfo (should be UYVY)

"PATH\python-3.9.6-embed-amd64\avfs" yuv422p10_alt1.vpy gives p210 according to mediainfo (should be v210)

using vspipe to ffmpeg
"PATH\python-3.9.6-embed-amd64\vspipe" --y4m yuv422_alt2.vpy - | ffmpeg -f yuv4mpegpipe -i - -an -f null NUL

ffmpeg reports Y42B, yuv422p . Normally it should say UYVY , uyvy422

[videoh]

1) download https://www.python.org/ftp/python/3.9.6/python-3.9.6-embed-amd64.zip
2) extract it to for example VS-R55-API4
...
I believe that's enough to get things working.

Thank you, DJ!

[lansing]

In VSScript.h there's a vsscript_finalize() to free the runtime, what do I do for this with VSScript4.h?

[Myrsloik]

In VSScript.h there's a vsscript_finalize() to free the runtime, what do I do for this with VSScript4.h?

You don't have to do anything when done now

[lansing]

This change in video format id comparison is getting too long?

// api3
if(cpVideoFormat.id == pfRGB24) {}

// api4
uint32_t formatId = m_cpVSAPI->queryVideoFormatID(cpVideoFormat.colorFamily,
cpVideoFormat.sampleType, cpVideoFormat.bitsPerSample, cpVideoFormat.subSamplingW,
cpVideoFormat.subSamplingH, m_pCore);

if(formatId == pfRGB24){}


Or, is this the correct way to use it?

int formatIdFound = m_cpVSAPI->getVideoFormatByID(&cpVideoFormat, pfRGB24, m_pCore);

if (formatIdFound != 0) {}

[lansing]

The field base enum in the VSConstants4.h is wrong


typedef enum VSFieldBased {
VSC_FIELD_PROGRESSIVE = 0,
VSC_FIELD_TOP = 1,
VSC_FIELD_BOTTOM = 2
} VSFieldBased;


The documentation (http://www.vapoursynth.com/doc/apireference.html) said that bff is 1 and tff is 2.

[Myrsloik]

The field base enum in the VSConstants4.h is wrong


typedef enum VSFieldBased {
VSC_FIELD_PROGRESSIVE = 0,
VSC_FIELD_TOP = 1,
VSC_FIELD_BOTTOM = 2
} VSFieldBased;


The documentation (http://www.vapoursynth.com/doc/apireference.html) said that bff is 1 and tff is 2.

Will fix. There are some comparison helper functions in https://github.com/vapoursynth/vapoursynth/blob/doodle1/include/VSHelper4.h#L93. I'll probably add some more to cover VideoFormat and pfConstant later.

[Myrsloik]

Does not seem to work correctly - I tried different combinations like putting it before/after set_output(),

VapourSynth-Editor-r19-mod-1 shows no error, preview ok


import vapoursynth as vs
core = vs.core
clip = core.std.BlankClip(width=1920,height=1080,format=vs.YUV422P8, length=300, color=[16, 128, 128])
#clip = core.std.BlankClip(width=1920,height=1080,format=vs.YUV422P10, length=300, color=[64, 512, 512])
clip = core.std.AssumeFPS(clip, fpsnum=30000, fpsden=1001)
clip = core.text.FrameNum(clip)
#vs.set_option('alt_output', 1)
#vs.set_option('alt_output', 2)
clip.set_output()
#vs.set_option('alt_output', 1)
vs.set_option('alt_output', 2)


bundled vspipe -v says
Core R54
API R4.0
API R3.6

using avfs
"PATH\python-3.9.6-embed-amd64\avfs" yuv422_alt1.vpy gives yv16 according to mediainfo (should be YUY2)
"PATH\python-3.9.6-embed-amd64\avfs" yuv422_alt2.vpy gives yv16 according to mediainfo (should be UYVY)

"PATH\python-3.9.6-embed-amd64\avfs" yuv422p10_alt1.vpy gives p210 according to mediainfo (should be v210)

using vspipe to ffmpeg
"PATH\python-3.9.6-embed-amd64\vspipe" --y4m yuv422_alt2.vpy - | ffmpeg -f yuv4mpegpipe -i - -an -f null NUL

ffmpeg reports Y42B, yuv422p . Normally it should say UYVY , uyvy422

Core R54 <= you've accidentally used an older version somehow

[poisondeathray]

Core R54 <= you've accidentally used an older version somehow


You're right :o

Core R55
API R4.0
API R3.6


But with Core R55, ffmpeg still reports Y42B, yuv422p for the yuv422 alt2 script .

Also, avfs now crashes when file is accessed, even a plain BlankClip script (without alt_output options) . It "mounts", but as soon as you touch or access the fake avi, it crashes no helpful error message

Calling old avfs with same plain BlankClip script does not crash

[Myrsloik]

In case anyone's wondering the part of the API in vsscript4.h is not considered stable. I predict that it will be rolled back closer to something along the lines of the old api.

[feisty2]

is there a functionality in API v3 that corresponds to setCacheOptions in API v4?

[Myrsloik]

is there a functionality in API v3 that corresponds to setCacheOptions in API v4?

The closest thing is if you manually insert a cache and pass a bunch of arguments to it. But the arguments are quite different in api3 so even that's questionable.

[lansing]

How do I get a map of functions from a plugin in api4?


VSMap * FunctionsMap = VSAPI->getFunctions(Plugin);

[Myrsloik]

How do I get a map of functions from a plugin in api4?


VSMap * FunctionsMap = VSAPI->getFunctions(Plugin);


Use getNextPluginFunction(plugin, nullptr)
Then pass the returned pointer to the function until it returns null. That means you've enumerated all plugins.

[lansing]

Use getNextPluginFunction(plugin, nullptr)
Then pass the returned pointer to the function until it returns null. That means you've enumerated all plugins.

Like this? Do I need to free the function every loop before getting a new one?


VSPluginFunction * function = VSAPI->getNextPluginFunction(nullptr, Plugin);

while (function != nullptr) {
# do stuff with function
function = VSAPI->getNextPluginFunction(nullptr, Plugin);
}

[Myrsloik]

Correct, you don't have to free anything.

[lansing]

What does the argument string returned by "vsapi->getPluginFunctionArguments(function)" look like? I didn't see any example in the api

[lansing]

How do I get the videoFormat name and uses it in one line like in api3?


myFunction(videoInfo->format.name);

[Myrsloik]

How do I get the videoFormat name and uses it in one line like in api3?


myFunction(videoInfo->format.name);


One line?
char name[32]; vsapi->getVideoFormatName(&videoInfo.format, name); myFunction(name);

But more seriously, use a lambda or something if you need it many times in the same function.

[Myrsloik]

Using that latest API4 R 55 portable RC1 version with latest portable python 3.9 on windows 7, running VSPipe.exe I get:
The program can't start because api-ms-win-core-path-|1-1-0.dll is missing from computer. Try reinstalling the program to fix this problem.
What do i have outdated? Older R51 portable is working.
Do I need Visual Studio 2019 for portable setup?

Install the most recent vs2019 runtime manually.

[lansing]

One line?
But more seriously, use a lambda or something if you need it many times in the same function.

This is another use case in api3:

infoString.replace("%format%", videoInfo->format.name);

[Yomiko]

core.add_cache is removed (although remains in the doc). May I know about a replacement for it?

[Myrsloik]

core.add_cache is removed (although remains in the doc). May I know about a replacement for it?

Cache insertion is now handled by the core so there's no direct replacement. If you want to change the automatic decision to add a lot of caching on your output simply use the SeVideoCache/SetAudioCache functions.

[Yomiko]

Thanks. Also I see all the arguments are required in these functions. Is it as intended?

[Myrsloik]

Thanks. Also I see all the arguments are required in these functions. Is it as intended?

No, they should obviously be optional

[Yomiko]

>>> import vapoursynth as vs
>>> core = vs.core
>>> clip = core.std.BlankClip(width=1921, height=1081, format=vs.RGB48)
>>> frame = clip.get_frame(2)
>>> frame[0] == frame[2]
True
>>> frame[0] == frame[1]
True
>>> frame = clip.get_frame(22)
>>> frame[0]
<memory at 0x0000021EE196B930>
>>> frame[1]
<memory at 0x0000021EE196B860>
>>> frame[2]
<memory at 0x0000021EE196B930>
>>>
Is it as intended in R55-RC1?

[Myrsloik]

>>> import vapoursynth as vs
>>> core = vs.core
>>> clip = core.std.BlankClip(width=1921, height=1081, format=vs.RGB48)
>>> frame = clip.get_frame(2)
>>> frame[0] == frame[2]
True
>>> frame[0] == frame[1]
True
>>> frame = clip.get_frame(22)
>>> frame[0]
<memory at 0x0000021EE196B930>
>>> frame[1]
<memory at 0x0000021EE196B860>
>>> frame[2]
<memory at 0x0000021EE196B930>
>>>
Is it as intended in R55-RC1?

Yes, it's how you access plane data now

[Myrsloik]

RC2 posted. Changes:

Reworked vsscript api to restore api3 functionality
AVFS fixes (packed RGB in Avisynth scripts)
Uses new and faster libp2p for packing
Removed lots of Python functions that have been deprecated since at least R51
API4 headers actually included in installer


Things you can do to help:

Update the __all__ tag in vapoursynth.pyx with all relevant symbols
Create vsscript4.h.rst and vshelper4.h.rst (these are simple and work 99% like the api3 counterparts)
Review pythonreference.rst (mostly updated by me already but probably missing some functions)
Review all other documentation.

[lansing]

The VSVideoFormat argument of isSameVideoPresetFormat() is missing the "constant" declaration?

[Myrsloik]

The VSVideoFormat argument of isSameVideoPresetFormat() is missing the "constant" declaration?

Correct. Will be fixed.

[poisondeathray]

was the vs.set_option('alt_output', X) removed from RC2 ?

[Myrsloik]

was the vs.set_option('alt_output', X) removed from RC2 ?

Yes, now alt_output is an optional argument to set_output(). Makes more sense that way.

[poisondeathray]

Yes, now alt_output is an optional argument to set_output(). Makes more sense that way.

It "works" , but same problems as last version ie. wrong fourcc for UYVY, avfs crashes



But with Core R55, ffmpeg still reports Y42B, yuv422p for the yuv422 alt2 script .

Also, avfs now crashes when file is accessed, even a plain BlankClip script (without alt_output options) . It "mounts", but as soon as you touch or access the fake avi, it crashes no helpful error message

Calling old avfs with same plain BlankClip script does not crash

[Myrsloik]

RC3 posted. Fixes the reported deadlock and avfs crashes on vpy script access (really tested this time).

[poisondeathray]

RC3 posted. Fixes the reported deadlock and avfs crashes on vpy script access (really tested this time).

avfs and alt_output issues fixed, thanks

Question about vspipe and alt_output - Does the alt_output fourcc emulation only work with avfs ? ffmpeg reports different fourcc with vspipe input using y4m headers (-c y4m), using same script that worked with avfs.

eg.
vspipe -c y4m yuv422_alt2.vpy - | ffmpeg -f yuv4mpegpipe -i - -an -f null NUL
reports Y42B, yuv422p instead of the expected UYVY, uyvy422

But,
avfs yuv422_alt2.vpy
reports the expected UYVY (and it works ok in programs like Premiere, gets YUV treatment, passthough)

(I don't think it matters that much, the alt_output /fourcc probably only be used for avfs - it's just a curiosity question)

[Myrsloik]

avfs and alt_output issues fixed, thanks

Question about vspipe and alt_output - Does the alt_output fourcc emulation only work with avfs ? ffmpeg reports different fourcc with vspipe input using y4m headers (-c y4m), using same script that worked with avfs.

eg.
vspipe -c y4m yuv422_alt2.vpy - | ffmpeg -f yuv4mpegpipe -i - -an -f null NUL
reports Y42B, yuv422p instead of the expected UYVY, uyvy422

But,
avfs yuv422_alt2.vpy
reports the expected UYVY (and it works ok in programs like Premiere, gets YUV treatment, passthough)

(I don't think it matters that much, the alt_output /fourcc probably only be used for avfs - it's just a curiosity question)

You're probably a bit format confused. Y4M is a super basic "container" that only allows planar yuv. Nothing else. It doesn't have the concept of fourcc or packed formats. That's why alt_output does nothing outside of avfs and vfw.

BUT for some reason FFmpeg decides to print out Y42B since that fourcc is effectively the same storage format as Y4M with the supplied headers. Pure coincidence.

[poisondeathray]

You're probably a bit format confused. Y4M is a super basic "container" that only allows planar yuv. Nothing else. It doesn't have the concept of fourcc or packed formats. That's why alt_output does nothing outside of avfs and vfw.


So vspipe rawvideo would not act any differently either ?

[Myrsloik]

So vspipe rawvideo would not act any differently either ?

Correct, there's no reason to. Everything that supports pipe input can accept pure planar nowadays.

[poisondeathray]

Correct, there's no reason to. Everything that supports pipe input can accept pure planar nowadays.

Thanks

[Yomiko]

It seems the VideoOutputTuple confuses existing python scripts that read vpy using runpy. For a video clip, in script.vpy
clip.set_output(2)
In python that handles both old and new api

runpy.run_path(script.vpy, dict(), '__vapoursynth__')
try:
clip = vs.get_output(2)[0]
except TypeError:
clip = vs.get_output(2)
vspreview may be modified to treat it as a tuple. Meanwhile in vseditor I don't see anything different from before.

May I know whether there will be further changes for this, like, backward compatibility for python or more breaking changes for vsscript?

[Myrsloik]

It seems the VideoOutputTuple confuses existing python scripts that read vpy using runpy. For a video clip, in script.vpy
clip.set_output(2)
In python that handles both old and new api

runpy.run_path(script.vpy, dict(), '__vapoursynth__')
try:
clip = vs.get_output(2)[0]
except TypeError:
clip = vs.get_output(2)
vspreview may be modified to treat it as a tuple. Meanwhile in vseditor I don't see anything different from before.

May I know whether there will be further changes for this, like, backward compatibility for python or more breaking changes for vsscript?

I think I'm done with all breaking changes now unless someone discovers something really broken.

I thought about adding an AudioOutputTuple but couldn't think of any additional information that'd be useful.

[Boulder]

Was the support for this old non-recommended style now removed? There are several utility scripts and tools like muvsfunc, mvsfunc and Zopti which use that and need some fixing.

import vapoursynth as vs
core = vs.core

[Myrsloik]

Was the support for this old non-recommended style now removed? There are several utility scripts and tools like muvsfunc, mvsfunc and Zopti which use that and need some fixing.

import vapoursynth as vs
core = vs.core

Uh no? core=vs.core should work

I only removed vs.get_core()

[Boulder]

Uh no? core=vs.core should work

I only removed vs.get_core()

Ah, I prolly looked too quickly and confused the two. The latter one is most likely still in use in a lot of older scripts but fortunately trivial to fix.

[lansing]

I ran into problem calling freeScript() in vsedit2. It kept crashing the program. Then I make a simple test with one button calling createScript() and one button calling freeScript():


VSScript * testVSScriptEnv;

void testCreateScript()
{
if(testVSScriptEnv == nullptr)
testVSScriptEnv = VSScriptAPI->createScript(nullptr);
}

void testFreeScript()
{
if (testVSScriptEnv) {
VSScriptAPI->freeScript(testVSScriptEnv);
}
}


If I go createScript -> freeScript -> createScript -> freeScript, the program will crash.

[Myrsloik]

I ran into problem calling freeScript() in vsedit2. It kept crashing the program. Then I make a simple test with one button calling createScript() and one button calling freeScript():


VSScript * testVSScriptEnv;

void testCreateScript()
{
if(testVSScriptEnv == nullptr)
VSScriptAPI->createScript(nullptr);
}

void testFreeScript()
{
if (testVSScriptEnv) {
VSScriptAPI->freeScript(testVSScriptEnv);
}
}


If I go createScript -> freeScript -> createScript -> freeScript, the program will crash.

Can't reproduce

[lansing]

Can't reproduce

I forgot the "testVSScriptEnv =" line in the above post, can you test again?

[Myrsloik]

I forgot the "testVSScriptEnv =" line in the above post, can you test again?

Obviously I adjusted it for those things. Still worked.

[lansing]

Obviously I adjusted it for those things. Still worked.

I found my problem. I was trying to reuse the temporary node reference. I passed it into a function, and then freed the node inside the function. And then when I free the environment, it will crash.


VSNode * pProcessingNode = m_cpVSAPI->addNodeRef(pOutputNode);
pProcessingNode = resizePlugin(pProcessingNode);
pProcessingNode = otherPlugin(pProcessingNode);

VSNode * resizePlugin (VSNode * a_pVideoNode) {
m_cpVSAPI->mapSetNode(pArgumentMap, "clip", a_pVideoNode, maReplace);
VSMap * pResultMap = m_cpVSAPI->invoke(pResizePlugin, "Point", pArgumentMap);
VSNode * pResultNode = m_cpVSAPI->mapGetNode(pResultMap, "clip", 0, nullptr);

m_cpVSAPI->freeNode(a_pVideoNode);
return pResultNode;
}


What is the correct way to do this?

[Myrsloik]

VSNode * pProcessingNode = m_cpVSAPI->addNodeRef(pOutputNode);
pProcessingNode = resizePlugin(pProcessingNode);
pProcessingNode = otherPlugin(pProcessingNode);

VSNode * resizePlugin (VSNode * a_pVideoNode) {
VSMap *pArgumentMap = m_cpVSAPI->createMap();
m_cpVSAPI->mapConsumeNode(pArgumentMap, "clip", a_pVideoNode, maReplace);
VSMap * pResultMap = m_cpVSAPI->invoke(pResizePlugin, "Point", pArgumentMap);
m_cpVSAPI->freeMap(pArgumentMap);
// use mapGetError(pResultMap) here if you need to check for errors
VSNode * pResultNode = m_cpVSAPI->mapGetNode(pResultMap, "clip", 0, nullptr);
m_cpVSAPI->freeMap(pResultMap);
return pResultNode;
}

At the end you have pProcessingNode and pOutputNode that needs to be freed.

[lansing]

At the end you have pProcessingNode and pOutputNode that needs to be freed.

Thanks I got it working.

[_Al_]

All working here so far, this is nice.
Also attempt to drag audio along with video, here in vapoursynth attaching audio to video frames using props.
But the code below does not work because ModifyFrame does not work with anode. vnode only. ModifyFrame (or FrameEval) is planned in future or is it a bad idea?

oh I see, n is not the same for vnode as is for anode, this could not work, only if bunch of anode frames (perhaps list) is attached to vnode and then reconstructed again, no idea how anisynth does it,

also deleting previous code to not confuse anyone

[_Al_]

testing about these, so it might help someone
API4 audio_test.py script:
import vapoursynth as vs
from vapoursynth import core
source_path=r"video.mp4" #stereo audio
v = core.lsmas.LibavSMASHSource(source_path)
a = core.bas.Source(source_path)
v.set_output(0)
a.set_output(1)
some line here are taken from somewhere else from this forum:
AUDIO
VSPipe -o 1 -c wav audio_tests.py - | neroAacEnc -ignorelength -lc -cbr 96000 -if - -of nero.m4a
VSPipe -o 1 -c wav audio_tests.py - | ffmpeg -y -i - ffmpeg_output.wav
VSPipe -o 1 audio_tests.py - | ffmpeg -y -f f32le -ac 2 -ar 48000 -i - ffmpeg_output.wav
VSPipe -o 1 -c wav audio_tests.py vspipe_to_wav.wav #but double size (32bit if original is 16bit)
VSPipe -o 1 audio_tests.py vspipe_to_wav.wav #but double size (32bit if original is 16bit)
VSPipe -o 1 -c wav audio_tests.py - | ffplay -
VSPipe -o 1 audio_tests.py - | ffplay -f f32le -ac 2 -ar 48000 -i -
VSPipe -o 1 -c wav audio_tests.py - | ffmpeg -i video.mp4 -i - -map 0:0 -map 1:0 -f AVI -c:v utvideo -pix_fmt yuv420p -colorspace bt709 -c:a pcm_s16le -y ffmpeg_utvideo.avi

VIDEO
VSPipe -o 0 -c y4m audio_tests.py - | ffplay -i -
VSPipe -o 0 -c y4m audio_tests.py - | mpv -
btw. can be two nodes (vnode,anode) piped into ffmpeg or ffplay in one line?

[Myrsloik]

No, to do that vspipe needs to wrap the streams into a more complex container like mkv.

[Myrsloik]

All working here so far, this is nice.
Also attempt to drag audio along with video, here in vapoursynth attaching audio to video frames using props.
But the code below does not work because ModifyFrame does not work with anode. vnode only. ModifyFrame (or FrameEval) is planned in future or is it a bad idea?

oh I see, n is not the same for vnode as is for anode, this could not work, only if bunch of anode frames (perhaps list) is attached to vnode and then reconstructed again, no idea how anisynth does it,

also deleting previous code to not confuse anyone

Frame props are nonsensical since audio frames are a very arbitrary thing. That's why I haven't really exposed the manipulation and I want audio support to mostly allow you to pass audio through in a convenient way.

I think what you should do it simply create wrapper functions for the trim, splice and other reorder functions and use those. Most filters don't reorder the clip. That would be my first attempt. I'll write an example some day of it.

Avisynth has a weird symbiotic relationship between one audio track and one video track. Have more than one audio track and things get iffy.

[feisty2]

you can attach audio frame(s), possibly audio frames from multiple audio tracks, to a video frame using the C++ API, it should be no more complicated than

auto [PictureFrame, EngAudioFrame, FrenchAudioFrame] = Node::AcquireFrameGiven(Index, GeneratorContex).From(Video, EngAudio, FrenchAudio);
auto CompositeFrame = Core.CopyFrameFrom(PictureFrame);
CompositeFrame["_Audio"] += EngAudioFrame;
CompositeFrame["_Audio"] += FrenchAudioFrame;

[Myrsloik]

R55-API4-RC4 (https://github.com/vapoursynth/vapoursynth/releases/tag/R55-API4-RC4)

Changes:

Fixes the expr issue causing weird values
Fixes lots of vfw and avfs bugs that snuck into RC3
No longer bundles any filters at all
Comes with a working vsrepo to help you with the previous point
Has a half-reworked documentation with a new theme
AverageFrames is now a core function


Will release an api3 build too with the expr and vsrepo fixes if it's working here.

[Boulder]

The expr issue seems to be fixed, MDSI produces normal scores.

[Yomiko]

How does VSFilterDependency help? If I invoke another filter to get a vnode, should I also include it in the dependency list for filter creation?

[Myrsloik]

How does VSFilterDependency help? If I invoke another filter to get a vnode, should I also include it in the dependency list for filter creation?

If your filter requests frames from the node it should be in the list. Simple rule.