VapourSynth - AUDIO SUPPORT AND NEW API BETA

[poisondeathray]

Yes, now alt_output is an optional argument to set_output(). Makes more sense that way.

It "works" , but same problems as last version ie. wrong fourcc for UYVY, avfs crashes



But with Core R55, ffmpeg still reports Y42B, yuv422p for the yuv422 alt2 script .

Also, avfs now crashes when file is accessed, even a plain BlankClip script (without alt_output options) . It "mounts", but as soon as you touch or access the fake avi, it crashes no helpful error message

Calling old avfs with same plain BlankClip script does not crash

[Myrsloik]

RC3 posted. Fixes the reported deadlock and avfs crashes on vpy script access (really tested this time).

[poisondeathray]

RC3 posted. Fixes the reported deadlock and avfs crashes on vpy script access (really tested this time).

avfs and alt_output issues fixed, thanks

Question about vspipe and alt_output - Does the alt_output fourcc emulation only work with avfs ? ffmpeg reports different fourcc with vspipe input using y4m headers (-c y4m), using same script that worked with avfs.

eg.
vspipe -c y4m yuv422_alt2.vpy - | ffmpeg -f yuv4mpegpipe -i - -an -f null NUL
reports Y42B, yuv422p instead of the expected UYVY, uyvy422

But,
avfs yuv422_alt2.vpy
reports the expected UYVY (and it works ok in programs like Premiere, gets YUV treatment, passthough)

(I don't think it matters that much, the alt_output /fourcc probably only be used for avfs - it's just a curiosity question)

[Myrsloik]

avfs and alt_output issues fixed, thanks

Question about vspipe and alt_output - Does the alt_output fourcc emulation only work with avfs ? ffmpeg reports different fourcc with vspipe input using y4m headers (-c y4m), using same script that worked with avfs.

eg.
vspipe -c y4m yuv422_alt2.vpy - | ffmpeg -f yuv4mpegpipe -i - -an -f null NUL
reports Y42B, yuv422p instead of the expected UYVY, uyvy422

But,
avfs yuv422_alt2.vpy
reports the expected UYVY (and it works ok in programs like Premiere, gets YUV treatment, passthough)

(I don't think it matters that much, the alt_output /fourcc probably only be used for avfs - it's just a curiosity question)

You're probably a bit format confused. Y4M is a super basic "container" that only allows planar yuv. Nothing else. It doesn't have the concept of fourcc or packed formats. That's why alt_output does nothing outside of avfs and vfw.

BUT for some reason FFmpeg decides to print out Y42B since that fourcc is effectively the same storage format as Y4M with the supplied headers. Pure coincidence.

[poisondeathray]

You're probably a bit format confused. Y4M is a super basic "container" that only allows planar yuv. Nothing else. It doesn't have the concept of fourcc or packed formats. That's why alt_output does nothing outside of avfs and vfw.


So vspipe rawvideo would not act any differently either ?

[Myrsloik]

So vspipe rawvideo would not act any differently either ?

Correct, there's no reason to. Everything that supports pipe input can accept pure planar nowadays.

[poisondeathray]

Correct, there's no reason to. Everything that supports pipe input can accept pure planar nowadays.

Thanks

[Yomiko]

It seems the VideoOutputTuple confuses existing python scripts that read vpy using runpy. For a video clip, in script.vpy
clip.set_output(2)
In python that handles both old and new api

runpy.run_path(script.vpy, dict(), '__vapoursynth__')
try:
clip = vs.get_output(2)[0]
except TypeError:
clip = vs.get_output(2)
vspreview may be modified to treat it as a tuple. Meanwhile in vseditor I don't see anything different from before.

May I know whether there will be further changes for this, like, backward compatibility for python or more breaking changes for vsscript?

[Myrsloik]

It seems the VideoOutputTuple confuses existing python scripts that read vpy using runpy. For a video clip, in script.vpy
clip.set_output(2)
In python that handles both old and new api

runpy.run_path(script.vpy, dict(), '__vapoursynth__')
try:
clip = vs.get_output(2)[0]
except TypeError:
clip = vs.get_output(2)
vspreview may be modified to treat it as a tuple. Meanwhile in vseditor I don't see anything different from before.

May I know whether there will be further changes for this, like, backward compatibility for python or more breaking changes for vsscript?

I think I'm done with all breaking changes now unless someone discovers something really broken.

I thought about adding an AudioOutputTuple but couldn't think of any additional information that'd be useful.

[Boulder]

Was the support for this old non-recommended style now removed? There are several utility scripts and tools like muvsfunc, mvsfunc and Zopti which use that and need some fixing.

import vapoursynth as vs
core = vs.core

[Myrsloik]

Was the support for this old non-recommended style now removed? There are several utility scripts and tools like muvsfunc, mvsfunc and Zopti which use that and need some fixing.

import vapoursynth as vs
core = vs.core

Uh no? core=vs.core should work

I only removed vs.get_core()

[Boulder]

Uh no? core=vs.core should work

I only removed vs.get_core()

Ah, I prolly looked too quickly and confused the two. The latter one is most likely still in use in a lot of older scripts but fortunately trivial to fix.

[lansing]

I ran into problem calling freeScript() in vsedit2. It kept crashing the program. Then I make a simple test with one button calling createScript() and one button calling freeScript():


VSScript * testVSScriptEnv;

void testCreateScript()
{
if(testVSScriptEnv == nullptr)
testVSScriptEnv = VSScriptAPI->createScript(nullptr);
}

void testFreeScript()
{
if (testVSScriptEnv) {
VSScriptAPI->freeScript(testVSScriptEnv);
}
}


If I go createScript -> freeScript -> createScript -> freeScript, the program will crash.

[Myrsloik]

I ran into problem calling freeScript() in vsedit2. It kept crashing the program. Then I make a simple test with one button calling createScript() and one button calling freeScript():


VSScript * testVSScriptEnv;

void testCreateScript()
{
if(testVSScriptEnv == nullptr)
VSScriptAPI->createScript(nullptr);
}

void testFreeScript()
{
if (testVSScriptEnv) {
VSScriptAPI->freeScript(testVSScriptEnv);
}
}


If I go createScript -> freeScript -> createScript -> freeScript, the program will crash.

Can't reproduce

[lansing]

Can't reproduce

I forgot the "testVSScriptEnv =" line in the above post, can you test again?

[Myrsloik]

I forgot the "testVSScriptEnv =" line in the above post, can you test again?

Obviously I adjusted it for those things. Still worked.

[lansing]

Obviously I adjusted it for those things. Still worked.

I found my problem. I was trying to reuse the temporary node reference. I passed it into a function, and then freed the node inside the function. And then when I free the environment, it will crash.


VSNode * pProcessingNode = m_cpVSAPI->addNodeRef(pOutputNode);
pProcessingNode = resizePlugin(pProcessingNode);
pProcessingNode = otherPlugin(pProcessingNode);

VSNode * resizePlugin (VSNode * a_pVideoNode) {
m_cpVSAPI->mapSetNode(pArgumentMap, "clip", a_pVideoNode, maReplace);
VSMap * pResultMap = m_cpVSAPI->invoke(pResizePlugin, "Point", pArgumentMap);
VSNode * pResultNode = m_cpVSAPI->mapGetNode(pResultMap, "clip", 0, nullptr);

m_cpVSAPI->freeNode(a_pVideoNode);
return pResultNode;
}


What is the correct way to do this?

[Myrsloik]

VSNode * pProcessingNode = m_cpVSAPI->addNodeRef(pOutputNode);
pProcessingNode = resizePlugin(pProcessingNode);
pProcessingNode = otherPlugin(pProcessingNode);

VSNode * resizePlugin (VSNode * a_pVideoNode) {
VSMap *pArgumentMap = m_cpVSAPI->createMap();
m_cpVSAPI->mapConsumeNode(pArgumentMap, "clip", a_pVideoNode, maReplace);
VSMap * pResultMap = m_cpVSAPI->invoke(pResizePlugin, "Point", pArgumentMap);
m_cpVSAPI->freeMap(pArgumentMap);
// use mapGetError(pResultMap) here if you need to check for errors
VSNode * pResultNode = m_cpVSAPI->mapGetNode(pResultMap, "clip", 0, nullptr);
m_cpVSAPI->freeMap(pResultMap);
return pResultNode;
}

At the end you have pProcessingNode and pOutputNode that needs to be freed.

[lansing]

At the end you have pProcessingNode and pOutputNode that needs to be freed.

Thanks I got it working.

[_Al_]

All working here so far, this is nice.
Also attempt to drag audio along with video, here in vapoursynth attaching audio to video frames using props.
But the code below does not work because ModifyFrame does not work with anode. vnode only. ModifyFrame (or FrameEval) is planned in future or is it a bad idea?

oh I see, n is not the same for vnode as is for anode, this could not work, only if bunch of anode frames (perhaps list) is attached to vnode and then reconstructed again, no idea how anisynth does it,

also deleting previous code to not confuse anyone

[_Al_]

testing about these, so it might help someone
API4 audio_test.py script:
import vapoursynth as vs
from vapoursynth import core
source_path=r"video.mp4" #stereo audio
v = core.lsmas.LibavSMASHSource(source_path)
a = core.bas.Source(source_path)
v.set_output(0)
a.set_output(1)
some line here are taken from somewhere else from this forum:
AUDIO
VSPipe -o 1 -c wav audio_tests.py - | neroAacEnc -ignorelength -lc -cbr 96000 -if - -of nero.m4a
VSPipe -o 1 -c wav audio_tests.py - | ffmpeg -y -i - ffmpeg_output.wav
VSPipe -o 1 audio_tests.py - | ffmpeg -y -f f32le -ac 2 -ar 48000 -i - ffmpeg_output.wav
VSPipe -o 1 -c wav audio_tests.py vspipe_to_wav.wav #but double size (32bit if original is 16bit)
VSPipe -o 1 audio_tests.py vspipe_to_wav.wav #but double size (32bit if original is 16bit)
VSPipe -o 1 -c wav audio_tests.py - | ffplay -
VSPipe -o 1 audio_tests.py - | ffplay -f f32le -ac 2 -ar 48000 -i -
VSPipe -o 1 -c wav audio_tests.py - | ffmpeg -i video.mp4 -i - -map 0:0 -map 1:0 -f AVI -c:v utvideo -pix_fmt yuv420p -colorspace bt709 -c:a pcm_s16le -y ffmpeg_utvideo.avi

VIDEO
VSPipe -o 0 -c y4m audio_tests.py - | ffplay -i -
VSPipe -o 0 -c y4m audio_tests.py - | mpv -
btw. can be two nodes (vnode,anode) piped into ffmpeg or ffplay in one line?

[Myrsloik]

No, to do that vspipe needs to wrap the streams into a more complex container like mkv.

[Myrsloik]

All working here so far, this is nice.
Also attempt to drag audio along with video, here in vapoursynth attaching audio to video frames using props.
But the code below does not work because ModifyFrame does not work with anode. vnode only. ModifyFrame (or FrameEval) is planned in future or is it a bad idea?

oh I see, n is not the same for vnode as is for anode, this could not work, only if bunch of anode frames (perhaps list) is attached to vnode and then reconstructed again, no idea how anisynth does it,

also deleting previous code to not confuse anyone

Frame props are nonsensical since audio frames are a very arbitrary thing. That's why I haven't really exposed the manipulation and I want audio support to mostly allow you to pass audio through in a convenient way.

I think what you should do it simply create wrapper functions for the trim, splice and other reorder functions and use those. Most filters don't reorder the clip. That would be my first attempt. I'll write an example some day of it.

Avisynth has a weird symbiotic relationship between one audio track and one video track. Have more than one audio track and things get iffy.

[feisty2]

you can attach audio frame(s), possibly audio frames from multiple audio tracks, to a video frame using the C++ API, it should be no more complicated than

auto [PictureFrame, EngAudioFrame, FrenchAudioFrame] = Node::AcquireFrameGiven(Index, GeneratorContex).From(Video, EngAudio, FrenchAudio);
auto CompositeFrame = Core.CopyFrameFrom(PictureFrame);
CompositeFrame["_Audio"] += EngAudioFrame;
CompositeFrame["_Audio"] += FrenchAudioFrame;

[Myrsloik]

R55-API4-RC4 (https://github.com/vapoursynth/vapoursynth/releases/tag/R55-API4-RC4)

Changes:

Fixes the expr issue causing weird values
Fixes lots of vfw and avfs bugs that snuck into RC3
No longer bundles any filters at all
Comes with a working vsrepo to help you with the previous point
Has a half-reworked documentation with a new theme
AverageFrames is now a core function


Will release an api3 build too with the expr and vsrepo fixes if it's working here.

[Boulder]

The expr issue seems to be fixed, MDSI produces normal scores.

[Yomiko]

How does VSFilterDependency help? If I invoke another filter to get a vnode, should I also include it in the dependency list for filter creation?

[Myrsloik]

How does VSFilterDependency help? If I invoke another filter to get a vnode, should I also include it in the dependency list for filter creation?

If your filter requests frames from the node it should be in the list. Simple rule.