Basic audio support for VapourSynth

[jackoneill]

https://github.com/dubhater/vapoursynth-damb


import vapoursynth as vs

cc = vs.get_core()

#cc.std.LoadPlugin("../libdamb.so")

src = cc.ffms2.Source("7 minutes 50 seconds video.mp4")
src = cc.damb.Read(src, "demuxed audio.wav")

a = src[0:720]
b = src[720:720*2]
c = src[720*2:720*3]
d = src[720*3:720*4]
e = src[720*4:720*5]
f = src[720*5:720*6]
g = src[720*6:]

src = f + d + a + g + c + b + e

src = cc.damb.Write(src, "edited audio.wav")

src.set_output()


I hope someone finds it useful.

For the curious, "damb" means "Dingoes ate my baby" (it's a band). I wanted to call it "ding" (because it deals with audio, right?) but there is already some software with that name.

[Myrsloik]

So it attaches the audio samples to each frame?

[jackoneill]

So it attaches the audio samples to each frame?

Raw audio samples, yes.

[foxyshadis]

Aww, and you could called the writer FriesAreDone() if you'd called it ding. :p

How does it deal with framerate/audiorate mismatches, where a sample overhangs the end or starts late? Or is that more of an "if it hurts, don't do that!" kind of thing? Fudging a few samples on 48kHz audio probably isn't going to hurt anyway...

[jackoneill]

Aww, and you could called the writer FriesAreDone() if you'd called it ding. :p

How does it deal with framerate/audiorate mismatches, where a sample overhangs the end or starts late? Or is that more of an "if it hurts, don't do that!" kind of thing? Fudging a few samples on 48kHz audio probably isn't going to hurt anyway...

It doesn't deal. Write only cares that a frame have audio samples attached. A frame with no audio attached will trigger an error. It will take the audio details (format, sample rate, channel count) from the first frame requested from it and use them to create the output file. It does not make sure that subsequent frames have the same kind of audio. Should it?

Read deals with mismatches in the audio and video lengths by returning an error if the audio is too short. That is, if the audio corresponding to a video frame does not exist at all (past the end of the audio file). Should it attach silence instead? It's okay if the last video frame has a bit fewer audio samples than the rest. If the audio is longer than the video, the extra samples at the end are ignored.

[kolak]

Silence would be good, but only at the end of the file.
If something happens in the middle than it rather should error.

[cretindesalpes]

You should add a frame property indicating the (sub-sample) delay or lead of the audio buffer, even if you don’t use it now. This could help handling mismatch between audio and video frame durations.

[jackoneill]

Here is v2, with a bit nicer behaviour and better documentation: https://github.com/dubhater/vapoursynth-damb/releases/tag/v2


Read:
* avoid desynchronisation due to accumulated rounding errors.

Write:
* reject clips with more than one type of audio
* detect the output format from the extension

[zerowalker]

So wait, can it read Audio from Video without demuxing? (Guessing not)
If so can it detect more tracks?

[tebasuna51]

So wait, can it read Audio from Video without demuxing? (Guessing not)
If so can it detect more tracks?
Not from Video (or container), from a external file with audio previously extracted and decoded:
src = cc.damb.Read(src, "demuxed audio.wav")

[zerowalker]

Ah ok.
But this is the only current way for VapourySynth to get Audio?
No "Avisource" way?

If so that's a bit of a downer, not often i have my tracks demuxed.

[jackoneill]

Ah ok.
But this is the only current way for VapourySynth to get Audio?
No "Avisource" way?

If so that's a bit of a downer, not often i have my tracks demuxed.

Currently, yes. There is no official audio support in VapourSynth yet, so none of the video source filters read the audio.

[zerowalker]

Ah ok too bad, but not a big minus as it's true gain is the Video processing power, hope for Audio though as it speeds things up;P

Thanks:)

[jackoneill]

Here is v3 (https://github.com/dubhater/vapoursynth-damb/releases/tag/v3). It was supposed to include Dissolve(), but I changed my mind about writing that.


Read:
* add support for delay
* return silence instead of aborting in case of errors (like reading past the end of the audio file)
* really reject unsupported formats
* fix detection of the sample type

Read and Write:
* add support for WAV with WAVEFORMATEX

[zerowalker]

Nice, WAVEFORMATEX.
You can look up 64bit WAV as well if you want and it's not supported, quite a forgotten format which is forcefully used when audio is too long:)

[kolak]

What about aiff? Easy to add?

If I add audio and pipe whole thing to ffmpeg will this pass audio?

[jackoneill]

What about aiff? Easy to add?

If I add audio and pipe whole thing to ffmpeg will this pass audio?

libsndfile supports aiff, so it's probably not much work for me. I'll just have to see how to handle the endianness.

No, audio will not be passed to ffmpeg. The audio is attached to video frames as properties. vspipe will only output the video itself. If your operating system has mkfifo, maybe you can avoid writing the audio to a real file.

[kolak]

Hmmm... ok.
Flame uses mkfifo on linux to pass audio to ffmpeg.

[lo1t3yu]

Can it synchronize audio and faulty video frames (delete mismatched duration)?

[jackoneill]

Can it synchronize audio and faulty video frames (delete mismatched duration)?

I have no idea what that means, but probably not.

[FNSCAR]

If video were vfr,would it work?

[jackoneill]

If video were vfr,would it work?

I don't think it would.

[xekon]

Here is v3 (https://github.com/dubhater/vapoursynth-damb/releases/tag/v3). It was supposed to include Dissolve(), but I changed my mind about writing that.


Read:
* add support for delay
* return silence instead of aborting in case of errors (like reading past the end of the audio file)
* really reject unsupported formats
* fix detection of the sample type

Read and Write:
* add support for WAV with WAVEFORMATEX



AWESOME! I have not done any video editing in quite a while, but this is really cool!

Now that there is a way to work with audio in vapoursynth, do you know of any audio waveform viewers for Vapoursynth? (such as audiograph or wavform)

http://avisynth.org.ru/docs/english/externalfilters/audiograph.htm
https://forum.doom9.org/showthread.php?t=165703

This would be useful for fixing sync issues.

[jackoneill]

AWESOME! I have not done any video editing in quite a while, but this is really cool!

Now that there is a way to work with audio in vapoursynth, do you know of any audio waveform viewers for Vapoursynth? (such as audiograph or wavform)

http://avisynth.org.ru/docs/english/externalfilters/audiograph.htm
https://forum.doom9.org/showthread.php?t=165703

This would be useful for fixing sync issues.

I'm not aware of any such plugins. You can hire me to port Waveform, if you want.

[poisondeathray]

If it attaches raw audio samples to frames, would it be possible to pipe to another application like ffmpeg ? or can you bypass the write by some other method ?

[jackoneill]

If it attaches raw audio samples to frames, would it be possible to pipe to another application like ffmpeg ? or can you bypass the write by some other method ?

In Unix-like systems you can use mkfifo. I don't know if Windows has something like that.

[Richard1485]

Many thanks for adding basic audio support for VapourSynth! I have cloned the GitHub and run ./autogen.sh, but upon running ./configure, I receive an error message:


configure: error: Package requirements (sndfile) were not met:

No package 'sndfile' found

I have successfully built libsndfile from source, but either that is not the package in question or it is not being detected.

EDIT: I'm running MX Linux.

Here is v3. It was supposed to include Dissolve(), but I changed my mind about writing that.

A pity! A basic Dissolve() function would have been a welcome addition. :)

[jackoneill]

Many thanks for adding basic audio support for VapourSynth! I have cloned the GitHub and run ./autogen.sh, but upon running ./configure, I receive an error message:



I have successfully built libsndfile from source, but either that is not the package in question or it is not being detected.

EDIT: I'm running MX Linux.


If MX Linux includes all the packages that Debian stable does, then you just need to install the package libsndfile1-dev.

If you must use the version you compiled, then first run this:

find / -name sndfile.pc


Assuming it prints this line:

/usr/local/lib/pkgconfig/sndfile.pc


then rerun configure like this:

PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/ ./configure


Then you can run make.

[Richard1485]

If MX Linux includes all the packages that Debian stable does, then you just need to install the package libsndfile1-dev.

It does. Thanks!

There was mention made earlier in the thread of using mkfifo to do the equivalent of piping the audio. It would be interesting to see how this can be accomplished. Despite having read examples of mkfifo usage online, what this would look like in VapourSynth remains opaque to me.

[jackoneill]

It does. Thanks!

There was mention made earlier in the thread of using mkfifo to do the equivalent of piping the audio. It would be interesting to see how this can be accomplished. Despite having read examples of mkfifo usage online, what this would look like in VapourSynth remains opaque to me.

Create the named pipe:

mkfifo audio.w64


Pass that name to damb.Write.

Insert it somewhere in your ffmpeg command as an input file:

vspipe script.py - --y4m | ffmpeg ... -i audio.w64 ... output.mp4


I assume it goes something like that. I never tried this.