Basic audio support for VapourSynth

[jackoneill]

https://github.com/dubhater/vapoursynth-damb

import vapoursynth as vs

cc = vs.get_core()

#cc.std.LoadPlugin("../libdamb.so")

src = cc.ffms2.Source("7 minutes 50 seconds video.mp4")
src = cc.damb.Read(src, "demuxed audio.wav")

a = src[0:720]
b = src[720:720*2]
c = src[720*2:720*3]
d = src[720*3:720*4]
e = src[720*4:720*5]
f = src[720*5:720*6]
g = src[720*6:]

src = f + d + a + g + c + b + e

src = cc.damb.Write(src, "edited audio.wav")

src.set_output()

I hope someone finds it useful.

For the curious, "damb" means "Dingoes ate my baby" (it's a band). I wanted to call it "ding" (because it deals with audio, right?) but there is already some software with that name.

[Myrsloik]

So it attaches the audio samples to each frame?

[jackoneill]

Originally Posted by Myrsloik

So it attaches the audio samples to each frame?

Raw audio samples, yes.

[foxyshadis]

Aww, and you could called the writer FriesAreDone() if you'd called it ding.

How does it deal with framerate/audiorate mismatches, where a sample overhangs the end or starts late? Or is that more of an "if it hurts, don't do that!" kind of thing? Fudging a few samples on 48kHz audio probably isn't going to hurt anyway...

[jackoneill]

Originally Posted by foxyshadis

Aww, and you could called the writer FriesAreDone() if you'd called it ding.

How does it deal with framerate/audiorate mismatches, where a sample overhangs the end or starts late? Or is that more of an "if it hurts, don't do that!" kind of thing? Fudging a few samples on 48kHz audio probably isn't going to hurt anyway...

It doesn't deal. Write only cares that a frame have audio samples attached. A frame with no audio attached will trigger an error. It will take the audio details (format, sample rate, channel count) from the first frame requested from it and use them to create the output file. It does not make sure that subsequent frames have the same kind of audio. Should it?

Read deals with mismatches in the audio and video lengths by returning an error if the audio is too short. That is, if the audio corresponding to a video frame does not exist at all (past the end of the audio file). Should it attach silence instead? It's okay if the last video frame has a bit fewer audio samples than the rest. If the audio is longer than the video, the extra samples at the end are ignored.

[kolak]

Silence would be good, but only at the end of the file.
If something happens in the middle than it rather should error.

[cretindesalpes]

You should add a frame property indicating the (sub-sample) delay or lead of the audio buffer, even if you don’t use it now. This could help handling mismatch between audio and video frame durations.

[jackoneill]

Here is v2, with a bit nicer behaviour and better documentation: https://github.com/dubhater/vapoursy...eleases/tag/v2

Read:
* avoid desynchronisation due to accumulated rounding errors.

Write:
* reject clips with more than one type of audio
* detect the output format from the extension

[zerowalker]

So wait, can it read Audio from Video without demuxing? (Guessing not)
If so can it detect more tracks?

[tebasuna51]

Originally Posted by zerowalker

So wait, can it read Audio from Video without demuxing? (Guessing not)
If so can it detect more tracks?

Not from Video (or container), from a external file with audio previously extracted and decoded:
src = cc.damb.Read(src, "demuxed audio.wav")

[zerowalker]

Ah ok.
But this is the only current way for VapourySynth to get Audio?
No "Avisource" way?

If so that's a bit of a downer, not often i have my tracks demuxed.

[jackoneill]

Originally Posted by zerowalker

Ah ok.
But this is the only current way for VapourySynth to get Audio?
No "Avisource" way?

If so that's a bit of a downer, not often i have my tracks demuxed.

Currently, yes. There is no official audio support in VapourSynth yet, so none of the video source filters read the audio.

[zerowalker]

Ah ok too bad, but not a big minus as it's true gain is the Video processing power, hope for Audio though as it speeds things up;P

Thanks

[jackoneill]

Here is v3. It was supposed to include Dissolve(), but I changed my mind about writing that.

Read:
   * add support for delay
   * return silence instead of aborting in case of errors (like reading past the end of the audio file)
   * really reject unsupported formats
   * fix detection of the sample type

Read and Write:
   * add support for WAV with WAVEFORMATEX

[zerowalker]

Nice, WAVEFORMATEX.
You can look up 64bit WAV as well if you want and it's not supported, quite a forgotten format which is forcefully used when audio is too long

[kolak]

What about aiff? Easy to add?

If I add audio and pipe whole thing to ffmpeg will this pass audio?

[jackoneill]

Originally Posted by kolak

What about aiff? Easy to add?

If I add audio and pipe whole thing to ffmpeg will this pass audio?

libsndfile supports aiff, so it's probably not much work for me. I'll just have to see how to handle the endianness.

No, audio will not be passed to ffmpeg. The audio is attached to video frames as properties. vspipe will only output the video itself. If your operating system has mkfifo, maybe you can avoid writing the audio to a real file.

[kolak]

Hmmm... ok.
Flame uses mkfifo on linux to pass audio to ffmpeg.

[lo1t3yu]

Can it synchronize audio and faulty video frames (delete mismatched duration)?

[jackoneill]

Originally Posted by lo1t3yu

Can it synchronize audio and faulty video frames (delete mismatched duration)?

I have no idea what that means, but probably not.