Audio Development

[forgetfool]

As far as I understand, BAS is Windows only solution, right?

Is there a Linux audio plugin for VapourSynth? "vs.core.damb.Read()" seems to have no effect, i.e. at does not attach sound to video.

[feisty2]

Video and audio are separate entities in vaporsynth. You may define an av container that associates an audio node with a video node, but you cannot attach an audio node to a video node.

[forgetfool]

As for the first part of the question, someone mentioned that I need to manually compile BAS (that it works for them on Ubuntu).

As for the second part, about DAMB, ok, I understand the point, but how would you actually produce a video with a sound? What is wrong with the code below? I feel I am still missing some basic understanding.

import vapoursynth as vs
src = vs.core.ffms2.Source(video.avi')
src = vs.core.damb.Read(src, "audio.wav")
src.set_output()

I am reading the documentation nere: https://github.com/dubhater/vapoursynth-damb

[feisty2]

class Muxer:
    def Render(self):
        self.VideoStream.set_output(0)
        self.AudioStream.set_output(1)

av = Muxer()
av.VideoStream = core.ffms2.Source('video.mov')
av.AudioStream = core.bas.Source('audio.wav')

av.Render()

edit:
and damb is obsolete (it is based on API v3 which has no native support for audio), don't use it.

[jackoneill]

Originally Posted by forgetfool

As for the second part, about DAMB, ok, I understand the point, but how would you actually produce a video with a sound? What is wrong with the code below? I feel I am still missing some basic understanding.

import vapoursynth as vs
src = vs.core.ffms2.Source(video.avi')
src = vs.core.damb.Read(src, "audio.wav")
src.set_output()

I am reading the documentation nere: https://github.com/dubhater/vapoursynth-damb

It's useful if you want to trim/splice the audio at the same time as the video. You use Read right after your video source filter, trim and splice the video as needed, then at the end of the script you use Write to export the audio to a file. vspipe will still output just the video. If you use only Read then the audio will be read from the file and nothing else happens.

[forgetfool]

Ok, so you are saying vspipe will never produce audio with video together, right?

I am now trying to use BestAudioSource plugin instad of DAMB (without much success either) -- see this thread: https://forum.doom9.org/showthread.p...=1#post1939019

[jackoneill]

Originally Posted by forgetfool

Ok, so you are saying vspipe will never produce audio with video together, right?

I am now trying to use BestAudioSource plugin instad of DAMB (without much success either) -- see this thread: https://forum.doom9.org/showthread.p...=1#post1939019

Maybe with BestAudioSource it will. I'm not sure how it's supposed to work.

[DJATOM]

BAS uses AudioNode entity to represent audio data. Literally it's the same as VideoNode, but for audio clips. To draw data from it, user has to assign node to certain output, for example, audio.set_output(1) - assuming that your audio clip named audio.

[poisondeathray]

Can you release an updated audio build when you get the chance ?

Thanks

[xekon]

Originally Posted by DJATOM

I've done some modifications to make possible to add few Trim instances with each other. The code is now looks like that

from vapoursynth import core, VideoNode, AudioNode
from typing import Sequence, Union


class Trim:
    def __init__(self, clip: Union[AudioNode, Sequence], start: int = None, end: int = None, fps: Sequence = None) -> Union[AudioNode, tuple]:
        """ Convenient wrapper for trimming audio samples by frame numbers """
        if not start and not end:
            raise ValueError('Trim: missing "start" and/or "end" options.')

        self.vclip, self.aclip = None, None
        self.mode = 'audio'

        if isinstance(clip, Sequence) and len(clip) == 2:
            self.mode = 'video_audio'
            self.vclip, self.aclip = clip
            assert isinstance(
                self.vclip, VideoNode), 'Trim: first clip must be video.'
            assert isinstance(
                self.aclip, AudioNode), 'Trim: second clip must be audio.'
            self.vclip = core.std.Trim(self.vclip, start, end)

        elif isinstance(clip, AudioNode):
            self.aclip = clip
            if not fps:
                raise ValueError('Trim: missing "fps" option.')
        else:
            raise ValueError(
                'Trim: clip must be audio type or sequence with video and audio clips.')
        if isinstance(self.vclip, VideoNode) and not (self.vclip.fps.numerator == 0 and self.vclip.fps.denominator == 0) and fps is None:
            fps = (self.vclip.fps.numerator, self.vclip.fps.denominator)
        samples_per_video_frame = self.aclip.sample_rate / fps[0] * fps[1]
        audio_start, audio_end = start * \
            samples_per_video_frame, end * samples_per_video_frame
        self.aclip = core.std.AudioTrim(self.aclip, audio_start, audio_end)

    def __add__(self, other):
        if not isinstance(other, Trim):
            raise ValueError(f'Trim: impossible to operate on non-Trim inputs.')
        if self.mode == 'video_audio' and other.mode != 'video_audio':
            raise ValueError(
                'Trim: add failed. Probably you\'re mixing inconsistent instances (audio + video and audio or so).')
        if self.mode == 'video_audio':
            self.vclip += other.vclip
        self.aclip += other.aclip
        return self

    def data(self):
        if self.mode == 'video_audio':
            return self.vclip, self.aclip
        else:
            return self.aclip

But with class rewrite we need to call method data() to return clips

vclip, aclip = (Trim((vclip, aclip), 18544, 18582) + Trim((vclip, aclip), 18687, 18724) + Trim((vclip, aclip), 18792, 18861)).data()

I tried to find a way to implicitly return data on assignment, but it seems that's too tricky.

WORKS GREAT! I was ripping my hair out trying to figure out how to encode some video and audio using trim/audiotrim in vapoursynth, the ONLY example I could find for working with the audio in vapoursynth was on this page but it does not include how to trim in the example:

https://www.vapoursynth.com/2020/01/...-how-it-works/

This function is excellent and in my opinion needs more exposure, just try searching google for this:

Vapoursynth AudioTrim Example (spoiler you wont find this thread)

I ended up searching for

Vapoursynth "bas.Source"

Which still did not find this thread, but it did find this one https://forum.doom9.org/showthread.php?t=177337 in which somebody linked to this thread, so I finally found it.

What I think would be absolutely killer is if all the functions had a small minimal example script of how to use it, even better is if these examples were right on https://vsdb.top/ so when you search for a function there you can find a practical example of how to use it, this part of the site could also be community driven like a wiki...

Now that I have it working I am very very very pleased with this Audio support and the ability to do all of my processing with VapourSynth, its absolutely wonderful!

[xekon]

trim.vpy:

import vapoursynth as vs
from typing import Sequence, Union
from vapoursynth import core, VideoNode, AudioNode

class Trim:
    def __init__(self, clip: Union[AudioNode, Sequence], start: int = None, end: int = None, fps: Sequence = None) -> Union[AudioNode, tuple]:
        """ Convenient wrapper for trimming audio samples by frame numbers """
        if not start and not end:
            raise ValueError('Trim: missing "start" and/or "end" options.')

        self.vclip, self.aclip = None, None
        self.mode = 'audio'

        if isinstance(clip, Sequence) and len(clip) == 2:
            self.mode = 'video_audio'
            self.vclip, self.aclip = clip
            assert isinstance(
                self.vclip, VideoNode), 'Trim: first clip must be video.'
            assert isinstance(
                self.aclip, AudioNode), 'Trim: second clip must be audio.'
            self.vclip = core.std.Trim(self.vclip, start, end)

        elif isinstance(clip, AudioNode):
            self.aclip = clip
            if not fps:
                raise ValueError('Trim: missing "fps" option.')
        else:
            raise ValueError(
                'Trim: clip must be audio type or sequence with video and audio clips.')
        if isinstance(self.vclip, VideoNode) and not (self.vclip.fps.numerator == 0 and self.vclip.fps.denominator == 0) and fps is None:
            fps = (self.vclip.fps.numerator, self.vclip.fps.denominator)
        samples_per_video_frame = self.aclip.sample_rate / fps[0] * fps[1]
        audio_start, audio_end = start * \
            samples_per_video_frame, end * samples_per_video_frame
        self.aclip = core.std.AudioTrim(self.aclip, audio_start, audio_end)

    def __add__(self, other):
        if not isinstance(other, Trim):
            raise ValueError(f'Trim: impossible to operate on non-Trim inputs.')
        if self.mode == 'video_audio' and other.mode != 'video_audio':
            raise ValueError(
                'Trim: add failed. Probably you\'re mixing inconsistent instances (audio + video and audio or so).')
        if self.mode == 'video_audio':
            self.vclip += other.vclip
        self.aclip += other.aclip
        return self

    def data(self):
        if self.mode == 'video_audio':
            return self.vclip, self.aclip
        else:
            return self.aclip

core = vs.core
video = core.ffms2.Source(source='/media/storage/vpy/over9000.mkv')
audio = core.bas.Source(source='/media/storage/vpy/over9000.flac', track=-1)

video, audio = (Trim((video, audio), 28406, 28491)).data()
#video, audio = (Trim((video, audio), 18544, 18582) + Trim((video, audio), 18687, 18724) + Trim((video, audio), 18792, 18861)).data()

video.set_output(0)
audio.set_output(1)

encode audio:

vspipe -o 1 -c wav /media/storage/vpy/trim.vpy - | ffmpeg -i pipe: -c:a aac -b:a 128k -ac 1 /media/storage/vpy/audio.aac
vspipe -o 1 -c wav /media/storage/vpy/trim.vpy - | ffmpeg -i pipe: -c:a libopus -b:a 96k -ac 1 /media/storage/vpy/audio.opus

encode video with previously encoded audio:

vspipe -o 0 -c y4m /media/storage/vpy/trim.vpy - | ffmpeg -i pipe: -i /media/storage/vpy/audio.aac -c:v libx264 -threads 1 -profile:v High \
 -level 4.1 -preset veryslow -crf 18 -x264-params "ref=6:bframes=16:merange=32:fast-pskip=0" -c:a copy "/media/storage/vpy/trim-x264.mkv"
vspipe -o 0 -c y4m /media/storage/vpy/trim.vpy - | ffmpeg -i pipe: -i /media/storage/vpy/audio.aac -c:v librav1e -qp 50 -speed 2 \
 -tiles 1 -tile-columns 1 -tile-rows 1 -c:a copy "/media/storage/vpy/trim-av1.mkv"

remove temporary audio files:

rm -rf /media/storage/vpy/audio.aac /media/storage/vpy/audio.opus

vmaf check:

ffmpeg -i /media/storage/vpy/trim-x264.mkv -i /media/storage/vpy/trim-av1.mkv -lavfi libvmaf='feature=name=psnr|name=ciede' -f null -

[Myrsloik]

I guess some easy way to trim audio along with video would be useful. My proposal is this:
Extend Trim like this:

std.Trim(vnode clip[, int first=0, int last, int length, anode[] audio])

If audio clips are supplied then the output in python will be a dict

{clip: <video here>, audio: <array of trimmed audio clips>}

[xekon]

A built in and officially supported method of trimming both the audio and the video with one or more range of frames would be wonderful!

I am sure however you decide to do it will be Awesome!

[PatchWorKs]

Very interesting.

Do you think some Python Audio Restoration Suite functions can be "ported" to VS ?

From thair git:

A set of tools to restore audio quality from a variety of old analog sources, such as tape, cassettes, acetates and vinyl.

Features

• Wow & Flutter Removal
• Speed matching to hum frequency
• EQ matching with differential EQ
• Spectral Temporal Alignment
• Automatic Dropout Restoration
• Spectral Expander / Decompressor

Hope that inspires !

[unix]

Originally Posted by Myrsloik

I guess some easy way to trim audio along with video would be useful. My proposal is this:
Extend Trim like this:

std.Trim(vnode clip[, int first=0, int last, int length, anode[] audio])

If audio clips are supplied then the output in python will be a dict

{clip: <video here>, audio: <array of trimmed audio clips>}

Can you elaborate and give us an example