Audio Development

[tebasuna51]

I don't want remove, I want insert silence before.

[TheFluff]

If you want to add a second of silence, you can create silence with BlankAudio() and splice. If you want an actual delay, as in you want to keep the audio samples the same but adjust their timing relative to the video, I think you'd need to manipulate the video timestamps instead (shifting all frame times backwards by one second). Hypothetically you could do that by manipulating the frameprops, but I don't think that's actually a supported thing. Video timestamps don't have to start at 0 in VS and they can be negative (since that happens in the real world, e.g. in MPEG TS where the timestamps just wrap around), so that's not a problem, but the docs say that only source filters are supposed to touch the _AbsoluteTime prop. I don't know if it'd actually work even if you were allowed to change it either - I'm just theorizing here.

e: the new VS API (V4) doesn't seem to have timing info for audio, it's just a bunch of samples AFAICT. BestAudioSource has some comments indicating that exposing an audio track's delay relative to a video track (or relative starting point) is or was a planned feature though.

[tebasuna51]

BlankAudio() is not a function documented, there are BlankClip but not BlankAudio.
With my method we only need know the samplerate to emulate the avisynth DelayAudio().
We can work with samples.

I only want make a 'Avisynth Audio functions and their VapourSynth equivalents (https://forum.doom9.org/showthread.php?p=1917206#post1917206)' table like there are in the /doc for video.

[TheFluff]

For reference BlankAudio is here (https://github.com/vapoursynth/vapoursynth/blob/doodle1/src/core/audiofilters.cpp#L868) in the source code.

My post was just a suggestion, how to accomplish what you were asking for with the currently existing tools. I think your idea of a DelayAudio-like filter is a reasonable one, but I'm not the person to talk to regarding that.

[DJATOM]

def DelayAudio(aclip: vs.AudioNode, delay: float = None):
if not delay:
raise ValueError("DelayAudio: missing delay value.")
sps = aclip.sample_rate
delay = delay * sps
return aclip[delay:]

Close enough?

[TheFluff]

def DelayAudio(aclip: vs.AudioNode, delay: float = None):
if not delay:
raise ValueError("DelayAudio: missing delay value.")
sps = aclip.sample_rate
delay = delay * sps
return aclip[delay:]

Close enough?

I don't think so? that only works in one direction (making the audio start earlier by cutting off the start) and it's also inverted compared to avisynth - setting delay to 1 will make the audio start one second earlier, not one second later like in avisynth. what tebasuna wants is delayaudio(1) to add 1 second of silence to the start of the track and delayaudio(-1) to do what your function does when you say delayaudio(1) - that is, cut the first second off the track

[Myrsloik]

This all reminds me to finish implementing BlankAudio and documenting it.

[DJATOM]

I don't think so? that only works in one direction (making the audio start earlier by cutting off the start) and it's also inverted compared to avisynth - setting delay to 1 will make the audio start one second earlier, not one second later like in avisynth. what tebasuna wants is delayaudio(1) to add 1 second of silence to the start of the track and delayaudio(-1) to do what your function does when you say delayaudio(1) - that is, cut the first second off the track

Sure, so delay must be inverted before slice operation.

[Myrsloik]

Thanks for your updates.

I have time to test now and I want explain the changes I see.

1) The command line I use now is:

VSPipe.exe -c wav zzaud.vpy zzD1sec.wav

Before vspipe need --wav

The zzaud.vpy I use is:


2) Now access to properties in a different way

And I obtain a output of:


3) How Use logging.info instead of print?
I obtain the messages but some duplicated and with the undesired prefix "Information: "

- Like you can see I try to emulate a delay of 48000 samples, or 1 second, maybe that can be abreviated with:
c = core.std.AudioTrim(-48000)

- In /doc I don't see any change with the precedent version.

Logging: core.logMessage(vs.MESSAGE_TYPE_INFORMATION, 'hello there')

I'll probably have the warning removed since I find it more annoying than helpful. Keep using print() if it works for you.

One second audio delay:
clip[0]*clip.sample_rate+clip

[tebasuna51]

Thanks for your answer.

[l33tmeatwad]

Seems like BestAudioSource repository got some updates but VapourSynth4.h is out of date and by just adding in the portion it's complaining about, nfMakeLinear, it will compile but Source will not show as a function (although bas registers).

[Myrsloik]

Seems like BestAudioSource repository got some updates but VapourSynth4.h is out of date and by just adding in the portion it's complaining about, nfMakeLinear, it will compile but Source will not show as a function (although bas registers).

I plan to make a proper RC-ish build soon. Just got distracted by actual paid work as usual. There are also a few annoying/boring issues left to fix.

[l33tmeatwad]

Cool, thanks for the update, I'll keep an eye out for it.

[tbjm]

I'm trying to import the CCD VirtualDub filter, but I'm encountering some errors.

core.avs.LoadPlugin(r"C:\vsplugins\VDubFilter.dll")
core.avs.LoadVirtualdubPlugin(r"C:\vdubplugins\ccd_64bit.vdf", 'CCD', 0)

Works fine using R52, but when using R51-audio5 I get the following error.

API MISUSE! Function 'CCD' failed to register with error: Argument 'c1' has invalid type 'clip'.
API MISUSE! Function 'CCD' failed to register with error: Argument 'c1' has invalid type 'clip'.
API MISUSE! Function 'CCD_2' failed to register with error: Argument 'c1' has invalid type 'clip'.
API MISUSE! Function 'CCD_2' failed to register with error: Argument 'c1' has invalid type 'clip'.

Possibly related to https://forum.doom9.org/showpost.php?p=1919973&postcount=93 ?

[unix]

There's no way to trim/cut the Video and audio in one/same script?!

[DJATOM]

Something like that should work
from vapoursynth import core, VideoNode, AudioNode
from typing import Sequence, Union

def Trim(clip: Union[AudioNode, Sequence], start: int = None, end: int = None, fps: Sequence = None) -> Union[AudioNode, tuple]:
if not start and not end:
raise ValueError('Trim: missing "start" and/or "end" options.')
vclip, aclip = None, None
if isinstance(clip, Sequence) and len(clip) == 2:
vclip, aclip = clip
assert isinstance(
vclip, VideoNode), 'Trim: first clip must be video type.'
assert isinstance(
aclip, AudioNode), 'Trim: second clip must be audio type.'
vclip = core.std.Trim(vclip, start, end)
elif isinstance(clip, AudioNode):
aclip = clip
if not fps:
raise ValueError('Trim: missing "fps" option.')
else:
raise ValueError(
'Trim: clip must be audio type or sequence with video and audio clips.')
if isinstance(vclip, VideoNode) and not (vclip.fps.numerator == 0 and vclip.fps.denominator == 0) and fps is None:
fps = (vclip.fps.numerator, vclip.fps.denominator)
samples_per_video_frame = aclip.sample_rate / fps[0] * fps[1]
audio_start, audio_end = start * \
samples_per_video_frame, end * samples_per_video_frame
aclip = core.std.AudioTrim(aclip, audio_start, audio_end)
return aclip if not vclip else (vclip, aclip)

And how to use
vclip = core.dgdecodenv.DGSource(r'....dgi')
aclip = core.bas.Source(r'....wav', track=0)
vclip, aclip = Trim((vclip, aclip), 18544, 18568)
vclip.set_output(0)
aclip.set_output(1)

[unix]

Thank you, but if I want to trim many parts how it gonna be?!

like:
vclip, aclip = Trim((vclip, aclip), 18544, 18568).Trim(20000,240000)

?

[DJATOM]

No, it probably will not work that way. But you could test Trim((vclip, aclip), 18544, 18568) + Trim((vclip, aclip), 20000, 240000), that might work.

[DJATOM]

I've done some modifications to make possible to add few Trim instances with each other. The code is now looks like that
from vapoursynth import core, VideoNode, AudioNode
from typing import Sequence, Union


class Trim:
def __init__(self, clip: Union[AudioNode, Sequence], start: int = None, end: int = None, fps: Sequence = None) -> Union[AudioNode, tuple]:
""" Convenient wrapper for trimming audio samples by frame numbers """
if not start and not end:
raise ValueError('Trim: missing "start" and/or "end" options.')

self.vclip, self.aclip = None, None
self.mode = 'audio'

if isinstance(clip, Sequence) and len(clip) == 2:
self.mode = 'video_audio'
self.vclip, self.aclip = clip
assert isinstance(
self.vclip, VideoNode), 'Trim: first clip must be video.'
assert isinstance(
self.aclip, AudioNode), 'Trim: second clip must be audio.'
self.vclip = core.std.Trim(self.vclip, start, end)

elif isinstance(clip, AudioNode):
self.aclip = clip
if not fps:
raise ValueError('Trim: missing "fps" option.')
else:
raise ValueError(
'Trim: clip must be audio type or sequence with video and audio clips.')
if isinstance(self.vclip, VideoNode) and not (self.vclip.fps.numerator == 0 and self.vclip.fps.denominator == 0) and fps is None:
fps = (self.vclip.fps.numerator, self.vclip.fps.denominator)
samples_per_video_frame = self.aclip.sample_rate / fps[0] * fps[1]
audio_start, audio_end = start * \
samples_per_video_frame, end * samples_per_video_frame
self.aclip = core.std.AudioTrim(self.aclip, audio_start, audio_end)

def __add__(self, other):
if not isinstance(other, Trim):
raise ValueError(f'Trim: impossible to operate on non-Trim inputs.')
if self.mode == 'video_audio' and other.mode != 'video_audio':
raise ValueError(
'Trim: add failed. Probably you\'re mixing inconsistent instances (audio + video and audio or so).')
if self.mode == 'video_audio':
self.vclip += other.vclip
self.aclip += other.aclip
return self

def data(self):
if self.mode == 'video_audio':
return self.vclip, self.aclip
else:
return self.aclip

But with class rewrite we need to call method data() to return clips
vclip, aclip = (Trim((vclip, aclip), 18544, 18582) + Trim((vclip, aclip), 18687, 18724) + Trim((vclip, aclip), 18792, 18861)).data()
I tried to find a way to implicitly return data on assignment, but it seems that's too tricky.

[unix]

Thank you DJ
I will try it later

[forgetfool]

As far as I understand, BAS is Windows only solution, right?

Is there a Linux audio plugin for VapourSynth? "vs.core.damb.Read()" seems to have no effect, i.e. at does not attach sound to video.

[feisty2]

Video and audio are separate entities in vaporsynth. You may define an av container that associates an audio node with a video node, but you cannot attach an audio node to a video node.

[forgetfool]

As for the first part of the question, someone mentioned that I need to manually compile BAS (that it works for them on Ubuntu).

As for the second part, about DAMB, ok, I understand the point, but how would you actually produce a video with a sound? What is wrong with the code below? I feel I am still missing some basic understanding.

import vapoursynth as vs
src = vs.core.ffms2.Source(video.avi')
src = vs.core.damb.Read(src, "audio.wav")
src.set_output()

I am reading the documentation nere: https://github.com/dubhater/vapoursynth-damb

[feisty2]

class Muxer:
def Render(self):
self.VideoStream.set_output(0)
self.AudioStream.set_output(1)

av = Muxer()
av.VideoStream = core.ffms2.Source('video.mov')
av.AudioStream = core.bas.Source('audio.wav')

av.Render()


edit:
and damb is obsolete (it is based on API v3 which has no native support for audio), don't use it.

[jackoneill]

As for the second part, about DAMB, ok, I understand the point, but how would you actually produce a video with a sound? What is wrong with the code below? I feel I am still missing some basic understanding.

import vapoursynth as vs
src = vs.core.ffms2.Source(video.avi')
src = vs.core.damb.Read(src, "audio.wav")
src.set_output()

I am reading the documentation nere: https://github.com/dubhater/vapoursynth-damb

It's useful if you want to trim/splice the audio at the same time as the video. You use Read right after your video source filter, trim and splice the video as needed, then at the end of the script you use Write to export the audio to a file. vspipe will still output just the video. If you use only Read then the audio will be read from the file and nothing else happens.

[forgetfool]

Ok, so you are saying vspipe will never produce audio with video together, right?

I am now trying to use BestAudioSource plugin instad of DAMB (without much success either) -- see this thread: https://forum.doom9.org/showthread.php?p=1939019&posted=1#post1939019

[jackoneill]

Ok, so you are saying vspipe will never produce audio with video together, right?

I am now trying to use BestAudioSource plugin instad of DAMB (without much success either) -- see this thread: https://forum.doom9.org/showthread.php?p=1939019&posted=1#post1939019

Maybe with BestAudioSource it will. I'm not sure how it's supposed to work.

[DJATOM]

BAS uses AudioNode entity to represent audio data. Literally it's the same as VideoNode, but for audio clips. To draw data from it, user has to assign node to certain output, for example, audio.set_output(1) - assuming that your audio clip named audio.

[poisondeathray]

Can you release an updated audio build when you get the chance ?

Thanks

[xekon]

I've done some modifications to make possible to add few Trim instances with each other. The code is now looks like that
from vapoursynth import core, VideoNode, AudioNode
from typing import Sequence, Union


class Trim:
def __init__(self, clip: Union[AudioNode, Sequence], start: int = None, end: int = None, fps: Sequence = None) -> Union[AudioNode, tuple]:
""" Convenient wrapper for trimming audio samples by frame numbers """
if not start and not end:
raise ValueError('Trim: missing "start" and/or "end" options.')

self.vclip, self.aclip = None, None
self.mode = 'audio'

if isinstance(clip, Sequence) and len(clip) == 2:
self.mode = 'video_audio'
self.vclip, self.aclip = clip
assert isinstance(
self.vclip, VideoNode), 'Trim: first clip must be video.'
assert isinstance(
self.aclip, AudioNode), 'Trim: second clip must be audio.'
self.vclip = core.std.Trim(self.vclip, start, end)

elif isinstance(clip, AudioNode):
self.aclip = clip
if not fps:
raise ValueError('Trim: missing "fps" option.')
else:
raise ValueError(
'Trim: clip must be audio type or sequence with video and audio clips.')
if isinstance(self.vclip, VideoNode) and not (self.vclip.fps.numerator == 0 and self.vclip.fps.denominator == 0) and fps is None:
fps = (self.vclip.fps.numerator, self.vclip.fps.denominator)
samples_per_video_frame = self.aclip.sample_rate / fps[0] * fps[1]
audio_start, audio_end = start * \
samples_per_video_frame, end * samples_per_video_frame
self.aclip = core.std.AudioTrim(self.aclip, audio_start, audio_end)

def __add__(self, other):
if not isinstance(other, Trim):
raise ValueError(f'Trim: impossible to operate on non-Trim inputs.')
if self.mode == 'video_audio' and other.mode != 'video_audio':
raise ValueError(
'Trim: add failed. Probably you\'re mixing inconsistent instances (audio + video and audio or so).')
if self.mode == 'video_audio':
self.vclip += other.vclip
self.aclip += other.aclip
return self

def data(self):
if self.mode == 'video_audio':
return self.vclip, self.aclip
else:
return self.aclip

But with class rewrite we need to call method data() to return clips
vclip, aclip = (Trim((vclip, aclip), 18544, 18582) + Trim((vclip, aclip), 18687, 18724) + Trim((vclip, aclip), 18792, 18861)).data()
I tried to find a way to implicitly return data on assignment, but it seems that's too tricky.

WORKS GREAT! I was ripping my hair out trying to figure out how to encode some video and audio using trim/audiotrim in vapoursynth, the ONLY example I could find for working with the audio in vapoursynth was on this page but it does not include how to trim in the example:

https://www.vapoursynth.com/2020/01/audio-support-and-how-it-works/

This function is excellent and in my opinion needs more exposure, just try searching google for this:

Vapoursynth AudioTrim Example (spoiler you wont find this thread)

I ended up searching for

Vapoursynth "bas.Source"

Which still did not find this thread, but it did find this one https://forum.doom9.org/showthread.php?t=177337 in which somebody linked to this thread, so I finally found it.

What I think would be absolutely killer is if all the functions had a small minimal example script of how to use it, even better is if these examples were right on https://vsdb.top/ so when you search for a function there you can find a practical example of how to use it, this part of the site could also be community driven like a wiki...

Now that I have it working I am very very very pleased with this Audio support and the ability to do all of my processing with VapourSynth, its absolutely wonderful!

[xekon]

trim.vpy:


import vapoursynth as vs
from typing import Sequence, Union
from vapoursynth import core, VideoNode, AudioNode

class Trim:
def __init__(self, clip: Union[AudioNode, Sequence], start: int = None, end: int = None, fps: Sequence = None) -> Union[AudioNode, tuple]:
""" Convenient wrapper for trimming audio samples by frame numbers """
if not start and not end:
raise ValueError('Trim: missing "start" and/or "end" options.')

self.vclip, self.aclip = None, None
self.mode = 'audio'

if isinstance(clip, Sequence) and len(clip) == 2:
self.mode = 'video_audio'
self.vclip, self.aclip = clip
assert isinstance(
self.vclip, VideoNode), 'Trim: first clip must be video.'
assert isinstance(
self.aclip, AudioNode), 'Trim: second clip must be audio.'
self.vclip = core.std.Trim(self.vclip, start, end)

elif isinstance(clip, AudioNode):
self.aclip = clip
if not fps:
raise ValueError('Trim: missing "fps" option.')
else:
raise ValueError(
'Trim: clip must be audio type or sequence with video and audio clips.')
if isinstance(self.vclip, VideoNode) and not (self.vclip.fps.numerator == 0 and self.vclip.fps.denominator == 0) and fps is None:
fps = (self.vclip.fps.numerator, self.vclip.fps.denominator)
samples_per_video_frame = self.aclip.sample_rate / fps[0] * fps[1]
audio_start, audio_end = start * \
samples_per_video_frame, end * samples_per_video_frame
self.aclip = core.std.AudioTrim(self.aclip, audio_start, audio_end)

def __add__(self, other):
if not isinstance(other, Trim):
raise ValueError(f'Trim: impossible to operate on non-Trim inputs.')
if self.mode == 'video_audio' and other.mode != 'video_audio':
raise ValueError(
'Trim: add failed. Probably you\'re mixing inconsistent instances (audio + video and audio or so).')
if self.mode == 'video_audio':
self.vclip += other.vclip
self.aclip += other.aclip
return self

def data(self):
if self.mode == 'video_audio':
return self.vclip, self.aclip
else:
return self.aclip

core = vs.core
video = core.ffms2.Source(source='/media/storage/vpy/over9000.mkv')
audio = core.bas.Source(source='/media/storage/vpy/over9000.flac', track=-1)

video, audio = (Trim((video, audio), 28406, 28491)).data()
#video, audio = (Trim((video, audio), 18544, 18582) + Trim((video, audio), 18687, 18724) + Trim((video, audio), 18792, 18861)).data()

video.set_output(0)
audio.set_output(1)



encode audio:

vspipe -o 1 -c wav /media/storage/vpy/trim.vpy - | ffmpeg -i pipe: -c:a aac -b:a 128k -ac 1 /media/storage/vpy/audio.aac
vspipe -o 1 -c wav /media/storage/vpy/trim.vpy - | ffmpeg -i pipe: -c:a libopus -b:a 96k -ac 1 /media/storage/vpy/audio.opus


encode video with previously encoded audio:

vspipe -o 0 -c y4m /media/storage/vpy/trim.vpy - | ffmpeg -i pipe: -i /media/storage/vpy/audio.aac -c:v libx264 -threads 1 -profile:v High \
-level 4.1 -preset veryslow -crf 18 -x264-params "ref=6:bframes=16:merange=32:fast-pskip=0" -c:a copy "/media/storage/vpy/trim-x264.mkv"
vspipe -o 0 -c y4m /media/storage/vpy/trim.vpy - | ffmpeg -i pipe: -i /media/storage/vpy/audio.aac -c:v librav1e -qp 50 -speed 2 \
-tiles 1 -tile-columns 1 -tile-rows 1 -c:a copy "/media/storage/vpy/trim-av1.mkv"


remove temporary audio files:

rm -rf /media/storage/vpy/audio.aac /media/storage/vpy/audio.opus


vmaf check:

ffmpeg -i /media/storage/vpy/trim-x264.mkv -i /media/storage/vpy/trim-av1.mkv -lavfi libvmaf='feature=name=psnr|name=ciede' -f null -

[Myrsloik]

I guess some easy way to trim audio along with video would be useful. My proposal is this:
Extend Trim like this:
std.Trim(vnode clip[, int first=0, int last, int length, anode[] audio])
If audio clips are supplied then the output in python will be a dict
{clip: <video here>, audio: <array of trimmed audio clips>}

[xekon]

A built in and officially supported method of trimming both the audio and the video with one or more range of frames would be wonderful!

I am sure however you decide to do it will be Awesome!

[PatchWorKs]

Very interesting.

Do you think some Python Audio Restoration Suite (https://github.com/HENDRIX-ZT2/pyaudiorestoration) functions can be "ported" to VS ?

From thair git:
A set of tools to restore audio quality from a variety of old analog sources, such as tape, cassettes, acetates and vinyl.

Features

Wow & Flutter Removal
Speed matching to hum frequency
EQ matching with differential EQ
Spectral Temporal Alignment
Automatic Dropout Restoration
Spectral Expander / Decompressor


Hope that inspires !

[unix]

I guess some easy way to trim audio along with video would be useful. My proposal is this:
Extend Trim like this:
std.Trim(vnode clip[, int first=0, int last, int length, anode[] audio])
If audio clips are supplied then the output in python will be a dict
{clip: <video here>, audio: <array of trimmed audio clips>}

Can you elaborate and give us an example