video source switch based on audio

[zenny]

New to avisynth. So bear with me if it sounds ridiculous.

The scenario is:

I have two video sources, say A and B and one sound source, say C.

I would like to replace video A (VSA) with B (VSB) when there is a certain pause detected in sound track C (STC) for a certain duration and again revert back to video source A.

VSA-->pause detected in STC-->replaced by VSB for certain duration or till next pause-->replaced by VSA till another pause-->and so on.

I could not figure out exactly whether and how this can be achieved in avisynth?

Appreciate any inputs from the avisynth veterans! Thanks in advance.

/zenny

[Guest]

I don't see any way to accomplish this automatically using Avisynth.

How many such switches are there in the stream? If it is just a few then you can do it manually.

Can you please describe the application so we can understand the problem better?

[zenny]

I don't see any way to accomplish this automatically using Avisynth.

How many such switches are there in the stream? If it is just a few then you can do it manually.

Can you please describe the application so we can understand the problem better?

Thanks neuron2 for taking time to reply.

If it is one I would have done it manually in an NLE. But I would like to do in a batch mode.

The application is I capture two videos of the same event with two cameras in two different locations, but with only one sound input (meaning both videos share the same sound track).

Let us say 100m race. I get a video stream from a camera from the rear end facing runners and another from the camera from the audience's side. Sound is recorded at the same time on a separate device. All two video input devices and sound input devices will be triggered at the same time and ends at the same time.

While compositing I would like to replace the video based on sound events (like a measureable amount of pause in milliseconds) either from video from the camera opposite of the runners or from the audience's side.

I ought to do that on a command line, that is why I am exploring avisynth. Somebody stated that MinMaxAudio plugin can do, but being new to avisynth, I could neither get a head nor tail of the plugin, just figured out that it is based on the frame counts.

This is to produce an automated composite. I hope I made it clear of application.

[Guest]

I still don't understand. Why do you think measurable amounts of audio pause will be correlated in any significant way with the video? What makes you think you won't just end up with a random composition?

[zenny]

I still don't understand. Why do you think measurable amounts of audio pause will be correlated in any significant way with the video? What makes you think you won't just end up with a random composition?

Syncing video replacement with sound is one of the requirements.

However, it would be nice to know how I can just composite with a random composition based on a specific duration for each video, but not ending with the side shots (video input from audience' perspective)?

Any avisynth script would be appreciated. Thus, it will give me a basis of improving what I am trying to achieve. Thanks again!

[Guest]

I guess I can't help you much if you don't answer my questions.

You can use Trims in the usual way for compositing with Avisynth.

[zenny]

I guess I can't help you much if you don't answer my questions.

You can use Trims in the usual way for compositing with Avisynth.

I guess I answered all your questions with an example. Do not exactly know what you want to get answered.

The sound based setup is one of the requirement for automation with avisynth script in compositing.

Let us say there is a quiet pause then I just want to replace the first video with the second and vice versa.

Trims is what I have been using so far without any customizations like the one I would like to achieve.

Say this is one of the recipe that I want to cook in avisynth kitchen, but this 'cook' is an 'apprentice' who may spoil the 'food' without support of the avisynth 'masterchefs'! ;-)

[Guest]

I asked why the video stream switches are correlated with audio pauses. You answer that by just saying it is a requirement. I was looking for an example of content where such correlations would be expected.

[Sir BlunderBrain]

Ok, this may not be exactly what you're after, but for what it's worth...

This sample uses ConditionalFilter together with the MinMaxAudio plugin to switch from clip a (black) to b (gray) when it encounters silence in the audio (after 4 sec in this case)

Histogram and the AudioGraph plugin (http://avisynth.org/warpenterprises/) are used for visualisation.

The drawback is of course that this operates on a per frame basis and can't take into account the length of the silence etc.

LoadPlugin("MinMaxAudio.dll")
LoadPlugin("AudGraph.dll")

a=BlankClip(color=color_black, pixel_type="YV12")
b=BlankClip(color=color_gray, pixel_type="YV12")

audio=Tone(type="Sine", frequency=3, level=0.7, length=4)+Tone(type="Silence", length=2)+Tone(type="Sine", frequency=3, level=0.7, length=4)

a=AudioDub(a,audio)
b=AudioDub(b,audio)

ConditionalFilter(a,a,b,"AudioRMS(0)", ">", "-48", show=true)

Histogram(mode="AudioLevels").ConvertToRGB32().AudioGraph(10)

You may also wanna want to check the documentation for the related runtime functions ScriptClip and ConditionalSelect (http://avisynth.org/mediawiki/ConditionalSelect)

Another approach may be to find a way to analyze your audio outside of AviSynth and use a combination of ConditionalSelect and ConditionalReader (http://avisynth.org/mediawiki/ConditionalReader) to do the actual mixing in AviSynth.

[IanB]

You could try a 2 pass approach. The first pass writes a log of audio levels per frame. You process the log to make your decisions. The second pass acts upon your decision output to select the video view.

Pass 1 :-LoadPlugin(".....MinMaxAudio.dll")

Colon=":" # Field Separator text

...Source("Video1....")
AudioDub(...Source("AudioTrack...."))

WriteFile("AudioLevel.log", "current_frame", "colon", "AudioRMS()")

Pass 2 :-
V1=...Source("Video1....")
V2=...Source("Video2....")

ConditionalFilter(V1, V1, V2, "ChoiceVar", "equals", "1") # Select Video based on ChoiceVar
ConditionalReader ("Choice.Log", "ChoiceVar") # Load ChoiceVar table

AudioDub(...Source("AudioTrack....")) # Add Audio
Of course you still need to write the magic to convert AudioLevel.log into Choice.Log. Which should be a fairly easy task in any text handling language like Perl, Basic, C, etc.

[vcmohan]

I asked why the video stream switches are correlated with audio pauses. You answer that by just saying it is a requirement. I was looking for an example of content where such correlations would be expected.

I got a personal Email from some one else on this very subject a week back. I assumed that in a case of video recording an interview with question and answer between two individuals and two cameras fixed on these individuals, such a requirement can arise. However as it was pointed out that type of switching may jumble up.
I have actually coded a special plugin for the purpose and is in testing phase. If I find it is satisfactory atleast in carefully prepared synthetic environment I will release it. How far it will be useful users need to judge.

[zenny]

@IanB and @SirBlunderBrain:
Thank you for your inputs. They are extremely helpful to explore. I shall try and let you know how it goes.

@vcmohan:
Thanks for your efforts to create a plugin, it would be extremly useful and helpful. Look forward to!

You all are such a wonderful bunch of helpful people. Appreciate it!

[vcmohan]

You may try my plugin SwitchByAudio, newly posted

[wonkey_monkey]

You may try my plugin SwitchByAudio, newly posted

We may, except that the page you linked to in your thread doesn't have a link to a download :)

But I found it here (http://avisynth.org/vcmohan/SwitchByAudio/SwitchByAudio.zip)...

David

[vcmohan]

sorry I corrected it now