Inheritance of Audio -- transition plugins

[vcmohan]

While authoring plugins for transitions and special effects I got some doubts as to whats happening to Audio.
In transitions I do not touch (process) audio at all. I have two input clips A and B. I return one frame for each get frame.Which Audio is retained in each of the following instances?
Returned frame from clip A :
Returned frame from clip B :
Returned Frame a NewVideoframe :
So far I was under the impression that when I return frame from A , audio from A will be retained and if NewVideoframe no audio will be present. However after reading the manual of OpenVIP NLE documentation I got this doubt. Can this be clarified?

[sh0dan]

No - audio will always be from the clip you supply as child in the GenericVideoFilter(PClip) constructor.

In this case:
YourFilter(PClip _child, PClip _child2): GenericVideoFilter(_child), child2(_child2) {....}

The audio from _child will be returned. You need to override GetAudio(...) to change this.

[vcmohan]

thanks
mohan

[sh0dan]

Inspired by your post I looked at the Dissolve code - and found that it only supported 16 bit audio. So I changed it to support both 16bit and float:

edit.cpp (http://cvs.sourceforge.net/viewcvs.py/avisynth2/avisynth/src/filters/edit.cpp?view=auto&rev=1.7&sortby=date&only_with_tag=MAIN) - look for "Dissolve::GetAudio".

There is also some conversion stuff:
GenericVideoFilter(ConvertAudio::Create(_child1, SAMPLE_INT16|SAMPLE_FLOAT, SAMPLE_FLOAT)),
child2(ConvertAudio::Create(_child2, SAMPLE_INT16|SAMPLE_FLOAT, SAMPLE_FLOAT))

This will ensure that samples are either 16bit or float format, but not necessarily the same - that will the following code do:


if (vi.HasAudio()) {
child2 = ConvertAudio::Create(child2, vi.SampleType(), SAMPLE_FLOAT);
// Clip 2 is checked to be same type as clip 1, if not, convert to float samples.
vi2 = child2->GetVideoInfo();

child = ConvertAudio::Create(child, vi2.SampleType(), vi2.SampleType());
// Clip 1 is now be same type as clip 2.
vi = child->GetVideoInfo();

if (vi.AudioChannels() != vi2.AudioChannels())
env->ThrowError("Dissolve: The number of audio channels doesn't match");

if (vi.SamplesPerSecond() != vi2.SamplesPerSecond())
env->ThrowError("Dissolve: The audio of the two clips have different samplerates! Use SSRC()/ResampleAudio()");
}

You are very likely to get in need of the ConvertAudio class of audio.cpp (http://cvs.sourceforge.net/viewcvs.py/avisynth2/avisynth/src/audio/audio.cpp?view=markup&rev=1.8&sortby=date&only_with_tag=MAIN). The header for this class is in avisynth.h. The link above is to a slightly older version, as I've just added 3DNOW/SSE conversion - but they haven't been tested enough for me to trust them completely yet.

[vcmohan]

Thanks. However it is not very clear to me. I have gone through the code of Dissolve and Audiodub .As below

void Dissolve::GetAudio(void* buf, __int64 start, __int64 count, IScriptEnvironment* env)
{
if (start+count <= audio_fade_start)
child->GetAudio(buf, start, count, env);

else if (start > audio_fade_end)
child2->GetAudio(buf, start - audio_fade_start, count, env);
----------
bool Dissolve::GetParity(int n)
{
return (n < video_fade_start) ? child->GetParity(n) : child2->GetParity(n - video_fade_start);
}
--------------------------------

AudioDub::AudioDub(PClip child1, PClip child2, IScriptEnvironment* env)
{
const VideoInfo& vi1 = child1->GetVideoInfo();
const VideoInfo& vi2 = child2->GetVideoInfo();
const VideoInfo *vi_video, *vi_audio;
-
vi = *vi_video;
vi.audio_samples_per_second = vi_audio->audio_samples_per_second;
vi.num_audio_samples = vi_audio->num_audio_samples;
vi.sample_type = vi_audio->sample_type;
vi.nchannels = vi_audio->nchannels;
}
-----------------------------
bool AudioDub::GetParity(int n)
{
return vchild->GetParity(n);
}


void AudioDub::GetAudio(void* buf, __int64 start, __int64 count, IScriptEnvironment* env)
{
achild->GetAudio(buf, start, count, env);
}
*******************************************************
It appears I need to have GetAudio and GetParity functions in my scheme. Unlike in GetFrame I do not see any IsWritable or MakeWritable functions. Is Audio always writable? In the main Audiodub filter various vi parameters are being set. Does one need to do this? This is not done in Dissolve.

[sh0dan]

Sorry for the late reply. Been very busy.

- In the GetAudio function you are given a buffer that you are supposed to fill with the appropriate samples. The sample is supposed to start at sample "start" and contain "count" samples (multiplied by the number of channels).

- The main difference between audiosub and dissolve is that audio is transfered from one clip to another in audiodub, while the audio must be the same on both input clips in dissolve.

There are many examples in audio.cpp (http://cvs.sourceforge.net/viewcvs.py/avisynth2/avisynth/src/audio/audio.cpp?view=auto&rev=1.10&sortby=date&only_with_tag=MAIN) - both simple and more advanced.

[vcmohan]

Originally posted by sh0dan

- In the GetAudio function you are given a buffer that you are supposed to fill with the appropriate samples. The sample is supposed to start at sample "start" and contain "count" samples (multiplied by the number of channels).

In Dissolve Filter GetAudio Function has the following code
if (start+count <= audio_fade_start)
child->GetAudio(buf, start, count, env);
From this I thought that Start+count is used to get the Frame number. Now your explanation is different and does not seem to whats in Dissolve. If this understanding of mine is incorrect then how does one access n in this function? For me the frame n decides which audio I need to return.

[sh0dan]

In the constructor, the following is defined:

audio_fade_start = vi.AudioSamplesFromFrames(video_fade_start);
audio_fade_end = vi.AudioSamplesFromFrames(video_fade_end+1)-1;

This takes the number of frames from the dissolve start and end, and converts it to the number of samples.

Start & count are always in samples. 'n' can never be accessed, as GetFrame and GetAudio have nothing with eachother to do.

What you request is "vi.FramesFromAudioSamples()" to get an approximate frame from a samplecount - but since it is rounded to the nearest frame, it is of little use. Operate on samples - not on frames.

[vcmohan]

In Avisynth.h there is num_samples in vi. What is this value? Is it total audio samples in the clip? While returning value for the audio_ samples_from_frames it uses samples per second, fps numerator and denominator. This value of num_samples could have been used also?

[sh0dan]

Originally posted by vcmohan
In Avisynth.h there is num_samples in vi. What is this value? Is it total audio samples in the clip?

Yes. The length of the sound does not have to be the same as the length of the video (thus the difference between aligned/unalignedsplice).

While returning value for the audio_ samples_from_frames it uses samples per second, fps numerator and denominator. This value of num_samples could have been used also?:confused: I have no idea what you mean.

vi.AudioSamplesFromFrames(int framenum) returns the number of samples that equals the length of 'framenum' frames.

[vcmohan]

Originally posted by sh0dan
:confused: I have no idea what you mean.

vi.AudioSamplesFromFrames(int framenum) returns the number of samples that equals the length of 'framenum' frames.

I mean the same value could have been derived by num_samples*framenum/num_frames

[sh0dan]

No - you need to know the speed of the frames also. The exact code:

(from avisynth.h)__int64 AudioSamplesFromFrames(__int64 frames) const {
return HasVideo() ?
(frames * audio_samples_per_second * fps_denominator / fps_numerator) : 0;
}

[vcmohan]

Originally posted by sh0dan
No - you need to know the speed of the frames also. The exact

t= frames/num_frames would be equal to the time at frames/total time of clip. fps will therefore be not required as it is constant through out (also audio samples per second is constant).Therefore multiplying t by audio samples per second should position audio correctly.

I have a few other doubts
1. What exactly is parity, since in different contexts this term is used in general differently. Also it appears to be frame dependant unlike audio.
2. In creating a plugin with "csib..." one can give names to these. There are also default values. Can some of the parameters named and ( for some preprocessing) and rest left without names? May be not as it may not understand which parameter is specified in the script. If no names are given it expects all parameters to be specified so the default will not exist. So are defaults only for the named parameters?

[stickboy]

Originally posted by vcmohan
1. What exactly is parity, since in different contexts this term is used in general differently. Also it appears to be frame dependant unlike audio.Parity specifies whether--in a field-separated clip--the frame represents a top-field or a bottom-field. It doesn't apply to audio because it makes no sense in that context.
2. In creating a plugin with "csib..." one can give names to these. There are also default values. Can some of the parameters named and ( for some preprocessing) and rest left without names?Exactly what do you mean? If the parameter string is "ci[foo]i[bar]i" then the first int is required and the second two are optional (and named). (You could leave the names out (e.g. "ci[]i[]i]") to make optional, unnamed parameters, but that's much less common, and there usually isn't a good reason for it.)

It's probably a good idea to put all of the optional arguments at the end, though.
So are defaults only for the named parameters?You can do whatever you want with the arguments given to your plug-in. It's your plug-in's responsibility to handle them, including setting default values to unused parameters, if you so desire.

The code to the built-in AviSynth filters is a good reference. Many of the third-party AviSynth plug-ins also have source code available.

[sh0dan]

Originally posted by vcmohan
t= frames/num_frames would be equal to the time at frames/total time of clip. fps will therefore be not required as it is constant through out (also audio samples per second is constant).Therefore multiplying t by audio samples per second should position audio correctly.

You are assuming that audio length = video length, which cannot be safely assumed.

I have a few other doubts
1. What exactly is parity, since in different contexts this term is used in general differently. Also it appears to be frame dependant unlike audio.

/**********************************************************************
* Reverse Engineering GetParity
* -----------------------------
* Notes for self - and hopefully useful to others.
*
* AviSynth is capable of dealing with both progressive and interlaced
* material. The main problem is, that it often doesn't know what it
* recieves from source filters.
*
* The fieldbased property in VideoInfo is made to give guides to
* AviSynth on what to expect - unfortunately it is not that easy.
* Fieldbased is only defined when video is separated using separatefields,
* or AssumeFieldBased() has been used, otherwise video is assumed framebased.
*
* GetParity is made to distinguish TFF and BFF. When a given frame is
* returning true, it means that this frame should be considered
* top field.
* So an interlaced image always returning true must be interpreted as
* TFF.
*
* SeparateFields splits out Top and Bottom frames. It returns true on
* GetParity for Top fields and false for Bottom fields.
* **********************************************************************/

2) (Adding to Stickboy's explanation)

You set default values like this:

int value = args[1].AsInt(255);

Here the default value is 255.

[vcmohan]

Originally posted by sh0dan
You set default values like this:

int value = args[1].AsInt(255);

Here the default value is 255.

My problem is if I do not name the parameters and do not specify them in script, the script processor rejects with an error. So the default has no function. Therefore only for the named params it seems to apply. Further if one names the params then, script processor does not type check the given param. Therefore one is forced to check with Defined() and IsWhatever(). Once this far is checked the default can be easily inserted by one more line of code.

Also probably I should not use a definition like "cs[xx]ii[bb]ii" as there can be problem as to which param is specified and which is not.
Hope my understanding is correct.

[stickboy]

Originally posted by vcmohan
My problem is if I do not name the parameters and do not specify them in script, the script processor rejects with an error. So the default has no function. Therefore only for the named params it seems to apply.You can define a plug-in to have optional, un-named parameters. Re-read my previous post.
Further if one names the params then, script processor does not type check the given param.This is not true; AviSynth does type-check named parameters. To handle optional parameters (named or unnamed), you almost always can do what sh0dan described.

(If you've experienced contrary behavior, please provide a reproducible example.)Also probably I should not use a definition like "cs[xx]ii[bb]ii" as there can be problem as to which param is specified and which is not.
Hope my understanding is correct.Yes, that parameter string can lead to ambiguity. As I said, you usually should put optional parameters at the end.

[vcmohan]

Thanks a lot for the clarifications. I vaguely remember an instance when for a bool inputting a number did not give an error, but since that was so many days back I am not too sure that I didn't make a mistake. So in effect IsWhatever() is not required except when you use a . as in "cs.ii"?

[stickboy]

Originally posted by vcmohan
I vaguely remember an instance when for a bool inputting a number did not give an errorI can't reproduce any type problems with boolean types, so if there ever was a problem, it seems fixed now...
So in effect IsWhatever() is not required except when you use a . as in "cs.ii"?That's the primary case I can think of, so yes.

[vcmohan]

In case of defaults, how exactly it works. For instance args[1].AsInt(30). If this is not a named argument as the argument has to be specified this has no purpose. In case of named argument in case it is not defined args[1] it self is non existant. So how can a dot operator access its position? There are two ways of coding I envisage in case of named parameter:

1.int Param=30; if it is defined make Param=args[1].AsInt(); while returning the function in place of args[1].... use Param;
2 just keep it as args[1].AsInt(30);

Which is correct?

[stickboy]

Originally posted by vcmohan
In case of defaults, how exactly it works. For instance args[1].AsInt(30). If this is not a named argument as the argument has to be specified this has no purpose. In case of named argument in case it is not defined args[1] it self is non existant. So how can a dot operator access its position?No. Look at avisynth.h and check how the Defined(), AsInt(), etc. methods work: a "non-existent" argument to an AviSynth function is an AVSValue with a special "void" type.

If your function parameter string looks like: "c[foo]i[bar]b", you can be sure that your argument array has 3 elements.

[vcmohan]

In my transition plugins I have tried to process audio as follows. If both clips have audio then mix them fading out the left clip and fading in the right clip. If only Leftclip has audio, fade out left clip. If only Right clip has Audio fade in the right clip audio. The first two seem to work ok but the third I think not working correctly. I set up flags regarding presence or not of audio in each of the clips. Then I put in (in constructor code) vi.num_audio_samples,vi.num_channels, vi.sample_type, vi.audio_sample_rate(or some such param)equal to those of right clip and then tried to process. I find that in GetAudio call the start and count both are zero. Is this a correct way to process? The changed Audio params will they affect earlier part of stream in the script? For my test I used two short clips one with .killaudio() and the other without.
I could not get help from the code in audiodub. In the mixing or dissolve the code tests for identity of the audio params of both clips and only if equal accepts while my intention is not so.
Or is this a wasteful excercise?

[Richard Berg]

It should be possible to write a filter to do as you describe. I'm not sure I follow your method -- Avisynth treats any number of input clips equivalently when passing them to filter classes. Can you post some code?

[vcmohan]

relevant parts of the code are below:

// include in class
void * abuf;
bool first;
bool LHasAudio, RHasAudio;
public:
//Definition of function if not already put
void __stdcall GetAudio(void* buf, __int64 start, __int64 count, IScriptEnvironment* env);





// include in constructor

char * Tname="Trans.....";
LHasAudio=false;
RHasAudio=false;
if (vi.HasAudio() && rvi.HasAudio())
{
// here check if all params are equal in both clips
}
else if(rvi.HasAudio())
{
LHasAudio=false;
RHasAudio=true;
vi.nchannels=rvi.nchannels;
vi.audio_samples_per_second=rvi.audio_samples_per_second;
vi.num_audio_samples=rvi.num_audio_samples;
vi.sample_type=rvi.sample_type;
}
else if(vi.HasAudio())
{
LHasAudio=true;
RHasAudio=false;
}

first = true;


Trans::~Trans()
{
// include in destructor
if(!first)
delete [] abuf;
// deallocate memory used.
}
void Trans::GetAudio(void* buf, __int64 start, __int64 count, IScriptEnvironment* env)
{
const VideoInfo& rvi= RightClip->GetVideoInfo();
__int64 audlimit;
const int ab=vi.BytesPerAudioSample();
int channels=vi.AudioChannels();
__int64 nsamples=vi.num_audio_samples;
// Fade out and fade in mix as dissolve
int nchn=vi.nchannels;

if(!LHasAudio&& !RHasAudio)
return;
if(LHasAudio^ RHasAudio)
{
if (LHasAudio)
{
// fade out code here works
}
if(RHasAudio)
{
// Fade in audio

RightClip->GetAudio(buf,start,count,env);

if(rvi.sample_type & SAMPLE_INT8)
for( __int64 i=0; i<count;i+=nchn)
for(int ch=0;ch<nchn;ch++)
*((char*)buf+i*nchn+ch)
=(*((char*)buf+i*nchn+ch)*(start+i))/(nsamples);
if(rvi.sample_type & SAMPLE_INT16)
for( __int64 i=0; i<count;i+=nchn)
for(int ch=0;ch<nchn;ch++)
*((short*)buf+i*nchn+ch)
=(*((short*)buf+i*nchn+ch)*(start+i))/(nsamples);
if(rvi.sample_type & SAMPLE_INT32)
for( __int64 i=0; i<count;i+=nchn)
for(int ch=0;ch<nchn;ch++)
*((int*)buf+i*nchn+ch)
=(*((int*)buf+i*nchn+ch)*(start+i))/(nsamples);
if(rvi.sample_type & SAMPLE_FLOAT)
for( __int64 i=0; i<count;i+=nchn)
for(int ch=0;ch<nchn;ch++)
*((float*)buf+i*nchn+ch)
=(*((float*)buf+i*nchn+ch)*(start+i))/(nsamples);
//env->ThrowError("%d %ld %ld %ld ", nchn,start, count,nsamples);

return;
}
}


sorry the code is not getting copied legibly. The last commented // env throw error I use to check values. I got start as 0, count as 0 and nsamples as 9600. The input stream has 16 bit stereo audio and 48khz 250 frames long. So must have large number of audio. Instead of fading in It appears to be high from begining. If you want the code to be sent as an attachment pl email me vcmohan@hathway.com

[Bidoche]

Just post it using code guards ([ code ] and [ /code ] without the spaces)

Like this :

class Dummy
{
BYTE * buffer;

public:

Dummy(int value);
~Dummy();

void DoSomething();
};


Anyway why don't you simply use Dissolve to make the audio fading ?

[scmccarthy]

My thought is that maybe it is impossible to have audio only in the right channel and not the left also. If there is only one audio channel, it will always be in audio channel number one, if there are two audio channels it will be in audio channel number one and two, etc.

Stephen

[vcmohan]

I am processing a transition where two clip segments of equal lengths are involved. I want to avoid long script each time a transition is required. Also the internal audio filters will only operate if both clips have identical audio parameters. I want to process if both have audio do a dissolve, if only child has then fade out and if only child2 has then fadein.I have coded as given but it did not work well in case of child2 only has audio. my script was:
avisource(....)
left=trim(last,0,249).killaudio()
right=trim(last,250,499)
mytransition(left,right,...)

The sound was pulsating and loud from begining. I checked the values of count and num audio samples. They were unexpectedly 0 and 9600. 9600 explains the pulsating sound but why such a low figure when I have reset the child vi audio params to those of child2 params?Since I reset these in my constructor, does kill audio resets it back to 0 and gives rise to problem? If so I expect that I should not get GetAudio callback from Avisynth at all!. But I get it.

May be the next step after my transition which normally be a splice will not accept if the input clips have different audio params and so my excercise is a waste. But I am curious whether such a processing can be done at all in Avisynth may be for a different filter!

[stickboy]

Originally posted by scmccarthy
My thought is that maybe it is impossible to have audio only in the right channel and not the left also.I think the "left" and "right" that vcmohan's been referring to mean "first clip" and "second clip" (i.e., as if you're looking at a timeline or horizontal filmstrip), not left/right audio channels.

[scmccarthy]

You're right stickboy, I see from his last post, that is exactly what he was talking about all along.

Stephen

[vcmohan]

I have further delved into my problem. Curiously I found that the formatted print statement of throw error is the problem. While printing __int64 numbers with either %d or %ld each number is printed as two numbers first part ok and second as zero mostly except probably when the number is zero (as in case of 'start' in the first instance) where its treated as one number. Even though I specified 5 numbers to be printed only 3 are being printed but dissected as 5. since the first number in the format is correctly coming out by shifting the list I could examine each number seperately and find that they were as expected. I am using VC++ 6 for my compilation.

Curiouly I found that the GetAudio call count had these numbers for a total of 384384 samples. 96000, 24000, 1602, 1602, 1601 ...1601. Is there some reason for splitting in this fashion? Why they are not of equal lengths?

[sh0dan]

"count" is usually changing for several reasons. First of all it is dependant on how much audio the client application requests at the time. Vdub likes to buffer up a bunch of audio before playing.

Furthermore filters like SSRC, SuperEq also like to do a bit of buffering. Finally the audiocache might also help creating differing count numbers.

[vcmohan]

Thanks for the clarification. Printing __int64 numbers still remains a problem. Is the behaviour of printing I mentioned above normal?
regards
Mohan

[stickboy]

Try %I64d for the printf format specifier.

Or break up the 64-bit value with bitshifts and print it out in hex.