Dumb questions about Avisynth audio support?

[trbarry]

I still don't even begin to understand how computer audio works but I've been fooling around with it a bit. For instance I understand that RIFF is just a container format, not a file format, and that you can throw RIFF headers on all kinds of things. But my main interest now is to be able to pass multichannel (5.1) AC3 audio through Avisynth. DVD2AVI -> MPEG2DEC -> Avisynth -> Virtualdub -> compressed avi file.

As an experiment I used the AC32WAV utility to put RIFF headers on one of my AC3 files. After doing this it seemed Virtualdub was able to use the "Open Wave File" command and mux it into my video. This was only with direct stream copy and not specifying any special options. But this was with normal Vdub (1.4c on that mschine) not with Nandub or Virtualdub_AC3.

So I also tried it in Avisynth using WavSource, which didn't work and couldn't read it.

Can anybody briefly explain what would be involved here? For instance, since I have the AC32WAV source I could probably change DVD2AVI to optionally save the AC3 files with RIFF headers and maybe even save the name of the file inside the D2v file. Then maybe there would be a way to pass the AC3 sound to Avisynth from a modified MPEG2DEC.dll.

And if we had the entire chain down pat then maybe we could get better control of the various sync errors that pop up from time to time.

But what else would be involved? How strongly is either Avisynth or Virtualdub wired to only 2 channel sound and what other gotchas might be involved?

Sorry if some of this may be unrealistic but I am only just starting to look at this.

- Tom

[Richard Berg]

This is one of our biggest priorities for v2.1 -- check some of the audio threads and you'll see we're thinking big :) (upgrading to 32-bit floating-point internally, moving to n channels instead of boolean mono/stereo, and much more)

I'm working on a RIFF parser myself. I've got the equivalent of mp3towav functionality mostly working -- can you send me the AC32wav program so I can get hacking on that too?

[trbarry]

Richard -

Actually I'd already posted the link in that ACM Audio thread. But see the source and binaries here for AC32WAV (http://mukoli.free.fr/ac32wav).

It appears to run in the DOS box with 2 parms, input and output.

- Tom

[dividee]

I'm not sure I understand you right, Tom.
Do you mean AC3 pass-through, which would serve the AC3 undecoded ? That's not how avisynth works. Sound has to be decoded on entry. Avisynth frameserve uncompressed frames and PCM samples only. Now that I think about it, it may be possible to implement pass-through for compressed sound, if you only want to filter only video, or serve compressed video, if you want to use avisynth to process audio only, but I'm not sure how well it fits in avisynth general architecure.

[trbarry]

Dividee -

I'm not sure which I meant. ;)

When decoding there are maybe normalization issues and maybe a performance hit but I don't know if these are significant, and I don't know if quality would be lost.

I guess I was mostly thinking pass thru if possible. But for my purposes it is not necessary to shrink the ac3 sound and I usually don't otherwise have to change it for frame rate conversions, etc. Other folks might have different needs. But I seriously wanted to pass though 5.1 sound.

BTW, I notice that when ac32wav puts the riff headers on the file it does not make it much bigger. Are the riff headers just a small block on the front of the file or interspersed throughout?

- Tom

[Richard Berg]

RIFF headers usually don't require more than 50-70 bytes at the front of the file. Additional chunks can be added before or after the main data chunk -- MP3TOWAV sticks another 72 bytes at the end of the file tagging it with their l337 kiddie signature, but that's it.

Woo, source code! Reverse-engineering is fun & all, but I've already done that once for MP3, which was quite enough...thanks!

There should definitely be a flag in AVISource (and MP3Source, AC3Source, etc. when they arrive) for choosing "full reprocessing" vs. "direct stream copy" vs. "fast recompress" vs....well you get the idea. Might be a very long-term prospect though, haven't looked at that code in a long time...

[sh0dan]

Passthrough sound very tempting, although I'm not sure how useable it will actually be - many filters will break the sound, and usually the sound is muxed in after the avisynth processing, if no Trim or other filters like that has been used.

It's both good and bad - A feature for 'direct-stream-copy' of audio (or video for that sake) could be a nice feature, but I don't like the consequences (how would it react to alignedsplice for example).

It could be a way of getting multichannel sound out of avisynth, if someone coded an ac3-encode-plugin, but it seems hacky.

[trbarry]

t's both good and bad - A feature for 'direct-stream-copy' of audio (or video for that sake) could be a nice feature, but I don't like the consequences (how would it react to alignedsplice for example).

Sh0dan -

I'm not sure about alignedsplice but it seems that things like ChangeFPS, AssumeFPS, and the explicit changes in video rate by things like IVTC filters ought to work with direct stream copy. But maybe I'm missing something.

- Tom

[Marc FD]

if you use direct-processing on video, you can't change anything, you cannot even cut where you want !!
(EDIT : maybe you can make fps-changes?)

I think avisynth is designed to do frame-serving, and not direct processing.
Decoding all the data to raw, filter/modify it, and then encode it.

That's avisynth spirit !

[Richard Berg]

Marc -- problem is, there's no good way to decode & reencode multichannel audio. Truth be told, letting scripts tinker with 6 independent channels from an AC3 source before rendering back to AC3 would be very cool, but not even the relatively mature Azid project is capable of the encoding step, nevermind the licensing/patent issues...

As far as breaking filters -- 90% of them don't override virtual GetAudio so all we'd have to do is change the base implementation. The other 10% are all internal (no plugins to my knowledge), and will have to be patched regardless once we decide how to handle floating-point audio.

[Marc FD]

i understand. knowing that, i think direct-audio is a good idea.

i'm not a pro but i don't think audio frames refer to other frames so we should be able to mix them like we want (if they are less than one video-frame long)

[trbarry]

if you use direct-processing on video, you can't change anything, you cannot even cut where you want !!
(EDIT : maybe you can make fps-changes?)

I thought at least Vdub with direct processing could cut as long as it was always before key frames. I know I've gotten away with it. But I still think of Avisynth as handling uncompressed frames for video but just sometimes dragging the audio along without doing much to it. At least if, as you asked, audio frames are not too long.

I seem to remember that Avisynth makes sure it doesn't cause cumulative errors but it could be off by the length of an audio frame, whatever that is.

I seriously have to learn how this works someday.

- Tom

[Richard Berg]

Well, the VideoInfo struct has a method for AudioSamplesFromFrames & vice versa.

Or do you mean that audio doesn't have the equivalent of P-frames? That's definitely true for PCM, but even really simple encodings like u-law inherently work in "derivative-space" for lack of a better term...no idea to what extent MP3 and AC3 are constructed similarly, but I get the general idea that they're divided into chunks which are themselves independent but use p-coding internally.

[Wow, that's the most unintelligible paragraph ever. Good luck, Euro folks :)]

To go back to an earlier comment -- there's no reason we couldn't do the equivalent of VDub's "direct stream" on video while allowing splicing & combining. VDub can do it with HuffYUV, a codec which will screw up your entire stream if one bit is removed/changed...that tells me that all it really does is bypass filters, not the conversion to/from raw RGB. If anything, we already support a better form of "direct stream" than VDub, since we can pass YUY2 internally and just not any filters on it.

In short, bad choice of terminology by me :) Supporting audio pass-thru is in fact a stronger claim than "direct stream copy." Whether it can be done depends on the intrinsics of the codec in question...something I've gotta look into, basically.

[dividee]

I had no problem understanding your paragraph :)

The chunks you talk about is what Tom & Marc refer to as audio frames. For instance, for MP3 (with conventional bitrates / samplerates) an audio frame is 1152 PCM samples in length (576 is also possible). As the duration of a video frame is usually not a multiple of the duration of an audio frame, you can't edit while keeping perfect synch.
I think Tom is referring to Virtualdub, not Avisynth, when he says that it "makes sure it doesn't cause cumulative errors but it could be off by the length of an audio frame".
Knowing the precision Avery likes, it wouldn't surpise me if Virtualdub does even better and the synch is never more than half a frame off.
An MP3 frame is small enough (1152 samples = 24ms @ 48 kHz), so it's not a problem.
I don't know the frame length for other encodings. I think that back when Nando was trying to add vorbis support to Nandub I read somewhere that vorbis frames are much bigger than MP3 frames.

[trbarry]

I think Tom is referring to Virtualdub, not Avisynth, when he says that it "makes sure it doesn't cause cumulative errors but it could be off by the length of an audio frame".

I was talking about Avisynth, but not necessarily based upon any certain knowledge. I had no idea for instance what the size of the "chunks" were or even if there are splitable chunks in .ac3 sound streams. But I would be willing to bet that there are, and that that they are not too large to always split on chunck boundaries, with or without rounding to the nearest possibility.

But beyond that I have never looked at Avisynth's audio code and would not yet know how to do frame rate adjustments with ac3 chunks. I'm just confident it is doable in much the same way as wave files are handled now, without reencoding anything. But I'll repeat, I am way out of my area of competance here.

And are these really called "chunks"? ;)

- Tom

[dividee]

Sorry Tom. As you were talking about audio frames, I was thinking you were refering to compressed audio. Since avisynth doesn't have (yet?) compressed audio support internally, that's what mislead me.
What were you referring to? Logically an audio frame would just be 1 samples for PCM (i.e. 4 bytes for 16bit stereo).

[trbarry]

What were you referring to? Logically an audio frame would just be 1 samples for PCM (i.e. 4 bytes for 16bit stereo).

Sorry. I guess I was referring to something that does not exist, which made it harder to find the referent. I had thought that even PCM was in a form of blocks (chunks?, frames?) and could not be easily split at arbitrary points that finely.

But it is my strong belief that I have to be willing to say stupid things, because otherwise no one will correct me and then I have to continue to believe stupid things, which is even worse. ;)

- Tom

[Richard Berg]

I think all good hackers are the same way :) But no, dividee's right -- a Wav file has no structure at all other than RIFF, and all the PCM data is in one RIFF chunk.

[Marc FD]

but is it doable to pass the _compressed_ audio stream trougth functions like ConvertFPS(), Dissolve() !
it seems really hard.
so direct-audio would be a somewhat "limited" mode

[trbarry]

I guess it would depend upon whether Avisynth already has any chunk-oriented AV interleaving code like Virtualdub does. I was previously assuming it did just because Avisynth could do PCM but if PCM is very fine grained then there might be some code missing here.

I'm still flailing around in the dard here without yet getting the time to go read the Avisynth audio code. Can anybody say what's missing?

I'm still anticipating passing through an arbitrary number of audio channels of anything with RIFF headers that Vdub could direct stream copy. I'm still assuming you could change the video frame rate, add/drop video frames, IVTC, etc.

Anyone want to crack my final illusions on this? ;)

- Tom