MeGUI: how to encode with avisynth audio manipulation

[Morte66]

A while back, when MeGUI started using avisynth audio processing, I checked the "use avisynth" box instead of "use BeSweet" and thought no more of it for a while since my encodes continued to work as before.

But then I wanted to do actual audio manipulation in avisynth before the encode, specifically PAL 25 to 24fps slowdown. And I wasn't really sure how we're intended to use this -- one script for audio and one for video, or one to do both, or...?

So anyhow, I tried a few things and found that I (apparently) needed to combine audio and video streams to get the slowdown and resample functions to work. I ended up generating a script like this, and loading it in both the audio and video sources on the main MeGUI input tab. Thus I am feeding a combined audio/video stream to the video input and again to the audio input.

LoadPlugin("C:\Program Files\DGMPGDec\dgdecode.dll")
loadplugin("C:\Program Files\avisynth 2.5\plugins\NicAudio.dll")
video=mpeg2source("D:\something.d2v",cpu=4)
audio=nicac3source("VTS_01_1 T01 3_2ch 448Kbps DELAY 0ms.ac3",2)
audiodub(video,audio)
AssumeFPS(24,1,true)
SSRC(48000,false)
My basic question is: am I doing it "right" (OK, "sensibly")? Or have I invented a square wheel? I'm particularly concerned about what happens if I use this approach with some heavy video filtering, i.e. wiould it get processed twice?

[ChronoCross]

I'm definitely not an expert in audio, but this seems like the right thread to ask my question as it is related. Do you really have to change the framerate? I always do NTSC 29fps rips and I convert the video to 24fps but I don't touch the framerate of the audio and it still plays perfectly in sync. I always thought that audio is related to time and not framerate?

Thanks for the reply.

[Morte66]

I'm definitely not an expert in audio, but this seems like the right thread to ask my question as it is related. Do you really have to change the framerate? I always do NTSC 29fps rips and I convert the video to 24fps but I don't touch the framerate of the audio and it still plays perfectly in sync. I always thought that audio is related to time and not framerate?

Thanks for the reply.

This is an NTSC vs PAL thing, it may not matter to you.

Over here in PAL land, where video runs at 25fps, they take the 24fps movie and speed video/audio up by 4%. 4% is enough for musos like me to worry about -- the music sounds obviously out of key, and the bass is just wrong. So I need to redefine the video and sound from 25fps/48000Hz to 24fps/46080Hz to get it to sound right. But sound cards tend not to like 46080Hz, so I need that SSRC filter to convert my newly-corrected audio up to 48000Hz whilst preserving the sound as accurately as possible.

OTOH NTSC is mostly done by duplicating video frames and leaving the audio alone, rather than speeding everything up like PAL. To play 24fps film on 29.97fps NTSC video, they first slow the film/sound by 0.1% to 23.976fps, then they duplicate one part in five of the video to take the speed up to 29.97fps NTSC (which is jerky because the duplicates are unevenly spread). To convert back, your software normally detects the duplicate information and discards it ("inverse telecine"), giving you smooth 23.976fps video. This leaves sound and video 0.1% slower than the original, but even nutters like me don't worry about 0.1%.

From there you could redefine the video as 24fps and it won't be perfectly in sync with the original sound, but it might be so close you can't tell. You could also convert the sound to true 24fps, basically doing the same resampling as me but one fortieth as much. You might lose more sound quality from the processing than you gain from the tempo correction, it depends how good your software is.

[ChronoCross]

That definitely explains my question. Thanks for the very informative answer. Wish I could help you wish your issues but as you see I'm not expert in these kinda things.

[Morte66]

...and as a follow-up: can I get AviSynth profiles to generate the audio part of the script, picking up the correct audio file etc?

It seems to me that if MeGUI is to use avisynth for sound, then the avisynth script generator window needs to take both a video file (D2V) and one or more Audio files as input, otherwise it cannot generate scripts to process the audio and video.

Or am I missing the point?

[berrinam]

...and as a follow-up: can I get AviSynth profiles to generate the audio part of the script, picking up the correct audio file etc?

It seems to me that if MeGUI is to use avisynth for sound, then the avisynth script generator window needs to take both a video file (D2V) and one or more Audio files as input, otherwise it cannot generate scripts to process the audio and video.

Or am I missing the point?
I agree with you, and there should clearly be some way of generating AviSynth audio scripts. There is no way to do it at the moment. I think a way will definitely come up sometime in the future. I've added it to the Feature Request list.

One thing which I think needs to be found out is whether adding video processing to a script slows down the audio encoding, and vice versa, because this determines whether MeGUI needs to generate separate scripts for video and audio or not. I know you raised that question in your first post, and no-one responded to that. Probably, the reason is that no-one knows.

[Sharktooth]

separate scripts will help the multi core/HT cpus... and the threading overhead on a single core/non-HT cpu is not as much to say it will slow it down.
that will avoid any possible slowdowns and will not load video filters in the audio encoding thread (and viceversa)

[Richard Berg]

One thing which I think needs to be found out is whether adding video processing to a script slows down the audio encoding, and vice versa, because this determines whether MeGUI needs to generate separate scripts for video and audio or not. I know you raised that question in your first post, and no-one responded to that. Probably, the reason is that no-one knows.
As far as speed, an audio filter is no different from a video filter that takes the same # of cycles. The average audio filter takes far fewer cycles than the average video filter because there's ~200X less data to process per second.

[berrinam]

Well, what I really meant is, if you are encoding audio from a script with, say, iiP in it, will it only encode at the audio equivalent of 1FPS because it still has to process the video, or will it ignore the video processing, as no GetFrame calls are being made (nothing is requesting the video in audio encoding). I've done some tests, and there seems to be no discernible difference in audio encoding speed between heavy video filtering and no filtering, and no video at all. So, I think it is a good option just to keep audio and video in the same script, so they can share things like FPS changes, trims, etc.

[Sharktooth]

if you have tested it then what's the problem?
if keeping a single script is better for trimming and stuff then go for it.

[berrinam]

Well, I think we should. I only raised the question because I hadn't done the testing at that time, so I didn't know. I just finished the tests when Richard wrote his reply.

[dimzon]

So, I think it is a good option just to keep audio and video in the same script, so they can share things like FPS changes, trims, etc.
there can be a really unsignificant overhead: parsing unnecesary script and initializing video filters. But it's really unsignificant so I prefer such way (audio & video together) too.

[Morte66]

Well, I think we should. I only raised the question because I hadn't done the testing at that time, so I didn't know. I just finished the tests when Richard wrote his reply.

Lone data point to be treated with some caution: I processed an episode of Deadwood (PAL/R2, progressive from D2V) twice, once with a combined audio/video script and once with separate inputs (.avs video, .ac3 audio). Both methods took essentially the same time, a few hours for the video and 8 or 9 minutes for the audio.

In other words, same story here.

[Morte66]

separate scripts will help the multi core/HT cpus...

One point: AFAICT (but I might be wrong), certain avisynth functions only work on a combined audio/video feed.

{edit} I think AssumeFPS might be one such -- it certainly affects both, and I've a feeling that one of the cryptic error messages I saw the other day implies that it requires both.

[dimzon]

One point: AFAICT (but I might be wrong), certain avisynth functions only work on a combined audio/video feed.
trim function need video bcz it's frame-based. In this case I'm creating fake 16*16 1000fps clip and audioDub it with audio to allow millisecond-accurate trimming ;)

[berrinam]

trim function need video bcz it's frame-based. In this case I'm creating fake 16*16 1000fps clip and audioDub it with audio to allow millisecond-accurate trimming ;)
Is there trimming going on in MeGUI? What for?

Or did you mean BeHappy or something else?

[dimzon]

Is there trimming going on in MeGUI? What for?

Or did you mean BeHappy or something else?
I mean BeHappy, there are millisecond-accurate trim controls

[berrinam]

Video cutting in the AviSynth creator for removal of ads, etc
Description: Since video and audio can be done through AviSynth cutting can be completely precise. Use should be done using the trim statement.
Status: No-one is working on this yet.
...
Rearrange the long dialogs -- AviSynth creator,
Description: Make them tabbed (preferably with Basic/Advanced, or some other structure) to make them less overwhelming, and also so that they can fit on a lower-resolution screen.
Status:
...
Some form of AviSynth script creation for audio, to allow the user control over advanced processing (and trimming)
Description: First we need to find out whether having audio processing in a video script slows down video encoding and vice versa. Then, we could either add audio input to the AviSynth Script Creator, or we could creat a separate script creator for audio. Although the current audio encoding system is able to add some audio processing, such as channel mixing, DRC and delay correction, it doesn't advanced processing, or trimming (which the video Avs creator also doesn't at the moment)
Status: Pending Doom9's approval

I'm rearranging the AviSynth window at the moment, and I am considering the other two changes I've quoted above: trimming and adding audio to the AviSynth creator. I have a few things to bring up first:

Is anyone else already working on these, in which case I will leave them alone?
Doom9, do you agree with the third one?
I think that, although audio should be loadable via the AviSynth window, as per the third request, it should still remain loadable directly in the main window. Unlike video encoding, audio often has no prepocessing, so a user shouldn't be forced to go through the AviSynth creator for a direct transcode.
If I enable audio filtering in the AviSynth window, but leave audio loading as it is in the main window, the audio would be filtered twice, as the AviSynth window would filter it, and then the audio encoder would also add its filtering. Perhaps there should be a special case for opening avisynth audio files -- the audio encoder would just directly pass that output onto the encoding executable. How would we make it clear to the user that this is happening?
I know that I posted the things I want in cutting, like scene change detection. However, that could be a fair bit more work. At the moment I propose just a slight modification of the way that zones are handled. Any suggestions on this?
What audio processing should be enabled? I think that everything offered by the baseAudioConfigurationDialog should be accessible (except for the BeSweet bits, obviously ;)). I looked through AviSynth's documentation, and the processing it has as part of its core filters is:

AssumeSampleRate
Amplify(dB) * (in baseAudioConfigDialog)
ConvertToMono
ConvertTo8/16/24/32/etc Bit
DelayAudio * (in baseAudioConfigDialog)
MergeChannels
ResampleAudio/SSRC *
SuperEQ
TimeStretch *

I've put an asterisk (*) next to the ones I think we could put into MeGUI, but I'm sure other people have many suggestions, as well.
EDIT: I've changed my mind. I no longer think that Resampling or TimeStretch should be offered by MeGUI. Resampling should be done through Avs templates, because you will probably always end up resampling to the same frequency (almost always 48kHz), and TimeStretch configuration in MeGUI would be too simple and too limited to be of any use to the people who actually need it. So basically, it would have the same filterset as the current AviSynth encoder, with the addition of trimming.

Comments?

[Morte66]

I'm rearranging the AviSynth window at the moment, and I am considering the other two changes I've quoted above: trimming and adding audio to the AviSynth creator. I have a few things to bring up first:

I think that, although audio should be loadable via the AviSynth window, as per the third request, it should still remain loadable directly in the main window. Unlike video encoding, audio often has no prepocessing, so a user shouldn't be forced to go through the AviSynth creator for a direct transcode.

Yeah. It feels right, even if it's kind of "impure".

If I enable audio filtering in the AviSynth window, but leave audio loading as it is in the main window, the audio would be filtered twice, as the AviSynth window would filter it, and then the audio encoder would also add its filtering. Perhaps there should be a special case for opening avisynth audio files -- the audio encoder would just directly pass that output onto the encoding executable. How would we make it clear to the user that this is happening?

I'd change the main input tab as follows, if possible...

- When you drag a combined audio/video AVS onto the GUI, or load it with either file input button, that script is used to populate both file inputs.

- You can overwrite either field of a combined source.

- A source which is been specified in the main input tab replaces any "bonus" stream from the other input.

Example:

I load "Buffy 101 audio-video.avs" into the video source and it populates both sources.

Then I load "Buffy 101 audio.mp3" into the audio source and it overwrites the audio field.

I hit AutoEncode, it queues the video part of "Buffy 101 audio-video.avs" and "Buffy 101 audio.mp3" for encode/mux with whatever encoder settings prevail.

I drag "Buffy 101 audio-video.avs" into the video field again, and it populates both again.

I hit AutoEncode, this time it queues the video and audio parts of "Buffy 101 audio-video.avs" for encode/mux.

I drag "Buffy 101 audio normalised.wav" into the audio field, it overwrites "Buffy 101 audio-video.avs" in that field only.

I hit Encode on just the audio, and "Buffy 101 audio normalised.wav" gets queued for audio encoding with whatever audio encoder is configured right now (e.g. FAAC VBR 100%).

I hit AutoEncode, it queues the video part of "Buffy 101 audio-video.avs" with "Buffy 101 audio normalised.wav" for encode/mux.


I've put an asterisk (*) next to the ones I think we could put into MeGUI, but I'm sure other people have many suggestions, as well.
EDIT: I've changed my mind. I no longer think that Resampling or TimeStretch should be offered by MeGUI. Resampling should be done through Avs templates, because you will probably always end up resampling to the same frequency (almost always 48kHz), and TimeStretch configuration in MeGUI would be too simple and too limited to be of any use to the people who actually need it. So basically, it would have the same filterset as the current AviSynth encoder, with the addition of trimming.
Comments?

I agree. The other options are characteristics of a type of job, not characteristics of a job, so they belong in the template system.

[Richard Berg]

TimeStretch is very necessary for processing analog captures, in order to fix sync. However, it can be done automatically. See the AlignAudio function I posted here (http://forum.doom9.org/showthread.php?t=106880).

[berrinam]

TimeStretch is very necessary for processing analog captures, in order to fix sync.I see. However, it only makes sense for MeGUI to have explicit control over it if it can do something clever (like suggesting a resolution, for resizing, or autocropping, or autodeinterlacing, or autodelaying audio). Otherwise, if the call is always the same, you are better to put it in a template in MeGUI, and if it does need tweaking, then MeGUI's control over it would probably not be enough, and you would be better to manually edit MeGUI's script, or edit it in some other tool.

Of course, when I say 'don't support it', I just mean don't explicitly add controls to manage it; that doesn't rule out templates.

[Richard Berg]

What's wrong with adding a checkbox for AlignAudio? I think people will find it very useful. All the hard work is done in the Avisynth function; MeGUI doesn't have to do anything except feed it a clip with audio & video.

[berrinam]

There's nothing fundamentally wrong with adding a checkbox for AlignAudio (I defer to Doom9's decision in the end, just making my case clear), just as there is nothing wrong with adding a checkbox for many other AviSynth filters. My objection to TimeStretch is that it is either a filter that you have to customise for each source, or it is a filter that you have the same values for each. If it is the first, then there is no point in having a set of controls for it, because MeGUI isn't designed to be the sort of environment for customizing that sort of thing. If it is the second, then it is something you can put in an one of the templates and be done with it. All you have to do is select the template and it will insert that filter for you. So basically, MeGUI shouldn't have a checkbox for TimeStretch because it isn't built to support the required tweaking, and it isn't important enough to warrant a checkbox for it (IMHO).

As to AlignAudio.... well, since it has no parameters, you could *still* just put it in your 'AnalogCaptures' template, if you have one. But that's just theory, and people aren't going to do that unless (a) someone writes a guide for analog captures which tells them to do add AlignAudio to the template, or (b) someone distributes specific avisynth profiles targetted for Analog captures, DVB sources, DVD sources, Anime, etc.

My only reason for objection to that suggestion is that I have no idea how relevant it is, and it doesn't make sense to add obscure filters just because someone requests them and there is a perfectly good way to add them yourself to all your encodes, through templates. That's exactly what AviSynth templates was supposed to help alleviate.

So, if you say that it's useful and recommended for people doing analog captures, then MeGUI endorsing it would surely be a good way to make people aware of it. I know nothing about analog captures, so if you tell me that is very useful for these, I will raise no further objections.

[Morte66]

One that just hit me: it's fairly common to handle more than one audio stream e.g. original audio plus director's commentary. They may have different delays, and commentary may be in a different format (e.g. AC3 2ch vs DTS), so they sometimes need to be treated separately. If you want to process them both via avisynth (assuming it can do it), that means picking them up in the script creator and feeding them through to the encoder main input tab.

Would you want MeGUI to cope with this, or does it fall under "do them separately and use the muxer" under the 80:20 rule?

If MeGUI is to cope with this, it seems like safety lies with the current main input tab becoming an encoder dialogue rather than an input dialogue, taking a single avisynth "compound input" which includes video and audio in various combinations, and applying its switches (choice of encoder profiles) to them. That compound input would be put together in the avisynth script creator, which can generate scripting for whatever deinterlace/denoise/delay/slowdown/etc you put on checkboxes and combos, then allow the user to edit the final script by hand for funky stuff.

If avisynth ever starts processing subtitles, I think my brain will explode...

[berrinam]

One that just hit me: it's fairly common to handle more than one audio stream e.g. original audio plus director's commentary. They may have different delays, and commentary may be in a different format (e.g. AC3 2ch vs DTS), so they sometimes need to be treated separately. If you want to process them both via avisynth (assuming it can do it), that means picking them up in the script creator and feeding them through to the encoder main input tab.Aaaah, geez. AFAIK, AviSynth can only handle one audio stream, so that basically forces multiple scripts. What do other people think?

[dimzon]

AFAIK, AviSynth can only handle one audio stream, so that basically forces multiple scripts
Yes, AviSynth can only handle one audio stream

[Doom9]

Doom9, do you agree with the third one?Is that necessary? I think we can safely assume that any kind of audio trimming / editing generally goes together with video. So the logic would be "set up my video chopping chain", then apply that to audio as well. And then of course there's the question where you put the chopping since there's already separate intro/credits placement and zone creation (and chopping is just a glorified zoning system).

[Doom9]

I haven't read page two until now.. but why all that complexity all of a sudden? What's wrong with treating audio completely separate, and only send it a cutlist in case its applicable? Audio encoding has been done like that for many years now. That case also causes no problems in the scenario morte66 outlines.. separate streams means separate delays.. the way it has always been with besweet for audio encoding. The only change that would make sense in the audio department is allow avs input so those that write their own script have the ability to create a script with video and audio and use it for both. And those that have elementary streams can do it as it's always been in megui.. set up the streams independently and process. And in case of automated encoding.. once there's a cutlist it can be added to the audio settings (or the AudioStream) so that the proper audio encoding script can be generated on the fly. I don't think it's a good idea to let people see the AviSynth scripts.. if they want a script for audio, they can write it themselves.. if not, let us do the dirty work and hide it from them just as we hide the avisynth script (only there because the video encoders need that kind of input we have to write it to your hd.. but nobody was ever forced to actually have a look at it).

[dimzon]

I haven't read page two until now.. but why all that complexity all of a sudden? What's wrong with treating audio completely separate, and only send it a cutlist in case its applicable? Audio encoding has been done like that for many years now. That case also causes no problems in the scenario morte66 outlines.. separate streams means separate delays.. the way it has always been with besweet for audio encoding. The only change that would make sense in the audio department is allow avs input so those that write their own script have the ability to create a script with video and audio and use it for both. And those that have elementary streams can do it as it's always been in megui.. set up the streams independently and process. And in case of automated encoding.. once there's a cutlist it can be added to the audio settings (or the AudioStream) so that the proper audio encoding script can be generated on the fly. I don't think it's a good idea to let people see the AviSynth scripts.. if they want a script for audio, they can write it themselves.. if not, let us do the dirty work and hide it from them just as we hide the avisynth script (only there because the video encoders need that kind of input we have to write it to your hd.. but nobody was ever forced to actually have a look at it).
Agreed in all points!

[Morte66]

but why all that complexity all of a sudden?

It started out with trying to find a neat, seamless way to build combined audio/video processing into the automated parts of the workflow. But then it turned out to be not so simple, and I guess it's not worth it.

done like that for many years now.

[Tangent. This is absolutely not a complaint about MeGUI. It's a complaint about the outrageousness of the universe.]

The Romans built their first roads in Britain about 2000 years ago, and rode chariots along them. Chariots were made to be just the width of two horses, to a standard design, so they all had wheels four feet eleven and a half inches apart. Over a couple of centuries, these wore grooves in the road.

The Romans left, but their roads remained in use. In later years, carts were made to fit those grooves exactly because both wheels in the groove is safe, but one wheel in the groove is an accident waiting to happen. Four feet eleven and a half inches became a standard for cart axles.

The first prototype railroad engines and carriages used cart axles. Four feet eleven and a half inches became British standard for railway track. So, when it came to building the 320km/h trains that run from London to Paris through the channel tunnel, their rolling machinery was carefully engineered to be exactly the width of two horses' arses.

In Europe, it is common to watch films at 25fps because a few decades ago that happened to be half the frequency of the electrical supply. It's technically irrelevant now, but we still view at 25fps with sped-up audio and video. In a sensible world, video encoding software producing computer files would come with a checkbox labelled "PAL to 24fps slowdown" that adjusted the video/audio/subtitles/chapters in one click. This would have been standard by the time Nero and (A)GK came along.

But life's not like that...

[Doom9]

well.. even though I don't see why you'd want to load the additional complexity upon you (well I do see.. .you want to load it on my back :) ), the whole "format conversion" (it's not really since TV resolutions play no role in the output MeGUI creates) bit is something I'm not dead set against..

[Richard Berg]

Avisynth has no limit on the # of channels it can process. What it does not do is group channels into "streams" the way container formats like MP4 do. So if you have a script where the main movie soundtrack is on channels 0-5 and the director's commentary is on 6-7, then it's MeGUI's job to encode & mux them separately. That seems natural to me -- it would be much more complex & error prone to add the notion of separate "streams" to Avisynth. The person (or program) that's writing the script already has absolute control over which inputs get put in which channels.

I haven't read page two until now.. but why all that complexity all of a sudden? What's wrong with treating audio completely separate, and only send it a cutlist in case its applicable?
That seems more complex to me, not less. We have to keep track of multiple temporary scripts within MeGUI, then find a way to search the "video" script for Trims & Splices, then write that info into each "audio" script using some sort of AudioDub(BlankClip, last) hack, then match up each audio script to its own encode & mux jobs? Possible, perhaps, but very difficult to make it do the right thing 100% of the time.

Back when we used Besweet, the user had to write separate scripts for audio & video. He also had to render out his audio script to WAV before he could even feed it to MeGUI. I thought the point of removing Besweet was to eliminate all of these unnecessary steps and put everything under Avisynth's control. When all of the desired processing is contained in 1 file, things are much simpler.

[Doom9]

We have to keep track of multiple temporary scripts within MeGUIThey are already created and destroyed on the fly and nobody is even the wiser (unless you go check the temp folder during actual audio encoding).. or is it even written down at all now with the new scriptserver? I haven't been following that development.
then find a way to search the "video" script for Trims & SplicesNo. Either you have your own manual script, in which case megui offers audio encoding from an avisynth script. Or you load your dts/ac3/mpa/wav/whatnot files as usual. Then you enter auto-mode, and you get your video cutlist there.. dimzon and me have long ago sorta agreed on a format for a cutlist that the audio encoder gets.. with the new script server, we don't even have to write the cutlist to a file anywhere.. just feed it to the scriptserver at the proper time.. the proper audio/video encoder will know how to turn a cutlist into the proper avisynth commands to be given to the scriptserver.
Whoever writes his own script should not expect special accomodations in megui.. we bend over backwards to accomodate the guys who give us their regular audio input, and those who want to cut their dgindex compatible input (eventually) and use all the build-in functionality.. but if somebody wants to do stuff like putting multiple audio tracks into a single avisynth script.. that's his problem.

[dimzon]

They are already created and destroyed on the fly and nobody is even the wiser (unless you go check the temp folder during actual audio encoding)..
MeGUI doesn't create temporaly AVS files at all - new AviSynthWrapper can parse (open) script direcly from string variable:

AvsReader.ParseScript("AviSource(...)\n\rBilinearResize(640,480)\n\r");

[Richard Berg]

Ok, I'm going to stop complaining about features that don't exist yet :)

Seriously, when the audio rewrite/refactor is done, I'll try it out and only then determine what changes (if any) are needed. Talking around & around about abstract workflows that don't exist yet is a waste of time, and I don't want to do that since you guys are the ones doing the most work right now.

[Doom9]

@dimzon: a word of warning.. I already added avs as supported input type for audio encoding, and raw aac as supported output type for the aac encoders. as far as I read the manpage, just setting the extension for the output filename should be enough for faac but I figure something needs to be done for nero.. just so that you can prepare (so far I haven't touched the avisynth audio encoder and I hope I won't have to).