Audio sync and realtime (Kludging live effects)

[NerdWithNoLife]

Hello. I'm trying to use AviSynth for something it was never designed to do, and not surprisingly have run into some problems. Hopefully a clever doom9's forum member can help me find a solution. The intent is to source a cheezy (public domain) film and my camera's input, and create Mystery Science Theater style silhouette effect - on the fly - for a live web show. Unfortunately, an audio sync issue has appeared, especially when the CPU is being taxed.

I am using this insane setup because: A) Budget is an issue so dropping cash on VidBlaster or Wirecast or a faster PC can only be done after exhausting other options. B) To do a live show, pre-capturing isn't an option.

Probably what's happening is when the processor gets too bogged down for realtime it continues playing the audio while waiting for the video, or the opposite. Since this effects work is being done on top of encoding live video, I'm very CPU conscious and have found the line where the CPU is at 70-90%, and try not to do anything special. The less I do, the better it stays in sync.

The workflow is: An AviSynth script that does this effects work and creates anamorphic video behind the back of >> ManyCam, which acts as the switcher then off to >> Adobe's Flash Media Live Encoder, which resizes and encodes a proper 1:1 16:9 live video. Here's the script:

## Assign image of theater seats
seats=ImageSource("seats.png",pixel_type="rgb32")

## Assign movie of the week
## Since this is an example we'll use a blue clip instead
#Cheese=AviSource("Gorilla-AVI.avi").ConvertToRGB32().AddBorders(40,0,40,0)
Cheese=Blankclip(seats,color=$0000FF).PointResize(240,240).AddBorders(40,0,40,0)

## Load live video from filter graph. The result is 30000/1001fps RGB.
## Since this is an example we'll comment that line out and use a sample image instead:
# DirectShowSource("live.grf",framecount=216000,audio=false)
ImageSource("Cam.png",pixel_type="RGB32")

## Clip "almost whites", consider everything else black
Greyscale().Levels(0,1,200,0,255)
Levels(254,1,255,0,255)
## Cropping so there's not a black line on the top
Crop(0,4,0,0)

## Shrink me
ConvertToRGB32().BilinearResize(100,76)
## Assign this to clip Cam
Cam=last

## Layer and position my silhouette over the movie
Layer( Cheese , Mask( Blankclip(Cam) ,Cam.Invert()) , x=186,y=164)
## Layer a row of movie seats on top of that
Layer(last,seats,x=-14)

The result looks like this:
http://img834.imageshack.us/img834/9982/examplel.png (http://img834.imageshack.us/i/examplel.png/)
Here are enough files to demonstrate it "working":
http://www.mediafire.com/?1y9hf0lwy8d9nnc

So knowing this is less than ideal, is there even a way to force AviSynth to operate in realtime, dropping frames or pausing if need be, but staying in sync? I'm interested to hearing some suggestions before I spend more money on software / equipment.

[Gavino]

Timing is under the control of the client application, not Avisynth, which just delivers video frames and chunks of audio when asked to do so. See IanB's recent post here.

The best you can do in Avisynth itself is to make sure your script is as fast as possible, so that it is capable of being rendered in real time.

[IanB]

As Gavino say the best you can hope for is to make the script as fast as possible.

Here is an example for your processing of the live component :-...
DirectShowSource("live.grf",framecount=216000,audio=false)

## Cropping so there's not a black line on the top
Crop(0,4,0,0) # Discard unused pixels early

## Greyscale it
ConvertToY8() # Zero cost from YV12

## Shrink me
BilinearResize(100,76) # Do the processing on the squished image

## Clip "almost whites", consider everything else black
Levels(199, 1.0, 200, 0, 255, True) # Why use 2 Levels when 1 will do

## Assign this to clip Cam
Cam=last
...Notice the order is about minimising the number of pixels processed overall.

If you are going to discard pixels, do it early. Don't process first them then ditch them.

If you just want greyscale data consider using Y8 format (2.6 feature) , particularly for masks etc. It is zero cost from all Planar formats, and still pretty fast from YUY2 and RGB.

If you are resizing in the workflow then do the processing on the smaller clip, i.e. after a resize to smaller, before a resize to larger. Okay, accuracy of processing may dictate that processing the original input is necessary. All rules have exceptions.

Look for ways to combine operations. Be cunning! Levels(199,1,200,0,255) is the same as Levels(0,1,200,0,255).Levels(254,1,255,0,255) but half the crunching.

Carefully choose your colour format. If the input is YV12 try to leave it YV12. Format conversions take time, and RGB is generally slower to process.

[NerdWithNoLife]

Thanks for the help. I now remember why I chose a bilinear resize after performing levels on the whole thing (considering the efficient way first). Making a binary decision on the smaller clip results in blocky fake-looking edges. Resizing the leveled image looks much more natural, but perhaps a blur on the small clip could achieve the same look while being less costly than levels on the larger one?

As for layering and masking, maybe there's a smarter way to do it in YCbCr land; I'm just not familiar with it. I use Layer and Mask because according to the documentation (http://avisynth.org/oldwiki/index.php?page=Layer): "There is no mask (alpha-channel) in YUY2, so the alpha-channel is assumed to be opaque everywhere."

But upon further investigation I don't think CPU is the issue here. With VirtualDub it plays just fine. I removed encoding from the equation and just played it back with Manycam. the desynchronization is noticable within a couple minutes, and horrid after a half hour, even with the CPU at %20 and under.

Apparently there's something off about the way this webcam software plays back AVS files. Not sure if anything can be done, but at least I know where the problem is.

[IanB]

Ahaaaa, you probably have a slight mismatch in the captured fps/audiorate, the rate defined in the script and the clock used by the final playback engine. If the audio sync drifts by a constant amount per period this is then the case. i.e. it starts in sync, after 10 minutes it is 2 seconds behind, after 20 minutes it is 4 seconds behind, etc.

It is usual to assume each video stage is 30000/1001 and slightly reclock the audio to compensate for what it is actually, or in this case it may be more practical to use AssumeFPS() to set the true exact (or measured) rate and then ChangeFPS(30000, 1001) to standardise it for playback. This will result in either the occasional dropped or duplicated frame in the realtime stream.

[Motenai Yoda]

As for layering and masking, maybe there's a smarter way to do it in YCbCr land; I'm just not familiar with it. I use Layer and Mask because according to the documentation (http://avisynth.org/oldwiki/index.php?page=Layer): "There is no mask (alpha-channel) in YUY2, so the alpha-channel is assumed to be opaque everywhere."
mt_masktools2 can create a clip mask and merge two clips according to the mask.

[NerdWithNoLife]

It took a lot of experimentation to find the correct rate, but this is what fixed it: AssumeScaledFPS(1001,1000).ChangeFPS(30000,1001)

The proper framerate was 30000,1001 but the playback program failed to get the message. To impede the troubleshooting process, the script either wasn't being loaded consistently (a delay in loading could cause an audio mismatch), or the changes wouldn't take because the program had an old script's framerate in memory. It was also important to do any trimming before the framerate conversion. Also I added seek=false to the directshow load of the live camera just for kicks.

I don't think this was so much an issue of maxing out CPU (it never went near 100%), but with interruption. If I don't touch it, the sync stays fine. But if I try adding other processes to the mix it can go off wildy. Perhaps what happens is Windows says "hold on a sec, I'll come back to you," but when it does come back there's now a discrepancy between where the playback software thinks it should resume and where AviSynth thinks it should resume. So I'm trying to outsmart the software, creating a time travel paradox. Edit: close, but the real solution is below.

Or else I'm just the one who's being outsmarted... :confused:

[NerdWithNoLife]

FOUND IT!!! When working with a 24000/1001 fps video I stumbled across the solution. Using a lower framerate solved the issue. But CPU wasn't even close to being the limiting factor.

I realized that the fundamental problem with this script was exceeding realtime. AviSynth cannot deliver the frames faster than the program operating in realtime requests, because it doesn't have the ability to travel into the future. Further investigation revealed that the offending program - ManyCam - thinks the framerate of 30p NTSC video is (near as I can tell) exactly 29.97. But we here at Doom9's Forum know it's actually 30000/1001 - which is technically faster than the rounded 29.97. A statistically small difference, but just enough to create a time travel paradox.

Simply adding ChangeFPS(2997,100) after loading the source video made audio sync issues disappear.