A script to help find lip sync DelayAudio value

[IanB]

Dear all,

I would like to share this script as an approach to finding the appropriate DelayAudio
value to use to correct lip sync errors in video material.

It has always been a pain trying to find the right delay offset by simply tweaking
the delay and watching the results over and over, I can never quite find the right
value before frustration takes hold.

The script below takes a short section of an input video which has some recognisable
sharp sound and its matching visual event. For an interlaced source I Bob deinterlace
it to double the video temporal resolution and also take the free ride to resize at this
point. I then have a function to return the clip with a specified DelayAudio value and
a subtitle identifying the delay used. I was finding that I was getting used to the
delayed audio and could no longer discern the best sequence, I found adding a FadeIO
helped the mind reset its zero reference between samples. I generate a series of these
clips and alignedsplice them all together. At the end of the process I then slow
everything down using AssumeFPS based on reclocking the audio to 8Khz.

Attached is a 200k .AVI showing the effect. (Wait for the Moderator)

Hope someone find this concept useful
IanB


Sample Script
==========
#
# Generate 11 variations of an input clip with varying audio delays
#
# Seperate into fields to double video temporal resolution
# Slow everything down to exagerate the Audio lipsync error
#
# Iterate the following values to find the optimal AudioDelay setting

Global Offset= 0.000 # Middle value of the sweep range being tested
Global Scale = 0.250 # Scale the sweep range from +/- 1 second

AVISource("Nemisis.avi", True, "YUY2") # Select the input file
# ===========
AssumeFrameBased() # Correctly describe frame/field attribute
AssumeTFF() # Correctly describe field order
Trim(6115, -30) # Define the range of frames to investigate

Bob(Height=240) # Double the FrameRate, Choose a good height

BiLinearResize(320, Height()) # Choose a good width

#
# AlignedSplice each prospective clip sample, specify a range of delays
#
DoDelayAudio(-1.0) ++ \
DoDelayAudio(-0.8) ++ \
DoDelayAudio(-0.6) ++ \
DoDelayAudio(-0.4) ++ \
DoDelayAudio(-0.2) ++ \
DoDelayAudio( 0.0) ++ \
DoDelayAudio(+0.2) ++ \
DoDelayAudio(+0.4) ++ \
DoDelayAudio(+0.6) ++ \
DoDelayAudio(+0.8) ++ \
DoDelayAudio(+1.0)

#
# Slow everything down, keep Audio and Video synced
# Audio will be 8kHz, Video will be slowed to match
#
AssumeFPS(8000.0 * FrameRate() / AudioRate(), True)

#
# Adjust the Audio Delay, Add a Subtitle, Fade In/Out
#
Function DoDelayAudio(Clip Clip, Float Delay) {
Delay=(Delay * Scale) + Offset
Return DelayAudio(Clip, Delay).Subtitle("Delay " + String(Delay)).FadeIO(6)
}

[onesoul]

Very nice work, thanks for sharing :).
A question to anyone: Once you find the delay, is it possible to generate silence by avisynth to put at the begining (or ending) of the video?

Regards

[scmccarthy]

@onesoulOnce you find the delay, is it possible to generate silence by avisynth to put at the begining (or ending) of the video? Just use the standard delay function in avisynth or virtualdub or besweet or HeadAC3he or azid or ac3dec. no need to add silence at the beginning. One thing to consider: if you stare at the actors lips too closely, you may be seeing a poor looping job on there part rather than an out of sync movie. Try to pick a scene that was not looped.

Stephen

[onesoul]

@scmccarthy
Oops, now I see I wasn't specific enough :). I already know (roughly) to add delay to sound but the problem I was trying to refer to is that if I don't add a silence wav to the gap left I will have synch problems at my stand alone player, of course at pc it wouldn't be a problem.

[sh0dan]

Nice - I added it to the ShareFunctions (http://www.avisynth.org/index.php?page=ShareFunctions) page at avisynth.org. I hope you don't mind (otherwise you can remove it yourself :-).

[scmccarthy]

@onesoul

Generally, if you add the delay separetly, it will be encoded directly into the audio. I love BeSweet. But you want to do in in avisynth. I'd still try re-encoding decoding the ac3 file with corrected delay settings.

Stephen

[IanB]

@onesoul

Crop(BlankClip(1) ++ DelayAudio(+0.123), 1, 0)

Would guarrantee the left edge alignment you are looking for, but I think the
AVS internals should look after this case already.


And it might be prudent to do the following for -ve DelayAudio's

Delta=-0.456 # Audio Delay value
# Add sufficient blank frames to the start to cover any left "overhang"
DelayAudio(BlankClip(Ceil(-Delta / FrameRate()) + Last), Delta)


And of course the same tricks might be applied to the right edge if that is a concern.

IanB

[onesoul]

@scmccarthy
I see what you mean, maybe besweet is the more easier way to do the delay without the "need" to insert silence, nevertheless it is nice knowing how to do it at avisynth. :)

@IanB
Thank you for the suggestions. It seems it will do the trick. :)

[DJ Bobo]

Just wanted to say, that this method may be only helpful for real movies.

For anime, especially japanese dubbed, it won't succeed.
The best solution remains, load the WAV in WaveLab or another application that shows graphs and millisecond timing, look in the video for a frame where something hits something else, note the timing, look for that place in the WAV graph and note the time where it happens there (it's quite easy to find, because when something hits, the graph goes high at that position)