Help me improve a mouth animating script, please.

[NerdWithNoLife]

I developed a method to make an oscilloscope of a WAV, then animate mouth movements based on the amplitude of that wave with images of the mouth 0/25/50/75/100% open. But the scripting isn't pretty and it sometimes results in crashes during encoding. And it's still pretty sloppy - sorry - as it is the result of much trial and error. Here is the script:

#Create a slug. Dimensions are only used to create the oscilloscope.
#Ideally the length of the clip in frames would be based on the audio length,
#but I did the math by hand because I don't know how to automate that.
a=blankclip(120,480,360,pixel_type="rgb24").assumefps(24)
Mouthlength=framecount(a)

#Load audio; make an oscilloscope and convert it to a mask
aud=wavsource("mouthaudio.wav") #Like my Homer Simpson impression?
a=audiodub(a,aud)
Osc=a.converttorgb32().audiograph(2).crop(193,148,-193,-152)
Osc=Osc.bilinearresize(2,60).crop(1,0,0,0).showalpha().pointresize(40,9).levels(254,1,255,0,255).Invert()
#View Osc if you want to see the concept in action.

#Determine whether the mouth should open 0/25/50/75/100%. Black=closed; white=open.
#This will be used as an alpha mask for each of the images.
#Take a sample pixel, then resize it to the size of the images. Point resize wouldn't work
#with very small areas, so I used stack vertical before the resize.
#Ideally, I'd like to simply evaluate whether the 0/25/50/75/100% regions are $000000 or $FFFFFF
m25m=stackvertical(osc.crop(0,3,2,1),osc.crop(0,3,2,1)).pointresize(200,192)
m50m=stackvertical(osc.crop(0,2,2,1),osc.crop(0,2,2,1)).pointresize(200,192)
m75m=stackvertical(osc.crop(0,1,2,1),osc.crop(0,1,2,1)).pointresize(200,192)
m100m=stackvertical(osc.crop(0,0,2,1),osc.crop(0,0,2,1)).pointresize(200,192)

#Load Images of the mouth 0/25/50/75/100% open. Mask each as either visible or transparent,
#based on the black/white binary masks above.  Ideally I would like to simply choose which image to
#use based on the above $000000 or $FFFFFF values, rather than layering all 5 images together,
#but I'm not smart enough and I don't know if there's a way to return a color value based on
#a given pixel in a clip.
m00=imagesource("00.png",pixel_type="rgb32").loop().trim(0,Mouthlength)
m25=mask(imagesource("25.png",pixel_type="rgb32"),m25m).loop().trim(0,Mouthlength)
m50=mask(imagesource("50.png",pixel_type="rgb32"),m50m).loop().trim(0,Mouthlength)
m75=mask(imagesource("75.png",pixel_type="rgb32"),m75m).loop().trim(0,Mouthlength)
m100=mask(imagesource("100.png",pixel_type="rgb32"),m100m).loop().trim(0,Mouthlength)

#Put it all together. Cross your fingers and hope it doesn't crash during encoding.
mouth=layer(m00,m25,"add",255)
mouth=layer(mouth,m50,"add",255)
mouth=layer(mouth,m75,"add",255)
mouth=layer(mouth,m100,"add",255)

audiodub(mouth,aud)

I tried to explain it as best as I can. Here is a zip with an example. It requires AudioGraph.

It would be neat if this could be refined so others can use it to save a lot of animating time.

[mikeytown2]

just off the top of my head, AudioGraph doesn't work all the time
http://forum.doom9.org/showthread.php?t=59412

Here's an alternative plugin that might be better
MinMaxAudio()
http://avisynth.org/mediawiki/Extern...#Audio_Filters
http://forum.doom9.org/showthread.ph...99#post1043099
Change the framerate before feeding the audio into that function and you should be able to achieve what you need.

Haven't tested your code yet, will play around with it this weekend.

[NerdWithNoLife]

I think you're right: the errors I got were similar to the ones mentioned in the AudioGraph thread. That's probably what's causing the crashing.

[tacman1123]

Very cool problem. I tried to solve a similar problem, and my solution required a lot of external functions that I'm now wondering if it can be done (or somewhat done) through Avisynth. Seems like there's always a way to do it in Avisynth!

Anyway, I wanted to find the long pauses in a video, so I could reduce them (probably blurring the video and sound on either side). My solution was to export the audio as a .wav, then run a script that read the .wav file and looked for periods of relative silence. I then had the frame ranges with sound, which I could work with using Trim and Splice.

What I'm wondering if there's a way using one of these filters to export the frame and min/max volume (or average, whatever), which I could then process and ultimately create a file for ApplyRange. Is there a way to export the audio analysis?

Thx,

Tac

[NerdWithNoLife]

I wish I knew enough to do this, but - in human language, not computer - what I want to do using MinMaxAudio is: if the volume in decibels is x, use image y. In other words,

If the volume is -6db or higher, use the 100% open image (100.png)
If the volume is -12db or higher, use 75.png
If the volume is -24db or higher, use 50.png
If the volume is -36db or higher, use 25.png
If the volume is less than -36db, use 00.png

You know, like:

db >= -6 ? m100
         : db >= -12 ? m75

Or something... this means there would have to be a way to assign the MaxAudio value to a variable named db on a framewise basis, then assign the frame an image based on that value. Any ideas?

[tin3tin]

Southpark style(Canadians):
Mouth_tin2tin.zip

Add the min max plugin from above to this folder.

[NerdWithNoLife]

LOL! Genius. I think the mouth was moving the wrong way, so adjusted the line with the overlay:

LoadPlugin("MinMaxAudio.dll")
face=ImageReader("Face.png",0,124,pixel_type="RGB32",fps=25).crop(1,1,0,0)
jaw=ImageReader("Jaw.png",0,124,pixel_type="RGB32",fps=25).crop(1,1,0,0)
a=wavsource("mouthaudio.wav").Normalize(1)

w = AudioDub(face,a) #.Histogram(mode="audiolevels")
#w = ScriptClip(w, "Subtitle(String(AudioRMS(0)))")

ScriptClip(w, "overlay(jaw,x=0,y=round((AudioRMS(0)*1/10)+6),mask=ShowAlpha(jaw))")

[Gavino]

Originally Posted by tacman1123

What I'm wondering if there's a way using one of these filters to export the frame and min/max volume (or average, whatever), which I could then process and ultimately create a file for ApplyRange. Is there a way to export the audio analysis?

You could use WriteFile to log the value you want for each frame (eg AudioRMS) to a file. You'd end up with quite a large file though, so you might want to refine the output with some conditional processing.

[tin3tin]

Maybe Papagayo is something for you?

Apparently there is a MOD too which exports video: http://www.lostmarble.com/forum/viewtopic.php?t=11373

[NerdWithNoLife]

I got it! It took loads of time tinkering (and educating myself) because MinMaxAudio seems to only work as a conditional filter. Never done anything like this before so to me this is really cool.

AudLength=5 #Set Audio Length in seconds

m00=ImageSource("00.png",pixel_type="rgb32").trim(0,-1)
m25=ImageSource("25.png",pixel_type="rgb32").trim(0,-1)
m50=ImageSource("50.png",pixel_type="rgb32").trim(0,-1)
m75=ImageSource("75.png",pixel_type="rgb32").trim(0,-1)
m100=ImageSource("100.png",pixel_type="rgb32").trim(0,-1)
Slug=BlankClip(m00).loop().trim(0,AudLength*24)

Aud=WavSource("mouthaudio.wav")#.Normalize(1) #If you need Normalizing.
Mouth=AudioDub(slug,aud)

ScriptClip(Mouth, "Overlay( AudioMax(1) >= -6 ? m100 \
: AudioMax(1) >= -12 ? m75 \
: AudioMax(1) >= -24 ? m50 \
: AudioMax(1) >= -36 ? m25 \
: m00 ) ")

AudioDub(last, WavSource("mouthaudio.wav"))

You just need the MinMaxAudio plugin, 5 equally sized images to represent the cartoon mouth 0/25/50/75/100% open, and a WAV file for your audio. Then just set the length of the audio in seconds and the script should do the animation for you. Normalizing the file caused an error in realtime playback for me though. Thanks again tin3tin for the genius idea of using overlay.

Edit: Added Gavino's line to prevent crashing.

[Gavino]

Originally Posted by NerdWithNoLife

... MinMaxAudio seems to only work as a conditional filter.

That's because it delivers its results on a per-frame basis (like AverageLuma etc), so it only makes sense within the conditional environment.

Even without the normalizing, THIS SCRIPT IS PRONE TO CRASHES, JUST LIKE AUDIOGRAPH IS. Dang.

I have a theory that these crashes (both here and in AudioGraph) are due to the audio source filter being accessed on the video processing path. If the Avisynth client uses a separate thread for video and audio, this could cause problems for non-thread-safe source filters.

Try duplicating the audio source and see if it helps. Add this line to the end of your script:

AudioDub(last, WavSource("mouthaudio.wav")) #2nd instance of WavSource

[NerdWithNoLife]

Originally Posted by Gavino

I have a theory that these crashes (both here and in AudioGraph) are due to the audio source filter being accessed on the video processing path. If the Avisynth client uses a separate thread for video and audio, this could cause problems for non-thread-safe source filters.

Try duplicating the audio source and see if it helps. Add this line to the end of your script:

AudioDub(last, WavSource("mouthaudio.wav")) #2nd instance of WavSource

It worked. I edited my post to include that line in the script. You just saved me a lot of aggravation.

[IanB]

Originally Posted by Gavino

I have a theory that these crashes (both here and in AudioGraph) are due to the audio source filter being accessed on the video processing path. If the Avisynth client uses a separate thread for video and audio, this could cause problems for non-thread-safe source filters.

No need to theorise, doing so definitely will crash! This is part of the reason for this little change item in 2.5.8

Changes:
...
* Added critical section to CAVIFileSynth class.

Of course this only protects clients using the AVIFile interface. Apps that directly use the IScriptEnvironment interface are free to do whatever they want and are responsible for protecting against unsafe concurrent accesses. Even TSP's MT builds are only thread safe in the video chain.

[mikeytown2]

Is there any reason why this won't work?

Aud=WavSource("mouthaudio.wav")
Aud.AssumeFPS(24).ConvertToRGB32().AddBorders(200,192,0,0)

I get this error
avisynth.AvisynthError: Input colorspace is not RGB24 or RGB32


Or is there different way to get the Wav's length?

[IanB]

WavSource("mouthaudio.wav") has no video stream. Use AudioDub(vid, aud) to add one. If you just want a canvas to Info() over use vid=Blankclip().

[Gavino]

Originally Posted by mikeytown2

Aud=WavSource("mouthaudio.wav")
Aud.AssumeFPS(24).ConvertToRGB32().AddBorders(200,192,0,0)

I get this error
avisynth.AvisynthError: Input colorspace is not RGB24 or RGB32

Arguably, this is a deficiency of AddBorders which does not check that the input clip has video.

Or is there different way to get the Wav's length?

Do you mean the length (in seconds, say) of the audio clip?
Use AudioLengthf()/AudioRate()

[mikeytown2]

Thanks Gavino, didn't even know about those audio functions
http://avisynth.org/mediawiki/Clip_properties


Here is a more generalized function

#Load Sources
Aud=WavSource("mouthaudio.wav")#.Normalize(1) #If you need Normalizing.
m00=ImageSource("00.png",pixel_type="rgb32",0,0)
m25=ImageSource("25.png",pixel_type="rgb32",0,0)
m50=ImageSource("50.png",pixel_type="rgb32",0,0)
m75=ImageSource("75.png",pixel_type="rgb32",0,0)
m100=ImageSource("100.png",pixel_type="rgb32",0,0)

#Set Parameters
FRate = 24
High = -6
Low = -36
Steps = 5

#Set up clip
Gap = (Low-High)/Steps
AudLength=AudioLengthF(Aud)/AudioRate(Aud)
Slug=BlankClip(m00).AssumeFPS(FRate).loop().trim(0,Ceil(AudLength*FRate))
Mouth=AudioDub(slug,aud)

#"Animate" Clip
ScriptClip(Mouth, \
"Overlay( AudioMax(1) >= High+Gap*0 ? m100 \
: AudioMax(1) >= High+Gap*1 ? m75 \
: AudioMax(1) >= High+Gap*2 ? m50 \
: AudioMax(1) >= High+Gap*3 ? m25 \
: m00 ) ")

#Fix ScriptClip Audio Glitch
AudioDub(last, WavSource("mouthaudio.wav"))

[IanB]

@mikeytown2,

Golden rule! Do not evaluate the script, evaluate only the decision! i.e.

...
m100= Overlay(Mouth, m100)
m75 = Overlay(Mouth, m75)
m50 = Overlay(Mouth, m50)
m25 = Overlay(Mouth, m25)
m00 = Overlay(Mouth, m00)

ScriptClip("AM=AudioMax(1)
  AM >= High+Gap*0 ? m100 \
: AM >= High+Gap*1 ? m75 \
: AM >= High+Gap*2 ? m50 \
: AM >= High+Gap*3 ? m25 \
:                    m00")
...

[NerdWithNoLife]

Added a framerate numerator and denominator setting to mikeytown2's script (for those of us in NTSC land). Modified the script to follow the golden rule:

#Load Sources
Aud=WavSource("mouthaudio.wav")#.Normalize(1) #If you need Normalizing.
m00=ImageSource("00.png",pixel_type="rgb32",0,0)
m25=ImageSource("25.png",pixel_type="rgb32",0,0)
m50=ImageSource("50.png",pixel_type="rgb32",0,0)
m75=ImageSource("75.png",pixel_type="rgb32",0,0)
m100=ImageSource("100.png",pixel_type="rgb32",0,0)

#Set Parameters
FRateNumerator = 30
FRateDenominator = 1
High = -6
Low = -36
Steps = 5

#Set up clip
Gap = (Low-High)/Steps
AudLength=AudioLengthF(Aud)/AudioRate(Aud)
Slug=BlankClip(m00).AssumeFPS(FRateNumerator,FRateDenominator).loop()\
.trim(0,Ceil(AudLength*FRateNumerator/FRateDenominator))
Mouth=AudioDub(slug,aud)
m100= Overlay(Mouth, m100)
m75 = Overlay(Mouth, m75)
m50 = Overlay(Mouth, m50)
m25 = Overlay(Mouth, m25)
m00 = Overlay(Mouth, m00)

#"Animate" Clip
ScriptClip(Mouth,"AM=AudioMax(1)
  AM >= High+Gap*0 ? m100 \
: AM >= High+Gap*1 ? m75 \
: AM >= High+Gap*2 ? m50 \
: AM >= High+Gap*3 ? m25 \
:                    m00")

#Fix ScriptClip Audio Glitch
AudioDub(last, WavSource("mouthaudio.wav"))

[Gavino]

@NerdWithNoLife: It probably doesn't matter much, but note that this new version (inherited from Mikeytown2) has slightly different parameters to your original one.

Your original used -6, -12, -24 and -36dB as the dividing points between the mouth positions, while the revised version uses -6, -12, -18 and -24. Since this uses a constant spacing, it's probably more appropriate, but it's not what you had originally (although perhaps it's what you meant!).

As it stands, it's a bit misleading to use the value Low=-36 in the script - it would be clearer to use
Low = -24
Steps = 3
which would have the same effect.