60 -> 30 fps flickering with minimum blur

[creaothceann]

Over at tasvideos.org (http://tasvideos.org/forum/viewtopic.php?t=11459) we're in a specific situation: videogame emulator recordings are 60fps, but Youtube allows only 30fps clips. So the first solution would be to drop every other frame. Unfortunately many games simulate transparency by turning off layers every other frame (e.g. when the player's sprite was hit by an enemy). On YT this shows up as either fully transparent or opaque sprites, depending on if the first frame was even or odd.

After some time we came up with TASBlend (http://tasvideos.org/EncodingGuide/TASBlend.html) (script and 60fps example clip (http://www.mediafire.com/?v7brz4a9bjgdf)). It keeps the flickering; the problem is a slight blurring of the whole screen. So...

Maybe we could somehow filter out the pixels that are the same color every other frame, but other colors in-between; the resulting mask could then be used to overlay the TASBlend-ed clip only to the detected (flickering) regions?

[-Vit-]

Edit: I originally put a solution up that only worked by making a particular assumption, but it wasn't robust in general. This version is more in line with the OP's request:


function TASBlend(clip c, float "ratio" ) {
# reduces framerate to 1/2 but leaves flicker effects partly visible
# blends frame pairs with alternating opacity (default is 2/3+1/3;1/3+2/3)
# optional "ratio" is the opacity of the first frame out of the four
ratio = default( ratio, 2.0/3 )
opacity = round( ratio * 257 )

# Blend flicker to half-rate
c.SelectEvery( 4, 1,0,2,3 )
blend = Layer( SelectEven(), SelectOdd(), level=opacity )

# Determine flicker is a simplistic way - compare each pixel with the previous and next two frames
Similar2 = 1 # How similar current pixel must be to the same pixel 2 frames forwards and backwards
Different1 = 8 # How different current pixel must be from the same pixel 1 frame forwards and backwards
maskf2 = Overlay( c, c.SelectEvery(1, 2), mode="difference", pc_range=true ).Greyscale().Levels( 128+Similar2, 1.0,129+Similar2, 0,255, false ).Invert()
maskf1 = Overlay( c, c.SelectEvery(1, 1), mode="difference", pc_range=true ).Greyscale().Levels( 128+Different1,1.0,129+Different1, 0,255, false )
maskb1 = Overlay( c, c.SelectEvery(1,-1), mode="difference", pc_range=true ).Greyscale().Levels( 128+Different1,1.0,129+Different1, 0,255, false )
maskb2 = Overlay( c, c.SelectEvery(1,-2), mode="difference", pc_range=true ).Greyscale().Levels( 128+Similar2, 1.0,129+Similar2, 0,255, false ).Invert()
maskf = Overlay( maskf1, maskf2, mode="multiply", pc_range=true )
maskb = Overlay( maskb1, maskb2, mode="multiply", pc_range=true )
mask = Overlay( maskf, maskb, mode="multiply", pc_range=true ).SelectEven()

# Only use blend where detected flicker
Layer( c.SelectEven(), blend.Mask(mask) )
}

Have to check both the previous and next frames, checking in one direction leads to flicker detection simply by something passing in front of a plain background [i.e. pixel goes black to color to black again]. Ideally this would suit motion compensation, but that would be difficult on such small, stylized and fast moving graphics. Consequently the approach loses flicker on fast moving objects. Tweak the balance between too much blending and too little with the Different1 and Similar2 values.

I'm sure there's a better way than all those Overlays & Levels, perhaps some TemporalSoften trickery, or convert to YUV and use MaskTools instead (since Overlay converts to YUV anyway), but I put this together quickly to correct my earlier mistake...

[creaothceann]

Thank you. :) I'll try that.

Wouldn't Layer work too instead of Overlay?

[jmac698]

You can still use Masktools on RGB with this simple preprocessing step:

function RGBintoYV12(clip v) {
#place r,g,b data directly into u,y,v planes
#this result has no visual meaning but is only for achieving arithmetic on the data
v
pointresize(width*2,height*2)#doublesize to allow full 4:4:4 color resolution with u,v
r=ShowRed.converttoyv12
g=ShowGreen.converttoyv12
g=g.pointresize(width*2,height*2)#this is destined for Y which must be bigger
b=ShowBlue.converttoyv12
YtoUV(r,b,g)
}

[-Vit-]

creaothceann: There is no "difference" mode in Layer afaik. There is "subtract", but that does something rather different. However, Overlay is only used to build the mask.
BTW the algorithm you proposed is reasonably effective, but has limitations. I already noted the issue with moving flickering objects - the pixels aren't in the same place frame to frame so the flicker is not detected. Also flickering objects that are also animating or changing color at the same time - individual pixels may be briefly mis-detected as not flickering because the color changes "cancel out" the flicker temporarily.

jmac698: Something like that + some masktools may well be more efficient. Though efficiency is not a major issue on such a small resolution.

[creaothceann]

Yeah, I was afraid of that... oh well. My only idea would be to increase the detected area somehow to compensate for moving sprites.

Anyway, here's some new code that shows the best results so far:
http://tasvideos.org/forum/viewtopic.php?p=286117#286117

[wonkey_monkey]

Over at tasvideos.org (http://tasvideos.org/forum/viewtopic.php?t=11459) we're in a specific situation: videogame emulator recordings are 60fps, but Youtube allows only 30fps clips.

Does it allow 20fps? I only suggest this for completeness, really, but at 20fps you'd be keeping every third frame, so flickering sprites would still flicker. You've already lost half your motion going to 30fps...

David

[creaothceann]

20 fps just has too jerky motion, imo. Besides, tool-assisted speedruns are created mostly frame-by-frame, so it's important to show as many fps as possible to not miss too much.

[-Vit-]

I see on the TAS site you are trying MVTools approaches. Below is a refined version of that. It works very well on the sample you gave above, but it may yet need tweaking. However, in general I would expect problems with any MVTools approach where there are fast animations or detailed backgrounds. The "Sensitivity" setting in the code might help, but... [Edit: I just tried it on the Roadrunner clip you have on the TAS site, a difficult example, had to cut the Sensitivity value to the range 0 to 20 for best results]

PointResize( Width()*2, Height()*2 )
ConvertToYV12() # Add matrix="PC.601" if you intend to convert back to RGB at the end

# How accurately to identify flicker. Increase if flicker is being lost, but with
# the increased chance of distortion or ghosting. Or decrease to reduce distortion
# and ghosting but with potential to miss flicker
Sensitivity = 75

clip = last
clipOdd = clip.SelectOdd()
w = clip.Width()
h = clip.Height()

srchClip = clipOdd.RemoveGrain(12).GaussResize( w,h, 0,0, w+0.0001,h+0.0001, p=2 ).Merge( clipOdd, 0.1 )
srchSuper = srchClip.MSuper( pel=1 )
fVec = srchSuper.MAnalyse( isb=false, blksize=16, overlap=4, truemotion=false )
bVec = srchSuper.MAnalyse( isb=true, blksize=16, overlap=4, truemotion=false )
fVecR = srchSuper.MRecalculate( fVec, blksize=8, overlap=2, truemotion=false )
bVecR = srchSuper.MRecalculate( bVec, blksize=8, overlap=2, truemotion=false )

oddSuper = clipOdd.MSuper( pel=1 )
newOdd = clipOdd.MFlowFPS( oddSuper, bVecR, fVecR, num=0, mask=2, ml=50 ) # Could tweak the ml value here too...

maskb = MMask( clipOdd, bVecR, kind=1, ml=Sensitivity+0.1 )
maskf = MMask( clipOdd, fVecR, kind=1, ml=Sensitivity+0.1 )
mask = mt_logic( maskb.SelectEven(), maskf.SelectOdd(), "max", U=-128,V=-128 )

Interleave( clip.SelectEvery(4,0), mt_merge( newOdd.SelectEvery(4,1), clip.SelectEvery(4,2), mask, luma=true ) )

# Add these lines to restore original sized RGB (chromaresample requires AviSynth 2.6)
#ConvertToRGB( matrix="PC.601", chromaresample="point" )
#PointResize( Width()/2, Height()/2 )

[2Bdecided]

This is a real challenge.

The perfectionist does the best they can on YouTube, but provides a link to a proper 60fps download at the top of the YouTube description. Mediafire seems to work well for this.

Cheers,
David.