DePan usage

[ajk]

Hey guys,

I need some help with DePan. I'm not sure if I'm just not reading the manual correctly or if there's something that is conceptually wrong with what I'm trying to do, but any help would be appreciated.

Let's assume there is a stream of frames. What I want to achieve is the following:

- for each even frame, (globally) compensate it to match the preceding odd frame (or the other way around, doesn't really matter)
- data from the neighbouring frames should be used to fill in the borders as much as possible
- then move on to the next pair of frames from a clean slate

In the end result for each pair of frames one would remain original and the other one would be matched up with that one. I need this for a sort of blend deinterlace filter. We can assume that a good match can always be found (mostly just need to compensate for panning), and no need to worry about scene changes or anything.

DePan seems like exactly the right tool, but I can't figure out how to achieve this type of thing. MVTools or something else is fine too if it's better for the job. Help would be most appreciated!

[pbristow]

OK, the only non-standard part of what you want to do is the middle step, i.e.:
[QUOTE- data from the neighbouring frames should be used to fill in the borders as much as possible[/QUOTE]

I think the process you want for that is:

0. Create a blank clip with larger frame size than the source video, i.e. big enough for the video itself plus whatever size border you want to try to fill-in.
1. Shift previous frame to be positioned same as current field, and overlay it on the blank clip;
2. Shift *next* frame to be positioned same as current field, and overlay it on result of 1, treating the border as transparent
3. Overlay current frame on top of result of 2.

This should give you a clip in which the dominant image is that last one added, i.e. the current frame, but with copies of the previous and next frame "sticking out from behind", supplying those extra bits of information in the borders. Is that what you want?

The trick is how to overlay the clips in such a way that (a) their edges don't get cropped to match the edges of source frame (which is why you have to add the big border to everything first), but (b) their new blank border doesn't overwrite useful information that's already in the new frame you're constructing. So the borders have to be treaed as *transparent*, rather than as black. For this you're going to need to look at using the Alpha channel and/or masktools, I think.

So the tools you want are probably DePan (using either it's own DePanEstimate function to do the motion estimation, or the alternative one in MVTools2), plus BlankClip and Overlay, and various others to handle the masking.

Complicated job. Good luck! :)

[ajk]

Yes, that is exactly what I want to do. :) I managed to do the other stuff but the border fill seems tricky. I thought it would be already built into DePan, as there is a mention in the DePanEstimate docs: "prev - max lag of some previous frame to fill empty borders (instead of black)" and the same for "next".

I'll have to give this some thought, perhaps I can just get away with some kind of soft mask that leaves the borders alone altogether. Can't zoom in the image unfortunately. Thanks for the tips, let's see what works best.

[pbristow]

What's your ultimate goal? To improve the look of one specific clip, or to create a better/faster/easier-to-use motion-compensated de-interlacer than the ones we already have, or just to teach yourself more about the issues & processes involved? Because reading through your first post again, you're really re-inventing the wheel here, starting at the "how many corners should it have?" stage. :) It's a useful educational exercise to go through the stages of figuring it all out, but it's not likely to lead to a better wheel than we already have.

[ajk]

It's for a specific purpose, not really deinterlacing per se, but frame rate reduction while still maintaining visual characteristics of the source. As far as I can imagine no regular deinterlacer should be able to help with this type of thing. I can post an example this evening when I get home. Nothing secret, I just wanted to try my idea out first rather than have the thread go sideways with other suggestions :) But I'm certainly open to other ideas if anyone can come up with something, so I'll post the sample later!

[ajk]

Ok, the full details as promised :)

I have a project where I want to record some footage of old video games. Many of these games have smooth 50Hz scrolling and capture nicely as such. However, YouTube and most similar video services don't work with 50fps material, so for the time being I need to make half-framerate versions.

The obvious way is to just drop every other frame and have the scrolling be a little jerkier. Many games however simulate transparency effects by flicking an image on and off (or between two images) each field/frame. Doing a SelectEven() in this case completely destroys the effect.

The next obvious way is to just blend the consecutive frames together to create actual transparency. This looks decent, but then the background is blurry most of the time due to the scrolling.

What I wanted to do was to in some reasonable way separate the local movement from the background scrolling and only apply the blending to that. This keeps the background reasonably crisp and retains the feel of the effects.

Here is a short Lagarith clip of one capture I made: source.clip.lags.avi (http://ajk.pp.fi/avisynth/source.clip.lags.avi)

This is the script I came up with:


AVISource("source.clip.lags.avi")

PartialBlend(Crop(0,32,0,-32))


function PartialBlend(clip i, clip cropped)
{
xm = 32
ym = 32
bt = 132

mdata = cropped.DePanEstimate(trust=0,dxmax=xm,dymax=ym)
depanned = i.DePan(data=mdata,offset=-1)

orig = i.SelectOdd()
comp = depanned.SelectOdd()

mask = Overlay(orig,comp,mode="difference").GreyScale()
mask = mask.mt_binarize(bt).Blur(1).Blur(1)

# Simulate a bit of flickering, can be skipped for neutral transparency
blended1 = Overlay(i.selecteven,i.selectodd,opacity=0.33)
blended2 = Overlay(i.selecteven,i.selectodd,opacity=0.67)

Interleave(blended1,blended2)
blended = last.SelectEvery(4,0,3)

return Overlay(i.SelectEven(),blended,mask=mask)
}


Here is a comparison of the different methods: tests.mkv (http://ajk.pp.fi/avisynth/tests.mkv)

It's not perfect, as often there is so much "local" movement that DePan can't lock on to the background reliably. In general however I think the background stays much crisper than with a straight blend, and the effects also retain their original appearance. I plan to improve the masktools stuff a bit more but this is the gist of it.

Any ideas for improvement of my method or better ideas for handling this kind of source?

[-Vit-]

There was a thread about this earlier (http://forum.doom9.org/showthread.php?p=1523722). The topic continued onto another forum and there was some further development. When I last read they were talking about tweaking the emulator to fix the problem more directly.

This is a more direct MVTools method of what you're trying to achieve (it converts from and back to RGB, which I guess you don't need). Haven't looked at your source, you may need to tweak the MVTools settings for your resolution. The remaining problems are, as ever, failures in motion analysis, where the algo can't tell the difference between complex motion and flicker. Also, the entire concept fails with rapid screen shake. I did some deeper analysis on this, looking at the motion between the every other frame to help. There's definitely scope for improvement, but this is often good enough.

YourSource(...)

# RGB to YV12 keeping accurate chroma
PointResize(Width()*2,Height()*2).ConvertToYV12(matrix="PC.601", chromaresample="point")

# How accurately the filter will identify flicker. Increase if flicker is being lost, will create larger blended areas
Sensitivity = 100

clip = last
w = clip.Width()
h = clip.Height()

# Create blended frames
c = clip.SelectEvery( 2, 1,0,0,1 )
blend = Merge( c.SelectEven(), c.SelectOdd(), 0.667 )

# Prefilter for motion analysis (straight out of TGMC)
srchClip = clip.RemoveGrain(12).GaussResize( w,h, 0,0, w+0.0001,h+0.0001, p=2 ).Merge( clip, 0.1 )
srchSuper = srchClip.MSuper( pel=1 )

# Motion analyze neighbor frames - flickering areas should not find a match
fVec = srchSuper.MAnalyse( isb=false, blksize=16, overlap=0, truemotion=false )
bVec = srchSuper.MAnalyse( isb=true, blksize=16, overlap=0, truemotion=false )
fVec = srchSuper.MRecalculate( fVec, blksize=8, overlap=0, truemotion=false )
bVec = srchSuper.MRecalculate( bVec, blksize=8, overlap=0, truemotion=false )

# Get SAD masks from current to neighbor frames - a measure of motion match confidence.
# Collect together those parts of current frame that *can* be found in the neighbors (maskInner)
# and those parts of the neighbor frames that can be found in the current (maskOuter)
# Combine to get a mask of the areas that are probably not flickering because they can be found in successive frames
maskb = MMask( clip, bVec, kind=1, ml=Sensitivity+0.1 ).mt_binarize( 128, mode="upper", U=1,V=1 )
maskf = MMask( clip, fVec, kind=1, ml=Sensitivity+0.1 ).mt_binarize( 128, mode="upper", U=1,V=1 )
maskInner = mt_logic( maskb, maskf, "min", U=1,V=1)
maskOuter = mt_logic( maskb.SelectEvery(1,-1), maskf.SelectEvery(1,1), "min", U=1,V=1 )
mask = mt_logic( maskInner, maskOuter, "min", U=1,V=1 )
mask = mask.PointResize( w/4, h/4 ).mt_inpand( U=1,V=1 ).BilinearResize(w,h) # Expand the areas identified as flickering a little

# Merge blended frames with normal frames based on mask
mt_merge( blend.SelectEvery(4, 0,3), clip.SelectEven(), mask.SelectEven(), luma=true )

# Restore to original sized RGB
ConvertToRGB(matrix="PC.601",chromaresample="point").PointResize(w/2,h/2)

[ajk]

Nice, I had missed that earlier thread, didn't really know what to search with. I'll give your script a try later. Mine is not recorded in an emulator so there is some noise and other stuff in addition, but it shouldn't make too much difference. Thanks!

[Gavino]

# RGB to YV12 keeping (almost?) accurate chroma
PointResize(Width()*2,Height()*2).ConvertToYV12(matrix="PC.601")
...
# Restore to original sized RGB
ConvertToRGB(matrix="PC.601",chromaresample="point").PointResize(w/2,h/2)

Since you must be using v2.60 here (as chromaresample is not available in 2.58), the initial conversion to YV12 should also have chromaresample="point" (or "bilinear") - otherwise it will use the default bicubic resampling which will mix the original pixels.

[ajk]

I gave that script a go and it does work pretty well. There is all kinds of little MVTools weirdness though, sometimes a single block in the middle of something doesn't get treated right etc., which can be quite distracting. Overall the motion tracking works really well though, which got me wondering if it would be possible to do global frame movement estimation with MVTools instead of DePan. Is there a way to get the average, or rather most common, movement vector and apply that offset to the entire frame?

[creaothceann]

Here is a short Lagarith clip of one capture I made: source.clip.lags.avi (http://ajk.pp.fi/avisynth/source.clip.lags.avi)

Are you going to do the whole game? I tried something quite similar. (http://www.mediafire.com/?r2s46ip2aj2htmk) :)

It should be fine to use ConvertFPS (or SelectEven) for the whole clip, and replace only the parts that need the special blending (http://tasvideos.org/EncodingGuide/TASBlend.html) code.

[ajk]

Nah, probably just some short clips. I agree about selective filtering, but unfortunately here it still basically means each time the character is shooting, which is most of the time :) I've been fine tuning my script and tried MVtools/MDepan too, I think the results are good enough. I can post a further comparison video tomorrow.

Nice work with your videos btw :) I assume that is the later DOS version of the game.

[creaothceann]

Yep.

Btw. have you tried this?

AVISource(...)
# AssumeFPS(50) # if it's not already 50.0
Interleave(last, last, last)
AssumeFPS(150)
SelectEvery(5)
# TimeStretch(rate=100.0 * (AudioLengthF / AudioRate) / (FrameCount / FrameRate))

#info