MonkeyDeint - initial results

[wonkey_monkey]

Hi,

A quick disclaimer - although I'm massively into digital video, I'm not massively into the digital video scene, so I don't really know much about what's being developed.

I work in TV and when I get to play with video (it's not really part of my job but it happens) it's almost always interlaced. That got me thinking about deinterlacers, and I found this page:

AviSynth Deinterlacers Comparison (http://heptium.sh.cvut.cz/~integra/deint/)

I wondered if I could write something to do better, and I've now got a very rough-around-the-edges program implementing my algorithm. Here are some results for you to peruse.

The first is a crop of the example on that page above. On the left is the original image, second is my current favourite deinterlacer, leakkerneldeint with default parameters, next is my deinterlacer (which for the time being I'll call MonkeyDeint), and on the right is MCBob, as mentioned by WorBry:

Note: MonkeyDeint is the third image.

http://www.channeltv.co.uk/david/football.jpghttp://www.channeltv.co.uk/david/mcfootball.jpg

As you can see, it has artifacts, but the central lines really show off what it might be able to do.

This next example came from a JPEG source, which explains the smeared colours, so please ignore those. The second image is not Leakkerneldeint, but Photoshop's deinterlacer. There's very little difference between the two because the interlacing on this image is so extreme.

http://www.channeltv.co.uk/david/pda.jpghttp://www.channeltv.co.uk/david/mcpda.jpg

This is an arguably better example, which is to be expected given the "cleaner" CGI source, although once again you can see the horizontal artifacts.

Another example:

http://www.channeltv.co.uk/david/sailing.jpghttp://www.channeltv.co.uk/david/mcsailing.jpg

This shows more of the problematic artifacts, particularly in the subtitles at the bottom of the picture, but you can also see increased detail over Leak in the furled sails and clothing.

The actual program is not fast, but it's not yet been optimised in any way and I'm not a brilliant C programmer. I'm using mingw to compile. I also have no idea how to go about writing an AviSynth plugin, although if I continue developing it, that would be my goal.

It might be worth noting that my algorithm (currently) entirely discards the other field and uses no data from it, so it could be used to restore a single separated field to full height.

So... any thoughts or suggestions?

David (aka Monkey)

[WorBry]

I wondered if I could write something to do better

Have you looked at MCBob?

http://forum.doom9.org/showpost.php?p=892518&postcount=31

[wonkey_monkey]

Not yet - I can't seem to get Huffyuv working so I can't look at the examples, but thanks for pointing me to it.

David

[Didée]

Those samples are encoded to "ffdshow-Huffyuv", which is not compatible with the original Huffyuv codec (ff-Huffy uses adaptive huffman tables, and supports YV12). You have to install a recent (i.e. not five years old) version of ffdshow to decode the samples.

[wonkey_monkey]

Thanks Didée - I used VLC to get the frames, but it's hard to tell much from the samples I looked at (the motion ones).

I've updated my first post to include MCBob examples. I used the default parameters, so it may not be an entirely fair comparison, but I've got some pretty good results (apart from the scratch artifacts).

David

[Didée]

Ouch! :D
(That's why MCBob still should be considered "experimental".) To get it deal better with the scrolling text, see
# I've seen at least one tickertape that needed below 'mt_merge' to be activated:
diff2prev12= diff2prev2 #.mt_Merge(diff2prev1,Vedge,U=2,V=2)
in the script.

And while your deinterlacer looks pretty good in still frames -- if you want to give it a hard time, try it on the "ToyTrain" or "Musicians" samples (linked here (http://forum.doom9.org/showthread.php?p=895006#post895006)) ...

Also, there is
And when compairing to this (http://img120.imageshack.us/my.php?image=sarahsecuretp1.jpg) or this (http://img140.imageshack.us/my.php?image=sarahmvbobzq3.jpg) or this (http://img78.imageshack.us/my.php?image=sarahtdeinteedilf5.jpg), I feel quite comfortable with getting this (http://img78.imageshack.us/my.php?image=sarahnewcg4.jpg) instead. Feels a bit like quadrature of the circle. :)
this circumstance ... I don't have the sample at hand now, but I can assure you that you won't have much fun on this, either. (Because it's practically an impossible thing to do for a spatial interpolator.)

[wonkey_monkey]

If by "ouch" you're referring to the corrupted text, that's from my deinterlacer, not MCBob. MCBob did a fine job - it's the fourth image in those examples. The text is actually static, not scrolling.

I got similar results to the rest of the deinterlacers on that image with the singer and the cross-hatched background.

ffdshow playback of those files is badly corrupted, and I can't get VLC to output all the images to disc, so I can't use those other examples - does anyone know what I can do?

First post updated to not use imgspot.com - has it always been this slow?!

David

[Guest]

Can you describe the algorithm you are using?

[wonkey_monkey]

I'll give it a try...

First the image is scanned, looking for what I think of as "high contrast" blocks - horizontal groups of 5 pixels whose total difference from their average is over a threshold. When one is found, a search is made for the closest matching block (least difference) in the vicinity, two lines above (within 8 pixels in either direction). This match is "confirmed" by reversing the search - seeing if the matched block's own best match (two lines below) is the original block. I hope this is making sense...

So now you've got two matching blocks separated by two vertical lines and a variable number of horizontal pixels. This separation (halved) is stored in a buffer, registered against those pixels on the line between the two blocks - this might be easier with diagrams, so I rustle some up later.

Once the image has been scanned, the gaps in the offset table are filled with a sort of cos-weighted interpolation between the determined values to get a smooth curve of offsets.

Then all that's left is to use those offsets to calculate which two pixels from the lines above and below to blend to fill in the empty field.

David

PS Still no luck with Huffy/ffdshow. VLC seems to be saving out 1 out of every 3 frames, despite me telling it to output every single one :(

[tritical]

Your method is pretty similar to what I currently use in eedi2, minus the enourmous number of checks it uses to try to prevent artifacts. Here is what eedi2 produces on the soccer image with defaults: eedi2_soccer.png (http://bengal.missouri.edu/~kes25c/eedi2_soccer.png) The current version tends to have a lot more problems at junctions than your method.

[jmac698]

Take a look at simplesample, http://forum.doom9.org/showthread.php?t=48261&page=7

[foxyshadis]

The trouble with this method is it looks great on stills, but using it in motion bobs all around, because it's just a (much better) dumb bob. (At half framerate it wouldn't really matter though.) That's why SecureBob and MV/MCBob start with eedi2 and then add lots of temporal checks around it, primarily to raise the detail level a bit. As you show, they're sometimes overly cautious, but the effect tends to be more pleasing in motion and easier on codecs, though not a great deal in some cases.

Sports are probably one of those areas where temporal stabilization shoud be lessened, because all those long straight court lines really give a lot of appeal to line-connection at any cost. MVBob and MCBob give you decent control over that, though I've never had to use them.

[Didée]

PS Still no luck with Huffy/ffdshow. VLC seems to be saving out 1 out of every 3 frames, despite me telling it to output every single one :(
I didn't ever try that with VLC, so no clue. Players using their own decoding routines are suspicious. ;)

If you have installed ffdshow, make sure that the codec "huffyuv" is set to be decoded by "libavcodec", both in

Start -> programs -> ffdshow -> video decoder configuration
(for DirectShow based playback in mediaplayers)

and in

Start -> programs -> ffdshow -> VFW configuration -> 'Decoder' tab
(for vfw decoding, e.g. VirtualDub)

There should not be any problems. If the problem persists, drag such a file into Graphedit, and report which filters hop in.


(At half framerate it wouldn't really matter though.)
Oh, really? Load the musicians sample, deinterlace to same framerate, and look close. IIRC around frames 30-40, you should note very visible jitter on the man's shirt. Just an example, sure ... one could go into lengthy explanations of [what] might happen [when] under [which] circumstances. Shortly, just anything might happen, and jitter may occur even when "only" using same-rate deinterlacing by spatial interpolation. (It's common, as soon as there's slow-speed vertical movement.)

[wonkey_monkey]

Start -> programs -> ffdshow -> VFW configuration -> 'Decoder' tab
(for vfw decoding, e.g. VirtualDub)

There should not be any problems. If the problem persists, drag such a file into Graphedit, and report which filters hop in.

Thanks for the suggestion. I'm using ffdshow 2004 alpha, the sse2 version. I found the VFW bit but Virtualdub still complains it can't find a codec. Graphedit loads the ffdshow filter, but plays back colourful noise (elements of the picture are just visible). VLC is outputting more (but still not all) images today so I might try bobbing a video sequence. I already know that MonkeyDeint does quite badly on the pianist's shirt...

David

[Didée]

I'm using ffdshow 2004 alpha, the sse2 version.
In the intended sense, that's a "five years old" version. ;)

Look here (http://sourceforge.net/project/showfiles.php?group_id=173941) for "current century" ffdshow builds.

[wonkey_monkey]

Thanks Didée, it works perfectly now.

David

[Terranigma]

This monkeydeint seems like it's going to be very promising :)

Edit:
Didée, why you gotta tease us with that bobber example? :o :D

[Terranigma]

davidhorman, any word on this deinterlacer? I haven't heard anything in quite some time. It would be awesome to see how it's progressing. :cool:

[wonkey_monkey]

It hasn't progressed at all for some quite some time, since I discovered mcbob - then my time was spent writing AVSFactory so I could make practical use of mcbob before the heat death of the universe ;)

Although monkeybob (sounds better than monkeydeint) appears to have done a better job than mcbob in some parts of the comparisons I posted, I now realise these are bad comparisons, as I was only giving mcbob one or two frames to work from.

That doesn't mean I've given up development - I may well revisit this, perhaps when I get my new dual core laptop and don't have to wait forever for the results :D The real test will be in comparing it to EEDI2, to which it is similar (I didn't know about EEDI2 until it was mentioned in this thread).

David

[Terka]

David,
the results look promising :) , please keep us informed about the progress!
PS:what about the speed?

[wonkey_monkey]

PS:what about the speed?

It's probably best not to ask ;)

David