deinterlace/resize on the fly

[skyline]

Hello all,
when I encode a DV (PAL) file using the interlaced option at 352x576, I would like to be able to playback the file at correct aspect/smooth motion(increased framerate) ie. 352x288x50 on the fly.
this is what MP6.4 does when I playback DV avi file ie. resize from 720x576x25i-->352x288x50p.
Is it possible for similar to be done with xvid dshow filter?
This is an avenue I am exploring after failing to eliminate "bobbing" effect when resizing 352x288x50iXvid-->720x576x25i interlaced DV avi. After all, Xvid now supports interlacing so why not use it!
Sorry for rambling.

Great work guys!

[avih]

this is beeing thought of.
i also would like to see something like that, however, the problem that i yet couldn't solve (i also didn't try yet) is how to double the framerate, since the framerate is written into the avi file, and the player application is expecting a specific framerate.

[trbarry]

Some folks on a number of different threads recently have stated that it is important to get a full 50 or 60 fps playback rate on deinterlaced material.

I admit that it looks a little bit better and we do manage it on DScaler. But how important is it to most people? I mean, if you go to the movies in the US it is only 24 fps. A mechanical projector may repeat each frame twice, but that is no different than letting it sit in the overlay buffer for 2 cycles on a PC. It is still only 24 fps but with less flicker.

And I notice that Fox is broadcasting a digital ATSC sub-channel now that claims to be 480i @ 60 but seems to be really 480p @ 30 and no one else has commented on it.

I don't know at all what techical obstacles there are to doubling the frame rate in dshow, even if the performance issues were handled. But I was just wondering how important it was to everyone.

- Tom

[avih]

trbarry, for one, as u understood by now, it's very iportant for me :) but i'm willing to spend the time and try to do something with it. nobody asked someone who's not interested to program this stuff for him :)

anyway, regarding 25/50 (or ~30/60 for ntsc) it's VERY important where the source material is truely interlaced. don't forget that when filming a 24fps film, the cammera shutter opens for longer time than a video camera that records 50 interlaced. that's why capturing truely interlaced material in 25 fps non interlaced seems quite 'jumpy' but capturing it as 25 interlaed, and playing as 50 de-interlaced gives such more rewarding playback.

don't forget that performance issue is not really that serious here. afterall, no more decoding power needed than decoding 640x480x25, it's just that the fields have to be splitted after that. and as u said, descaler does that very well, and can apply quite cpu intensive deinterlacing techniques. but for really low cpu resources, bob should be enough. bob is also the most 'correct' display mode to emulate interlaced display.


avi

[trbarry]

Okay, if Bob is good enough then I agree there is not extra performance penalty. And there is even a smart bob built into DirectX/DirectShow these days. I think that is what you get when you choose the 'Scaler Bob' deinterlacing choice in DScaler.

But at least on my graphic card it jitters something horrible on still scenes.

- Tom

[skyline]

@ trbarry

>It is still only 24 fps but with less flicker.

Yes, but I am dealing with truly interlaced source material (DV PAL) and as I understand it, that is 50 fields per second. What may be compounding the problem for me is that much of my footage is shot using high shutter speeds with lots of action ie. jetski footage etc.
Looks REALLY bad IMHO at 25p.
I wasn't aware of dscaler. I am downloading now and will give it a burl shortly.
Am curious as to why when I play DV file in MP6.4, is seems to be scaled to 352(?)x288x50, yet when i compress using xvid with interlaced option, the same file plays at 720x576x25 with all of the interlacing artefacts visible.
Why dont they both play using the same method as they are both the same dimensions and both compressed?

[-h]

Am curious as to why when I play DV file in MP6.4, is seems to be scaled to 352(?)x288x50, yet when i compress using xvid with interlaced option, the same file plays at 720x576x25 with all of the interlacing artefacts visible.

I don't believe that's happening, it's just that Microsoft's decoder (by default) plays DV avi files at quarter size. Right-click on the image and go into the DV decoder preferences, and check out the options. The frame rate is still 25.

Why dont they both play using the same method as they are both the same dimensions and both compressed?

Because that's the way Microsoft set their defaults :)

All XviD's interlaced mode does is turn on some coding techniques that improve the compression of interlaced content.

-h

[avih]

Am curious as to why when I play DV file in MP6.4, is seems to be scaled to 352(?)x288x50, yet when i compress using xvid with interlaced option, the same file plays at 720x576x25 with all of the interlacing artefacts visible.

i had similar problem a while ago. it turned out one of the mpeg2 decoders floating on the web does that :) i really can't remember which one, but try uninstalling (through control_pannel/dxman (from analogx)/other) all mpeg2 decoders/filters, and install each of them in a row, till it's working ok.

again, that's not the exact problem i had since xvid always played at the resolution i encoded. only mpeg2 changed the resolution while playing. could be that the resolution was changed already during decoding mpeg2 and encoding!!!

but it won't hurt giving it a try :)

also, some of the mpeg2 decoders have deinterlace options. and if on 'auto' it can choose bob instead of weave. from what i saw, their bob is not really true, it just 'select odd lines' (or even), hence no artifacts seen (at least powerdvd decoder does this). it's important to know what your filters are doing ;).

a good way to check that out will be using graphedit to load your questionable file.


cheers
avi.

[skyline]

@ -h
>I don't believe that's happening, it's just that Microsoft's decoder (by default) plays DV avi files at quarter size. Right-click on the image and go into the DV decoder preferences, and check out the options. The frame rate is still 25.

options tell me "half PAL 360x288"
So that explains the size. It is using MainConcept vfw DV codec 2.04.
Video renderer acheived 24.38fps
Strange still that it appears less jerky than watching the 352x288p xvid clip and nowhere near as smooth as the silky 352x288x60 encode.
It is this 352x288x60 encode that I would love to successfully recreate a DV file from. So far I can't de-bob.

I changed properties to full PAL and now I get same playback as the full-res interlaced xvid (Interlace aretfacts and all).

[MfA]

Originally posted by avih
and as u said, descaler does that very well, and can apply quite cpu intensive deinterlacing techniques.
Still pretty piss poor deinterlacing techniques though, compared to what is commonly used even in consumer electronics nowaday's.
but for really low cpu resources, bob should be enough. bob is also the most 'correct' display mode to emulate interlaced display.
Dunno about correct, it certainly isnt the best looking one by a long shot.

Modern televisions also have higher framerate's to reduce flicker, this gives them the same problems as we have on our monitors. The best way to handle it is just as those TV's handle it, use modern deinterlacing technique's to do framerate upconversion together with deinterlacing. The reason why we have to resort to bob'ing display is not because its correct, its because we dont have access to proper deinterlacing techniques (motion compensated).

IMO this also makes it hard to make definitive judgements on how much worse deinterlaced coding is as interlaced coding ... because its hard to determine how much of the quality difference is caused by the temporal resolution (interlacing of course has better temporal resolution, but coding efficiency is lower) and how much by the lack of quality in the deinterlacing.

[avih]

@mfa:
you can't beat bob with correctness. theoretically, it's the only correct way for displaying interlaced cources. any other way will result in a loss of information, or modifications of that information (even though it indeed sometimes looks better).
bob is showing the source information exactly as it was captured by the video camera. period.

>" Still pretty piss poor deinterlacing techniques though, compared to what is commonly used even in consumer electronics nowaday's."

consumer electronics is ELECTRONICS, meanning it can use hardware. dscaler gives software only solution, and it by far offers the most advanced software deinterlacer techniques as of today (of which, bob is the simplest one it offers).

the most advanced hardware deinterlacers use hardware motion compensation, which is quite hard to do in real time in software with current computers and cpu power.

yet, some people (i.e. trbarry which done much work with dscaler) is working on motion compensated deinterlacing implementation in software.

seeing that u do understand something about deinterlacing and computer limitations, i wouldn't expect you to trash current efforts as u did. instead, give constructive feedbacks if u have some.

[MfA]

The resizing/interpolation of bob deinterlacing already modifies the information, it add's incorrect information (thats why it bob's). The only correct way to display interlaced material is with interlaced display as it was scanned, and the phosphors on a monitor are simply way too fast to even consider doing that.

Say you had an original sequence of frames, and from those you created an interlaced sequence by dropping the odd/even lines for each frame... the only objective way to say which method is the most correct way of recreating the odd/even lines is the one which when compared to the original sequence gives the smallest difference, and bob aint gonna be it.

Motion estimation can be done at a couple times realtime now, IMO its not so much a question of computational complexity ... moreso a question of just how damn difficult and how much work the task of implementing it is. Just because I call the technique used piss poor compared to whats possible doesnt mean the work is piss poor, I just call a spade a spade.

[avih]

my statement that bob is the most correct way assumes we're using a progressive monitor, with higher resolution than the captured source.

so resizing is inevidable in this case, and interlaced monitor is not a factor. we're talking about current monitors.

however resizing can be done 'stupidly' like doubling the lines, or smart, by interpolating between them. both methods are implemented in dscaler.

and i'm not sure a camera capture progressive and deletes avery other line in each frame. but even if it does, there're arcticles explaining the FACT that u can't interpolate perfectly to get the full original scene. therefore i was saying that it most correctly recover the image that was captured by the camera, not the actual scene.

it's trues, as i mentioned that advanced deinterlacing techniques can give better percieved visual quality than bob.

[MfA]

Its not how the camera works exactly, but thats not really the point. It will give you almost the same result as if the scene was recorded with a camera which does use interlaced scanning, and give you some extra samples to work with if you want to objectively test a deinterlacing method.

Each pixel represents a sample, if we create new samples then the most correct way to do so will give results closest to what you would get if the sample was taken directly from the original scene. Any error will make the new sample clash with sample's from the same point (in the scene, not the screen) in other frame's which are truely sampled, thats why bob bob's.

Anyway, glad to hear people are working on motion compensated deinterlacing ... best of luck.

[-h]

Does anyone have pointers to documents about motion compensated deinterlacing?

It sounds as though you just perform a typical motion search of the top field against bottom (or whatever field is first temporally), and if a match is found, move the lines from top and bottom fields to be vertically adjacent in some intermediate position. I can think of many complications.

I've got tomorrow off (and if I don't finish this field-based motion estimation stuff then, I probably won't before I leave ;)), and it sounds like a simple modification from the scant info I've found. Some specifics would help though.

-h

[trbarry]

-h -

Gerard de Haan has published extensively about it and many of his papers are available here. (http://www.ics.ele.tue.nl/~dehaan/publications.html#Books_and_book_chapters)

It sounds as though you just perform a typical motion search of the top field against bottom (or whatever field is first temporally), and if a match is found, move the lines from top and bottom fields to be vertically adjacent in some intermediate position. I can think of many complications.

I think it works better if you match the middle (bob'd, interpolated) field with searched for blocks. And it also gives more choice if you search against both the (temporal) prev and next fields, and possibly against their motion adjusted average.

I'm very disappointed with my own efforts in motion compensated deinterlace so far. The results of using a macro block match that compresses well doesn't necessarily make the best deinterlaced visual results. I've tried a number of (ad hoc) ways but so far I don't get visually pleasing results if pixels move by more than 1 or 2, even when I can find them.

So I'm pretty convinced so far that motion compensated deinterlace would have to be combined with a per-pixel adaptive step for real life hi rez results.

But I often write a prototype before properly doing my research, just to get a feel for things. And of course this means I usually throw away the first version of everything. ;)

- Tom

[avih]

-h.
some video papers, including about motion compensated deinterlacing, human visual perception and many others:

homepage:
http://www.ics.ele.tue.nl/~dehaan

subpage with links to publications:
http://www.ics.ele.tue.nl/~dehaan/publications.html

actual pdf directory:
http://www.ics.ele.tue.nl/~dehaan/pdf/


cheers
avi.

[MfA]

Alexis also has a paper on it, and if you can read Dutch the recent phd thesis from Dimitri Van De Ville (http://www.elis.rug.ac.be/~dvdevill/pub/) also has something to say about it (also a list of his publications there in english, but not online ... for english papers in electronic form you'll need IEEE Xplore access).

Trbarry, what you are saying about the prev/next fields reminds me a bit of this paper (http://eeserver.korea.ac.kr/~bk21/publication/sjko/sjko_s3.pdf). What kind of interpolation filter are you using BTW?

[trbarry]

MfA -

Thanks for the reference. I'm only slowly digesting it but at least some versions of my rapidly evolving but not very functional program are much like that.

I've been playing with doing a bidirectional motion search like that but on a 4 level hierarchy. The idea was to be able to pick up motion of up to 40-50 pixels with reasonable (but not yet real time) performance.

But so far I get pretty good results with motion of 1 or 2 pixels and might not have bothered with the rest. I'm probably overlooking something.

I'm actually a newbie to ME but have been writing DScaler per-pixel adaptive deinterlace functions for a couple years so I'm more comfortable in that area.

So far this has been alternately fun and very frustrating. I'll maybe add a couple comments on the adaptive part of that paper (mostly medians) after I've thought about it some more.

- Tom

edit: Not to challenge your opinion but have you ever actually tried my Greedy (High Motion) deinterlace in DScaler? I have a Toshiba HD RPTV with built-in line doubling/deinterlace and IMHO DScaler looks a bit better on all NTSC sources. But I admit to being somewhat biased here. ;)

- Tom