New motion compensated deinterlace filter (TomsMoComp)

[trbarry]

I've just released my new TomsMoComp.dll deinterlace filter.

This is a first cut at doing motion compensated deinterlace of video source material, but still doing it at a reasonable speed.

There are actually 2 versions in the zip file; one each for Avisynth and DScaler 4.0.

See the ReadMe file at www.trbarry.com/Readme_TomsMoComp.txt or get the source and DLL at www.trbarry.com/TomsMoComp.zip.

The Avisynth filter also offers a fancy new line doubling option to verically double the size of a (progressive) clip in a way that I think offers a bit more apparent detail than regular scaling. I'll post some samples and some more about that later.

Note this is NOT for film source, only video (ATSC, PAL, or NTSC). Use ForceFilm, DeComb, or GreedyHMA for telecined film material.

This is brand new so any feedback appreciated.

- Tom

[avih]

great tom :)
i'll test it.
how about moComp temporal filter now? ;)

best regards
avi

ps
add '.txt' to your readme link

[trbarry]

Avih -

Thanks. Link fixed now.

- Tom

[UGAthecat]

Too sweet!
I'm with avih also about the MC-temporal filter (as previously begged for here http://forum.doom9.org/showthread.php?s=&threadid=28138 :) )

I did some simulations in photoshop, and the really noisy source I referred to in the post above went from about 40% pixels being 'locked' with a threshold of 20, to about 90% of the pixels being locked when the previous frame was moved to compensate for motion in the new frame.

[Milkman Dan]

tom, I get a "Internal Error: MMX state left on: T:\projects\virtualDub_old\main\VideoSource.cpp:1101"

I have no idea where that path comes from. It's not present on my system.

I have a SSE-enabled Athlon.

[trbarry]

tom, I get a "Internal Error: MMX state left on: T:\projects\virtualDub_old\main\VideoSource.cpp:1101"

oops. Maybe in spite of my warning everyone else constantly about this I may have terminated without issuing an emms instruction. ;) Avery/Vdub has encountered this so often that he has now put in helpful checks for it.

I'll look for it.

But be aware it can also come from BiCubicResize in some versions of Avisynth.

- Tom

[trbarry]

Milkman Dan -

I can't find an exit from the program that has this error. (of course I'm as blind as the next guy)

Could you verify that it happens with all other filters removed?

- Tom

[Milkman Dan]

Sorry for taking so long. I removed all other filters. The error still occurs.

I didn't notice before that there is an AVIsynth read error just before the MMX state error. The address changes each time though.

[dividee]

Just tested your filter. Quality seems very good.
However, I had some access violations too. It only occurs with some combinations of parameters and clip properties, but I didn't find a pattern. For example, with res of 720x480 it only works with TopFirst=0 and SearchEffort<6. If I resize it to 704x480 it works for other parameters, but if I crop it to the same resolution it doesn't. Maybe pitch related? Also it sometimes cause a line of gargabe pixel at the bottom, particularly in "double height" mode (SearchEffort=-1) but not only.

[trbarry]

Grrrrrrrrr......

I have not been able to reproduce the error yet, much less fix it.

Could someone post a script that causes the error? And say what kind of machine and input file it is?

I certainly believe that it happens, it just does not yet happen here.

- Tom

[dividee]

LoadPlugin("TomsMoComp.dll")
BlankClip(width=720,height=480,pixel_type="YUY2")
TomsMoComp(1,0,1)
#TomsMoComp(0,6,1)
#TomsMoComp(0,5,1)
#BilinearResize(720,240).TomsMoComp(0,-1,1)

With the first two set of parameters it crash, the third one works. The last one gives me a colorful dotted line at the bottom.
I have an Athlon XP.

[trbarry]

dividee -

Thanks. I'll try that right away. Hopefully it will crash for me here also (what a thing to say ;) )

BTW, what release of Avisynth are you running?

- Tom

[dividee]

I was using the CVS code from about 24h ago.
Will try with older versions and report back.

[EDIT:] Same behavior with 1.0b7, 1.0b6, 1.0b5 and latest CVS

[trbarry]

Okay, I can crash it here too.

It looks like I process too many lines and fall off the end of the buffer. Unless it's the -1 doubling option. Then I process too few.

Fix soon.

- Tom

[tenebrenz]

how about moComp temporal filter now?

I'm still hoping that someone will write a filter that does temporal smoothing in still areas and spatial smoothing in moving areas which should be a little easier to code than motion-compensated temporal smoothing. I think if I mention it every few months maybe someone will :)

[Wilbert]

There is a VirtualDub filter who does exactly this (I think):
Use Freese website: Time Based Deinterlacer (http://www.uwe-freese.de/vdubfilters.html)

[Koepi]

Ah, too bad, that's a deinterlacer too. I thought you'd come up with a motion compensated temporal smoother ;)

trbarry, can you adopt the code from avisynth's temp smoother (or was it dividee's code?) to work with the found motion? :) (well, and be scene-change adaptive as well is somewhat important i guess...)

*hide*

Regards,
Koepi

[Wilbert]

Ah, too bad, that's a deinterlacer too. I thought you'd come up with a motion compensated temporal smoother :)

Oeps, sometimes I can't read :) Sorry ...

[trbarry]

Sorry guys, the bug is still not fixed. I mostly played hookey yesterday.

And I don't think I can easily adapt this as a temporal smoother.

- Tom

[Defiler]

Just checking in here re: 60fps output (from the CPU Optimization thread.)
Wanted to get on e-mail notify for this thread.
trbarry: Occasionally when I use Force FILM on a DVD that shows 99% film in DVD2AVI, I still get choppy pans. What do you think about leaving such a thing at 29.97fps and using TomsMoComp on it? Stupid idea?

[UGAthecat]

I think the only way us beggars are going to get a M.E. Based Temporal filter is if somone adapts the M.E. algos from XVid :rolleyes: . Just my guess, but I don't think Tom uses full on motion estimation :confused: .
Just out of curiosity, is this another fruit of the labor of porting DScaler stuff?
I just wish I had a portion of the C knowledge you guys are working with, I'd be able to devote all of this surfing/posting time to making something productive.

UGAthecat

PS - If anyone has good links or info on getting started with C++, which programs to use (Ive got VC++/VStudio6), and info on getting started on making filters for avisynth, please PM

[Defiler]

Originally posted by UGAthecat
PS - If anyone has good links or info on getting started with C++, which programs to use (Ive got VC++/VStudio6), and info on getting started on making filters for avisynth, please PM Seconded.

[poptones]

There's a bit on Graft's website. There's also a tutorial in the veedub "plugin sdk." The pixel formats are pretty easy, and the filters already in there provide a good working model.

That's how I learned to do it. And I can barely even drive the compiler (I still haven't a clue how to use the debugger; I just open VS and hit F7 when I'm ready to test). I'm pretty good in jscript, but I never spent much time learning c.

I still don't know much at all about gettings frames from media and such, but filters themselves aren't terribly difficult. Just pick one (I used "subtract" to figure out how to make the layer filter) and start hacking away at it. The rest of it is just figuring out how to do the math for the effect you want. And that's not specific to avisynth - that's just stuff you gotta google.

[trbarry]

I finally fixed the crash and green line bugs (Avisynth & (maybe) DScaler).

As before see the ReadMe file at www.trbarry.com/Readme_TomsMoComp.txt or get the source and DLL at www.trbarry.com/TomsMoComp.zip .

Now Version 0.0.1.2 though I forgot to update the Readme fix log.

- Tom

[trbarry]

Just out of curiosity, is this another fruit of the labor of porting DScaler stuff?

UGAthecat -

On this one I actually wrote the Avisynth version first as it is easier to test, but my coding habits and pieces-parts are mostly geared to writing for both these days.

The first (unreleased) version of TomsMoComp was using a block heirarchical scheme for larger motion search, and then a single pixel smaller brute force search. But I only released the 2nd half of it, for performance reasons. The rest didn't seem to help much more for deinterlace quality and probably would have needed a bunch more work.

I'll get back to it later on a faster machine. ;)

- Tom

[trbarry]

For anybody thinking about writing more filters see Ben RG's Writing Avisynth Plugins (http://math.berkeley.edu/~benrg/avisynth-extensions.html) page.

That's where I first started, with a copy of his Invert() filter.

- Tom

[dividee]

It doesn't crash anymore, but for two of the four test cases in the script I posted earlier it now gives a green line on top.
There are the cases with parameters (0,6,1) and (0,5,1).
At first sight, it seems it only happens when Topfirst=0.

[EDIT:] It's not the topmost scanline, but the third one from the top. (second line of the top field)

[trbarry]

Darn.

I tested with all your test cases but I guess I kept tinkering as I went along. Oops.

- Tom

[trbarry]

dividee (and all) -

It appears the problem was only with Bottom First and VF turned on.

If so it's fixed again. Pleas try Again.

- Tom

[dividee]

Yep, it seems all right now!
Thanks a lot for your great work! It still has to prove itself (anyone up for a comparative test?), but I think it's the best deinterlacer known to man ;)

[trbarry]

Thanks a lot for your great work! It still has to prove itself (anyone up for a comparative test?), but I think it's the best deinterlacer known to man

dividee -

Thanks. I wasn't making that claim for it it, but I can't afford a Teranex. ;)

- Tom

[Defiler]

Well, I hope you've got broadband, because I'm going to link to two test files I just created.

The first was made using the following AviSynth script:
LoadPlugin("C:\Bin\AviSynth\mpeg2dec.dll")
mpeg2source("D:\Projects\FUTURAMA_DISC_3\ep1\futurama.d2v")
LoadPlugin("c:\Bin\AviSynth\TomsMoComp.dll")
BicubicResize(640,480,0,0.5)
TomsMoComp(1,15,1)

I fed it to VirtualDub, and used XviD in 1-pass Quality, set to 85%.


The second was made with this script:
LoadPlugin("C:\Bin\AviSynth\mpeg2dec.dll")
mpeg2source("D:\Projects\FUTURAMA_DISC_3\ep1\futurama.d2v")
SeparateFields

I fed that into VirtualDub in Full Processing mode, and used the Deinterlace - Smooth filter to convert it to a progressive 50fps file.
After that, I fed it to Resize (Precise Bicubic A=0.6), 640x480.
Once again, 1-pass Quality, set to 85%.

Caveats: Clearly the resize process uses slightly different settings, and occurs at different stages, meaning that the two deinterlace methods are operating on input frames of different sizes. That shouldn't affect the smoothness of the output, though, and that's what I am interested in. I've fixed that in my latest test files, but I don't want to upload them again. Heh.

Also, this is hardly how I would normally encode these files. I'd be cropping off the borders, retaining more resolution, and doing two-pass with a higher bitrate, etc, etc. Still, they'll do for a simple test. They also serve to illustrate the file size impact of doubling the framerate. You will need at least an 800MHz CPU to play the second clip, unless I am much mistaken.

Without further ado, here are the test files.

1:
http://www.hellninjacommando.com/temp/tom-1-15-1.avi
2:
http://www.hellninjacommando.com/temp/tom-50fps.avi

I am so impressed with this plugin. I've got some real torture tests to feed it tonight, but it seems to give great results. Cheers!

[trbarry]

Defiler -

I've downloaded the files, but it's been a long long day and I'm heading off to bed now.

But you should never resize (except maybe InterlacedResize if absolutely necessary) before any IVTC or deinterlace filter. A resize operation can vertically blend field information before the IVTC or deinterlace filter gets a chance to decide which fields or pixels to use. It will confuse the process somewhat.

But I'll take a look at the rest of it tomorrow.

- Tom

[Defiler]

I realized that after I made the files. I repeated the test with A=0.6, and with the resize after the deinterlace, and it looked much the same. Not a big enough difference to bother reposting.

[WarpEnterprises]

@trbarry: Could you please explain in short what exactly means
that uses motion compensation .
I did a short test and the results where really good, but nonetheless I'm interested in the working principle and the difference to e.g. SmartDeinterlace.
I guess one difference is the way you decide which part of the image is moving.
But when it is moving which pixels do you use?

[trbarry]

WarpEnterprises -

Movement is determined not in blocks but a single pixel (actually a single byte, say luma) at a time. This happens in a very small area that more or less corresponds to the size of the SearchEffort parm.

So say SearchEffort=3. Then TomsMoComp would check only 3 possibilities for the value of a given weave pixel.

1) It moved one space to the right

2) It moved one space to the left

3) It did not move.

4) Otherwise we don't know. Use WierdBob.

Assume x,y,t refer to horiz, vert, and time.

For the case where it moved one space to the right then the value from the previous field at P[x-1,y,t-1] should match the value from the next field P[x+1,y,t+1] within some threshhold. If it matches then the average of the 2 is used. But even then there is a sanity check to ensure it does not create visible weave artifacts. (a vertical median filter)

If the value for that pixel doesn't closely enough match then we use the pixel value generated only from surrounding values in the current frame at t=0 using the WierdBob code.

The WierdBob algorithm is a variation of something I found in a paper on motion comp deinterlacing linked by someone in the Xvid forum when I first mentioned I was working on motion comp deinterlace. I'll go look for the reference but it was part of something called an ELA (or ELF?) algorithm, can't remember now. It was just a passing reference, even in that paper, but it dovetailed nicely into something I was already doing.

But I didn't use any of those sophisticated ideas for motion search. After writing a semi functioning heirarchical block motion search it appeared that for deinterlacing I got best real time results only detecting small motion, and for that I could go faster just brute forcing it in assembler on single pixels, but processed in MMXSSE 8 bytes at a time.

I've never looked at the code for Smart Deintelace so I can't explain the differences from that.

- Tom

edit: The WierdBob algorithm is really an ELA (Edge-based Line Averaging) technique. I can't find the link but I got the idea where it is discussed (but not originated?) in:

Y. -Y. Jung, B. -T. Choi, Y. -J. Park, and S. -J. Ko, "An effective de-interlacing technique using motion compensated interpolation," IEEE Trans. on Consumer Electronics, vol. 46, no. 3, pp. 460-465, Aug. 2000.

[WarpEnterprises]

I still have two problems.
1. The speed of the filter seems to be NOT depending on the search effort
2. A more general problem. (My test-material is self-made Hi8-recordings.)
The resulting video is quite sharp,
but when I view it on TV it looks "jumpy" (as if the shutter speed of the camera would have been to short).
In fast movements the information of one field is lost.
Is there a possibility to simulate a longer shutter speed so to get an effect like a speed blurring, but only in those image parts where the movement is so fast?

[trbarry]

WarpEnterprises -

The speed should depend upon the search effort but that currently max's out at about 27 or so. And in DScaler I can get 60 fps out of it with a SeachrEffort=13 setting. So maybe you can't notice the difference as much with all the other decoding/encoding overhead.

The jerkiness is maybe because under Avisynth it will take 50 (60 ntsc) fields / second and output 25 (30) full frames. Under DScaler it can output up to 60 full frames / second but I foolishly didn't support that for Avisynth. I'll try to add that later.

- Tom

[Defiler]

I've experienced the same "jumpiness" with this filter. Smooth -> Jerk -> Smooth -> Jerk.
No such problem with decomb.dll's FieldDeinterlace()
I've tried various search levels, mostly 21, like the Readme uses for an example.
I can post a short test clip to illustrate, if you'd like.

[trbarry]

I can post a short test clip to illustrate, if you'd like.

Defiler -

Yes, please if possible. I'm about to release a new version of this with horizontal doubling but because of all the various flavors and options it is a pain to regression test it. So if there is something I should fix first I'd really like to see it.

But please also post your avs script.

- Tom

[trbarry]

I don't usually encode clips at 60 fps because you have to store all the darn things, which take up more space. And at higher resolutions it even takes too much CPU to play them.

But for those of you determined to encode at 60 fps and who have machines capable of playing it, maybe you might experiment with the following scrips. These are based upon one by Poptones over in this thread (http://forum.doom9.org/showthread.php?s=&threadid=28568&pagenumber=3).

But try either:

# This script is only for TopFirst material
LoadPlugin("d:\save-oe\MPEG2DEC2.dll")
LoadPlugin("F:\TomsMoComp\Avisynth\TomsMoComp.dll")
Mpeg2Source("D:\VCR\Its_All_Here.d2v")
Top=AssumeFrameBased().ComplementParity()
Bottom=Top.SeparateFields.Trim(1,0).weave
return Interleave(Top.TomsMoComp(1,15,0),Bottom.TomsMoComp(0,15,0))


or

# This script is only for BottomFirst material
LoadPlugin("d:\save-oe\MPEG2DEC2.dll")
LoadPlugin("F:\TomsMoComp\Avisynth\TomsMoComp.dll")
Mpeg2Source("D:\VCR\Its_All_Here.d2v")
Bottom=AssumeFrameBased()
Top=Bottom.SeparateFields.Trim(1,0).weave
return Interleave(Bottom.TomsMoComp(0,15,0),Top.TomsMoComp(1,15,0))


Just an experiement. But if you preview both quickly in vdub it should be pretty obvious which one works best for your clip. Just step through a few frames with motion, one frame at a time.

- Tom

[poptones]

Is there a more up to date dll? When I try that script on my P2/450 I get a mofo of a crash.

[fisix]

not doing 60fps, but the plugin works beautifully and flawlessly on a boxing match captured off tv. the filter=1 looks nice.

i looked at the difference between a couple different options in fielddeinterlace by graft and tomsmocomp... and i could see obvious jaggs in diagonal edges in the progressive frames served by fielddeinterlace that WERENT THERE with tomsmocomp.

i thought at first it was a problem (in grafts) with which field came first, but in the process of ruling that out, i noticed that there didn't seem to be a difference in using field=1 or field=0. ? i guess the jags might just be an error in fielddeinterlace, but i'm concerned with not easily seeing a difference in field=0 or field=1 in tomsmocomp.

outstanding filter, tom. thanks.

-fisix

p.s. if yould like to see the frames, just let me know and i'll send them to you in email and you may post them (i can't webserve). your choice of gif, jpeg, whatever.

[WarpEnterprises]

Me too saw only very small differences between 1 oder 0. Try looking at lines, in one setting they look smoother, in the other a little bit staircase-like.

[Defiler]

Originally posted by trbarry

Yes, please if possible. I'm about to release a new version of this with horizontal doubling but because of all the various flavors and options it is a pain to regression test it. So if there is something I should fix first I'd really like to see it.

But please also post your avs script.Didn't notice your reply before I left for work today. I'll link an example this evening.

[trbarry]

Is there a more up to date dll? When I try that script on my P2/450 I get a mofo of a crash.

poptones -

There have been a couple versions, all listed above. You should be able to compare version info by right clicking on the dll, or just comparing the file dates of yours with the current zip. It's so small you might as well download again to be sure. But yes, the very 1st version would crash sometimes.

But if you are running the current version please email me. You are the only known P-II tester. I thought I wrote it so P-II was supported but something might have slipped through. But I'm sure we can find it and I'll send you another test version once we have an idea what's happening. I've now got the horizontal expansion (SE=-2) now working also but wanted to test a bit more before releasing.

All the source and project is in the zip. Can you compile it under VS6? If so, does it blow up on an unsupported P-II instruction or just fall off the end of the buffer like it did before?

- Tom (trbarry#trbarry. com #=@)

[trbarry]

Me too saw only very small differences between 1 oder 0. Try looking at lines, in one setting they look smoother, in the other a little bit staircase-like.

I should have mentioned in the Readme (but didn't) that the first (TopFirst) parm will be ignored and will have ZERO effect when SearchEffort is < 0 and only a very tiny effect when SE=0.

But for SE > 0 (when actually doing motion search) you will find that if you specify the TopFirst parm wrong then the motion search will more often fail to find a match and more often bob slow moving pixels. So the picture will be slightly more jaggie with slightly less detail.

Ah, a table: ;)


SE < 0 Doubling only, no motion search, ignore TopFirst
SE = 0 WierdBob only, no motion search, TopFirst effect small
SE > 0 Motion comp deinterlace, TopFirst matters


For those of you that have also downloaded my older GreedyHMA deinterlace filter there is some TopFirst discussion in the readme file and a couple AVS scripts in that zip to determine what the value of TopFirst should be. See www.trbarry.com/GreedyHMA.zip .

One of these days I'm going to have to see how Donald Graft made it automatic on some of his filters.

Avisynth actually keeps track of which it thinks is the dominant field but it often gets it wrong so that info is not reliable. Maybe instead we just need a generic AutoSetParity() filter the could look at the data on the fly and keep it correct. But I'm not sure how easy that would be. It can even change within clips due to edits, etc.

- Tom

[vlad59]

Hi Tom

I found a good interlaced source to test your script : it's The Lodoss Chronicles (an anime). And I found something strange :confused:

My script is following (it's simple) :

mpeg2source("C:\Temp\dvd\DVDVOLUME\lodoss.d2v")
TomsMoComp(0,10,1)


I found that bottomfirst works better for me (especially on scene change) but I got strange result (I've attached an png image in a zip)

The stranger thing is if change the searcheffor to 15 this buggy frame is good but I found another one buggy.

If I change the setting to top field first I don't have any problem (but the scene change are more crappy)

I can send you a 4 or 5 frame long hulfyuv if you want.

EDIT : removed attachment to save bandwidth (Tom already have a hulfyuv clip)

[trbarry]

I can send you a 4 or 5 frame long hulfyuv if you want.

Vlad -

Thanks, yes, I'd like to look at it.

- Tom

[Billy|B-A]

This may be a stupit question, but I am new to Avisynth and editing/making scripts. (truthfully i never knew that a avs file could be opened in notepad)I have a few questions about how to use it. Please think of me as a 5yr old child when explaining this.

1. Instead of using Avisynth can I just edit the avs file made by GordianKnot include TomsMoComp.dll?

sample script (http://personal.atl.bellsouth.net/atl/p/e/perdue3\opening.txt)

2. I installed Avisynth, so by doing this am i really just using the program anyway?

3. What do the setting mean? i.e.(1,0,1)

4. Can Avisynth read d2v files? (After a little pratice I may be able to skip GordianKnot

5. What commands are available? What do they do?

edit - Please disregard the settings, just doing so "how to kill this file" testing.