Decomb 5.0.0 beta 15: Fix for crashing problem

[Guest]

A beta of New Generation Decomb is now available for your testing. Please carefully read the accompanying tutorial and reference manual as a lot of things have changed in Telecide(). FieldDeinterlace() and Decimate() are unchanged.

Get the beta here:

http://shelob.mordor.net/dgraft/decomb/decomb500b15.zip

As always, feedback will be greatly appreciated.

[HomiE FR]

Thanks neuron2. :)

I'll give it a try and I'll come back when I have some results.

EDIT :
I've tried Decomb 5.0.0 beta 1 with some anime source that I had, reading your tutorial to set the parameters. But I have a question : is the possibility to match the current frame with the next one gone ? I ask this because new generation Decomb has problems where the good old Decomb 4.10 beta 2 chooses to match the frame with the next one. And with show=true, Decomb 5.0.0 beta 1 doesn't show any information about "n" anymore, just about "c" and "p".

I've taken 2 screenshots, the first with Decomb 4.10 beta 2 which chooses to match the frame with the next one, and the second with Decomb 5.0.0 beta 1 which chooses to match with the previous one (it seems it doesn't care about next frame).
Sorry but the images are PNGs which are about 400 kb.
Decomb 4.10 beta 2 (http://perso.wanadoo.fr/homie.fr/decomb_410b2.png)
Decomb 5.0.0 beta 1 (http://perso.wanadoo.fr/homie.fr/decomb_500b1.png)

I hope you'll understand my problem. It seems obvious to me that Decomb 4.10 beta 2 performs better than Decomb 5.0.0 beta 1 on this source (I have this problem very often on the anime source tested).

The function calls were :
- Decomb 4.10 beta 2 : Telecide(post=false,blend=true)
- Decomb 5.0.0 beta 1 : Telecide(order=1,post=0,show=true)

It doesn't perform any better if I use guide=1 (I have some 3:2 pulldowns on this source). I have chosen order=1 accordingly to the test suggested in your Decomb 5.0.0 tutorial.

[cipher]

Damn great, DG!!!I've been waiting for this since I read your journal.
Infinite thanks!!
I'm gonna give it a shot, right away!:D

=======
Edit: So far it's just great!
I assume "vthresh" is similar to the "threshold" in v4. right? It seems to me that vthresh becomes more accurate now!(Maybe because it has more divisions than "threshold"?:)).
Edit in this edit: Beyond that, what previously version4 had to decide as combed frames (even at a very low threshold) now have been "decombed" as clean frames, progressive but no interlaced, and in my eyes they are indeed clean now:D

But one thing, DG, is "dthreshold" actually working? When I wasn't touching it, it was fine; but when I specified dthreshold=7 in Telecide(), vdmod reported "...Telecide does not have a named argument 'dthreshold'..."
My script was this:
LoadPlugin("MPEG2Dec3.dll")
LoadPlugin("Decomb.dll")
mpeg2source("01.d2v",cpu=4)
ConvertToYUY2()
Telecide(order=1,guide=1,vthresh=13,dthreshold=7,post=2,show=true)
Decimate(cycle=5,quality=3)

By the way, my material is a normal movie(well, an mtv), not an anime.
=======

~~~~~~~
2nd Edit:
Comfirmed by my friend, "dthreshold" didn't work. Instead, he found out that "dthresh" worked(at least no error reported by vdmod). The manual should've been updated now.:)

[wotef]

hi some feedback and questions

why is field dominance/parity now important?
is it working correctly? on some bottom-field PAL material i have, telecide(order=0) seems to leave more combed frames than telecide(order=1)

also, is telecide(post=1) meant to display some metrics? i don't see any

[Guest]

@cipher

1. Yes, dthreshold is now dthresh. It's mentioned in the Tutorial but I forgot to update the reference manual. Thank you for pointing it out.

@wotef

vmetric is displayed only when post != 0. When post=1 vmetric is calculated and displayed but not used.

Are you reading the journal at my web site? It describes why field order is now important. Basic summary: it's faster and less error-prone to do 2 comparisons rather than 3. Did you verify the field order using the technique I described in the tutorial? If so, and you still think there is a problem, please make a source clip available for my inspection.

@HomiE FR

Yes. The forward match is gone. At some scene changes you may generate a combed frame that will be deinterlaced. Is this occurring for you only at some scene changes? If not, please make a source clip available for inspection. I personally do not see this as a big deal as long as it is limited to some scene changes and since postprocessing is now so fast.

[HomiE FR]

@neuron2 : These problems don't only show up on scene changes, but all over the source (sometimes it's really visible, sometimes not) and postprocessing (deinterlacing?) can't do anything good to these frames. I've tried to lower the vthresh so that these frames are postprocessed, but it is not any better...
And I haven't any knowledge on deinterlacing, but the forward match wasn't supposed to be useful elsewhere than only some scene changes ? On the source I'm talking about, the forward match "n" was used very often ! And now the frames where "n" was used with Decomb 4 have the problems I've described above (maybe not all, but many since I can see the problem on different scenes).

I'm ready to give you a part of the vobs I'm playing with (a Z1 anime) but I don't know how you want the source clip. Do I have to cut the vob and give you a part, ar do I just compress the part unprocessed with MPEG-4 (XviD) @ quantizer 2 ? If you want a vob file, I don't know how to cut easily a vob where I want to (Google didn't find what I was looking for) so I will need your advice.

Thanks for checking my problem ! :)

[Guest]

@Homie FR

It's really important to get me this clip!

I need a good-sized VOB fragment. Use VOB splitter or other tool, available on doom9 download page.

Thank you!

[wotef]

didn't know about your journal til your reply - look forward to reading it!

i did some more testing on 10 random/different clips (all verified bottom-field dominant) and take back what i said

on 8 of these clips, specifying order=0 gave better frames

on the other 2 clips, there was no difference in quality between specifying either order (both left some residual combing at default settings) - and for these two, tomsmocomp seemed to fare better than telecide - presumably these were "true video" clips

[Guest]

@wotef

I can only comment on specific clips if you make them available for inspection.

You can tell if a clip is video by doing separatefields and then stepping through it. If there is motion only on every other field (or every second or third for 3:2 pulldown), then it is progressive. If there is motion on successive fields, it is video.

[wotef]

ah, thanks, ok, that helps a lot, yes they were both video

[killingspree]

Originally posted by neuron2
FieldDeinterlace() and Decimate() are unchanged.


so does this mean there is no difference in those two to their predecessors in v4? or did you just not change any parameteres?

thanks for the great new filters, going to do some testing now (:

steVe

[JuanC]

Originally posted by neuron2
... At some scene changes you may generate a combed frame that will be deinterlaced. Is this occurring for you only at some scene changes? For me it is ocurring at not some but almost every scene change. Also, I get many interlaced (and "out of pattern") frames in static scenes where only small objects/parts are moving...

Tweaking GThresh: The show / debug options output include two metrics: p and c. I understand they are Previous and Current. Am I right? It would then be great to see in the show/debug output the metrics for Predicted and Calculated. I've been using different values for Gthresh: 0, 10, 100, even 1000, and haven't been able to undestand the difference in behavior.

In latest betas of Telecide v4.10 I was using mm=2: matching mode considering only Current and Next. In general it was giving me better matches for my TV captures. Will there be an option to do something like that in Decomb 5.xx?

Great improvements! Thanks again, :J

[HomiE FR]

@neuron2 : I've cut a part of a vob where the artifacts are visible. It's about 72 MB. But I don't know how I can give it to you. We can do it through IRC I guess, or through ICQ or anything you want. My connection is a DSL line, so 128 kbps UP (16 KB per second).

My ICQ number is 76019744 if you want to contact me directly.

[Guest]

@killingspree

They are identical to the previous versions. Is that clearer than unchanged? :)

@JuanC

You have to provide a source clip if you want me to look at any issue.

P = Previous, C = Current

I will add display of the predicted and calculated metrics. Gthresh sets how big of a difference between those two can exist without preventing a pattern override. Set gthresh=0 and all overrides will be prevented. Set it to 100 and none will be prevented.

Setting mm=2 in old Telecide is exactly the same thing as not doing the backward match operation, i.e., it makes Telecide field order dependent. If you are having difficulty with a specific clip in this regard, please make the source clip available for inspection.

@Homie FR

I will send you FTP access details in a PM.

[JuanC]

Originally posted by neuron2
Setting mm=2 in old Telecide is exactly the same thing as not doing the backward match operation, i.e., it makes Telecide field order dependent. If you are having difficulty with a specific clip in this regard, please make the source clip available for inspection. As soon as I get home back from work I will chop a relevant part of one of my unprocessed mpeg2 tv captures, about 10-20MB, and I would also appreciate if you send me a PM with details on temporary FTP access. (I´ve got a slow cable connection)

Thanks, :J

[Guest]

It's not my FTP site and I have to go ask for permission every time.

Yours is small enough to break into two pieces using WinRAR and mail to me. Please do that and make each piece less than 10 Meg. Also, please retain the pieces until I acknowledge getting them all. Thank you. neuron2@attbi.com

[HomiE FR]

@neuron2 : I just sent you a PM. The vob file should be ok. Thanks for trying to locate the problem, I'll keep checking this thread for news ! :)

[DoW]

I was wondering about the handling of fade in/out in the new decomb, so I tested it on "Zone of the Enders: Idolo" and noticed that while the frame was mostly black with the white intro credits fading in, the vmetric was very small (started at vmetric=0.000000), and slowly increased as the credits bacame brighter (up to vmetric=31.xxxxx). The effect was that even though the credits had combing, telecide wouldn't detect them as combed, but when they reached their full brightness, it would detect them as combed, event though they were not.


Telecide(order=1, guide=1, vthresh=33, post=2, blend=true, chroma=true, show=true)
Decimate(cycle=5, mode=2, quality=3)


Not sure if this is a special case or what have you, but I figured Id point it out and be on the safe side. If Im wrong, please feel free to bring out the tar and feathers.

[Guest]

Originally posted by DoW
Not sure if this is a special case or what have you, but I figured Id point it out and be on the safe side. If Im wrong, please feel free to bring out the tar and feathers. Does Decomb 4 handle it any better?

[killingspree]

Originally posted by neuron2
@killingspree

They are identical to the previous versions. Is that clearer than unchanged? :)

oh well... thanks for the info... i guess /me was stupid once again. i just checked the new version on some captured pal material and it worked well.

thanks for your work
steVe

[DoW]

@neuron2:
Yeah, it handles it better. Using Decomb 4.10b2, I had to use threshold=4 to detect the combing in the intro credits. The vmetrics for the credits are high in comparison to the rest of the movie in that, the static credits (non-combed) have a vmetric of 31ish, while in the rest of the movie, the highest vmetric that a non-combed frame has is 22. Im kinda busy, so later tonight Im gonna look though ZoE some more and see if I notice anything else.

[Shayne]

Would just like to say excellent Tutorial and excellent results. Great thanks for all that hard work.

[Guest]

@Shayne

Thank you for your fedback.

@DoW

Can I get the unprocessed source clip of the fade?

@HomiE FR

I cannot duplicate your isses, especially that strange PNG image. I use this script and everything looks just wonderful with the VOB you sent me:

telecide(order=1,guide=1,post=2,vthresh=75)
decimate()

Please try beta 2 with post=2; perhaps the issue with video passthrough was biting you. Bear in mind that clip has some frames with detail that really looks like combing! You need a relatively high vthresh to handle that.

@killingspree

I didn't mean to imply you are stupid! :o
You've been very helpful to me in the past and I always appreciate your feedback.

@all

I have removed beta 1 as beta 2 is coming shortly.

[Guest]

I have released beta 2. See the link at the first post in the thread.

The post parameter has changed:

-----
post (0-5, default 2) controls whether and how Telecide performs postprocessing to clean up frames that come through the field-matching still combed:

post=0: Use this to totally disable postprocessing.

post=1: Use this to enable the metrics calculation and to display vmetric but not perform deinterlacing.

post=2: Use this to enable deinterlacing. Note that in this mode, the field matching occurs normally and the best matched frame is deinterlaced and delivered.

post=3: This is the same as post=2 except that the deinterlacing motion map is displayed in blue on the deinterlaced frames.

post=4: This is the same as post=2 except that instead of using the best field match for the frame, the original frame is deinterlaced. You would use this to pass video sequences through when you have a hybrid clip.

post=5: This is the same as post=4 except that the deinterlacing motion map is displayed in blue on the deinterlaced frames.
-----

Previously post=4 behavior was implemented by default. Now it has been divided into post=2 and post=4 (for hybrid clips). post=2 is the way Decomb classic works.

[DoW]

I split off a bit of the VOB (~56MB) that has the intro credits in it, plus a few frames that generate artifacts similar to what homie FR was describing earlier. I can toss it on my FTP (~16k/sec UL) if you want, PM me to let me know how you want me to get it to you.

EDIT:
b2 fixed on issue I noticed with a frame selection.

[Guest]

I'm wondering if the problem you and HomiE FR are reporting is coming from doing random timeline movements. Does it happen if you just play the clip straight through from the beginning, or just after moving around on the timeline?

[DoW]

-> forwarding

[Guest]

I've now duplicated the HomiE FR problem. It follows timeline navigation. I am withdrawing the beta for repairs. Please stand by.

[HomiE FR]

@neuron2 : Thanks, thanks, thanks. I'm standing by, no problem ! :) I was a bit "disappointed" (maybe not the right word, since it was actually working right) that you didn't see the arfifacts I get, but since you found them and understood where they come from, that's great (and fast also).

[Guest]

@HomiE FR

You're my man, HomiE!

I found the bug. I'm so stupid. I'll release a new beta this weekend. I want to also add a great new feature that will fix scene changes. If we detect a frame as combed, try the forward match before giving up! This gives us the benefit of (most of the time) doing only the one-way match but without messing up some scene changes.

I also want to look at the clips submitted by DoW and JuanC.

@all

I must say, you dom9'ers really are "the cat's meow" when it comes to testing.

[killingspree]

Originally posted by neuron2

@killingspree

I didn't mean to imply you are stupid! :o
You've been very helpful to me in the past and I always appreciate your feedback.

oops i guess i sounded a bit harsh (:
i wasn't offended in anyway!

oh and now i'm looking forward to the new release... i somehow missed beta2 so yes... i'll see what is coming up next...

off to do some other testing :-P

steVe

PS: sorry OT: @neuron2: have you seen/read the new and improved version of the capture guide?

[Guest]

Originally posted by killingspree
PS: sorry OT: @neuron2: have you seen/read the new and improved version of the capture guide? Is that the one that you translated and I proofread? :)

[killingspree]

" Is that the one that you translated and I proofread? "
that was the basis... but we released the new version (2.0) of the guide a few days ago. we added the complete NTSC part, postprocessing with avisynth and gknot, logo removal and many many more things (:
partly you'll hardly recognise it again!

steVe

[HomiE FR]

@neuron2 : :) I'm always ready for some testing, and when I mean that you're the one to thank I mean it. That's you who give us this great tool (I wouldn't be able to code one line of C++) ! I'm really happy that you found the problem and if you can even improve the scene changes with forward match, I think New Generation Decomb will beat up my anime source easily ! ;)
Standing by for your new version...

EDIT : bad typo...

[N_F]

Great work Donald! I haven't had any good material to really test it on, but from what I've read it seems really good. One thing though:

Have you considered how this top/bottom field thing one needs to examine will affect automazation programs like Gordian Knot?

[Guest]

Originally posted by N_F
Great work Donald! I haven't had any good material to really test it on, but from what I've read it seems really good.Be sure to use the beta 3 I am now posting!

Have you considered how this top/bottom field thing one needs to examine will affect automization programs like Gordian Knot? Sure. They will either require the user to specify the field order, or they will continue to use Decomb classic.

[Guest]

Decomb 5.0.0 beta 3 is now available. Please see the first post in the thread for the link.

It fixes the matching bug reported by HomiE FR and it will now try the forward match when a frame is still combed after current/backward matching and pattern guidance, thereby fixing the scene change problem. That only works when post != 0, but postprocessing is so blazingly fast that there's no reason to turn it off.

This kicks butt on HomiE's VOB:

telecide(order=1,guide=1,post=2,vthresh=75)
decimate()

I will now have a look at the clips sent by DoW and JuanC.

Note that I am still not fully happy with the pattern guidance and anticipate improvements there. But so far I think the field matching and postprocessing are really looking good, and FAST!

[Guest]

@JuanC

Your ducks clip is handled just fine by beta 3 with this:

Telecide(order=1,guide=1,post=2)

And if you set guide=0, your Friends clip looks fine.

[JuanC]

Originally posted by neuron2
@JuanC

Your ducks clip is handled just fine by beta 3 with this:

Telecide(order=1,guide=1,post=2)

And if you set guide=0, your Friends clip looks fine. Yes, It's a whole lot better now! :cool: My friends clip got almost zero interlaced frames now, better and faster!, than with previous versions of Telecide.

However I could use any advice on how to deal with the almost static scenes where only small parts are moving: if you take a look at frames from 88 to the end, on the clip with the ducks, you’ll see that it’s telecined, following the telecine pattern, but all frames are detected as progressive, “out-of-pattern” and thus passed intact. With the betas of Telecide 4.10 if I use guide=0,mm=2, they are correctly matched… Well, it misses just a couple o frames.

Again, thanks for your great, generous work. :) Juan

[Guest]

@JuanC

Almost zero is not acceptable. :)

Two things are happening. First, your clip is a noisy capture. If you set nt=50, it gets fixed. This script looks great:

Telecide(order=1,guide=1,post=2,nt=50)
Decimate()

But there is also the pattern guidance issue. I wrote above: "Note that I am still not fully happy with the pattern guidance and anticipate improvements there."

For now, nt=50 is your ticket. I'm wondering if I should have some noise tolerance by default. Decomb classic had some nt by default; that's why it worked.

Also note that pattern guidance is disabled for the first and last 9 frames of all clips.

Thank you once again for your valuable feedback.

[Guest]

@DoW

One of my basic philosophies is not to worry about things that cannot be perceived!

If you set vthresh=25, the worst frame is imperceptibly combed when viewed normally. OK, if you stop the clip, zoom in by 2 (which is a no-no for assessing combing!), you can see FAINT combing. No human being would ever notice it playing normally.

So others can see my point, here is the WORST frame in the fade; subsequent ones are deinterlaced. And realize that the fade is very quick; there's no way anyone will perceive any combing.

http://shelob.mordor.net/dgraft/misc/fade.jpg

If you handled that by using threshold=4 with Decomb classic, then you obviously ruined the rest of the movie.

[DoW]

Hmmm... wonder about ZoE. Might try a little nt there as well to see what happens. ZoE is such a weird source, I can see why most people hesitate to work on it. Havent gotten a chance to try 5.00b3 yet, thats on my list of things to do tommorow. Thanks for all your effort neuron2.

EDIT:
Actually, I used trim to do FieldDI on only that section, so it applied to only the credits (/me grumbles about ApplyRange sillyness, still working on that).

[Guest]

Originally posted by DoW
Actually, I used trim to do FieldDI on only that section, so it applied to only the credits (/me grumbles about ApplyRange sillyness, still working on that). That makes more sense, but I still say why bother? Anyway, you can force it much more easily with a Telecide override file.

[JuanC]

Originally posted by neuron2
... Two things are happening. First, your clip is a noisy capture. If you set nt=50, it gets fixed. This script looks great:

Telecide(order=1,guide=1,post=2,nt=50)
Decimate() Wow! This is just great! it works wonders!!!

Thanks again, Juan

[HomiE FR]

@neuron2 : I would like to apologize for something : my bug reports were NOT totally justified... :( The new beta version improved things, but when I tried again this morning I saw one more time these ugly lines when there was high motion (the artifacts that you didn't see first). Now I understand why you didn't see as much artifacts as I did : my brand new Avisynth 2.5.1 RC4 from sh0dan's page had a flawed ConvertToYUY2()... :( When you told in your last post that the new beta worked well with my clip, I did not question your statement and asked myself how things could go wrong with such a simple script. And ConvertToYUY2() was the only possible guilt. I used then YV12toYUY2() from MPEG2Dec3 1.06 from Nic and these artifacts went away suddenly (I checked a few times to be sure).

I'm sorry for having made a "false" bug report, but since it helped in some way (because of other artifacts in this clip) to improve New Generation Decomb I'm not so ashamed ! :) Moreover I think the scene changes look better now.

Thanks again neuron2, the clip is just perfect now. I'm now able to use Decomb on the full source, be sure that I'll come back if I spot any new problem (even though I'll double check my script and my Avisynth version before I do some bug reports ;) ).

[Guest]

@all

Beta 4 is released. It fixes a crash when the last frame needs to be deinterlaced. See the first post in the thread for the link.

@HomiE

Are you sure the ConvertToYUY2() is broken? Did you use ConvertToYUY2(interlaced=true) as required? If it is really a bug, have you notified sh0dan? Please advise me about this, as it affects my testing too!

Finally, why not avoid the conversion by using mpeg2dec (instead of mpeg2dec3) until YV12 support is added? You should really do that as it is much faster.

[HomiE FR]

@neuron2 : Too bad it was my fault. You're right, I had forgotten to put "interlaced=true"... :( I have first notified sh0dan about my "bug", but he told me to put "interlaced=true" just like you. And you are both right, the artifacts that were still there are now gone with "interlaced=true".

But about using MPEG2Dec instead of MPEG2Dec3, I will follow your advice. Thanks.

HomiE

[Guest]

@HomiE FR

Thank you for the clarification. Your feedback has been very helpful and even your mistakes are instructive and will help others. ;)

@all

There seems to be some confusion about post=2/3 versus post=4/5. Therefore, I have added the following text to the tutorial. I hope it clarifies things for you.

"There is one more important thing to say about postprocessing before we move on. There is a subtle difference between post=2 and post=4 which might affect you. The reference manual describes the differences in detail. Setting post=2 will often avoid making gross frame blends at scene changes, but it can make deinterlaced sequences jerky. Setting post=4 retains the smoothness of deinterlaced sequences, but will make frame blends at all scene changes."

[Guest]

I've started a FAQ for Decomb 5. Please see the first post in the thread for a link.

I will appreciate receiving suggestions for FAQ additions.

[DoW]

Originally posted by neuron2
That makes more sense, but I still say why bother? Anyway, you can force it much more easily with a Telecide override file.
Cause Im a crazy perfectionist (by virture of what I do in real life, 'cause mistakes are B.A.D. in the chem lab (when a chemist makes a mistake, the word 'evacuation' is usually used:eek: ). Seriously, thanks for all the work neuron2. As for the override, Im still kinda trying to learn that.