Decomb 5.0.0 beta 15: Fix for crashing problem

[Guest]

Originally posted by dsmith
Having trouble with the override file. David keeps me honest on the overrides for pattern matching. :)

Get this:

http://shelob.mordor.net/dgraft/decomb/decomb500b8.zip

[dsmith]

Quick fix :)

Next:
There's no indication in the Show text when a frame has had the postprocessing aspect overridden. You can tell by the absence or presence of a motion map on post=3, but there's nothing for post=1.


--
David

[Guest]

Originally posted by Bach
is it possible for you optionally to get the order parameter from the d2v file instead of simply from "order=1(or 0)"? Sorry for the delayed response; I've been thinking about it. :rolleyes:

First, not all source clips come from DVD2AVI, for example, my off-air Sopranos captures.

Second, strangely enough, a clip that is fully 3:2 does not have enough information in the D2V file! The numbers repeat 0 1 2 3 and if you think about it, it just isn't enough. If you see some interpolated all 0 or all 2 sequences, fine, but if not...

Third, I'd prefer users to understand the issues and affirmatively set a field order. That's why it throws an exception if the order parameter is omitted. :devil:

[Guest]

Originally posted by dsmith
There's no indication in the Show text when a frame has had the postprocessing aspect overridden. You can tell by the absence or presence of a motion map on post=3, but there's nothing for post=1.
That's true. My reasoning was that the user asked for it, do we need to tell him?

Actually, the show function was getting complicated... :D

How badly do you want it? You've earned some good brownie points so far.

[gizmotech]

Originally posted by neuron2
EDIT: It occurred to me that maybe you are misinterpeting "override" in this context. If the pattern guidance thinks a blind field match is not what it predicts based on pattern analysis, it tries to override the blind field match. The override is allowed if the mismatch is less than the gthresh.

Ahh, you're right, I wasn't understanding correctly. This is a much better explination. (though I'm still wrapping my mind around a few things).

Perhaps it would be better to describe gthresh as a limiter, as it limits how large the mismatch can be when overiding the blind match(assuming I understand).

At any rate, I'm quite happy with this version, now I just need some yv12 to test with and you've got yourself best filter of the year. (no more silly 2k override files for me! w00t!)

Gizmotech

[Guest]

@gizmo

I'll clarify the reference manual and add a FAQ entry. If you misunderstood, it must have been badly expounded. :)

Thank you for the feedback.

If nothing serious comes up in the next few days, I'll start the YV12 support. Then I want to integrate the new Decimate(mode=4) hybrid-handling functionality and the new edge-directed interpolation (to fight jaggies on interpolated areas).

[Guest]

Here's the new FAQ I made in response to gizmo's point:

How does gthresh affect the field matching when pattern guidance is enabled?

If the pattern guidance thinks that the calculated (blind) field match is not what it predicts based on pattern analysis, it tries to override the blind field match. The override is allowed if the pattern mismatch is less than gthresh. Think of gthresh as a limiter: it limits how large the mismatch can be before a predicted match is overruled by the actual calculated match. If your gthresh is too high, it may cause a new pattern in the clip to be erroneously overridden by the previous pattern. There is a happy medium that allows new patterns to be locked onto quickly while still allowing the pattern guidance to correct mismatches due to noise, etc. The happy medium is usually in the range 5-15%, but it will be clip-specific.

[cipher]

Okay, I've just done 4 music videos, and the results are awesome as expected!

Three of them contained a lot of fades, and one of these three was a little noisy, but I used "nt" to beat it; the last one was much more like normal films, clean and well-made.
The result was absolutely exciting. And Decombe 5 beta7 had just done what its previous versions never did.:D

[JuanC]

Great! Betas 7/8 are giving me better results than previous betas! :) I have some questions though:

1. GThresh: What does (technically) a “pattern mismatch=100.00%” mean? I see this frequently in some parts of my video clips.

2. VThresh: Many times in my clips, the vmetric is higher than VThresh, but the frame is shown as being “[progressive]” (and not post processed) ??? In fact these frames are not interlaced, but :confused:

3. VThresh: Also, I have seen that lowering VThresh, Telecide, instead of deinterlacing the best match (P or C), would make a match “using N”, resulting some times in additional duplicate frames (Exact duplicate of the next frame). Could it be possible in a new release of Telecide to tweak this behavior? (Perhaps using a new threshold?) So that some times it could deinterlace (blend) it?

4. VThresh: Finally, when I used VThresh=255 to see what the vmetric was originally for a set of frames, then I saw that vmetric is always zero???

Thanks for your invaluable help,

Juan

EDIT: Numbered the questions to be referred in my next post.

[cipher]

JuanC, the manual has been updated.:)
Vmetric higher than the Vthresh is not necessarily considered being combed, but ONLY vmetric higher than 48 is defined as combed.
If you lower Vthresh, vmetric will be calculated to a higher value, if it goes up beyong 48, then it is shown as "interlaced".

Edit: So 48 is the unique real threshold, but vthresh acts more like a calibration for every specific movie.

[JuanC]

Thanks cipher, I missed it :rolleyes:. It's in the tutorial!!.

There are the answers for my questions 2 & 4. So the only valid, remaining, questions after reading "DecombTutorial.html" are 1 & 3.

Thanks again, :J

EDIT: I noticed that using nt>0 will also lower the vmetric values.

[Guest]

@JuanC

Pattern mismatch of 100% means either that the pattern analysis did not result in any prediction at all, or that the frame was combed after pattern guidance.

Regarding the forward match: I will consider your request. Of course, Decomb classic would always output the forward match, wouldn't it?

The nt parameter should be used with caution. It is appropriate only for really noisy clips and will do harm otherwise.

[cipher]

The nt parameter should be used with caution. It is appropriate only for really noisy clips and will do harm otherwise.

I know "nt" affects the decision made by telecide(), when you said "do harm", did you mean nt may not only give a wrong judgement, but it may also decrease the image quality?

[Guest]

"Do harm" just means that the field match decision may be made wrong.

[JuanC]

Originally posted by neuron2
Regarding the forward match: I will consider your request. Of course, Decomb classic would always output the forward match, wouldn't it?
@neuron2, Thanks for your answers.
Honestly, I don’t know how to answer that, since I get better results for my TV captures using Telecide v4.10b4 (mm=2): matching fields from only Current and Next frames. Here the options I use:

Telecide(guide=0,Threshold=25,blend=true,chroma=true, mm=2, y0=350, y1=430, show=true)

And the ones I use for v5.0b8:
Telecide(order=1,guide=1,Gthresh=10,Vthresh=12,post=3,blend=true,y0=350,y1=430,nt=50,show=true)

[Guest]

@JuanC

What do you mean by "better results". Can you provide a source clip that demonstrates it?

You have guidance enabled in one and not the other. Please make proper comparisons.

Are you SURE you have the field order correct?

[JuanC]

Originally posted by neuron2
What do you mean by "better results". Can you provide a source clip that demonstrates it? I mean the best you can get with that version (4.10b4). It is important to note that I have only tried video clips I capture from analog, noisy TV.

These clips are hybrid mostly film. I have tested this morning the same clips I sent you before. I could send them to you again, you just have to say so. Honestly, they look better to me with those options in v4.10b4

Originally posted by neuron2
You have guidance enabled in one and not the other. Please make proper comparisons. You suggested me to disable guidance with Telecide 4.10 to disable pattern hinting to Decimate; It works wonders. In v5.0b disablig guidance would "Do harm" to the field matching. (in my clips)
Originally posted by neuron2
Are you SURE you have the field order correct? Yes I am, I used your very illustrative tutorial to make sure.

[Guest]

@JuanC

I ask what you mean by better and you respond by saying they look better! Can you please say what is better? Are the scene changes better? Is the field matching generally better? Are there more combs? WHAT???

When I get home tonight I will retry your clips.

[HomiE FR]

@neuron2 : Hi, thanks for the new betas. :) I think there is a bug when using ovr parameter in Telecide in beta 8. I think it's not my fault because the only line there is in "telecide.txt" is :
3044,3214 nppcc
and I'm sure that Decomb reads the line because I've got an Avisynth access violation at frame 3044.

Moreover this is only happening when using "n" frame in ovr. But I need these frames in some "travellings" (pannings in goog english??), I can send you the scene where ovr is needed in order to get correct field matching.

Please correct this small bug, Decomb is close to reaching the top ! :) Thanks in advance.

HomiE

[Guest]

@HomieFR

I can't duplicate this. Please provide a clip, script, and override file that causes it to happen. Also, please provide a clip showing the need for the overrides (if it is not the same as the one just requested). Thank you.

[JuanC]

Originally posted by neuron2
Can you please say what is better? Are the scene changes better? Is the field matching generally better? Are there more combs? WHAT??? OK, you're right. When I say v4.10b4 look better, I mean: The field matching is better in general using (mm=2) for my noisy, hybrid mostly film, Analog TV captured clips. I don't really know if this is very common for capture cards. I use an ATI AIW 8500 with catalyst3.4 & MMC8.5, interlaced MPEG2 @8mbps. And, as a result of this:

There are less, almost zero "combed" frames escaping from field matching/ post processing (however there are more frames being Post-Processed/ blended than with v5.0); and
There are no additional duplicate frames resulting from pattern changes making only occasional C/N matches.

The scene changes are fine with both versions (4.10b4 & 5.0.0b8) of Telecide (There are some sporadic new duplicate frames with 5.0). Any way, latest beta 8 of v5.0 is an enormous improvement over previous 5.0 betas.

I have seen an issue (Telecide v5.0) with horizontal patterns in some frames being detected as combing and thus getting blended.

I wish there could be a not very far future release of the new Telecide with the ability to tweak any override or option to get a behavior similar to mm=2 in the old Telecide.

Thanks yet again, :J

[HomiE FR]

OK no problem, here it comes :

1. The clip comes from the same anime that I already sent you before to test previous matters. But it's another scene, which I already cut in order to send it to you. The vob file is 23.8 MB.

2. The script is just what is written below (just the directory of the vob file is different) :
MPEG2Source("[...]\override.d2v")
ConvertToYUY2(interlaced=true)
Telecide(order=1,post=0,ovr="telecide.txt")
Decimate(cycle=5)
ConvertToYV12()
The problem happens no matter if Decimate is used or not, and no matter if ConvertToYV12 is used or not.

3. The override file is called "telecide.txt" and is put in the same directory as the script which is written above. There is only one line is the file :
52,773 nppcc
The problem appears with "nppcc" as the manual field matching. If I put "pppcc" there is no avisynth access violation, but there is combed frames every 5 frames, as you'll probably see when looking at the clip.

The problem is, to be precise : "Avisynth read error: Avisynth: caught an access violation at 0x..., attempting to read from 0x..." (the two numbers ... are different).

This clip is, as you thought, the one where I believe that Decomb is unable isn't able to do the correct field matching without some help. But I hope you'll prove me wrong ! :)

I'm not sure about how I should send you the clip : do I use the ftp account that you provided last time ? I'm waiting for an answer to send the file.

HomiE

[Guest]

Yes, use the ftp account.

[Guest]

@JuanC

I keep telling you, Decomb 5 already emulates mm=2 when order=1!

Please, let's get specific. I have your clips and your scripts. Point me to the clip and frame numbers you think are handled better by Decomb 4. Leave off Decimate so we can use the direct frame numbers from Telecide.

I'm sorry to be so exacting but generalities just don't help us.

Thank you!

[JuanC]

Originally posted by neuron2
Please, let's get specific. I have your clips and your scripts. Point me to the clip and frame numbers you think are handled better by Decomb 4. Leave off Decimate so we can use the direct frame numbers from Telecide. OK, I'll do so when I get home. Thanks.

[HomiE FR]

@neuron2 : I've got an answer. If post=0, it's impossible to play with "n" field matching. It leads to the bug described above. Moreover during some scene changes I can see some combed frames just after the change caused by the use of another field matching instead of "n". If post=2 or post=4, it's possible to play with "n" in an override file, and the scene changes are perfect, with the "n" field matching detection where it needs to.

However I'll still send you my clip when I get the pass to the FTP, cause I still need overrides to make it perfect.

HomiE FR

PS : I don't know if what I said above is totally obvious, but at least in the reference manual it isn't obvious that "n" field matching should only be used when post!=0.

Edit : I have another problem with beta 8, but I don't think this is a bug... I have problems with some scene changes. First of all this scene change needs a forward match "n" on the frame after the change in order to be uncombed.
- if post=0, "c" is used instead of "n" and the result is combed,
- if post!=0 AND vthresh=10 (default), "n" is used ! :) which is great. But with such a low vthresh I have many progressive frames which are deinterlaced. So I have to increase vthresh.

1st question : why isn't field matching acting the same if post=0 or post!=0 ? I thought that field matching and deinterlacing were two different operations, done one before the other. In my case I would love to have the scene changes at "n" even without postprocessing (cause my clip is really clean).

Then is I increase vthresh so that no (or very few) progressive frames are processed, the scene change becomes falsely detected (field matching at "c" instead of "n")... Could you explain (please) why Decomb works this way ? It seems that in my case I can't have a perfect setting : on the one hand you can have good field matching on scene changes BUT too many frames processed, and on the other hand you can have false field matchings on scene changes BUT few progressive frames processed.

Thanks in advance neuron2. I will also send you the part with this scene change so that you can test it (if you want and have time to).

[Guest]

Originally posted by HomiE FR
@neuron2 : I've got an answer. If post=0, it's impossible to play with "n" field matching. It leads to the bug described above. Confirmed. I will fix it for the next beta.

Moreover during some scene changes I can see some combed frames just after the change caused by the use of another field matching instead of "n". If post=2 or post=4, it's possible to play with "n" in an override file, and the scene changes are perfect, with the "n" field matching detection where it needs to. I need the clip that demonstrates this.

PS : I don't know if what I said above is totally obvious, but at least in the reference manual it isn't obvious that "n" field matching should only be used when post!=0. I'll make sure it is prominent in the documentation.

Edit : I have another problem with beta 8, but I don't think this is a bug... I have problems with some scene changes. First of all this scene change needs a forward match "n" on the frame after the change in order to be uncombed.
- if post=0, "c" is used instead of "n" and the result is combed,
- if post!=0 AND vthresh=10 (default), "n" is used ! :) which is great. But with such a low vthresh I have many progressive frames which are deinterlaced. So I have to increase vthresh. I need a clip that demonstrates it.

1st question : why isn't field matching acting the same if post=0 or post!=0 ? I thought that field matching and deinterlacing were two different operations, done one before the other. In my case I would love to have the scene changes at "n" even without postprocessing (cause my clip is really clean). For speed reasons and to avoid spurious n matches, which shouldn't be necessary except at scene changes, Decomb does not look at the n match *unless* the matched frame is determined to be combed. You can get what you want by setting post=1. My concern is that you are claiming more than just scene change frames are affected. I really need to see the clip that demonstrates that. Thanks.

Then if I increase vthresh so that no (or very few) progressive frames are processed, the scene change becomes falsely detected (field matching at "c" instead of "n")... Could you explain (please) why Decomb works this way ? It seems that in my case I can't have a perfect setting : on the one hand you can have good field matching on scene changes BUT too many frames processed, and on the other hand you can have false field matchings on scene changes BUT few progressive frames processed. The combing detector should be able to achieve what you want. I need the clip to comment further.

I need the clip(s) that demonstrate all these points plus scripts and instructions for creating them. Thank you.

[Guest]

By the way, I just published a description of MetricX in my journal.

[InfoCynic]

Edit: This took so long to post, dgraft posted TWICE while I was
working. Sheesh. :) Well, keep in mind your reply wasn't available when I posted this.

So I've spent about 8 hours in the past 2 days playing with Decomb 5 and everybody's favorite whipping-boy, TNG. Yes, I have no life, thank you all for caring. :) This has led to some interesting revelations...

1. Some of the exterior/space scenes in later seasons are FILM, with an identifiable 3:2 pulldown pattern. Not all, and the pattern can be hard to get decomb to match... but important to know.
2. Post-processing plus pattern-guidance is a very bad mix. Since TNG contains large portions of 3:2, guide = 1 made sense, and with some tweaks to the gtresh, I was very happy with the matching I was getting. Until I moved from post=0 to post=1 to find vthresh. I'll put up a series of screenshots for this in a bit.

With postprocessing on, if a frame is detected as interlaced, it gets a 100% mismatch and is automatically out-of-pattern. This makes some sense in theory, but when reality is that our vthresh is going to mark some progressive frames as interlaced, and that these frames are going to be in-pattern, it's a little annoying. I think Homie FR was getting at something like this in his post above.

I want to be able to keep guide=1 on, because with the new pattern guidance algorithms (which are working quite well with post=0, I must say), I can use hints=true and pass it off to mode=1 or mode=3 decimate (side note: I'd like to see that new mode 3 algorithm as an option soon, so I can test it on TNG while I'm working on it anyway).

3. Deinterlacing of frames detected as combed, and noticably combed, is not nearly strong enough, even with dthresh=0. I'll post a screenshot of that too.

Overall, I think Decomb 5 may finally be able to get us where we need to be in order to be able to handle hybrid material like TNG or Babylon5. Given that the problem has always been knowing FILM from VIDEO and that the new pattern guidance can work with it, at least in some cases, is very encouraging.

Telecide(order=1,guide=1,gthresh=17,post=4,vthresh=4,dthresh=10,show=true,hints=true)

Screenshots at http://infocynic.no-ip.org/TNG/
Changes from that for specific screenshots:

TNG01.PNG: post=5, shows areas being detected as interlaced
TNG02.PNG: post=4, vthresh=5. frame is no longer detected as interlaced. But I need vthresh=4 if I want to catch most of the interlaced frames...
TNG03.PNG: exactly as stated. shows frame flagged interlaced, out-of-pattern
TNG04.PNG: post=0. shows frame without any postprocessing. Looks good to me, and it's even in-pattern (which it should be, it's clearly part of a 3:2 cycle if you saw its neighbors).

TNG05.PNG: post=0. shows another frame that needs deinterlacing. this is the "before" postprocessing.
TNG06.PNG: post=4, dthresh=0. This is the "after." Not much improvement.
TNG07.PNG: post=5, dthresh=0. And yet... to see this, you'd think it would have done a better job.

Let me know if you need anything more, I can probably cut a vob sample for you, since the old one you have is a whole different set of problems, which I don't want to address in this post.

[gizmotech]

Though it's largely useless to say this, I can confirm what Homie is saying as I have noticed this behaviour of scene changes being matched to C at vthresh's higher then 10. I had assume it was intentional, however I have identified 1 or 2 such scene changes where the scene needs to be matched to N but has been matched to C.

GizmoTech

[Guest]

@Info

TNG06 looks very well deinterlaced to me. What are you talking about?

Anyway, the ONLY way I want to address problems is with a VOB, a script, and a text message telling me what frames you object to. I hope you can appreciate the reason for that. Thank you.

[dsmith]

On the request about showing the Post override state in the Show text - that's just an "it would be nice if" request. Not really needed, just makes things a bit easier sometimes.

Read over your journal entry for today. Very nice setup for the algorithm :) However, it also explains why I'm having trouble with getting it to recognize when it needs to do the deinterlacing. You're requiring an additional comb match to the left or right of a detected pixel. The combs that I have the most trouble with are usually set up somewhat like this:

xx
xxxx
xxx
xxxxx
xxxx
xxxxxx
xxxxx

I have to move the vthresh down to around 3 or so before it will push those frames over 48. Obviously it's because the current algorithm isn't really designed to find that kind of combing. To be fair, with a normal vthresh in the 8 to 10 range the vast majority of the problem frames are caught, and I can still always set up the override files for this. My question is, what sort of artifacts show up if you look for adjacent combed pixels above and below? And what about checking the diagonally adjacent pixels, to find the stairstepping problem? (assuming you're only checking every other line, perhaps require either TL+BR or TR+BL adjacent pixels be combed)

Anyway, taking a look at some files that I had set up for v4... These are counts of the number of combed frames from pattern mis-matches during a particular scene. All of them except for one frame in the v5 set 3 in Ep 19 would fall under the "mouth" problem. The one out of place frame has combing across the entire face (seen from profile). No pattern guidance was used because I need to be sure there is never any frame uncertainty (I use these for editing).

Ep. 1 of BGC2040:

v4 override set 1: 5 corrections
v5 override set 1: 4 corrections

v4 override set 2: 8 corrections
v5 override set 2: 0 corrections

v4 override set 3: 7 corrections
v5 override set 3: 0 corrections


Ep. 19 of BGC2040:

v4 override set 1: 1 corrections
v5 override set 1: 1 corrections (different frame)

v4 override set 2: 6 corrections
v5 override set 2: 1 corrections

v4 override set 3: 38 corrections
v5 override set 3: 7 corrections


Overall an immense improvement. Also tried it on the Dual! creditless opening, and it was completely solid (though admittedly not much of a challenge; I keep it around for testing compression). Going to try a couple more series that I remember as being problematic.


--
David

[Guest]

Originally posted by gizmotech
Though it's largely useless to say this, I can confirm what Homie is saying as I have noticed this behaviour of scene changes being matched to C at vthresh's higher then 10. I had assume it was intentional, however I have identified 1 or 2 such scene changes where the scene needs to be matched to N but has been matched to C.

GizmoTech How am I supposed to do anything if you don't provide a clip?

C'mon guys, I'm here busting my butt for you, and you can't help me?

[troy]

Can someone take a big step back and help a newbie. What does this filter do. What is it good for. How do you use it.

[Guest]

It converts telecined and hybrid material to fully progressive material. The documents in the distribution answer all your other questions.

I was so tempted to answer: It's no good. It only eats CPU cycles. And nobody has figured out how to use it yet, much less its author.

[MasterYoshidino]

clip, clip, clip :(
would be nice haha but the most recent beta corrected my bad field matching on Chobits hybrid scenes where the "NEXT" button (3D Coreography @ 30fps ) overlayed on a FILM anime.
nonetheless the beta8 version seems to be very promising.

[gizmotech]

@neuron2

ohh come on.. it does more then just eat cpu cycles :P

All kiding aside it works really well. I'm just running my source through one last test run here, and if the scene change problem still exists for me I will post a vob segment for you to fiddle with. I think I might have been remembering from beta 6 however, as I didn't see the error where I thought I did before.

Gizmotech.
*Who really appreciates all of neuron2's hard work.

[DoW]

I am having some issues with ZoE and the new beta 8. First off is the fact that even though the vthresh is = 10, scenes with vthresh of 40 or so are passed though with post processing tellim me that they are progressive (the intro zoom in with the comb like effects have vmetric of up to 40 or so). Secondly, field matching issues that beta 6 resolved have cropped back up in this beta (see frames 931 and 932, 932 was okay in b6, but 931 was bad still). I think that is for some reason, b6 uses n for some of these frames, while b8 seems to be slightly stricter about using c and p only. Slightly more scarry are some frames where the new vmetric is lower for the combed frame than the rest of the scene (frames 894, 895). There is also a rather funny thing I noticed when going through the video, frame 828, vmetric = 0, p=0, c=0, pattern mismatch = -1.#J%. If any of this is repeat, Im sorry. Thanks for all the effort, I hope that this can all get ironed out, even if it involves calling me a nub. Note: tried with and without nt value.

AVS:
LoadPlugin("D:\Filters\mpeg2dec.dll")
LoadPlugin("D:\Filters\decomb500b8.dll")
MPEG2Source("D:\zoe idolo\ZoE test.d2v")
Telecide(order=1, guide=1, post=2, nt=20, vthresh=10, blend=true, chroma=true,
show=true)

I think you still have the cut I sent you, if you need it again, Ill send again. I can post a pic of the the last issue on my ftp if you want.

Edit: Stupid code to big, fixed

[Guest]

@DoW

Thank you, DoW! Now that is something I can work with. A clip, script, and description that demonstrates problems. :)

I've a very good idea what is happening and what to do about it.

@HomiE FR

I am advised that you are using the wrong password to login. The password is the doom9 screen name of the owner of the site. PM me if you can't deduce it from that. I post this here because you are not picking up your PMs, it seems.

EDIT: Everybody stand down. The same thing that is affecting DoW is affecting you all. Fix coming...

[HomiE FR]

@neuron2 : Ok the FTP account is working like a charm now, but yesterday when using the same login/password, it didn't work at all.

About the clip I'm sending you now (not finished yet, but in less than 30 minutes), it's a scene where I think that a few frames (during a fading) need an override field matching to be detected correctly ("n" field matching needed, and these frames are NOT detected as combed, far from that, you'll see why ! :) ). But I've never seen any filter (old Decomb, Uncomb, IT,...) which can handle it right.

Then about the scene change matter, the clip I provided last time should suffice to illustrate the problem. Just check the frame 153 (just after the scene change), it is typically the kind of frame where "n" field matching is needed. This is with this frame that I did my tests, so I don't think another clip is necessary.

Finally THANK YOU for your hard work. I think I write it in each post, but it seems it isn't enough. :p Of course Decomb is a great plugin, nobody will question that here. It's already from far the best out there. Sorry to have been a little late, but I just had to sleep a bit.

HomiE FR

EDIT : The file is up. Use override_good.vob (you can discard override.vob, since I had a little problem when uploading the file for the first time). Thanks.

[InfoCynic]

This is accurate as of beta 8.

I'll post the FTP to you by PM. My script and a file complaints.txt which details specific frames. I only have 25k upload and it's a 90mb vob (when I tried cutting it smaller, Telecide just would not synch up at all...), so be patient, but there's a lot of good examples of problems in there. The 1 or 1-2 frames here and there that are marked interlaced and out-of-pattern when they're progressive in-pattern, I'm not overly concerned about... realistically they probably won't matter... and the thing is, they are correctly flagged with post = 0 (every frame and range I found tagged wrong with post = 4 was right with post = 0).

The larger sets, and the interlaced frames not getting caught is annoying. If I use a lower vthresh to catch them, I get more and more problems with progressive frames marked interlaced out-of-pattern.

From a theoretical standpoint, does pattern guidance really have to be tied to the progressive/interlaced decision?

[Mentar]

I've played around with Decomb5 beta6 alot (didn't check D5b8 yet, but will now) and have made one little discovery which slightly confuses me. Please help me if I'm misunderstanding something here, but after careful stufy of the Reference manual I'm still none the wiser :p

When selecting the best field match for a clip, p and c are assessed and measured with a metric. The smaller result is arguably the better match, so it's being chosen and processed. Then the vmetric value is generated, which assesses how "combed" the result is, to decide whether or not it's interlaced and needs to be postprocessed.

Now my question: Shouldn't the "better" p or c value also result in a better vmetric result?!? Obviously this is not necessarily the case, see here:

http://www.earth-alliance.org/Combed.png
http://www.earth-alliance.org/Clean.png

c gets the better result than p, but the vmetric is much worse (and rightfully so, since the frame is combed)

This is the avs script I used to generate it:

LoadPlugin("C:\Tools\GORDIA~1\mpeg2dec3.dll")
LoadPlugin("C:\Tools\GORDIA~1\decomb.dll")
mpeg2source("Z:\noir vob1\Noir1.d2v")
ConvertToYUY2(interlaced=true)
Telecide(chroma=true,post=1,order=1,debug=true,show=true,
vthresh=26,ovr="override.txt")

After lots of manual tinkering, it seemed obvious that the vmetric was awesome. It correctly identified all combing in the resulting frame, later in the day I'd stop through the avs and only watch the vmetric to find bad field matches. After entries in the override file I was able to clean up the entire clip, and _always_ the manual overrides seemed to result in a better vmetric.

So here's my second question: Wouldn't it be feasible to code a Decimate mode which generates the vmetric value from _all 3 matches (p, c and n) and then selects the frame with the best vmetric? I wouldn't mind if such a mode was slow as a snail, it would ALWAYS be faster than manual interaction necessary for the perfectionists.

If any test clips are needed, I would be gladly willing to oblige.

Once again thanks a TON for the latest decomb, the results have never been THIS good already :D

[Mentar]

Just a quick addition with decomb5 beta8:

The effect is the same, just the values are a bit different:

1) For the "normal" mode without manual override:

frame 23324:
vmetric=46
p=318618 c=308882
[using c] [progressive]

2) For the manual override forcing p:

frame 23324:
vmetric=0
p=318618 c=308882
[forcing p] [progressive]

Both frames look just the same as in the other pics, so I chose not to post them again.

thanks to gizmotech :)

[Guest]

Time to come clean... :)

In beta 8, chroma processing was disabled, so chroma=true was ignored. I did that because MetricX was picking up too much chroma interlacing on my test clips and making it impossible to find a good vthresh value. I thought, I'll release the beta and repair that in the next beta and no-one will notice. Well, you did notice, e.g., DoW's combed frames are due to lack of chroma combing detection.

Now here's the astonishing discovery I made. I added proper chroma handling with MetricX. Immediately I started picking up totally unexpected chroma interlacing. I had a clip which was PERFECTLY matched field-wise, forcing the other ways made obvious gross luma combing. But the chroma was combed! So I separated fields and this is what I saw. First, one field of the matched frame:

http://shelob.mordor.net/dgraft/misc/chromashift2.jpg

Now the other field:

http://shelob.mordor.net/dgraft/misc/chromashift1.jpg

Look at the head of the man on the right. It's obvious why the frame is chroma combed! This is a revelation to me. I knew that chroma could be messed up due to bad Y/C separation (the rainbow effect), but I was unaware it could just drift apart like this on fast motion. MetricX brought it to light!

Here's the kicker. If I performed the forward match for the frame I got a good match with no combing. So this is an explanation of why forward matching may still be required, even if it is theoretically not required in the absence of such gross stream errors. But we live in the real world and we have to deal with these things. So it looks very much as if the forward match must be restored! I'm sure many of you will be happy to hear that, but for me it means a major re-write.

So, while I assess my options given this new information, please do not deluge me with reports and questions on beta 8 and before, as things are obviously going to change a lot. However, if you have any data or other experience about such chroma screw-ups, it would help me to know about them.

Thank you all for your excellent testing results. I hope you are learning as much as I am from the experience.

EDIT: After further testing it's pretty clear to me that suppressing the forward match test makes field-matching less reliable, and not more reliable (as I expected for theoretical reasons). Now I am seriously considering going back to a three-way match on the bottom field, and if that produces a combed frame, try the additional two matches on the top field. This would be the best possible matching strategy, since it would always output a progressive frame if one could be found. It would also give the best performance for clips with blended fields. Ah well, live and learn. :D

Still, we've made some gains that will carry over to the full 5-way match solution. MetricX is a great combing dectector. The idea of piggybacking combed frame detection on the field matching remains. And of course, the field matching itself is better now. Finally, the pattern guidance is better.

[Guest]

@HomiE FR

Thank you for the very interesting clip.

What is happening is that there is a clean 3:2 pattern and it fades into a clean 3:2 pattern with a different phase. So, this override file does OK to match the sections (only bothered about the first fade) but for the frames in the middle of the fade there is still a problem (e.g., frame 77):

0,79 cccpp
80,170 ccppc

If you do this you get frame 77 OK:

0,76 cccpp
77 n
78,79 cccpp
80,170 ccppc

So you have successfully provided another example of a case where real world material requires the forward match. Bravo!

It is not guaranteed that this will always help in this kind of case. It helped here because the phases were only off by 1. But, if there is a good match for a frame WE SHOULD DELIVER IT. That is why I am now going to the full 5-way matching strategy. This version of Decomb has taught us a lot, and much of it will be re-used, but it is now withdrawn.

[gizmotech]

@neuron2

Ok, so now that decomb is being recoded for a new version this doesn't really apply, but ok.

Homie's forward matching on scene changes is ok, however if you want to watch the true ******* child of forward matching anime, go rent disc 2 of Crest of the Stars. Episode 6 has more forward matching in it then any other source I've seen. I've attempted to observe a patern, however it just seems to randomly change between when it decides a forward match would be a good idea.

If nescessary I can try to post a vob for you. pm w/ details if interested.

Gizmotech

[Richard Berg]

Look at the head of the man on the right. It's obvious why the frame is chroma combed! This is a revelation to me. I knew that chroma could be messed up due to bad Y/C separation (the rainbow effect), but I was unaware it could just drift apart like this on fast motion. MetricX brought it to light!
It's admirable that you're willing to rewrite Decomb to handle situations like this, but speaking from my own habits, I always run something like Guava Comb before Decomb to remove such broadcast artifacts. (Of course, I can't say in advance how much success this would have on your test clip.)

P.S. Did the thread on new hybrid-clip handling disappear, or is it just me?

[^^-+I4004+-^^]

i've been doing a comparation of different deinterlacers,and i have reached such conclusion:
unlike every (hmm,didn't tried greedy,but..) other deinterlacer,fielddeinterlace is taking the bottom field and interpolating on that (all other deinterlacers use top field to make interpolation)_IF I SAW CORRECTLY!_

(i'm basing the asumption on 2 things i saw:comparing the frame deinterlaced by fielddeint. to other deinterlacers (especially in static/logo areas,and comparing fielddeinterlaced video to the source_looking for same areas as a test patterns)
even the smartdeinterlacer is building interpolation on the TF basis...

(and i guess i've been bashing fielddeint. before was because of this effect)

so,graft here's a question for you:would it be possible to make this an configurable option (to choose what will be the skeleton for the interpolated parts) so anyone can choose........
(although i see that fielddeinterlace is not real "spatial" deinterlacer as it interlaces the complete image..and not combed parts only...unlike it's counterparts)

i'm asking this because i would like to compare this deinterlacer with others,but this thing is keeping me from doing it properly.....(also it's twice as fast as smoothdeinterlace()...heheh),quality is there too,but bottom field?

also to throw this in too (before i am thrown out again...lol!)i have read something about that hybrid challenge,tickers and all(on web of yours),so i wondered do you need any pal "tickers" or you solved that already?(or is it ntsc stuff only?as you see i didn't read it all etc.)
some nice ones over here......

[Guest]

@I4004

It *is* a spatially adaptive deinterlacer. If you get that wrong how can anyone lend credence to your comparisons?

Why do you think it is important to interpolate only on the top field? So your misinformed comparisons can proceed?

[Guest]

Originally posted by Richard Berg
It's admirable that you're willing to rewrite Decomb to handle situations like this, but speaking from my own habits, I always run something like Guava Comb before Decomb to remove such broadcast artifacts. (Of course, I can't say in advance how much success this would have on your test clip.)Interesting. I'll try it but I still want to rewrite NGD (New Generation Decomb). Turns out my design is quite flexible and it is not so major as I thought.


P.S. Did the thread on new hybrid-clip handling disappear, or is it just me? Some idiot (who I have spent much time helping in various threads) inexplicably decided to **** on my thread, so because I can't just selectively delete people's posts, I deleted the whole thing. :)

Is there anything in there that you need?

EDIT: Tell you what. Since I'm in a take-no-prisoners mood, I will restore the thread to the Development forum with the **** removed.