Decomb 5.0.0 beta 15: Fix for crashing problem

[Guest]

Warning: guide=2 causes crashes. Fix coming tonight.

[OUTPinged_]

Duh, just came to report guide=2 is buggy.

DG is too fast for me :-)

[Pojken Purken]

I need some advice regarding the "order" parameter of Telecide. I'm trying to encode the old television series "Cosmos" which is even worse than TNG. The field order changes several times through the episode, even within a single scene. This seemed to be handled quite well with the earlier version of decomb. How should I handle this scenario in my AVS-script?

Regards
PP

EDIT: My problem seems related to DVD2AVI so I've started a thread about it in the correct forum. Sorry for bringing it up here in the debug thread. I've got a workaround so I'll start testing beta 13 now :)

[JohnMK]

Indeed I'd like to find the ultimate parameters for Cosmos as well. :) You can guess I'll be paying close attention to this thread.

[Guest]

Beta 14 fixes guide=2 mode. See the first post in the thread for the link.

[bouis]

@neuron2

This is most curious. The file I'm using isn't a vob; it's a .m2v and it can't be split by chopper_xp or tmpegenc. Creating a short .d2v of the problem scene change (with 5000 frames before it, no less) does not cause a crash. I'm using YUV2.

The problem filter is telecide with guide=1; with guide=0 or previous Decomb 5 versions it works fine. It does not work with betas 11-14.

The frame responsible is 100,052 (http://www.datasync.com/~bouis/badframe-100-052.JPG) and it follows an instant scene change from 100,049 (http://www.datasync.com/~bouis/scenechange-100-049.JPG). It is reported as 100% film by DVD2AVI.

If I attempt to use alt + right arrow to skip 50 frames in vdub past the problem, it crashes as well.

Yes, I use the first fielddeinterlace for debugging. I do a lot of westerns and they generally have too much false deinterlacing. I just do manual overrides. It can be tedious -- so, how about a fielddeinterlace option to only output frames which are detected as combed, with no deinterlacing? So if I have 100 detections, I get a 100 frame avi which I can easily skim over.

[Guest]

@bouis

Please get this DLL and tell me if it crashes. We can isolate the problem this way if you are willing to test a series of DLLs.

http://shelob.mordor.net/dgraft/decomb/decomb_bouis1.zip

Regarding your request, let me think about how best to achieve your goal.

[bouis]

@neuron2

Decomb_bouis1.dll does not crash.

Edit: Oops! yes it does. Forgot to un-comment the loadplugin line.

Thanks for considering my feature request.

[bouis]

#1 does crash, I just screwed up. I'm sorry.

[Guest]

@bouis

How big is the m2v file? I'm thinking it would be much easier if I can get it.

[bouis]

It's 4,924,495,566 bytes. I imagine that I can crop it and put it on a DVD-R. You're in the US right? E-mail me your address and I'll USPS it first class on monday, if you'd like.

[Guest]

Sounds good. Check PM. Thank you.

[bilu]

@neuron2

A question about field-order dependency: how is it related to the 2-way match? :confused:

Just to point why I have doubts:
Movie -> 2:3 -> Telecide()
==========================

a b c d e f g h i j k l -> a b b c d e f f g h i j j k l -> a b c d d e f g h h i j k l l
a b c d e f g h i j k l a b c d d e f g h h i j k l l a b c d d e f g h h i j k l l

c c n n c c c n n c c c n n c

Wrong Field Order
=================

a b c d e f g h i j k l a b c d d e f g h h i j k l l a b c d d e f g h h i j k l l
a b c d e f g h i j k l -> a b b c d e f f g h i j j k l -> a b c d d e f g h h i j k l l

c c n n c c c n n c c c n n c
It seems to me that there's no need for field-order dependency if we match current/next (2-way) only.
Maybe I'm thinking wrong, but I'd like to know why.

I got this doubt after reading in Decomb FAQ:
Why was Decomb 5.0.0 made field-order dependent?

It's faster and less error-prone to do 2 comparisons rather than 3. But Decomb 5.0.0 will still try the third match if a frame is still combed after the 2-way match. So no extra combed frames are emitted compared to Decomb 4. The tutorial provides a really easy way to determine a clip's field order.

And this got me thinking (Decomb Reference Manual) that the main reason why you need field-dependency could be because:

- it's matching bottom-field with top current/next when order=1?
- it's matching top-field with bottom current/next when order=0?
APPENDIX A. Telecide() Theory of Operation

Here's what Telecide does by default. When it receives a request for a frame it gets access to the next frame and the requested one (called the current frame). Here is what he sees (where N=next, C=current, t=top field, b=bottom field):

Ct Nt
Cb Nb

Telecide makes 2 combinations of frames from the available fields. Here are the combinations:

Nt
Cb

Ct
Cb

Now both of these frames are checked for combing. The least combed frame is the output frame (assuming no pattern guidance). It is that simple.

If I'm right about actual field matching, how about:

- On a TFF stream, after matching Cb -> Ct/Nt, we should try Ct -> Nb.
If failed we should then try Cb -> Pt and finally Ct -> Pb;

- On a BFF stream, after matching Ct -> Cb/Nb, we should try Cb -> Nt.
If failed we should then try Ct -> Pb and finally Cb -> Pt;

But to get better performance, we should decide between both methods based on the 3 last successful ones or something like that.

I believe that following these methods could reduce stuttering.

Insanity or ignorance? :)
Please give me some feedback.

Best regards,
Bilu

[JohnMK]

Donald,

What are your current Star Trek TNG recommendations for this latest beta? I came up with:

Telecide(order=1,guide=1,post=4,vthresh=18,hints=true)
Decimate(mode=3)

But I don't have an eye for such things. Can you or anybody else think of a better combination of parameters? Just a quick look at my test file and it appears video sequences are too jerky for my taste. I know that's to be expected from decimation however.

Just looking for some input, thank you.

[Guest]

@bilu

You wrote (for BFF):

a b c d d e f g h h i j k l l
a b b c d e f f g h i j j k l --> c c n n c c c n n c c c n n c

If you use order=1 for this, on the third frame you compare like this:


c d
|/
bThere ain't no match. Why do you show an 'n' match?

Then you wrote:

- On a TFF stream, after matching Cb -> Ct/Nt, we should try Ct -> Nb.
If failed we should then try Cb -> Pt and finally Ct -> Pb;

Consider a TFF stream:

... [a b] [c d] [e f] ...

Matching Ct -> Nb would match c to f. But they are 3 fields apart. How can they match?

[Guest]

Originally posted by JohnMK
What are your current Star Trek TNG recommendations for this latest beta? Are you talking about the clip from the "Star Trek Hybrid Challenge"?

[JohnMK]

Sure. Or any generic Star Trek The Next Generation clip with video & film.

[bilu]

Originally posted by neuron2
@bilu

You wrote (for BFF):

a b c d d e f g h h i j k l l
a b b c d e f f g h i j j k l --> c c n n c c c n n c c c n n c

If you use order=1 for this, on the third frame you compare like this:


c d
|/
bThere ain't no match. Why do you show an 'n' match?


Because I compared

c
|\
b c

It was a Ct -> Nb match. And [Ct Cb] [Nt Nb] they are 3 fields apart and match. And every time you match Cb -> Pt you're matching 3 fields apart too: [Pt Pb] [Ct Cb].

I don't know if my logic makes sense, but what I'm saying is try on a TFF stream to match as it was BFF before matching previous.

Do you think there could be streams that deserve this treatment?


Bilu

[bilu]

TFF
===

a b b c d e f f g h
a b c d d e f g h h

flags= 01230123

BFF
===

a b c d d e f g h h
a b b c d e f f g h

flags= 23012301
It would deal OK with 00001230123.

But what about 22223012301 ?a b c d d e f g h h -> a b ? ? d e f ? ? h
a b b c d e f f g h a b b c d e f f g h? will match previous here.

It seems it wouldn't make a difference from matching Ct -> Nb after all, the only difference would be between:

A B B C D and A B C D D

Both would create a duplicate to be removed.
Sorry to waste your time :rolleyes:

EDIT: Do you see any advantage in matching Ct -> Nb after matching
Cb -> Pt and choose the best match? Would it improve over the blended fields syndrome?

Bilu

[Guest]

Originally posted by bilu
Because I compared

c
|\
b c

It was a Ct -> Nb match. And [Ct Cb] [Nt Nb] they are 3 fields apart and match. No, they are 1 field apart. Because for BFF, you have [Cb Ct] [Nb Nt].

And every time you match Cb -> Pt you're matching 3 fields apart too: [Pt Pb] [Ct Cb]. No, it's one field. Perhaps this can help you. Number the fields in temporal order by 1, 2, 3, 4, etc. Then for TFF:

1 3 5 7
2 4 6 8

For the (e.g.) second frame, you only want to compare the adjacent fields, 2-3, 3-4, 4-5. That is Pb-Ct, Cb-Ct, Cb-Nt.

For BFF:

2 4 6 8
1 3 5 7

For the (e.g.) second frame, you only want to compare the adjacent fields, 2-3, 3-4, 4-5. That is Pt-Cb, Cb-Ct, Ct-Nb. Contrary to your assertion above, the Cb and Pt fields are one field apart.

I don't know if my logic makes sense, but what I'm saying is try on a TFF stream to match as it was BFF before matching previous.It makes no sense (for validly formed streams) to try to match fields more than one field apart.

[Guest]

Originally posted by bilu
Do you see any advantage in matching Ct -> Nb after matching
Cb -> Pt and choose the best match? Would it improve over the blended fields syndrome? I won't speculate on the analysis you show because I disagree with the premise. But I'll answer this. If I am considering a Cb-Pt match it must be a BFF stream: ...[Pb Pt] [Cb Ct] [Nb Nt]... Actually, Telecide() starts by comparing Ct-Cb and Ct-Nb. It can revert to the Cb-Pt match if either a) triple=true, or b) the two-way match is combed.

Have you read my journal? I clearly explain there why it is dangerous to do anything beyond two-way matching (except when the result is combed). The triple option is there to help in the blended fields case, but it can cause jerkiness for the reason I describe in the journal.

[bilu]

I have read your journal. I also understand why to make a third match if the 2-way match result is still combed.

Field match could be adjacent using the correct field order or 3 fields away if using the wrong order.

That's why I was proposing to decide between Cb -> Pt and Ct -> Nb and select the better match. When needed, of course.

Trying the wrong field order next match before trying to match previous, and then choose the best result. If you have to deal with blended fields, it may help having choices.

I'm not bashing or criticizing, just trying to help and/or learn. :scared:


Bilu

[Guest]

@bilu

I've decided for reasons of speed and match reliability to require the field order to be specified. It cannot validly change during a clip. Given that decision, there is no need to test for matches between fields that are more than one field apart.

The fallback backward match test (and the triple option) already allows a blended field to be avoided for matching.

The only way to have all three things you want (1. order independence, 2. correct edit cuts, and 3. blended field avoidance) is to have a full 5-way matching:

Pt-Cb
Pb-Ct
Ct-Cb
Ct-Nb
Cb-Nt

But this is dangerous because it can emit spurious duplicates and drop frames.

Decomb 4 did this:

Pt-Cb
Cb-Ct
Cb-Nt

It satisfies goals 1 and 2 but not 3.

Uncomb does this:

Pt-Cb
Ct-Cb
Pb-Ct

It satisfies goals 1 and 3 but not 2.

I've decided it's better to have safe matching by sacrificing goal 1, while satisfying goals 2 and 3. The fact that it is faster is a welcome benefit. As I said, once you accept specifying the field order, there is no need for more than the two-way match with fallback.

[bilu]

Understood, sir :)

Unfortunately forum was down for a long time and I couldn't ask you to explain me something that may have been the cause of my confusion:

Is a 2301 flag sequence like this:

At Bt Ct Dt Dt
Ab Bb Bb Cb Db

or this?

At Bt Bb Cb Dt
Ab Bb Ct Dt Db


Best regards,
Bilu

[Guest]

Originally posted by bilu
Is a 2301 flag sequence like this:

At Bt Ct Dt Dt
Ab Bb Bb Cb Db

or this?

At Bt Bb Cb Dt
Ab Bb Ct Dt Db
Neither one! Your t's and b's are redundant; it's top if it's on the top. :) Instead, I will add numbers giving the temporal field sequence. But you can't just pick out the 2301 and say what is it, without knowing what comes before it! Suppose the D2V *starts* with 2301. Then we have:

A1 B3 B5 C7 D9
A2 B4 C6 D8 D10Note that it's TFF because the first frame byte in the D2V is a 2 or a 3 (the 'Graft Conjecture').

But look what happens if the D2V starts as 012301... Now your isolated 2301 sequence is rendered differently.

A2 B4 C6 D8 D10 E12 F14
A1 B3 B5 C7 D9 E11 F13 F15Note also that the field order is now BFF.

[bilu]

Hmmm... I made confusion between top and bottom fields :)

From the conjecture thread:

0 (bottom first without repeated field)
1 (bottom first with repeated field)
2 (top first without repeated field)
3 (top first with repeated field)


Now if I understood well:

3= At At 1= At
Ab Ab Ab


And this means B3=B5 on the BFF and TFF streams you presented :)


Bilu

[Guest]

Originally posted by bilu
Now if I understood well:

3= At At 1= At
Ab Ab Ab
Not quite. It could also be this way:


3= At At 1= At
Ab Ab Ab
As I said, it depends on what has come before.

And this means B3=B5 on the BFF and TFF streams you presented. Yes. That is why they are both B. :)

[bilu]

Originally posted by neuron2
Not quite. It could also be this way:

3= At At 1= At
Ab Ab Ab
As I said, it depends on what has come before.
1313
====

At Bt Bt Ct Dt Dt
Ab Ab Bb Cb Cb Db

At B is your flag 3 example.

3131
====

At At Bt Ct Ct Dt
Ab Bb Bb Cb Db Db

At B is your flag 1 example. Again at B :)

EDIT: Specifying order here shows what it's worth:

On a BFF stream starting with 1313, it would allways do 3-way match if we had chosen order=1. Correct match would allways be previous and chance is greater to get a wrong match.

But on the TFF stream starting with 3131 it would allways match current/next. :)

EDIT 2: Another example01231313 (BFF)
========

At Bt Ct Dt Dt Et Ft Ft Gt Ht Ht
Ab Bb Bb Cb Db Eb Eb Fb Gb Gb Hb

Using order=0 becomes

Ab Bb Bb Cb Db Eb Eb Fb Gb Gb Hb
At Bt Ct Dt Dt Et Ft Ft Gt Ht Ht

and allways match current/next


23013131 (TFF)
========

At Bt Bt Ct Dt Et Et Ft Gt Gt Ht
Ab Bb Cb Db Db Eb Fb Fb Gb Hb Hb

allways match current/next with order=1It took me all this to understand the value of the field dependency.
Let's hope this info is valuable for others as well.

Thanks for your patience, neuron2 ;)

And sorry for filling your beta thread with this :D

Bilu

[Guest]

Originally posted by JohnMK
What are your current Star Trek TNG recommendations for this latest beta? I came up with:

Telecide(order=1,guide=1,post=4,vthresh=18,hints=true)
Decimate(mode=3)
This works good for me for the challenge clip:

telecide(order=1,guide=1,post=4,vthresh=45,hints=true)
decimate(mode=3,threshold=0.6)

Two points: 1. You need to get the threshold right for mode=3. 2. It's important not to have too low a vthresh with post=4, because it will force backward matches that declare out-of-pattern and thus force video decimation.

I'm planning a new attack on hybrid clips. But first we have to nail this crashing problem with guide=1 for betas 11 and beyond.

[gizmotech]

Hmm...

I've got an interesting situation with the new decomb I was wondering if it could be explained.

I am having the following numbers reported to be on the scene in question:
(in YV12)
matches: 86777 , 88607 , 93231
vmetrics: 35 26 23 [chosen 35] (p)

(in YUY2)
matches: 92700 93857 97355
vmetrics: 37 38 37 [chosen 37] (p)

Now in theory with a guide=0 and triple=true which value will it use to determine patern matching? the lower "matches" value or the lower "vmetrics" value?

The reason I ask is in this frame (which I have sent you neuron2) the current and next matches are the correct matches, (either one will accomplish the task) however decomb seems to be insisting to match previous.

I've noticed this effect several other areas of the source as well. Now I've adjusted my nt value down and fixed the issue for the moment, however I do wonder which value it would be judging the matches off of, as the vmetrics usually seem more reliable then the matches values.

On another note, When overriding a series of pans (scenes with 100+ frames being pattern matched manually) I seem to get completely random slow downs in processing. I wonder if the recent evaluation of the patern override might be the cause of the problem, as it was never there before that was added (me and my stupid ideas eh?)

Gizmotech

[Guest]

@gizmo

In triple mode, the lowest match value will be used. This is one reason why I do not recommend triple mode.

Using vmetrics is better for some streams and using match values is better for others. In my experience, match values are better overall.

I've noticed the slowdown (without any overrides). Does it occur only when show=true? That is the case for me.

[gizmotech]

@neuron2

Of course now that you've asked me this I can't replicate the problem. As I mentioned it's quite random, but it's a VERY noticeable slowdown. Now that you mention it though, I have yet to notice it when show=true is disabled, however as I mentioned... very random.

I know I'll get flogged for asking but is there a way to "switch" which value set we can use for matching? As you've just mentioned vmetric works better in some cases, where as matches works better in others, it might be of some benefit to have them be switchable in the telecide arguemnts. If it's too much work, don't worry about it...

At any rate, I'm off to figure out the differences between the two sets of numbers and see if there is a way to average them out to create a better matching tool *Thinking idlely to him self....(if one is better then the other, and the difference is less then 10% then use the other one for evul :)

Gizmotech

UPDATE: I have managed to duplicate the error (through shear luck) and it would appear to be there regardless of the show=true variable being set or not.
Again: It appears to be at it's worst when there is a patern match and then a vtresh override as well, forinstance:
24356,25346 cccnn
24356,25346 v 30

This type of action will cause the slow down to occur.

[Guest]

Please get beta 15 from the link in the first post in the thread.

It was crashing whenever the source clip had > 100000 frames and either pattern guidance or pattern overrides were used. Guess how big my frame metrics cache is. :)

I also changed the default vmetric to 35 as it seems a generally better value for most clips.

Thanks to bouis and manono for making available clips that let me duplicate the problem.

[toraneko]

thanks for the new release ! going to test it this evening !

to make FieldDeinterlace() easier to switch, could it be possible to display in FieldDeinterlace(show=true) the image "threshold" value, just as it is done with "vthresh" in Telecide(show=true) ?

thanks in advance

[bilu]

Originally posted by neuron2
This works good for me for the challenge clip:

telecide(order=1,guide=1,post=4,vthresh=45,hints=true)
decimate(mode=3,threshold=0.6)

Two points: 1. You need to get the threshold right for mode=3. 2. It's important not to have too low a vthresh with post=4, because it will force backward matches that declare out-of-pattern and thus force video decimation.

I'm planning a new attack on hybrid clips. But first we have to nail this crashing problem with guide=1 for betas 11 and beyond.


I'm looking forward to see this new approach in your Journal :)

Just being curious, do you know other kinds of hybrid than film/video?

Example: PAL->NTSC conversion / video


Bilu

[manono]

Hi bilu-

Just being curious, do you know other kinds of hybrid than film/video?

I saw a short silent film recently that had 3 different framerates within the same film (18fps, 20fps, and 22fps). Does that count? I thought that was pretty strange. I just made the whole thing 22fps.

[Cyberia]

@neuron2: It just occurred to me that the reason I like to use Blend over Interpolate is because the output of Interpolate looks 'jagged'

It just doesn't fit in the image quite right. (I'm generalizing, I admit, sometime it works great)

Would you consider adding some kind of anti-aliasing option to Interpolate? Hopefully that would produce a best of both worlds situation. (More detail but softer edges)

Did I make ANY sense?

[Richard Berg]

I believe there's a bug in the documentation downloaded from the link in the first post.

Mode=1 will leave the clip at 30fps while mode=2 will leave it at 24fps. It is your (difficult) decision about whether you want the final stream at 24fps or 30fps.

...should read "mode=3"

As a frequent processor of hybrid material, I can't wait to see what mode 4 entails!

[Guest]

Originally posted by Cyberia
Would you consider adding some kind of anti-aliasing option to Interpolate? Hopefully that would produce a best of both worlds situation. (More detail but softer edges)

Did I make ANY sense? It makes sense to ask for a solution to jaggies that result from simple interpolation. But anti-aliasing is not the way to do it. I have mentioned that I am working on edge-directed interpolation. I think that is the way to go.

@Richard

Thank you for the correction.

[bilu]

Guys, neuron2 is leaving this forum, but not development.

Please read this thread for further details:

http://forum.doom9.org/showthread.php?s=&threadid=57337


Bilu

[asifanwar]

@ neuron2:

I just wanted to give you a HUGE thanks for the new Decomb. I have a telecined source (a certain space trilogy on LD...) and I have been trying to IVTC it. I had used Doubleweave with Pulldown and got mixed results and then I tried Decomb5. At first I got progressive frames but they were defintitely slightly jumpy.
So I sat down and read the Reference Giude very carefully and kept trying different things (3 hours worth of time) and now it is PERFECTLY smooth and purely progressive!

I realise a lot of people probably complain about Decomb not being able to handle their video, simply cos they cant be bothered to read your Guide, but I would tell them that it IS worth it. I learned a lot of new stuff AND I got the result I wanted - simply by taking the time.

Great filter! Definitely an improvement over Decomb4.

Kind regards.

[EDIT] Oops I just read neuron2 is leaving the forum? Uh oh.

[Shayne]

You could ask your decomb question here i am sure Donald would be happy to answer you

http://neuron2.net/ipw-web/bulletin/bb/index.php

[kastro68]

Could someone provide a mirror for decomb 5 beta 15?


I can't seem to download from the link provided on 1st page.

thanks in advance.

Warpenterprises page only has decomb 406b7

[asifanwar]

Latest version is beta 16.

Here (http://neuron2.net/decomb/decomb500b16.zip )