New Decomb beta release

[Guest]

I am making available here a beta of the next version of Decomb. This revision adds a new mode to Decimate that greatly improves the accuracy of decimation when there are repeated duplicates in the clip.

Previously, Decomb had great trouble handling such clips. These clips can arise whenever motion is rendered such that changes do not occur in every frame. For example, if a 12fps animation is doubled up to 24fps and then 3:2 pulldown is applied, the resulting clip has sequences of 2 and 3 duplicates. Decomb should always decimate from the sequences of 3 duplicates to avoid introducing jerkiness. Previously, Decomb simply removed the frame most similar to its predecessor. The new mode of decimation allows Decomb to remove the duplicate from the longest string of duplicates.

This new mode works well with any source of repeated duplicates, including low frame rate animations (8fps, 12fps, etc.) and clips where the motion is slow enough that not every frame changes. It works best in conjunction with pattern guidance (guide=1).

Also, the default for simple deinterlacing was changed to interpolation rather than blending, and the default threshold for pattern guidance was changed to 50.

manono kindly did initial testing of this new version, but any bugs that might remain in the beta are my fault. :)

Please get the beta here:

http://shelob.mordor.net/dgraft/decomb39beta1.zip

I will appreciate feedback on this new version.

[Guest]

I decided I prefer the default for deinterlacing to be blend after all. So, please get this one:

http://shelob.mordor.net/dgraft/decomb39beta2.zip

[poptones]

I don't have any animations to test, but I'm curious how you are doing pattern guidance now? In the old code I used to make the "de-NTSC" filter for the PAL vids I was recording off satellite, I used a fairly simple process (OK, not so simple) of looking at the debug output until I was getting good sequences of zeros (ie 11101 means remove the fourth frame from this set of five) and then my AVS script generator would simply parse the debug output looking for zeros in groups of five. The debug stream I was using looked like this:

11101
11010
11000
11011
11010
00000
11010
11111
11111
11010
10010
10101
10100
10111
10100
11011
11011
10111
10111

You get the idea. Anyway, I just kept a memory of the pattern and used a simple filter to determine when it had changed. If the change is less than one position, it changes instantly. If the cadence is more than one, it has to accumulate two groups before it switches.

In the above, the first group would lose frame 4, the second frame 3, the third frame 3, the fourth frame 3, and so on down to group 12 where the cadence definitely changed. Since the next several groups all detect frame 2, it switches to that cadence until it gets more than two clusters in a row with only one detected match, switches again, and so on. Thusly:

11101 cadence=11101
11010 cadence=11011
11000 cadence=11011
11011 cadence=11011
11010 cadence=11011
00000 cadence=11011
11010 cadence=11011
11111 cadence=11011
11111 cadence=11011
11010 cadence=11011
10010 cadence=11011
10101 cadence=10111
10100 cadence=10111
10111 cadence=10111
10100 cadence=10111
11011 cadence=11011
11011 cadence=11011
10111 cadence=10111
10111 cadence=10111

This worked very, very well at detecting 25fps sources in 30fps NTSC video. It loses cadence at blends, of course, but at those transitions it's not so noticeable. And obviously it doesn't work at all with sources that detect to odd patterns - like 24fps sources that leave you with this

11011
11101
11110
11110
11101
11011
10111
01111
01111
10111

Anyway, I'm just curious how you do this now. Just thought I'd let you know what I came up with, and have had very good success (Berg has two of the videos I made like this - Natacha Atlas, and Sultana - and will likely vouch for my assertion). The method that was being used in that older code was much more simple, and while I'm sure you've evolved it your mention of it here reminded me I never shared that bit of my own with anyone (including you).

I wasn't using C to do all this - I just tweaked the detection stage in your filter (I completely changed the threshold algorithm) and then used a jscript to parse the debug.txt file after I "played" the video. The jscript output an avs with all the necessary delete() commands. Yeah, I know it would have made more sense to do it in the filter, but I was just learning c back then (I'm still not great at it) and I am a bit of a wiz with jscript, so I used what I know. Anyway, I probably still have the scripts around here if you're interested. I don't use this contraption anymore because I no longer have PAL satellite sources to decode... long story short: direcTV "quality" sucks ass.

[Guest]

@poptones

Telecide and Decimate have always been decoupled, and still are even in the new version (despite my recent posting about experimenting with coupling them). Telecide has the field matching functionality that can be guided by the prevailing pattern. But it still outputs all frames, including duplicates, and Decimate independently decides which one to trash. This is one approach to IVTC: not the only one and not necessarily the best for all situations. But it does have the advantage of being versatile. Telecide is useful for PAL and other situations where decimation is not wanted. And Telecide is extremely robust to field order changes.

The Decomb 3.9 beta release does not change field matching/pattern guidance at all. Exactly the same patterns will be emitted by Telecide. What has changed is how Decimate chooses the right 0 frame (duplicate) to delete. If we had:

10010 00100 10001

...we want Decimate to remove the first 0 in the second group. But the way mode=0 worked was that it would remove the frame that was most similar to its predecessor. Due to noise, etc., that may or may not be the right frame. The new mode=2 will see that the second group is "attached" to two strings of dups, and will remove the frame from the longest string.

I find that Telecide(guide=1) with Decimate(mode=2) now works very well with all my NTSC 3:2 material. It breezes through stuff like the region 1 Princess Mononoke. Nevertheless, I think that an integrated algorithm could be faster, and I am planning some experiments in that regard.

Hopefully I have answered your question.

[Guest]

@poptones

It occurred to me you may have been asking how pattern guidance works in Telecide. It simply stores the last five match decisions and if that group matches a known valid pattern, then the next match is predicted according to that pattern. Then the next match is calculated and if it is within threshold of the prediction, then the predicted match is used. If it is too far out the new match is used. If the previous five decisions don't match a known pattern, then the calculated match is used.

This is certainly not as effective as an integrated algorithm with lookahead could be, and it is probably not as fast either. I am experimenting with an idea that manono gave me that might result in a good integrated solution. The existing integrated solutions seem to be not very robust to field order changes.

[Guest]

@poptones

>direcTV "quality" sucks ass

As a digital video engineer specializing in DirecTV systems, I would have to say I resemble that remark. :)

Can you elaborate a little for me, please?

[poptones]

You sure you want me to?

I had dtv when I lived in LA, and it was alright. It wasn't exactly what I considered a "value" tho as I was spending 12-14 hours a day elsewhere, so I let it go.

When I moved back here I spent an assload of money on a Philips 6000 tivo unit and a spare receiver for the other room. Unroftunately, with a 29" monitor (albeit one of those Princeton 800x600 units) it became really, really obvious how bad the picture really was. And perhaps ironically, the stations I cared most about (Dark Angel and the tuesday lineup on Fox, M2, WorldVision and the "artsy" movie channels) were the absolute worse of the lot.

Don't take this personally. I don't know who runs the compressors for Fox, but it seems they are unable to set a decent black level. I have video here I'll be glad to share to illustrate this. And ALL the channels are so compressed and at so low a resolution, it's just not worth it. One of my fave channels, WorldVision, degraded over time to the point it was, by my standards, quite literally not worth watching. I've seen internet video that looked better.

I don't have the caps anymore, but I once posted caps from a Bjork video taken both from M2 and from the DVD I bought long ago. If you can track down the thread at Ars (I think it was an old lounge thread) 'tho the pictures are gone, the remarks of the people who saw the difference with their own eyes should prove my point well.

I'm sure the HD channels (which were not available when I had the service) are quite nice. The quality of the regular channels, combined with the fact DTV was not, at that time, even discussing carrying the one network I cared most about (UPN) added up to major suckage of service, IMO.

The 45GB of storage now resides in my HTPC. In RAID0 the 30GB shared partition is ALMOST as fast as my old 20GB deathstar. It's my $600 reminder to spend my entertainment dollars more wisely in the future.

[fisix]

i thought tivo recorded lower quality than original broadcast, even at best quality settings.. right?

-fisix

[OUTPinged_]

DG, you da man \o/

[hakko504]

Any chance of you introducing multiple decimation directly into Decimate? I'm thinking in terms of going directly from 30 to 12fps and 60 to 25 fps (useful for NTSC2PAL).

In any way, I'm eager to test the new version. It's been a pleasure to work with the old version, even though I've mostly used it for bad NTSC2PAL transfers and to correct field order shifts. (I'm in PAL country and usually doesn't need IVTC.)

[poptones]

i thought tivo recorded lower quality than original broadcast, even at best quality settings.. right?

The Philips 6000 is a direcTivo unit. It has no encoding capabilities at all; it simply records the bitstream EXACTLY as delivered from DirecTV.

And BTW: the standalone tivo in highest quality mode uses MORE disk space per hour than a DirecTivo recording direcTV. That means a SA tivo actually has a higher potential quality than a directivo unit recording an "all digital" signal. People in the AVS forum have reported good experiences using a SA tivo in HQ mode to record anamorphic widescreen movies from HD broadcasts (ie it's basically "true DVD quality). You can't even do that with a DirecTivo unit - they have no HD capabilities at all (or at least they didn't) and they cannot record from an external source because they have no MPEG encoding capabilities.

To be fair: the picture was no worse than the digital cable I had in Tucson. But then, that ain't saying much. I remember watching "Headless Horseman" and realizing the quality of the divx rip I downloaded from the internet was better than I was watching on cable.

It's the old paradigm: GIGO.

[Guest]

@hakko504

>Any chance of you introducing multiple decimation directly into
>Decimate? I'm thinking in terms of going directly from 30 to 12fps
>and 60 to 25 fps (useful for NTSC2PAL).

Sure. Is it just to avoid multiple Decimate calls? Can you suggest a proposed syntax?

[hakko504]

Originally posted by neuron2
Is it just to avoid multiple Decimate calls?
Yes and no.
Yes, because otherwise you need Decimate(cycle=2).Decimate(cycle=5) to go from 30 to 12 fps.
No, because a single call would make it easier to select the total sequence that would give the most fluent video.

And as for syntax: You could add a new parameter DecimationRate (default value =1) that decides how many frames in a cycle that should be removed. That would make the conversion from 30 to 12 fps: Decimate(cycle=5,decimationrate=3)
and from 60fps to 25 fps: Decimate(cycle=12,DecimationRate=7)

[Guest]

@hakko504

Thank you for the explanation. Yes, that shouldn't be too difficult. I'll have a go at it.

[poptones]

What on earth do you cut to 12fps?

[MNale]

My results with this new beta have extremely jumpy/jerky video.

My source...

SDI Video Quicktime 4:2:2 8 bit Component Y'CbCr exported via Premiere and VideoServer to CCESP 2.5. Video is captured from Sony Digibeta DVW-A500 with Targa 2000 DDR card to Premiere 6.0.

My .avs script for CCE...

LoadPlugin("f:\captures\misc\decomb.dll")
AVIFileSource("g:\IPCServer.avi").BicubicResize(720,480)
ConvertToYUY2()
Telecide(guide=1)
Decimate(cycle=5,mode=2,threshold=50)
ResampleAudio(44100)

[Guest]

@MNale

Couple questions:

1. Are you sure your source material is telecined 3:2 pulldown material?

2. Is it still "jumpy/jerky" with Decimate(mode=0)? If yes, is it still "jumpy/jerky" with Decomb 3.8?

3. Can you make available for download a short clip that shows the problem?

BTW, there is no point in setting threshold with mode=2 as it is relevant only for mode=1. If you in fact set threshold2 to 50, then try it again at its default of 100.

[MNale]

The material is telecined using Philips DataCine Spirit.

Using Decimate(mode=0) with 3.9b is less jumpy in parts.
My best results have occured using Decomb 3.8 with...
Decimate(cycle=5,mode=1,threshold=50), but there is still frame dropping causing jerkiness.

I will gladly post a link to my ftp site where you to see the .m2v or if you want I can let you see the source Quicktime (although it would be quite large since it is component SDI.) Email me as to what you want.

[Guest]

@MNale

You've offered to get me a source clip and I've emailed you about that.

>My best results have occured using Decomb 3.8 with...
>Decimate(cycle=5,mode=1,threshold=50), but there is still frame
>dropping causing jerkiness.

With mode=1 Decimate NEVER drops frames.

I am thinking your problems lie in the input clip and not the processing done by Decomb. However, when I get a source clip I'll be able to verify that.

[Dreassica]

I get this error messgae when loading AVS with new decomb in Vdub:
"Avisynth read error:Telecide: Malloc failure. What does that mean??

[Guest]

@Dreassica

Does it happen with Decomb 3.8 and the same AVS and clip?
It means that Telecide could not allocate memory it needed.

[Guest]

@MNale

The clip you provided was captured off air and contains a mixture of telecined material and straight video material. As described in the help file, this leads to a problem. If you decimate to fix the telecined content, you will also decimate the straight video, making it jerky. You can get acceptable results, however, by decimating like this:

Decimate(mode=1,threshold=50)

There is nothing wrong with Decomb 3.9.

[Koepi]

heh,

respect Donald,

I often wish I could handle such cases with so much patience as you.

I just always feel so ignored if those things happen.

Just wanted to let you know that I have highest respect for your work!

Regards,
Koepi

EDIT: sorry for the off-topic

[Guest]

@koepi

Thanks for the generous words. Not sure "ignored" fits in that sentence. Maybe you're not a native English speaker?! :)

In any case, I always try to be patient because you never know when someone might find some obscure issue that I'd like to know about. I always say I'll listen to anybody, even the street sweeper, because there is always a chance that they have happened upon some hidden piece of information, or have had a unique insight.

Example: the gent that pointed out he got a malloc failure in Telecide. I *could* just dismiss it because Telecide didn't change in version 3.9. But I'm anxious to follow up on it because it is not something that should happen. Maybe his system was low on memory. Maybe Telecide could detect that and warn about it! ;)

[Guest]

@OUTPinged_

> DG, you da man \o/

Judging from an exhaustive review of all your postings, and from a thorough and serious analysis of all personality and character traits suggested therein, not to mention a diligent attempt to interpolate between the lines of your impressive body of published material, I have been able to tentatively conclude that you, too, are "da man".

I stand humbly before your shining brilliance, ever more filled with a sense of the sheer greatness of this world, one that is so elevated by your existence.

[Koepi]

Erm, yupp, sorry for my lack of english skills :-/ (so, and now translate all this thread to german, report to me, and after that, I'll allow you to go to bed! ;) )

No really, I think it's hard sometimes to stand those flames, and you really manage that in a -for my terms- a perfect way and you deserve special respect for that.
(Well, plenty of other Mods here, too. Me seems to be the only one who can't stand ignorance of written down, researchable information).

Btw, I finally got my hands on the Platform SDK (and installed it), so now I can play around with avisynth stuff as well. I hope you all have working combat suites as my code sure will blow everything up ;)

I'm looking forward to learn from your MMX/XMM/3dNow/and everything else with no Intel Names/ ... on it! :)

I think I'm in the right mood to start a contest:

writing the slowest filter for avisynth ever, which has _real_ results, no NOPs.

I think that's another thing where we can learn from, because - hitting max speed is as complicated as hitting lowest speed.

Regards,
Koepi

[Guest]

Not the whole thread...

Erm, yupp, traurig für meinen Mangel an englischem Fähigkeiten:- / (so und übersetzen Sie jetzt dieses ganzes Gewinde zum Deutschen, Report zu mir, und nach dem, erlaube ich Ihnen schlafen zu gehen! )

Kein wirklich, denke ich ihn bin hart manchmal, jene Flammen zu stehen, und Sie handhaben wirklich das in a - für meine Bezeichnungen verdienen eine vollkommene Weise und Sie speziellen Respekt für das.
(gut, viel von anderem Mods hier, auch. Ich scheine, das einzige wer nicht Unwissenheit von notiert stehen kann, researchable Informationen zu sein).

BTW, erhielt ich schließlich meine Hände auf der Plattform SDK (und brachte sie) an, so jetzt ich kann mit avisynth Material herum außerdem spielen. Ich hoffe Sie, die, alle Arbeitskampf Suites da mein sicherer schlag des Codes Willensalles oben haben

Ich schaue vorwärts, um von Ihrem MMX/XMM/3dNow/and alles zu erlernen sonst ohne INTEL Namen/... auf ihm!

Ich denke mich bin in der rechten Stimmung, zum eines Wettbewerbs zu beginnen:

den langsamsten Filter für avisynth überhaupt schreiben, das _ reale _ Resultate hat, kein NOPs.

Wir von ich denke, der ist eine andere Sache, der erlernen können, weil - Höchstgeschwindigkeit zu schlagen ist so schwierig wie, niedrigste Geschwindigkeit schlagend.

Respekt,
Koepi

[Koepi]

Ouch :-(

[stickboy]

Donald,

I emailed you a couple months ago about maybe adding another mode to Decimate to blend duplicated frames together instead of discarding them completely. (One of the thoughts was it might make hybrid clips that are mostly film smoother.) Have you given any more thought to this?

[ChAoS Overlord]

I have this trailer on dvd, it seems to be composed of:

frame 1 = frame 2
frame 3 = frame 4
frame 5 = frame 6
...

but they are interlaced, I want to add the odd and the even lines together, so that the framerate gets cut in half, and de lines get shared. I'd like to do it with decomb, but perhaps there are even better solutions?

[Guest]

@stickboy

How does your suggestion differ from the existing Decimate(mode=1)?

From the help file:

"If mode=1, instead of discarding the most similar frame, Decimate()will either replace it with a frame interpolated between the current frame (usually a duplicate of the preceding frame) and the following frame, or it will pass the frame through as is. The choice between these two depends on the threshold parameter setting and on how different the frame is from its preceding frame (see below). Decimate(mode=1) is useful for hybrid clips where you do not want to reduce the frame rate but want to ameliorate the effect of duplicate frames that are emitted by Telecide() (frames that are normally removed with mode=0)."

@ChAoS Overlord

Looks like a 12fps segment doubled up to PAL and then captured with the field phase off by one. Is it PAL or NTSC frame rate? Probably you just need a simple one field phase shift. Try using Telecide by itself without Decimate to fix that.

[ChAoS Overlord]

It is an NTSC Frame rate.

[Guest]

What do you mean by "add the odd and even lines together"?

Can you post a small source clip and say exactly what you want the processing to do to it?

[stickboy]

Originally posted by neuron2
How does your suggestion differ from the existing Decimate(mode=1)?It'd be the opposite of mode 1. Whereas mode 1 scales film frame rate to match video, for material that's mostly film, I think I'd rather scale the video frame rate to match film.

In other words, given a frame sequence F1,F2,F3,F4,F5, if F3 and F4 are considered most similar, then the output frame sequence would be F1,F2,(F3+F4)/2,F5.

(I think I explained this better in my email, but I munged my mailbox.)

[ChAoS Overlord]

Originally posted by neuron2
What do you mean by "add the odd and even lines together"?

Can you post a small source clip and say exactly what you want the processing to do to it?

frame 1 = even frames of picture 1
frame 2 = odd frames of picture 1
frame 3 = even frames of picture 2
frame 4 = odd frames of picture 2

Like that, now I just want to combine those two frames

[stickboy]

Chaos Overlord: Can't you use the Weave function?

[ChAoS Overlord]

Originally posted by stickboy
Chaos Overlord: Can't you use the Weave function?

Does this do the thing I mention? Sorry but I have very little expertise when it comes to ivtc and framedropping stuff. :(

[Guest]

@ChAoS Overlord

>Does this do the thing I mention? Sorry but I have very little
>expertise when it comes to ivtc and framedropping stuff.

This is all the more reason to submit a small source clip as I requested. Your description borders on incoherent and you don't understand a simple suggestion! If you really want help, post a small source clip and let us figure it out.

[ChAoS Overlord]

OK, I want to post a clip, but all I have is the vob file, can I cut a sample out of a vob? If so, how?

[Guest]

Rather than splitting a VOB, which can be problematic, you can open the VOB with DVD2AVI, mark the in and out points, and then save as AVI, using the HUFYUV codec. Make sure Force Film is off.

There is an app called vobsplitter that you could try.

If you have an ftp site, I can download 100meg without difficulty.

[Guest]

@ChAoS Overlord

Thank you for making the VOB available.

You have the classic problem of blended fields, which is described in the help file in the section "Notes on Field Matching". It it were my clip, I would just bite the bullet and deinterlace, because there are too many blended frames for Telecide to do any good for you:

FieldDeinterlace(full=true)

You could also try:

FieldDeinterlace(full=true,blend=false)

...and see which you like best. Or try TomsMoComp.

There just is no way to reassemble things correctly when you have blended fields.

[ChAoS Overlord]

I think it is possible cause every frame is an exact duplicate of the previous one except for the fact that it is te other scanlines. (sorry for my poor explanation).

anyway: this (http://membres.lycos.fr/chaosoverlord/VARIA/Prob.01.avi) is the download link of a very short sample (open in vdub).

Small question: How come I can open this HUFFYUV file in MWP & VDUB but not in bsplayer?

[Si]

Looking at the clip, I'd add the following comments

1. The quality is very very poor.

2. It seems to me the material has had level changes applied on a field by field basis which causes the effect of scanlines appearing when viewing on a computer monitor.

3. No chance of being recovered :-(

regards
Simon

[Guest]

@ChAoS Overlord

You could have made clear that this new clip you've linked is not the same as the VOB you sent me, and it has a completely different problem. That leaves your thread disjointed and your new opening statement "I think it is possible..." dangling. To which clip were you referring when you said that?

I agree with Simon about the cause of the problem with this new clip: level changes. It might be possible to SeparateFields(), then apply my AntiFlicker filter, and then recombine fields, but it would be much better to solve the problem of why your fields are oscillating in luminance.

I respectfully suggest that you need to learn to use the simple analysis tools available to you, such as SeparateFields().

[Guest]

@stickboy

In other words, given a frame sequence F1,F2,F3,F4,F5, if F3 and F4 are considered most similar, then the output frame sequence would be F1,F2,(F3+F4)/2,F5.Thank you for the explanation. That is very interesting, because usually the 30fps parts are small so it might be better to keep the final frame rate at the decimated rate. I will try an experiment to see how it looks.

[ChAoS Overlord]

Originally posted by neuron2
You could have made clear that this new clip you've linked is not the same as the VOB you sent me

Excuse me, I never sent you a vob...

I will upload a new sample of the same file now...

I am overwriting the file now, same filename other sample, please look into it.

[Guest]

@ChAoS Overlord

Sorry. I've confused you with another person who sent me a VOB. It's hard when people use handles without their names here and then send email with their real names and without the handles! I can't reliably match them up. :)

You see, your post said you'd try to send a VOB. Then I got an email with a VOB saying "I'm the guy from doom9." So I don't even know who it was now! But they did have blended fields.

Oh, I am so confused.

[Guest]

@chaos

Exactly the same problem. The fields alternate light and dark. Something is wrong with your capture card.

[ChAoS Overlord]

Originally posted by neuron2
@chaos

Exactly the same problem. The fields alternate light and dark. Something is wrong with your capture card.

I'm not using a capture card, this is a dvd rip from my dvd rom drive.

[Guest]

@chaos

Then the mastering is bad. You can easily prove this to yourself using Avisynth and SeparateFields().