IVTC4 plugin for Avisynth

[daxab]

Time for me to give something back.

I've written an IVTC plugin for Avisynth. It's tuned for two kinds of source: Anime and noisy TV captures.

It's called IVTC4 because it's the fourth one I know of, after IVTC, GreedyHMA, and Decomb. It's not better than these, just different.

The code is all new.

The goal is correctness, not speed, but interlacing frames still emerge, if your source has edits or interlaced sections (e.g., 30fps animation overlaid on 24fps telecined video). If this is a problem I recommend following IVTC4 with Donald Graft's FieldDeinterlace().

http://www.geocities.com/daxab88/

[Guest]

@daxab

Do you plan to release the source code? If not, are you willing to describe the principles of operation? Thank you.

[daxab]

The source is still pretty rough -- I'd like to make it about twice as fast as it is now. But the basic operation is simple.

It works by assuming each group of 5 double-woven frames contains 2 interlaced frames, 2 unique progressive frames, and one duplicate progressive frame. Of course, this isn't true if there are edits, CG overlays, etc., but it assumes this anyway (most of the time).

It picks the two interlaced frames using a metric over the luminance values (only works on YUY2). The metric is imperfect and gets fooled by things like noise and anime. If it doesn't see a strong interlacing metric it then goes into a pattern mode, where it assumes frames are occuring in a repeating pattern. It also has a mode where it looks for an edit point -- three interlaced frames, instead of two. This rarely occurs and it rarely thinks this is happening. The 'edit' parameter enables this last bit of processing, since it's risky; it might produce a stutter.

Once it has the two progressive frames it serves them as output. It does not try to solve the problem of deinterlacing.

[Blight]

daxab:
I think this system has been proven not to work well.

You can never assume a pattern as TV sources seem to change it every scene change, which makes it very unstable and the reason why the original AVISynth IVTC didn't really work properly.

Frankly, the sources are usally extremely messed up which is why the blindfolded match-fields system works quite well, since it doesn't really expect anything.

[daxab]

I think this system has been proven not to work well.

You can never assume a pattern as TV sources seem to change it every scene change,...@Blight

I'm not sure what "system" exactly you're referring to, but I agree that a purely pattern-based approach will run into problems at edit points. That is why IVTC4 is not primarily pattern based.

If you're saying that IVTC4 doesn't work well, that's another matter.

Have you tried it?

Although I'm planning on improving both speed an accuracy, I'm already pretty happy with its performance on actual NTSC TV captures (with edits, CG overlays, and noise).

[meleth]

I did an encode over the night with it. Didn't have time to check how it turned out other than very very briefly before i had to leave for work. What i did notice though was that there were quite a few interlaced frames and that the avi was really undersized. Dunno why it turned out that udnersized though, might be i forgot to change something in my hurry to get started.

[daxab]

@meleth

If you see a lot of interlaced frames, there are a few possibilities. One is that you don't have 24fps telecined source. Another is that you left out the DoubleWeave() command. Another is that your parity is wrong. Your script should look like either this:
AVISource("...")
AssumeFrameBased()
DoubleWeave().IVTC4("c:\ivtc4.cache")or this:AVISource("...")
AssumeFrameBased().ComplementParity()
DoubleWeave().IVTC4("c:\ivtc4.cache")Try both. One of these will produce 99+% interlaced frames, and one will produce 99+% progressive frames.

That is, as long as your source is actually 24fps telecined.

[AllTimeSToneD]

... i'm a bit Baka =)

I tryed your IVTC4 filter on a Source material where the Video was 24fps and has been telecined to 30fps, then double-woven to 60fps.

The Movie is a Anime so i wanted to start some tests. I tested all the settings u recommend here and on your page. But the material is still Interlaced.

With the "IVTC" plugin it works fine just few scenes r Interlaced.

My question is: Did i anything wrong ?

[daxab]

@AllTimeSToneD

With the default settings, most frames should be progressive, so if most frames are interlaced, then something is wrong.

Can you post your .avs file and arrange for me to get a short clip that shows the problem?

[Guest]

@daxab

Your approach is very interesting and appears to work well on several torture clips that Decomb has problems with. Would you mind sharing your combed frame detection metric with us?

[daxab]

Anyone using IVTC4, please download version 1.01 which has a slightly improved frame metric.

http://www.geocities.com/daxab88

[daxab]

@neuron2

What I use in IVTC4 is the tried-and-true sum of the squares of differences. In this case, the differences between luminance (and sometimes also chrominance) values of adjacent horizontal lines. In pseudo-code:metric = 0
for (y=0; y<...; y+=2) {
for (x=0; x<...; x+=useChroma?1:2) { // YUYV
metric += (frame[x][y] - frame[x][y+1]) ^ 2;
}
}Pros: This metric has the nice property of being easy to compute, requiring no thresholds -- which are hard to set properly -- and being mathematically well-behaved. And it amplifies large luminance deltas, giving them greater weight.

Cons: It gets fooled easily by the usual gremlins: noise, low motion, low luminance. A noisy progressive frame, especially with horizontal lines of noise, can have a higher metric than an adjacent interlaced frame, if the interlaced frame is cleaner and has low motion.

Is this similar to the metric you use?

[High Speed Dubb]

Would that also be sensitive to the amount of vertical detail in the frame?

[daxab]

@High Speed Dubb

Yes, right. So a frame of something like a screen door or horizontal blinds will give an artifically high metric. But an adjacent interlaced frame should have an even higher value.

[High Speed Dubb]

I think I get it &mdash; You&rsquo;re comparing the statistic with the analagous measure from other fields, not against a fixed threshold. If there&rsquo;s a strong indication of which fields are paired, you go with that; otherwise you assume the best recently seen pattern has continued. Is that right?

Sounds like a good method to me. I came up with a slightly different heuristic a couple weeks back:

local_comb =
0 if frame[x][y] is between frame[x][y+1] and frame[x][y-1]
Min(|frame[x][y]-frame[x][y+1]|, |frame[x][y]-frame[x][y-1]|)^2 otherwise

but that&rsquo;s more difficult to compute, and very possibly not any better.

Depending on some assumptions, the optimal way to weight the current data with neighboring frames/fields is called a &ldquo;hidden Markov model.&rdquo; It&rsquo;s a pretty standard pattern detection method &mdash; A Google search will turn up way too much about it.

[daxab]

@High Speed DubbIs that right?Yes, more or less. There are a lot of nitpicky details, but that's the general idea. Of course, once you fall back to the pattern you can easily be wrong (a big problem around a fade-out / edit / fade-in sequence). So I'm still trying to improve the metric to the point where the pattern is used very, very rarely.

I've just been trying something identical to your heuristic. In my code it looks like:

minDiff(frame[x, y], frame[x, y+1], frame[x, y+2]) ^ 2

Where minDiff(a, b, c) is min(|a-b|, |b-c|). I also tried scaling (to compensate for luminance level), and making minDiff(a, b, c) = ( |a-b| < |b-c| ) ? ( |a-b| / min(a,b) ) : ( |b-c| / min(b,c) ). This actually does work better, but starts to get quite expensive, since now you're doing a floating point division (or a scaled integer division).

[kramerdog]

I just tried IVTC4 with a noisy TV cap. It worked great. I've had a lot of trouble in the past with the other three IVTC plugins. IVTC4 handled the scenes where only a very small portion of the frame was interlaced such as a blink of an eye, or small mouth movements.

Great job! Thanks :D

[Guest]

@daxab and High Speed Dubb

Thank you for the information about your combing metrics. I am very interested in this. The metric I currently use is:


comb = (value[y-1][x] - value[y][x]) *
(value[y+1][x] - value[y][x]);
if (comb > 0) sum += comb;

I do this for each candidate match and then pick the one with the lowest result.

I have tried methods similar to the ones you describe and I find that each method tends to work pretty good but have problems with certain frames. One method might do well on test frame A and bad on B, while another may do the reverse. Finding a universally good method is proving difficult. :(

[DivxBr]

And if....

comb = (value[y+1][x] - value[y][x]) ^ 2;

if (ABS(value[y+1][x] - value[y][x]) >
ABS (value[y+2][x] - value[y][x]) ) sum += comb;

.. or something like that, this could (I think) avoid the edges - checking if there´s low frequency in the even/odd lines (same field) when there´s high frequency in the frame (two fields).

:)

[b0b0b0b]

@daxab:

I really like your idea for a cache. I'm glad someone finally implemented this.

[High Speed Dubb]

@daxab

It&rsquo;s very interesting that you see an improvement when you scale the measure depending on the lighting. Noise is additive, not multiplicative. So to avoid noise sensitivity, it would be better not to scale at all. On the other hand, by scaling by luminance, you&rsquo;re reducing sensitivity to lighting changes.

Now that I think about it, all of these measures listed (except your scaled one) are going to be sensitive to changes in lighting, since that changes the amount of vertical detail.

Since there are only a few possible divisors, maybe you could do the division with a lookup table? i.e., keep a 256 length array of reciprocals, then multiply with the appropriate one. You could even use that to improve the scaling a bit, by using a minimum divisor even when the luminance is near 0.

[High Speed Dubb]

@neuron

That&rsquo;s very, very close to the formula used in DScaler. (That might not be a coincidence &mdash; I don&rsquo;t know where DScaler&rsquo;s formula comes from.) DScaler uses the exact same equation. But for each pixel, it compares that value with a threshold, and adds 1 (and not the comb value) to a running total wherever the threshold is exceeded.

The problem with this is that the senstitivity to noise depends very closely on the amount of vertical detail. That&rsquo;s because the slope of the comb statistic near value[x][y] varies quite a bit depending on |value[x][y-1] - value[x][y+1]|.

[Guest]

@High-Speed

The first use of this metric that I have seen was by Gunnar Thalin in his area-based deinterlacer, I adopted it from him and added it to my Smart Deinterlacer as the "field-mode differencing". Previously, I had only frame-mode differencing, simply comparing corresponding pixels in adjacent frames. Early in the history of DScaler, I was approached and provided the source code for Smart Deinterlacer. So the metric appears to originate with Gunnar. He also later added an additional factor based on abs(value[y-1][x] - value[y+1][x]) that reduces the comb metric. This tended to eliminate edges. I have not included that for performance reasons, but now I am wondering if it really is required.

[High Speed Dubb]

Here&rsquo;s a potentially useful trivia fact: If you look at the distribution of vertical absolute differences, it looks a whole lot like a lognormal pdf, and a little like an exponential. That seems to be true regardless of the material. There&rsquo;s probably some way to exploit this.

@neuron

Yeah, that measure was odd enough that I didn&rsquo;t think too many people would have come up with it independently. I&rsquo;ll enter a comment into the code so people will know the origin.

[High Speed Dubb]

He&rsquo;s already credited for the Video Bob deinterlacer. Maybe that was just copied over for the &ldquo;comb factor&rdquo; calculation.

[daxab]

That metric has a lot of nice qualities.

@kramerdog

Great to hear it! I do a lot of testing with noisy TV caps. It's good to hear that it's working for someone else.

@b0b0b0b

I didn't really want to make speed a priority (although I'm surprised at how fast IVTC4 actually is) but I really don't like paying twice (or four times) if I do a VBR encode. The only problem I'm having is that it plain doesn't work if the user doesn't pass in a full path name. I think this could be fixed pretty easily but I haven't gotten around to it.

@High Speed Dubb

The reason I was scaling was precisely to handle scenes like a fade-in, where the overall luminance was changing over 5 or 10 frames. This made the metric seem to increase, since most metrics I've tried increase with overall luminance.If you look at the distribution of vertical absolute differences, it looks a whole lot like a lognormal pdfTo exploit this wouldn't the distribution need to look different in a combed frame? Maybe a significantly different mean, or two maxima? Do you know if that's the case?

@DivxBr

That's a good idea.

[Guest]

@daxab

Are you ready to release your source code to selected people, e.g., me? The reason I ask is that the results I get from applying your stated metric don't match what your filter is outputting. :)

I'd like to understand why. Perhaps you are overruling the match decision based on the prevailing pattern. I'm really curious to know if, when applied to my torture clip, the results are coming purely from blind field matching or whether your pattern guidance is getting involved.

@all

Here's another metric that runs really fast when coded in MMX (no multiplies and you can do abs() really fast, plus of course doing multiple pixels in parallel):


if (abs(value[y-1][x] - value[y+1][x]) < E)
{
comb = abs(value[y-1][x] - value[y][x]);
if (comb > T) p += comb;
}

E and T both equal to 10 works nicely. This is like the simple field differencing that daxab described but qualified to eliminate edges.

[daxab]

@neuron2

I'm still sitting on the source code. It's not much, but it's all I've got :).

...the results I get from applying your stated metric don't match what your filter is outputting....Perhaps you are overruling the match decision based on the prevailing pattern.Yes, I use a pattern when the metric doesn't work due to noise, edits, etc. I'm sure this is what you're seeing.

[High Speed Dubb]

@daxab

The fit between a lognormal and the histogram of absolute vertical differences isn&rsquo;t close enough to let you distinguish between progressive and interlaced fields. That&rsquo;s especially true because there&rsquo;s a decent fit with the lognormal when the putative frame is really interlaced.

The exponential seems to distinguish a bit better -- but it&rsquo;s a lousy fit for either toward the high end of the distribution. (Though some of that is noise which should be ignored.)

But I don&rsquo;t think checking the fit to a distribution will help much. My goal here was just to find a robust estimate of the real vertical difference distribution. With that, approximating the likelihood P(Data | progressive) would become possible.

I&rsquo;ve attached the fit of an exponential to a progressive frame. The pink is the maximum likelihood exponential distribution; the white is the vertical absolute difference distribution. It&rsquo;s on a log scale.

[High Speed Dubb]

I wrote
I&rsquo;ve attached the fit of an exponential to a progressive frame...Or maybe I didn&rsquo;t... Well, I&rsquo;ve posted it on my site &mdash; Here&rsquo;s a link:

http://students.washington.edu/ldubb/computer/ProgressiveSitcom.jpg

And while I&rsquo;m at it, here&rsquo;s a link to an exponential fit to an interlaced &ldquo;frame.&rdquo;

http://students.washington.edu/ldubb/computer/InterlacedSitcom.jpg

[daxab]

@High Speed Dubb

Interesting stuff. I assume you are not doing this real-time, and that it's more to provide info about what real-world frames are structured like.

[High Speed Dubb]

@daxab

No, it&rsquo;s in real time in DScaler. Noise filtering, comb filtering, and deinterlacing were running at the same time. :)
(The histogram filter worked on the current field before the other filters, so they shouldn&rsquo;t mess up the statistics.)

I&rsquo;d be happy to email out the source for the edited histogram filter, if it would be helpful. But it&rsquo;s in DScaler 4 format, so you&rsquo;d need to compile that in order to use the filter. And unless you have a Bt8x8 card, you&rsquo;d need to convert it over to AVISynth (or other) format, instead. (But the filter is really pretty simple &mdash; It might be easier to write this kind of tool yourself than to convert my quickly kludged code.)

I think my next step try will be to see what the histogram of |log(value[y])-log(value[y-1])| looks like. That probably won&rsquo;t work in real time. :(

With luck, some of these curves will give some hints about good comb statistics to use.

[meleth]

@daxab, i was thinking. As i understood it your main purpose with this filter is reducing the fps while keeping a smooth motion.

What would happend if instead of discarding the frame it got blended in to the previous or next frame? Would the ghosts be visible when watching the clip?

[Guest]

@meleth

You can already do this with Decomb by setting the mode=1 option to Decimate(). Generally, the result is quite satisfactory and only an occasional really bad blend gets to your consciousness. Blight has suggested an improvement, only blend the frames that are duplicates; that way real 30fps content will not be affected.

[meleth]

@neuron

Oh, gonna have to check that out when i get home.

I guess blight's suggestion is ok, if it's optional. Cause i'm quite sure that atleast when you are dealing with anime you might come across 5 frame patterns that don't have any duplicate frames

[Guest]

@meleth

It will be optional and there will be a threshold for deciding when a frame is a duplicate.

But if there are no duplicates then why wouldn't you want to pass them all, i.e., why would you want to blend them as they represent 30fps content?

Remember that Decimate(mode=1) leaves the clip at 30fps.

The ideal is that when the content is 30fps, i.e., no duplicates, then nothing is done; when it is film content, i.e., duplicates appear, then the duplicates are blended. This allows hybrid 30fps/film clips to be more appropriately dealt with. This is exactly what Blight suggested, so I do not understand your point.

[meleth]

@neuron, what i was thinking was this. Say it is a 30fps clip. I still want to take it down do 24fps in order to save file space. Normally there's a big chance you'll get choppy playback. But if you instead of dropped a frame you blended it together with the next or previous one.

[Guest]

@meleth

So then if you have:

A B C D E

You want to make (say):

A B C+D E

That is an interesting idea. I will try it and see what it looks like. Thank you for your clarification.

Of course, you could take this to its extreme, reduce the frame rate to 6fps, and output:

A+B+C+D+E

:)

[daxab]

@meleth et al

Yes, use Decomb for that. IVTC4 (and 5, if I ever finish it) is strictly for material that really is, underneath, 24fps. So there's no motivation for blending -- the frame being dropped is an honest duplicate.

[daxab]

@High Speed Dubb

Thanks for your offer of source code, but I'm avoiding looking at other people's source right now. I've found that it's too easy to fall back on something you've seen, once you've seen it -- it lodges in your brain as an established way to do things.

But one day I might take you up on that -- DScaler sounds like a very interesting animal.

[Guest]

@daxab and meleth


Yes, use Decomb for that. IVTC4 (and 5, if I ever finish it) is strictly for material that really is, underneath, 24fps. So there's no motivation for blending -- the frame being dropped is an honest duplicate.
Yes, meleth, please post further followups on this to my IVTC thread. Let's not hijack this one. :)

[High Speed Dubb]

@daxab

Considering this source code, you&rsquo;re making the right decision for your sanity. ;)

I&rsquo;m not sure this next graph is entirely on topic, but it&rsquo;s not entirely off topic, either. It&rsquo;s a histogram of the frequency of each brightness value of the image, on a log scale. See:

http://students.washington.edu/ldubb/computer/ColorBarBrightnessHistogram.jpg

It shows a couple interesting things. The peaks all look quadratic on this log scale, which means they&rsquo;re Gaussian on a linear scale. That&rsquo;s what you&rsquo;d expect, but it&rsquo;s nice to see it confirmed in the data.

Probably more interesting is that although the heights vary, the shape of the quadratics all look very similar (except on the far left, which represents very dim pixels). That implies the variance of the noise is the same regardless of the brightness &mdash; except where the screen is really dim. That&rsquo;s a big help when trying to get rid of (or avoid the affects of) noise.

[opp]

I don't know how the cache file works but I have a suggestion to reduce the cache size, assuming the cache stores a bunch of true false things for frames. There are lots of patterns in movies, and if the cache stores has a list of common patterns they could be encoded. Ontop of that, if the pattern repeats, the number could be added. for example 10110 is pretty common. and could be stored as A, then something like 11010 could be B, and a movie has AAAABABBAA for the first group of frames. This could be transformed to 4ABA2B2A and could help free up space if it is needed. If there are some big irregularities, there could be a sort of flag that causes it to store the information uncompressed. Well, I am curious to find out how the cache file works, maybe something can be found to make it smaller and maybe even require the plugin to refer to it less to help increase speed.

[Lord Sandwich]

Just tried IVTC4, and I have to say it works wonders on my copy of Transformers The Movie.

I did encounter a problem within the first few frames, though: IVTC4 would crash if I used anything other than the default parameters. This part--they were mostly pure black frames, btw--had to be rendered separately from the rest of the movie.

Other than that, everything's peachy. Not a single combed frame in sight. Great work, daxab!

[daxab]

@Lord Sandwich

Thanks for the feedback. I don't know what would have caused the crash of IVTC4 but I'm glad you were able to work around it. I'm planning on releasing a significantly modified version which should be more robust. Until then...

[Lord Sandwich]

@daxab

It seems that the errors I experienced were caused by the "neighbor" parameter. Anytime I change this things become very unstable--with motion at "1.20" or so, VirtualDub reports errors past the first two frames and refuses to output anything. With motion at up to "2.00" (not recommended, I know :)) VirtualDub may hang during heavy seeking.

[daxab]

@Lord Sandwich

Well, the neighbor parameter is gone in IVTC5, so that should fix that :).

As far as VirtualDub pausing during seeks, this is expected behavior, and it may pause longer when the motion threshold is raised, depending on the source video. Random access is supported, but real time playback is not.

The overall time taken (during an encode, for instance) isn't increased by the pause, since when it pauses, it's precomputing the results in surrounding areas. Later, when you get to those areas, it returns frames immediately.

[Lord Sandwich]

@daxab

Good to know the problem's moot. I turned off the neighbor option and used a full FieldDeinterlace to get rid of the stray combs.

BTW, IVTC4 did a MUCH better job of eliminating duplicate frames than Decomb 3.4. The difference really is astounding. :)

[Guest]

>BTW, IVTC4 did a MUCH better job of eliminating duplicate frames
>than Decomb 3.4. The difference really is astounding.

Not sure what you mean by this, since Decomb's job is not to "eliminate duplicate frames". In any case, would you be willing to make available to me a test clip that you think Decomb has problems with? Thank you.

[daxab]

@Lord Sandwich

That was one of my main goals with IVTC4: to avoid "stutter" caused by duplicate frames coming through. I found that stutter was really visible even during casual viewing. Glad to hear it's working.