How to fix jaggy diagonal lines introduced by QTGMC?

[cagali]

The source is a clip with mixed progressive/interlaced frames.
There are purely progressive frames, purely interlaced frames and frames contain both p and i area, like this example.

The area showing a person with a moving hand is progressive, and on the other end, it is an interlaced text block moving in from outside the frame.

If you look at the hand, the edges become jaggy after processed by QTGMC.
I tried to replace QTGMC with nnedi3 and yadif, the jaggy edges are gone, the edges are almost as smooth as the source but some other artifacts known to these filter appear instead.

I tried to set QTGMC to use nnedi3's default for interpolation, and also set sharpness=0, but maybe I had set them wrongly or for other reasons, the jaggy edges does not disappear.

How should QTGMC's setting be tuned for this kind of mixed frame, or for these moving edges?

Thank you very much.

Source:
https://c1.staticflickr.com/1/681/22778210276_b646ccf442_b.jpg

TDeint + QTGMC:
https://c2.staticflickr.com/6/5687/22385837528_a5ce63b653_b.jpg

The QTGMC is called in this way:

TDeint(mode=0, full=false, cthresh=6, chroma=false, MI=60, edeint=QTGMC(Preset="Placebo", FPSDivisor=2, SourceMatch=3), order=1, emask=TMM(mode=0, order=1), slow=2)



Edit: A sample is uploaded.
http://www.mediafire.com/download/3k335p5caonwea9/HKT_05C3.7z

[TheSkiller]

Some of QTGMC's parameters can be tweaked to aid this, but without a sample I cannot give a more specific answer than this:

• Rep0
• Blocksize
• Search
• Searchparam
• DCT
• ThSAD1
• ThSCD1 and ThSCD2

are candidates for tweaking here (most likely only two or three need adjustment).


Edit: Btw, the images in your post don't work.

[cagali]

Thanks TheSkiller. The image are updated and should be viewable now.

Some of QTGMC's parameters can be tweaked to aid this, but without a sample I cannot give a more specific answer than this:

• Rep0
• Blocksize
• Search
• Searchparam
• DCT
• ThSAD1
• ThSCD1 and ThSCD2

are candidates for tweaking here (most likely only two or three need adjustment).


Edit: Btw, the images in your post don't work.

[johnmeyer]

Still frames are completely misleading when dealing with anything interlaced, whether partially interlaced or completely interlaced. You have to upload an actual, unprocessed, video clip.

[TheSkiller]

The jaggies are not caused by QTGMC itself, they are caused by TDeint.

Simply using your QTGMC call alone, without TDeint, results in a jaggie-less video.

The problem is you are wrapping QTGMC in TDeint. I suspect it is caused by the fact that by default QTGMC does not leave the original fields unaltered. TDeint does, so the interpolated fields by QTGMC do not match the original ones that TDeint injects, resulting in jaggies.

I would skip TDeint altogether.

[cagali]

The jaggies are not caused by QTGMC itself, they are caused by TDeint.

Simply using your QTGMC call alone, without TDeint, results in a jaggie-less video.

The problem is you are wrapping QTGMC in TDeint. I suspect it is caused by the fact that by default QTGMC does not leave the original fields unaltered. TDeint does, so the interpolated fields by QTGMC do not match the original ones that TDeint injects, resulting in jaggies.

I would skip TDeint altogether.

I tried again using QTGMC only, the jaggies have gone. Thank you! :thanks:
If I use qtgmc without using tdeint to find combing first, the default inputtype=0 should be used?
Then I think it may handle those purely progressive frames a little less suitable that inputtype=1 (para for progressive input).

Now there are at least 3 options.
1. tdeint + nnedi3 (need to live with nnedi3/other filter's artifacts in p/i mixed frames)
2. tdeint + qtgmc (need to solve the field problem)
3. qtgmc inputtype=0, for the whole clip (need to live with some slightly overprocessed progressive sections)

I often use qtgmc as my first choice of deinterlacing, would you suggest another high quality deinterlacer?

[TheSkiller]

If I use qtgmc without using tdeint to find combing first, the default inputtype=0 should be used?Yes. But there is no need to find coming using TDeint. QTGMC is a full grown deinterlacer, it works well on it's own. ;)


Then I think it may handle those purely progressive frames a little less suitable that inputtype=1 (para for progressive input).I think it still handles them a lot better than TDeint does, don't you think?


3. qtgmc inputtype=0, for the whole clip (need to live with some slightly overprocessed progressive sections)Use Lossless=2 and SourceMatch to reduce the "processed" look. And if that's not enough use EZKeepGrain.


I often use qtgmc as my first choice of deinterlacing, would you suggest another high quality deinterlacer?Personally, no. QTGMC all the way.

[cagali]

Yes. But there is no need to find coming using TDeint. QTGMC is a full grown deinterlacer, it works well on it's own. ;)


I think it still handles them a lot better than TDeint does, don't you think?


Use Lossless=2 and SourceMatch to reduce the "processed" look. And if that's not enough use EZKeepGrain.


Personally, no. QTGMC all the way.

Yea. Maybe just use QTGMC(Preset="Slower", FPSDivisor=2, SourceMatch=3, Lossless=2) to save the trouble.

[cagali]

An alternative is to use TCombMask+IsCombedTIVTC+ConditionalFilter instead of TDeint, so only combed pixels are processed by QTGMC as usual and no cpu cycles are wasted on pure progressive frames.



I have tested your code, the result is great!
Only the "combed pixels" are processed, the progressive frames are totally untouched.

Actually this thread is kind of a spin-off problem from another thread.
http://forum.doom9.org/showthread.php?p=1745311#post1745311

Now with proper tuning of IsCombedTIVTC and mt_merge's parameter,
I could even save quite some time from trimming the source clip by setting a pretty low combing threshold (athreshL=2), and still keep an acceptable quality for resulting clip.
:thanks: