Moire Effect & TFM(pp=0) Solution

[Danette]

I have DVD-sourced videos that, even in their source format, exhibit moire effect on tight patterns, such as checked coats.

Using TFM(pp=0), this effect is nearly eliminated.

My question, after searching extensively, is: what is the downside in using the pp=0 setting? The default setting is pp=6. I almost wonder why pp=0 isn't the default.

[poisondeathray]

If they had moire "even in their source format" on those patterns , but not with TFM(pp=0), it means whatever you're using to view the DVD is also applying a deinterlace on that section (instead of only field matching, decimation)

pp=6 is the default to treat residual combing. Combing is common for some sources, especially hard telecined older ones (orphaned fields from edits on in interlace timeline, interlaced fades). TFM pp=6 applies a simple cubic deinterlace (essentially resizing a field) when combing is detected. A single field resize causes that moire pattern - you only have 1/2 the spatial information. You can change what deinterlace is used with clip2 (but any deinterlace will be worse than if the "partner" field pair is available) , or the detection thresholds (unlikely to get a setting that will ignore this problem automatically), or set manual overrides

lines, moire patterns are often detected false positives, because they "look" like combing to an algorithm . The downside of pp=0 is residual detected combed frames are not processed. Maybe one day there will be some smart "AI" program that is able to accurately distinguish between them... but not today.

[Danette]

Thanks.

If they had moire "even in their source format" on those patterns , but not with TFM(pp=0), it means whatever you're using to view the DVD is also applying a deinterlace on that section (instead of only field matching, decimation).

I’ve watched this video in VDub2, AvsPMod and MPC-HC and all three show the moire effect prior to any processing, but not after applying pp=0. Are you saying that it could be the various monitors that have an interlace function built in? My TV also shows this difference.

The downside of pp=0 is residual detected combed frames are not processed.

Are you aware of any script that can analyze a file to identify any residual combed frames? This would be after processing the file with pp=0.

[poisondeathray]

I’ve watched this video in VDub2, AvsPMod and MPC-HC and all three show the moire effect prior to any processing, but not after applying pp=0. Are you saying that it could be the various monitors that have an interlace function built in? My TV also shows this difference.


A display can have processing built in, but usually not a computer monitor.

But how did you open it ? Vdub2/AvspMod/MPCHC with what source filter ? What input driver ? There might be additional processing that you are not aware of

Or, if you're using MPEG2Source (DGIndex) only , with honor pulldown flags, there is no processing at all. If there is moire before TFM(pp=0) , but not after using TFM(pp=0) where it becomes completely clean - it suggests the fields were not matched or out of phase (top field and it's "partner" bottom field are not aligned on the same frame) .



Are you aware of any script that can analyze a file to identify any residual combed frames? This would be after processing the file with pp=0.

There are various comb detection scripts, some are motion adaptive , some are masks to be used with other functions. None of them are "perfect" . eg. tcombmask, combmask, dcomb (iscombed) ,...a few others

For very fine residual combing, there are also non selective "brute force" approaches which do not degrade the image that much, such as vinverse, vinverse2

[Danette]

But how did you open it ? Vdub2/AvspMod/MPCHC with what source filter ? What input driver ? There might be additional processing that you are not aware of

Or, if you're using MPEG2Source (DGIndex) only , with honor pulldown flags, there is no processing at all. If there is moire before TFM(pp=0) , but not after using TFM(pp=0) where it becomes completely clean - it suggests the fields were not matched or out of phase (top field and it's "partner" bottom field are not aligned on the same frame) .

Well, just focusing on AvsPMod, I used the d2v file and DGI is set to honor pulldown flags. If there is a way to test to see if the fields are out of phase, let me know. They are TFF and so is the script.

For very fine residual combing, there are also non selective "brute force" approaches which do not degrade the image that much, such as vinverse, vinverse2

Perhaps adding one of these, when using pp=0, is the best clean-up approach. However, I have watched a few of the shows with the pp=0 and I donÂ’t notice any deficiencies in them, other than the moire effect being gone.

[johnmeyer]

Since when does interlacing have anything to do with moiré? Moiréis an interference pattern, and can be caused by many different things. The only thing you get when interlaced material is shown on a progressive display is "teeth" and those only show up when there is a lot of motion. By contrast, moiré happens even with nothing is moving.

A clip would make it a lot easier to know what it really going on.

[poisondeathray]

Since when does interlacing have anything to do with moiré? Moiréis an interference pattern, and can be caused by many different things. The only thing you get when interlaced material is shown on a progressive display is "teeth" and those only show up when there is a lot of motion. By contrast, moiré happens even with nothing is moving.

A clip would make it a lot easier to know what it really going on.

DEinterlacing , such as when TFM detects combing and applies deinterlacing - has a lot to do generating moire patterns

When you deinterlace (in this context meaning interpolation of a single field , such as resizing to a full frame) , a fine pattern like checker cloth, grids, it generates moire patterns. Simple interpolation like cubic resizing is prone to generating more severe patterns than other deinterlacers



Well, just focusing on AvsPMod, I used the d2v file and DGI is set to honor pulldown flags. If there is a way to test to see if the fields are out of phase, let me know. They are TFF and so is the script.


You separatefields() and look for field pairs that belong to the same progressive frame . TFM(pp=0) should align them if they are out of phase . TFM() default settings will also align them but also apply pp=6 deinterlacing if combing is detected as discussed above. So a false positive like checkered clothing is the prototypical example, or horizontal blinds, some type so fabrics, clothing etc.... and it will flare up with bad moire patterns. You can change clip2 to QTGMC, but it's still less ideal if it's not really interlaced (false positive fabric pattern)


Perhaps adding one of these, when using pp=0, is the best clean-up approach. However, I have watched a few of the shows with the pp=0 and I donÂ’t notice any deficiencies in them, other than the moire effect being gone.

The DVD's where you will often have problems are older DVD's many types of anime, ones that have been edited on an interlaced timeline (instead of removing pulldown or on a progressive timeline) , or some TV shows that have been edited.

[johnmeyer]

OK, I can see how deinterlacing would create moiré patterns. I guess I thought the OP was talking about interlaced video from a DVD.

[Danette]

DEinterlacing , such as when TFM detects combing and applies deinterlacing - has a lot to do generating moire patterns

A clip would make it a lot easier to know what it really going on.

If you are interested, here is a sample from the source file:

https://www.mediafire.com/file/48ryrqvdwwnskdj/Moire_Test.m2v/file

Pay attention to the jacket worn by the older man.

[poisondeathray]

It's not completely clean to leave pp=0 on that clip

at ~11.5-12.5 seconds there are about 6-7 frames few frames TFM gets wrong. The guy in the background is combed, it's easy to see because he's in motion. There is actually no problem in the fields (it's all perfect 3:2), it's just a result of a bad TFM match. Some of them get fixed slow=2, but not all. There might be other settings you can tweak to get it right , I didn't spend much time playing with them

This is an example where default adaptive TIVTC shoots you in the foot, probably because of the foreground patterns. A manual, set cycle, gets this perfect. But you'd have to check other segments, in case the cadence was broken


MPEG2Source()

manualIVTC(last,2)

function manualIVTC(clip v, int p)
{
v=loop(v, p, 0,0)
v=v.SeparateFields().SelectEvery(10, 0,1, 2,3, 5,6, 7,8).Weave()
v=trim(v,int(p*3/5),0)

return v.assumeFPS(24000,1001)
}

[johnmeyer]

If you are interested, here is a sample from the source file:

https://www.mediafire.com/file/48ryrqvdwwnskdj/Moire_Test.m2v/file

Pay attention to the jacket worn by the older man.As I expected, the moiréis in the video and has nothing to do with interlacing or deinterlacing. Here is one field (not frame) showing the moiré on Ray Walston's jacket:

https://i.imgur.com/iXhjdVe.png

Since it is a single field, the moiré is not caused by any interaction between fields.

"My Favorite Martian" was shot on 35mm film and then telecined when shown on TV. It did NOT originate on video and therefore interlacing doesn't enter into the equation.

The DVD you took this from either has that telecine "baked in" or when you extracted it from the DVD, the software didn't ignore the pulldown flag. If you want to do anything further you should remove the pulldown using TFM/TDecimate.

Either way, I'm pretty sure that TFM and its various pp modes have nothing to do with solving the problem and there is nothing you can do about this other than to find a better source. This probably happened during the scan of the film because moiré happens when you get beats between fine repetitive patters and the spacing of the scan lines or the "distance" between adjacent pixels.

"My Favorite Martian" is available for streaming via Amazon Prime or IMDB, with commercials. A quick search reveals that you may be able to get it via other streaming options. I think Hulu has it for free.

[poisondeathray]

TFM deinterlacing has everything to do with worsening moire patterns. The OP said when PP=0, the effect is nearly eliminated

When you deinterlace , instead of field matching, when a match is available = bad (!)

If you examine the clip with TFM(display=true) , vs TFM(pp=0, display=true) , it should be clear what is being discussed in this thread

TFM(display=true)
https://i.postimg.cc/KzT1gbQt/tfm-display-true.png (https://postimages.org/)

TFM(pp=0, display=true)
https://i.postimg.cc/2yF32xs6/tfm-pp-0-display-true.png (https://postimages.org/)

[Danette]

This is the script that I've been using:

AssumeTFF()
TFM(pp=0,slow=2)
TDecimate()
QTGMC(InputType=1,Preset="Medium",EZDenoise=0)
LimitedSharpenFaster(strength=200)
Lanczos4Resize(640,480)

In fact, it is what I use on nearly all backups, with minor variations, e.g.; sometimes decimation is not needed, sharpening and denoising varies.

As poisondeathray mentioned, pp=0 dramatically reduces the moire effect.

It's not completely clean to leave pp=0 on that clip

In this case, as you mentioned, would adding vinverse or vinverse2 clean it up?

[johnmeyer]

OK, I IVTC'd it and used pp=0 for TFM. That makes most of the frames look good.

I don't want to get tied up in semantics (what is interlacing vs. what is telecine).

The PP value being used is a red herring. The post processing is not what is eliminating the artifact. Instead, the trick is getting the matching to work every time. For instance, this produces a slightly better result than TFM(pp=0)/TDecimate:

TFM(pp=6,mode=2,cthresh=50,mChroma = false,MI=150,blocky=8,blockx=8)
TDecimate()

The pp value makes no difference. Mode and the threshold settings determine the goodness of the output. I was not able, in a limited time, to tune the script, and I expect that you'll have to feed it manual overrides.

[edit]Why are you using QTGMC? For noise reduction? The clip is progressive after it goes through TDecimate, so deinterlacing is not needed.

[poisondeathray]

In this case, as you mentioned, would adding vinverse or vinverse2 clean it up?

I would not. The problem in this case is the field matching - it's matching the wrong fields , so there is combing. Vinverse/2 or QTGMC or anything is always going to worsen the moire patterns and soften other parts of the image. TFM trying to be too smart, but this clip is perfect 3:2 cadence. See post #10, that's a perfect IVTC (the last frame isn't because of the way the sample is cut)

Usually the best solution (for anything) is to fix the actual underlying problem

There might be some TFM settings to fix everything, I didn't have time to play with it)

[poisondeathray]

The PP value being used is a red herring. The post processing is not what is eliminating the artifact. Instead, the trick is getting the matching to work every time. For instance, this produces a slightly better result than TFM(pp=0)/TDecimate:

TFM(pp=6,mode=2,cthresh=50,mChroma = false,MI=150,blocky=8,blockx=8)
TDecimate()

The pp value makes no difference. Mode and the threshold settings determine the goodness of the output. I was not able, in a limited time, to tune the script, and I expect that you'll have to feed it manual overrides.


The reason this is a slight improvement is a cthresh of 50 essentially means pp=0 . The threshold is so high, that it never deinterlaces. Remember, the default cthresh is 9

The end result is that same bunch of combed frames (around ~10-13 see) mentioned above get passed through , and the moire is worsened on those frames because of the wrong matches

This clip is solvable without those problems, see post #10. But non adaptive IVTC is potentially "dangerous" . The OP has to check that the cadence is not disrupted in other parts

[Danette]

OK, I IVTC'd it and used pp=0 for TFM. That makes most of the frames look good.

I don't want to get tied up in semantics (what is interlacing vs. what is telecine).

The PP value being used is a red herring. The post processing is not what is eliminating the artifact. Instead, the trick is getting the matching to work every time. For instance, this produces a slightly better result than TFM(pp=0)/TDecimate:

TFM(pp=6,mode=2,cthresh=50,mChroma = false,MI=150,blocky=8,blockx=8)
TDecimate()

The pp value makes no difference. Mode and the threshold settings determine the goodness of the output. I was not able, in a limited time, to tune the script, and I expect that you'll have to feed it manual overrides.

[edit]Why are you using QTGMC? For noise reduction? The clip is progressive after it goes through TDecimate, so deinterlacing is not needed.

Thanks, but I want to avoid fine-tuning to that level. I have ~100 episodes, so I need the one-size-fits-all aspect of the pp=0 (I can live with whatever the minor difference may be).

I use QTGMC, in this case, not for deinterlacing (thus the InputType=1). QTGMC does other beneficial things that helps me avoid more of that fine-tuning need.

[johnmeyer]

If mine works better, you should use it. Why wouldn't you use something that does a better job.

Your choice, of course.

[Danette]

If mine works better, you should use it. Why wouldn't you use something that does a better job.

When I compared the two, I could not see a difference.

[real.finder]

this method https://forum.doom9.org/showthread.php?t=176657 seems work

[Danette]

this method https://forum.doom9.org/showthread.php?t=176657 seems work

This did not do anything differently than the script I've been using. What is it that you think is different about it?

[poisondeathray]

This did not do anything differently than the script I've been using. What is it that you think is different about it?

No, it's different . It doesn't have the wrong matches like TFM, so it has less moire and combing and is much cleaner. Any post processing - even QTGMC in progressive mode - will worsen the moire patterns compared to pp=0, and much worse compared to a clean IVTC

That method produces perfect results like the function in post #10, it's 100% clean. But both those methods will fail if the cadence is interrupted, such as edits while still interlaced - there will sections where lots of frames are combed because it's out of phase

If you can't see the difference, interleave the results, or compare in avspmod tabs. Very different

[Frank62]

As I often said: I'd prefer the solution "by hand". You can use it adapted to the three possible patterns in this case. Usually one episode can be fixed with three or four pattern changes - if very much more, of course, not.
Or the best way: You buy the 100% Pulldown-free PAL-DVDs we released about two years ago. :)

[Danette]

No, it's different . It doesn't have the wrong matches like TFM, so it has less moire and combing and is much cleaner. Any post processing - even QTGMC in progressive mode - will worsen the moire patterns compared to pp=0, and much worse compared to a clean IVTC

That method produces perfect results like the function in post #10, it's 100% clean. But both those methods will fail if the cadence is interrupted, such as edits while still interlaced - there will sections where lots of frames are combed because it's out of phase

If you can't see the difference, interleave the results, or compare in avspmod tabs. Very different

Since I can’t see the difference (although I will scrutinize it more, later), and since there seems to be no downside to it, I will take your word that it is better than TFM. Can it be used interchangeably with TFM on all interlaced material, i.e.; is it superior to TFM or, at least, no worse?

[poisondeathray]

Since I can’t see the difference (although I will scrutinize it more, later), and since there seems to be no downside to it, I will take your word that it is better than TFM. Can it be used interchangeably with TFM on all interlaced material, i.e.; is it superior to TFM or, at least, no worse?

The bad frames just jump out at you. Some are as bad as the screenshot in the earlier post with deinterlacing. TFM bad matches are when the top field from one frame is matched with the bottom field of another frame. Instead of returning the original film frame (both fields from the same frame) . That wrong matching manifests/appears as combing, and if there is significant motion between the frames, such as when the suit guy is moving, they manifest as terrible moire patterns

Adding QTGMC in progressive mode makes almost every frame look worse with the suit pattern (degrading all the frames, even the previously clean ones TFM got right), everything is softer and no amount of added sharpening can return the lost details

I already mentioned the downsides - potentially it's very dangerous because those two approaches are not adaptive. The 2nd one is slightly adaptive (in that you don't have to choose the part of the cycle), but can still return the wrong matches for a changing or interrupted cycle. For someone that doesn't want to overlook things , possibly fix the phase for a certain segment, it's like playing with fire. But if all your DVD's are perfect 3:2, no breaks, then yes it will work and perfectly if it's the same phase everywhere. If it's any of the types of DVD's mentioned earlier, it's problematic

[VoodooFX]

What if for pp you run QTGMC to eliminate that residual combing, using conditional with something like qCombed (http://forum.doom9.net/showthread.php?p=1776501) or IsCombed to heal only combed frames.

[johnmeyer]

The bad frames just jump out at you. Some are as bad as the screenshot in the earlier post with deinterlacing. TFM bad matches are when the top field from one frame is matched with the bottom field of another frame. Instead of returning the original film frame (both fields from the same frame) . That wrong matching manifests/appears as combing, and if there is significant motion between the frames, such as when the suit guy is moving, they manifest as terrible moire patterns<snip>I'm now totally on the same page as you. Thanks for the education.

Once I understood that the artifact happend because of bad matching, I realized that you have to either tune TFM, or find another approach. It will be really interesting if Katie's work finally finds a use.

[real.finder]

and since it black and white film, it's better and faster to use Converttoy8/Converttoy

also I already made Katie method as function KGIVTC() in AnimeIVTC.avsi if someone care

[Danette]

I’ve scrutinized the differences and do agree that the clean Katie approach (removing my use of QTGMC) seems perfect on that small sample I provided and also seems to do the same on the full episode.

The 2nd one is slightly adaptive (in that you don't have to choose the part of the cycle), but can still return the wrong matches for a changing or interrupted cycle. For someone that doesn't want to overlook things , possibly fix the phase for a certain segment, it's like playing with fire. But if all your DVD's are perfect 3:2, no breaks, then yes it will work and perfectly if it's the same phase everywhere. If it's any of the types of DVD's mentioned earlier, it's problematic

This causes some confusion for me:
- Which is the “2nd one” to which you refer, the Katie or pp=0?
- Is there some way to determine where or how many breaks occur in a 3:2 or if the cadence is interrupted?

I don’t want to have to nurse these 100 or so TV episodes, frame-by-frame, so I would opt for the process that would quickly identify which script (among pp=6, pp=0 or Katie method) would be most likely to deliver the better results.

Are you aware of any analysis tools, such as DGIndex summaries, GSpot, MediaInfo or ReStream, that are able to give a good indication of the “perfect 3:2”, interrupted cycle or some other measure that could be used to decide between Katie, pp=0 or pp=6?

and since it black and white film, it's better and faster to use Converttoy8/Converttoy

How would this be applied? I tried adding it to the Katie method, but TDecimate seems to not accept the colorspace. Is Converttoy8/Converttoy, essentially, the same as Greyscale()?

[real.finder]

How would this be applied? I tried adding it to the Katie method, but TDecimate seems to not accept the colorspace. Is Converttoy8/Converttoy, essentially, the same as Greyscale()?

you need to update tivtc https://github.com/pinterf/TIVTC/releases

[poisondeathray]

I’ve scrutinized the differences and do agree that the clean Katie approach (removing my use of QTGMC) seems perfect on that small sample I provided and also seems to do the same on the full episode.

Lucky you! :)


This causes some confusion for me:
- Which is the “2nd one” to which you refer, the Katie or pp=0?


The function in post #10 is the 1st with the strict cycle and phase, Katie's approach is the 2nd . They both give the same results on that sample (the 1st keeps the last extra frame, and it's combed, but it's because the way the sample was cut) . But if you try them on typical hard telecine DVD's with issues (e.g. any 90's TV shows DVD's, older anime DVD's), you're in trouble.


- Is there some way to determine where or how many breaks occur in a 3:2 or if the cadence is interrupted?

I don’t want to have to nurse these 100 or so TV episodes, frame-by-frame, so I would opt for the process that would quickly identify which script (among pp=6, pp=0 or Katie method) would be most likely to deliver the better results.


Not automatically; you have to choose based on the pros/cons .

If these specific DVD's are from the same series, same set, same studio, same transfer process - there is a higher probability that you could just set and forget if you've checked this one and it was ok (but then again, you said you couldn't see differences the first time, and yet there were major differences....)

If there are edits/ out of phase / not perfect cycle - you're in trouble and you will see combing artifacts (and if all your DVD's have guys with suits featuring fine patterns, moire too)

One way might be to run it, and run a combed detection script (either for fallback, or treatment) . But you run into the exact same fundamental problem - there is no way an algorithm can 100% accurately determine between "combing" and say horizontal patterns like window blinds, patterns on a suit in the year 2020. This is why TFM is getting the wrong matches in the first place (It's generally good overall, but that's why it has manual overrides). You need human eyes. (Maybe some advanced neural net or "AI" in the future)


Are you aware of any analysis tools, such as DGIndex summaries, GSpot, MediaInfo or ReStream, that are able to give a good indication of the “perfect 3:2”, interrupted cycle or some other measure that could be used to decide between Katie, pp=0 or pp=6?


Only when it's soft telecine. It will indicate 100% film and you can "force film". Not when it's hard telecine . There has to be some way to analyze the actual content accurately. ie. your eyes.

[hello_hello]

TFM works fine on the sample, as long as you don't let it de-interlace, disable MicMatching, and restrict it to only using the top 140 lines of the frame for it's field matching decisions. :)

TFM(y0=140, y1=480, pp=0, micmatching=0)
TDecimate()

Why are you using QTGMC? For noise reduction? The clip is progressive after it goes through TDecimate, so deinterlacing is not needed.

He's running QTGMC in progressive mode. It can work very well as a denoiser, and can also be useful for stabilising the picture. It's not too often I encode a DVD without running QTGMC in progressive mode on it (assuming it's not interlaced). Unfortunately QTGMC isn't the right choice for denoising Uncle Martin's jacket though. Most denoisers probably aren't going to do well with it. TemporalDegrain tends to blur a bit where there's motion, but it doesn't do too badly with the jacket. Maybe someone else will have a better idea.

TFM(y0=140, y1=480, pp=1, micmatching=0)
TDecimate()
TemporalDeGrain()
GreyScale()
CropResize(0,0, 10,2,-10,-2, InDAR=15.0/11.0, ResizeWO=true)
Gradfun3()

https://i.ibb.co/xFsKWMK/Martian.png (https://ibb.co/jJ59mL9)

[StainlessS]

HH, have you tried SpotLess at all [just curious whether it would do any good].
(After un-moire, I have not looked at clip)

[hello_hello]

HH, have you tried SpotLess at all [just curious whether it would do any good].
(After un-moire, I have not looked at clip)

No I haven't. I'll check it out.

(I added a screenshot to my previous post)

[hello_hello]

This is the script that I've been using:

AssumeTFF()
TFM(pp=0,slow=2)
TDecimate()
QTGMC(InputType=1,Preset="Medium",EZDenoise=0)
LimitedSharpenFaster(strength=200)
Lanczos4Resize(640,480)

Nearly all 4:3 DVDs have an ITU or mpeg4 pixel aspect ratio.
(ITU and mpeg4 pixel aspect ratios are virtually the same, but the mpeg4 PAR's are easier to work with)

Pixel aspect ratios:
https://forum.doom9.org/showthread.php?p=1058927#post1058927

The mpeg4 display aspect ratio equivalents of 16:9 and 4:3, are 20:11 and 15:11, but if you crop 16 pixels from the width, what you're left with is exactly 16:9 or 4:3, so...

Crop(8,0,-8,0)
Lanczos4Resize(640,480)

The sample needs a couple more pixels cropped each side to make it clean, and because I'm OCD about the display aspect ratio being a minimum of 4:3, I cropped a couple of pixels top and bottom too for my previous screenshot. It'd be the equivalent of:

Crop(10,2,-10,-2)
Lanczos4Resize(636,476)

That's just me though.

Shameless plug:
CropResize (https://forum.doom9.org/showthread.php?t=176667)

CropResize(0,0, 10,2,-10,-2, InDAR=15.0/11.0, ResizeWO=true, Info=true)

https://i.ibb.co/DfbGd4g/Moire-Test-avs-snapshot.png (https://ibb.co/DfbGd4g)

[johnmeyer]

TFM works fine on the sample, as long as you don't let it de-interlace, disable MicMatching, and restrict it to only using the top 140 lines of the frame for it's field matching decisions. :)

TFM(y0=140, y1=480, pp=0, micmatching=0)
TDecimate()
Why didn't I play around with micmatching? I should have known better, but I guess it's been fifteen years since I did all the work with TFM and had to manipulate all the matching parameters in a spreadsheet.

As much as I like it that Katie's work finally found a home, I still think the OP is better off following hello^2' advice because, as we all know, the cadence is likely to change at some edit point, and then you need a tool which can adapt.

[poisondeathray]

Why didn't I play around with micmatching? I should have known better, but I guess it's been fifteen years since I did all the work with TFM and had to manipulate all the matching parameters in a spreadsheet.

As much as I like it that Katie's work finally found a home, I still think the OP is better off following hello^2' advice because, as we all know, the cadence is likely to change at some edit point, and then you need a tool which can adapt.

Disabling the micmatching is not doing the "heavy lifting" .

The reason that works is the lower ~3/4 of the image is not used for the 1st part of the metrics calculation because of the y0/y1 exclusion band. The reason why you have to disable micmatching in addition to the exclusion band - is the exclusion band does not actually bypass all checks.
See this for more info
https://forum.doom9.org/showthread.php?p=1913224#post1913224

If the suit guy moved to the top of the frame, camera tilted, etc.. , TFM would make the same types of mistakes. You'd have to animate the exclusion band somewhere else wherever the suit guy wasn't. Moreover, it's also potentially dangerous to field match based on the top 1/4 of the image only

[johnmeyer]

Disabling the micmatching is not doing the "heavy lifting" .

The reason that works is the lower ~3/4 of the image is not used for the 1st part of the metrics calculation because of the y0/y1 exclusion band. The reason why you have to disable micmatching in addition to the exclusion band - is the exclusion band does not actually bypass all checks.
See this for more info
https://forum.doom9.org/showthread.php?p=1913224#post1913224Interesting information about the exclusion band not bypassing all checks. I did not know that.

I did consider setting up an exclusion area, but didn't want to spend five minutes fiddling with the syntax, e.g., is the origin in the upper left or lower right; am I excluding things inside the box or outside the box, etc. I would eventually figure it out, but it was more time than I wanted to put into it.

[Danette]

After much back-and-forth experimenting with the above suggestions, it looks, to me, that the “TFM(y0=140, y1=480, pp=1, micmatching=0)” approach is superior in this case. I will probably be using it as the default for other DVD’s as well, unless I notice a problem. It seems that every approach is fraught with potential peril, leaving me with no highest-probability for avoiding problems.

I could not get the “Katie” method to work. It handled the moire effect well (not quite as good as aforementioned approach), but leaves what appears to be heavy interlace artifacts in many sections. If anyone wishes to address this, I’d be pleased to re-evaluate it. Examples are below.

Unfortunately, my normal use of QTGMC, in progressive mode, worsened the moire effect in all experiments. So, I’ll have to leave that out.

The script that seems most pleasing, to me, is:

TFM(y0=140,y1=480,pp=0,micmatching=0,slow=2)
TDecimate()
VagueDenoiser(chromaT=0)
KNLMeansCL(d=1,a=1,s=4,h=4)
LimitedSharpenFaster(strength=400)
Greyscale()
Lanczos4Resize(640,480)

Gradfun3()
Was this added because you found it useful in my particular case, or do you generally add it as a just-in-case tool?

Katie vs. TFM(y0=140,y1=480,pp=0,micmatching=0,slow=2) problem examples:

https://i.postimg.cc/QC3hJ2yB/Katie.jpg
https://i.postimg.cc/c4ddgN32/TFM.jpg

[johnmeyer]

The whole reason some of us weren't keen on Katie's approach is that it fails as soon as the pulldown cadence changes, which it often does at scene changes or cuts, if the video has been manipulated at some prior time. TFM/TDecimate is almost always the best way to go, although as this thread shows, getting it "tuned" can sometimes be a chore. Having said that, most of the time the defaults work fine.

[hello_hello]

The script that seems most pleasing, to me, is:

TFM(y0=140,y1=480,pp=0,micmatching=0,slow=2)
TDecimate()
VagueDenoiser(chromaT=0)
KNLMeansCL(d=1,a=1,s=4,h=4)
LimitedSharpenFaster(strength=400)
Greyscale()
Lanczos4Resize(640,480)

I still think the aspect ratio will be a little off if you don't crop.

Crop(8,0,-8,0)
Lanczos4Resize(640,480)

And if the only thing TFM has a problem with is that particular jacket, it'd consider splitting the episodes where he's wearing it and using TFM normally for the the scenes where he isn't. It's more work, but it's safer. Just make sure the Trim ranges are always multiples of five.
This sort of thing.

Trim(0,874).TFM(slow=2).TDecimate() ++ \
Trim(875,1289).TFM(y0=140,y1=480,pp=0,micmatching=0,slow=2).TDecimate() ++ \
Trim(1290,1574).TFM(slow=2).TDecimate() ++ \
Trim(1575,1834).TFM(y0=140,y1=480,pp=0,micmatching=0,slow=2).TDecimate() ++ \
Trim(1835,0).TFM(slow=2).TDecimate()

VagueDenoiser(chromaT=0)
KNLMeansCL(d=1,a=1,s=4,h=4)
LimitedSharpenFaster(strength=400)
Greyscale()
Crop(8,0,-8,0)
Lanczos4Resize(640,480)

Or you could IVTC first, and splice in the second TFM/IVTC where you see problems. Because the trims are post IVTC the frame numbers will be different, but I didn't bother changing them as I was just making up numbers anyway. It shouldn't be necessary to Trim in multiples of 5. Just switch between the two clips as required. Or add a third clip with different TFM settings or different IVTC if need be.

A = TFM(slow=2).TDecimate()
B = TFM(y0=140,y1=480,pp=0,micmatching=0,slow=2).TDecimate()

A.Trim(0,874) ++ \
B.Trim(875,1289) ++ \
A.Trim(1290,1574) ++ \
B.Trim(1575,1834) ++ \
A.Trim(1835,0)

VagueDenoiser(chromaT=0)
KNLMeansCL(d=1,a=1,s=4,h=4)
LimitedSharpenFaster(strength=400)
Greyscale()
Crop(8,0,-8,0)
Lanczos4Resize(640,480)

return A

# use return A to check the standard IVTC and find any problem frame ranges
# remove it to encode when you're done


Gradfun3()
Was this added because you found it useful in my particular case, or do you generally add it as a just-in-case tool?

I add it to the end of every script. It helps to prevent banding (https://en.wikipedia.org/wiki/Colour_banding) for 8 bit encoding by dithering. Noise naturally prevents banding, so noise removal can increase the likelihood of banding, and for B/W it can be even more likely as there's only luminance values. If there's enough noise left behind to prevent banding, and/or you're encoding at a very high quality, or using 10 bit h265 encoding, you probably won't need gradfun3.

Gradfun3 is part of the DitherTools package (http://avisynth.nl/index.php/Dither_tools).

[Frank62]

-Kathie's approach does work sometimes. As long as the frames (fields) to decimate are detected 100% correctly (which seems nearly impossible for a whole episode).
If not, there will be "interlacing" plus jerkyness. Late edits won't break Kathie's approach working well, the decimates MIGHT work before and after an edit well, maybe one frame keeps interlaced. But there will be scenes where decimation won't work 100%, as always, no matter how you set thresholds. You will have to change those and splice a lot of times.

-The TFM approach will work better, as always, because the probability is higher that fields are matched correctly first, and later decimation will also work better, than decimating fields before 100% correctly. But the jacket problem shows that also field matching can fail a lot. Ok, you can splice again, with different parameters, in this case not thresholds but exclusion areas like jackets. In the end there will still - as always - be some "interlaced" frames plus some dups left. About 3-10 per episode typically, with, without micmatching a. s. o.

-The by-hand solution is by far the best. You also will possibly have some splices to do by hand, but the result will be 100% clean, no mismatched fields, no dups, no jerkyness. We typically work about 5 minutes on an episode to find the right patterns (there are only three possible), that change 0-5 times typically per episode. Automatisms are good, when you HAVE to use them, if patterns change too often plus too many episodes.
So, where is the problem to use this approach?

[Danette]

-The TFM approach will work better, as always

Vs the Katie approach, this is what seems correct, to me, also.

The by-hand solution is by far the best. We typically work about 5 minutes on an episode

So, where is the problem to use this approach?

I think you hit on the answer to your question about "where is the problem" to using the by-hand solution. I acknowledge that it would be best, but 5 minutes per episode is 3 hours for a typical season and it is just too much time for too little benefit. I miss several frames every time I blink, so maybe the little combing that occurs will appear when my eyes are closed. Occasional combing is nothing as compared to the persistence and distraction of the moire effect.

[Frank62]

Of course your decision. I just wanted to point out that, IF you had to splice parts with different parameters of the two other solutions, THEN the by-hand solution would be the best by far.

[Danette]

Of course your decision. I just wanted to point out that, IF you had to splice parts with different parameters of the two other solutions, THEN the by-hand solution would be the best by far.

Yes, thank you for the better suggestion. It’s the time issue only, although the time issue may not be too big a problem because I believe this clip is not interlaced, but is telecined only, and there are few segments where the by-hand approach would be needed. For example, when I run this script on the entire episode:

CombTest=BlankClip(Last,Color=color_red)
TFM(Clip2=CombTest)
TDecimate()

I see red frames only in the moire segments when pp=6. When pp=0, there are no red frames and, as I understand it, pp=0 turns the deinterlacing function off and this should result in many combed frames if the video is truly interlaced.

I still think the aspect ratio will be a little off if you don't crop.

Crop(8,0,-8,0)
Lanczos4Resize(640,480)

Like you, I am very sensitive to the AR in my videos. However, I have found that cropping borders, simply because they are there, doesn’t always work. With each TV series, I do spend some time trying to verify the AR. I do this by trying to find a known circle or square object in the video that is not angled, depth wise, and ensure that the diameter is consistent. I suspect that, in many cases, that the studios crop the original edges because they are bad and then add the border to ensure the correct AR. It becomes a real pain when they seem to do this episode-by-episode, but this is less common. In the case of this series, the borders do seem to provide the correct AR.

[hello_hello]

Like you, I am very sensitive to the AR in my videos. However, I have found that cropping borders, simply because they are there, doesn’t always work.

That's the very reason I crop borders.... because they're there..... :)
It drives me nuts when the picture edges aren't clean, which is why I'd crop a bit more than 8 pixels from each side of your sample.

In the case of this series, the borders do seem to provide the correct AR.

It's very hard to tell just from your sample, but looking the globe in the background, I'm still inclined to lean more towards 15:11 than 4:3. Not that the difference is huge, but I'm not sure I've ever met a 4:3 DVD that didn't have an ITU or mpeg4 aspect ratio (15:11). It's a different story for 16:9 DVDs though.

CropResize(640,480, InDAR=4.0/3.0)

https://i.ibb.co/25W1b9L/4-3.png

CropResize(640,480, 8,0,-8,0, InDAR=15.0/11.0)

https://i.ibb.co/vHtXstj/15-11.png

[Frank62]

It's very common that left and right are blacked areas due to often worse picture quality near borders, so they're cropped. Has nothing to do with AR, that of course is 4:3. The master sources we had for PAL were HD, so I think the ones they used for the NTSC-DVDs will be pretty much the same.

[Danette]

What both of you DON'T have is the entire episode. In the opening credits there is a circle that I used determine if cropping gave better dimensions than uncropped. To me, it appeared that, by cropping, the circle was out of proportion (I mark the diameter on a piece of paper and then compare it when turned perpendicularly).

He's running QTGMC in progressive mode. It can work very well as a denoiser, and can also be useful for stabilising the picture. It's not too often I encode a DVD without running QTGMC in progressive mode.

I, too, like to run QTGMC in progressive mode on all videos for this reason. Is there some setting that can be turned off in QTGMC (placed after TFM/TDecimate) so that QTGMC can do all it's nice work, but without re-introducing the moire effect?

[hello_hello]

I've seen quite a few DVDs where the aspect ratio for the opening credits was wrong. Like these. I found them on the internet and don't know where they came from, but I've seen the poor MGM lion being tortured on DVDs in the past. These are both 1080p so it's likely they weren't resized.

https://www.logolynx.com/images/logolynx/fe/fe75bd653ac72fa98dd103e310d06826.jpeg
https://www.logolynx.com/images/logolynx/d1/d18edcad22bbd71a473fe4a190eab9d4.jpeg

Anyway... I can only go by the sample and guess what looks right to me. It's your encode though, so whatever looks right to you.

[hello_hello]

I, too, like to run QTGMC in progressive mode on all videos for this reason. Is there some setting that can be turned off in QTGMC (placed after TFM/TDecimate) so that QTGMC can do all it's nice work, but without re-introducing the moire effect?

Not that I know of. To the best of my knowledge InputType=1 doesn't de-interlace, but QTGMC still does everything else it'd do in order to repair a de-interlaced picture. That's why it can be great at stabilising, but I guess this is one example of when that has a negative effect.