TDeint and TIVTC

[tormento]

Originally Posted by PoeBear

Your output would match how the US Blu-ray was done

That has terrible rendering of the 60i frames.

[flossy_cake]

I'm working on a realtime solution for such "mixed cadence" content, where it will output all sequences at 60p, such that the 24p sequences become 3:2, 30p sections become 2:2, and 60i sections are 1:1. All in a 60p "container". hello_hello managed to solve this already by making it into a VFR file using timecode file, but it wasn't ideal for me as I wanted a realtime solution that didn't require transcoding (generation loss).

The logic of it is actually simpler than I was expecting: just choose either TFM.TDecimate.ChangeFPS(60000,1001) or TFM.BWDIF(field=-2, thr=3) depending on whether the last 5 consecutive frames of the output of TFM are unique, i.e if the YDiff or CFrameDiff are all above a certain value for the last 5 frames. The hard part is trying to get Avisynth to do what I want it to do on a per-frame basis given the quirks and limitations of its runtime functions, but, I think I've finally got it doing what I want using a combination of ScriptClip and FrameEvaluate. But I'm not optimistic as I really need to "look ahead" 5 frames otherwise the transition between 24p and 30p/60i sections could have a stutter. And the per-frame evaluations need to be done in a controlled order, which is difficult given how Avisynth handles multithreading (the docs warn that messaging through global variables is "unsafe" inside a runtime function due to this).

What would really help though is if Tritical's CFrameDiff had a parameter to calculate the diff between current frame and n frames ahead, just like the internal YDiffToNext does. Because I have a feeling YDiffToNext is not going to be sufficient & Tritical is using more robust metrics with CFrameDiff (and even that gets some false positives eg. if there's composite video dotcrawl that changes every field -- coincidentally that's why sometimes the duplicate frame in the TFM stream looks ever so slightly different -- because it's matching it with a field that although correctly matched in the sense it comes from the same frame, it comes from a different "copy" of that field, i.e the third of the "3" in "3:2" sequence instead of the first).

[PoeBear]

Originally Posted by tormento

That has terrible rendering of the 60i frames.

Then you can edit out all the 60i parts by hand, and use that Dogway script on them. It's always going to look rough unless you interpolate more frames before bringing back down to 23.976

[Dogway]

Originally Posted by tormento

The video is an "intentional" mix of 23.976p (animations, the 80% of the video), 29.97p (pannings) and 60i (CGI), all soft tagged as 29.97p. The author tells us that that was the best way to preserve the "feeling" and "crispness" of the anime.

I'd like to have a uniform fps, possibly a standard one, such as 23.976p being the most used there, without sacrificing quality too much (CGI has different images at every 60i frame).

Can you, please, help me with a proper script to deal with it?

Here is a sample.

Now your explanation made it more clear and with the source I could have a look.
Interesting practice as it has been a very long time since I had to deal with fields...

The script below treats each mode separately, augmenting them to 60fps progressive with duplicates.

DGSource("Bebop.dgi",cl=236,ct=0,cr=236,cb=0)
src=last


### NTSC ###
N=Interleave(src,src)

### FILM ###
DGTelecide()
TDecimate(mode=1) # anime
Interleave(last,last,last)
TDecimate(1,1,6) # Decimate on longest strike so the pace shift ratio is minimal
F=last

### INTERLACED ###
src
QTGMCp(tr2=1,preset="very slow",sourcematch=2,sharpness=0.2,MatchEnhance=0.0,MatchPreset="Slow", MatchPreset2="Slow",border=false)
I=last


ClipClop(F,N,I,sCMD="N(0,5125); F(5126,5455); N(5456,9180); I(9181,9349);")

# VFR in 60fps CFR progressive

In the end you run the fabolous ClipClop to stitch them back together.
The way to look for frame numbers in this 60fps mashup is to take the source and simply:

src
Interleave(last,last)

in another tab in AvsPmod, then go through it and mark with bookmarks the frames where each mode starts.
A script for detection could be possible I think. Simply look for combed frames, if all progressive it's 29.97, if all combed it's 60i, otherwise it's telecine. It's not that easy since you have to keep into account long streaks of no movement, and also scene changes.
As I said yesterday I'm a bit busy finishing my scripts but many people can help on this regard.


A remark on augmenting FILM to 60fps, this works only because it's anime. As I observed the animation pattern goes mostly 3x3, sometimes 3x2, and rarely 2x2. The framerate augment is not linear so the animation pace can be altered but on the good side the shift is minimal 1/60 = 0.017 seconds at most every 10 frames. Or for a 3x3 pattern, multiplied by 2.5 gives 7.5x7.5 or 8x7 after 1 out of 6 decimation.
The only problematic area is when there's 30fps content going on through the windows (the asteroids), there you'll see 3 dups followed by 2 dups, and this can cause some stutter, but it would be the same if played on TV.

In my opinion this is better than interpolation, which doesn't work nice for anime.
It would involve interpolating 24p to 120p, then selectevery(4,0).Interleave(last,last)

timecode VFR I don't recommend as the TV will be jumping through refresh rates.

[tormento]

Originally Posted by flossy_cake

I'm working on a realtime solution

That would be really nice! Please keep us informed about progress.

[tormento]

Originally Posted by Dogway

The script below treats each mode separately, augmenting them to 60fps progressive with duplicates.

Great! Does it keep the original duration? I don't want to mess with audio or subs.

Originally Posted by Dogway

in another tab in AvsPmod, then go through it and mark with bookmarks the frames where each mode starts.

This point is obscure to me. Using interleave(last,last) outputs a 59.97 video, how is possible to find the exact places where fps changes?

Originally Posted by Dogway

A script for detection could be possible I think.

It would be much better, please That script could be really usefulm even in ExTools, to have a swiss knife to deal with "variable" fps sources.

[Reel.Deel]

@tormento

There's NTSCanalyze, from experience it does an OK job, but it's prone to some errors on some frames. There was a discussion on how to improve it here.

[flossy_cake]

Yeah the big issue is dealing with false positives and false negatives. And @Dogway mentioned also: "It's not that easy since you have to keep into account long streaks of no movement, and also scene changes". If the content is like 90% 24p and 10% 30p/60i then I suppose you could bias the error in the direction of assuming 24p in such "when in doubt" scenarios. eg. if YDiff or CFrameDiff of last 5 frames are ALL below some threshold, then treat it as 24p. In this case 24p sections should look good, but slight stutter when there is very small amount of motion on 30p sections.

[flossy_cake]

There seems to be a bug with TFM's motion adaptive deinterlacing, eg. when PP=6 (the default). It seems to drop the entire frame to half vertical resolution when it detects combing, when it should only be doing that for moving parts of the image, as that is the whole "motion adaptive" feature (otherwise I think it would have been called field discarding or drop field or something like that). To reproduce it set cthresh=-1 so that TFM thinks everything is combed.

It seems we can work around this though with eg. clip2=bwdif, which then seems to activate TFM's better internal logic of only using deinterlaced pixels for moving parts of the image, and those pixels then become bwdif's pixels. edit: actually now that I think about it, it's probably just replacing the entire frame with bwdif's pixels when isCombed, but I can't tell for sure. If it were the case, it just means TFM's motion adaptive deinterlacing is extremely primitive... but that doesn't seem likely either given the author's other work with TDeint. Still, I don't know for sure.

Also on the other issue of scene changes having a repeat frame due to orphaned field, that seems workaroundable with scthresh=100, which causes combed frames on scene changes to stay combed, so that they get picked up by PP=6 & therefore clip2=bwdif'd. With this setting, image quality now is on par with my Sony DVD player for such soft telecined content which orphaned fields on scene changes - good.

[flossy_cake]

Originally Posted by flossy_cake

I'm working on a realtime solution for such "mixed cadence" content, where it will output all sequences at 60p, such that the 24p sequences become 3:2, 30p sections become 2:2, and 60i sections are 1:1. All in a 60p "container". hello_hello managed to solve this already by making it into a VFR file using timecode file, but it wasn't ideal for me as I wanted a realtime solution that didn't require transcoding (generation loss).

Wow it looks like TFM can do this all on its own!

Test clip: https://drive.google.com/file/d/1KdE...ew?usp=sharing

source = "480i 3-2 and 1-1.mkv"

LWLibavVideoSource(source=source, stream_index=-1, repeat=true, cache=false)

TFM(last.ChangeFPS(60000,1001), mode=5, PP=6, cthresh=3, MI=80,
	\ scthresh=100, slow=2, ubsco=false, mmsco=false, micmatching=4,
	\ clip2=bwdif(last, field=3, thr=3), display=true)

LanczosResize(720,540)

The above assumes you have the latest BWDIF which has the new thr param: https://github.com/Asd-g/AviSynth-BWDIF/releases. Also its negative field param values won't work anymore if the file has _FieldBased=0 (progressive) which soft-telecined content seems to have. That's why I've set it to 3 instead of -2 (this particular clip is hard-telecined though).

Still have the issue with combing detection not being quite good enough on actors mouths when they are sitting perfectly still (eg. the female newsreader in the clip) their mouth will get combed cause there just isn't enough motion. That's why I've lowered cthresh from 9 to 3, and it still fails a little bit there. If your content has a lot of 60i you might want to lower it to 2 and then lower MI as well to 48-64 or something. But then you'll probably get more false positives on 24p sections and have resolution dropping to 240p on motion when it shouldn't.

edit: BWDIF and yadifmod2 seem to have an issue where they can't detect dynamic changes to the field order (git issue). TDeint(mode=1) and Bob() seem to be fine and can detect dynamic field order changes. The test clip in this post is not affected as the field order is static.

[flossy_cake]

imo the combing detection is the real weak point with TFM and TDeint. I've tried DG's isCombed (qCombed) as well and that is not reliable enough either. It needs per-scene threshold tweaks to work reliably. Even on one scene I raised the threshold to 50 and it was still declaring the whole frame as combed. I need a better metric, and it has to be fast for real time use.

Or maybe I need to come up with a whole new strategy like a new type of mask with some hysteresis characteristic. Or a mod of IsCombedTIVTC which assigns each 16x16 block a boolean isCombed and only deinterlaces blocks with true.

[flossy_cake]

I'm finding TDecimate's mode2 seems to be doing a better job at managing bad edits and orphaned fields in problematic 3:2 content. In mode2 we don't specify cycle & cycleR but instead just give it a frame rate value like 23.976 and it uses an alternate algorithm to decide which frames to remove. Whereas with mode0 & mode1 I was finding there would be some scenes where there would be 2 candidate frames for decimation within a cycle of 5 frames, and it would pick the wrong one. Not necessarily by virtue of bad logical reasoning, as in one case it picked the one with the lower framediff value, but it just so happened the frame with the slightly lower framediff value was the real frame and the other was the duplicate. It's a tricky situation.

[tormento]

And what about duration? Does all that lead to audio desync? I could be really hard to fix it per scene based.

[flossy_cake]

Originally Posted by tormento

And what about duration? Does all that lead to audio desync?

I don't see any reason why it would, since 60 frames are going in & 60 frames are coming out. Nothing is getting re-timed, it's just doing field matching and deint on the 60 frame input.

For some reason TFM needs micmatching=4/5 to work when the input framerate is duplicated from 30 to 60, and I'm not sure why. I don't even know what micmatching is & couldn't find any documentation explaining it. It looks like something to do with the field matching process -- if anyone knows the details please enlighten me.

[flossy_cake]

Originally Posted by StainlessS

I think that is why its best to use

d2v = "..."
Mpeg2Source(d2v)
TFM(d2v=d2v,...) # maybe PP = 0 for no post processing

It seems the clip already contains the necessary flags to handle it with ScriptClip.

Test clip (laundry sequence, soft telecine with bad edits on scene change): https://drive.google.com/file/d/1dwa...usp=share_link

file = "soft telecine.mkv"

LWlibavVideoSource(file, repeat=false)

ScriptClip(last, 
\ """

	fieldbased = propGetAny("_FieldBased")
	parity = GetParity(current_frame)
	
	if (!IsInt(fieldbased)){fieldbased="undefined"}
	else if (fieldbased==0){fieldbased = "progressive"}
	else if (fieldbased==1){fieldbased = "bottom field first"}
	else if (fieldbased==2){fieldbased = "top field first"}
	
	if (!IsBool(parity)){parity="undefined"}
	else if (parity==true){parity = "top field first"}
	else if (parity==false){parity = "bottom field first"}

	SubTitle("_FieldBased: " + fieldbased + "\n" + "GetParity:   " + parity, font="courier new", size=24, lsp=10)
	
\ """)

LanczosResize(720, 540)

Notably the _FieldBased frame property stays "progressive" the whole time, except for those combed frames on scene change.

It seems LWLibavVideoSource is the only source filter which actually sets the _FieldBased and parity values for each frame.

But it should be a trivial matter of returning a deinterlaced clip when _FieldBased != progressive.

Hard telecined is still problematic but I'm already getting ideas
edit: it looks like hard telecined doesn't contain any information to detect orphaned fields -- both _FieldBased and Parity remain locked at the same value for all frames.

[flossy_cake]

Here's an updated version which handles the combing detection issue with a latching system where if a scene contains 4 or more combed frames, the remainder of the scene is treated as combed until the next scene at which point the latch is reset.

# Convert 30i to 60p
# 3:2 sections IVTC'd to 24p and 3:2'd
# 2:2 sections weaved to 30p and 2:2'd
# 1:1 sections deinterlaced to 1:1


global isCombed = false
global combedCount = 0
global yDiff = 0
global sceneThresh = 35
global lastScene = 0
global lastSceneYDiff = 0
global lastFrameNum = 0
global latchDeint = false


file 			= "https://drive.google.com/file/d/1KdE0np8016zr7ovRassdbH7OtbKU7e-a/view?usp=sharing"
global orig30		= LWLibavVideoSource(source=file, stream_index=-1, repeat=true, cache=false)
global bwdif60		= bwdif(orig30, field=3, thr=2)
global tfm24as60	= TFM(orig60, mode=5, PP=0, cthresh=9, MI=80, scthresh=100,
			\ slow=2, ubsco=false, mmsco=false, micmatching=4, display=false)
		
Select(tfm24as60)
LanczosResize(720,540)


function Select(clip c)
{
	c = ScriptClip(c, 
	\ """
	
		global yDiff = YDifferenceToNext(tfm24as60, -2)
		global isCombed = IsCombedTIVTC(tfm24as60, cthresh=9, MI=64)
		
		if (yDiff > sceneThresh || current_frame > lastFrameNum+2 || current_frame < lastFrameNum-2 ){ 
			global latchDeint = false
			global lastScene = current_frame
			global lastSceneYDiff = yDiff
			global combedCount = 0
		}
		
		if (isCombed){
			global combedCount = combedCount + 1 
		}
		
		if (!latchDeint && isCombed && combedCount >= 3){
			global latchDeint = true
		}
		
		global lastFrameNum = current_frame
		
		debug = "frame: " + String(current_frame)
		\ + "\n" + "yDiff: " + String(yDiff)
		\ + "\n" + "isCombed: " + String(isCombed) + "\n" + "combedCount: " + String(combedCount)
		\ + "\n" + "latchDeint: " + String(latchDeint) + "\n" + "lastScene: " + String(lastScene)
		\ + " (" + String(lastSceneYDiff) + ")" 
							
		if (latchDeint || isCombed) { 
			bwdif60.SubTitle("BWDIF \n" + debug, font="courier new", size=24, lsp=10)
		}
		else { 
			tfm24as60.SubTitle("TFM \n" + debug, font="courier new", size=24, lsp=10)
		}
		
		
	\ """ , after_frame=true)
	
	return c
}

edit: I'll be reworking this considerably over the next few days. I'll try to make something more generic as well so it should work for any 25i or 30i file, hopefully it can function as a "one size fits all" general purpose deinterlacer.

[SaurusX]

I've mentioned it before, but a feature of TFM I'd love to see is a "pattern bias" feature. It would attempt to keep to a defined telecine pattern as it compares the combing metrics for each field combination. The pattern would be adhered to if the matches pass the deinterlacing test even if another field combo has a slightly better metric. I'm hoping this would make field matching for shaky animation telecine transfers better and not have those random chunky frames where the two matched fields are basically identical.

[flossy_cake]

Originally Posted by SaurusX

and not have those random chunky frames where the two matched fields are basically identical.

That sounds like an issue I'm having when using bwdif on 3:2 sections. For some reason it generates a weird looking aliased frame every now and then, as if it had just copied field 1 into field 2 or something. Here are 2 consecutive frames coming out of bwdif:

https://i.ibb.co/pnRHYjy/2-bad.png
https://i.ibb.co/68MdYtP/1-good.png

The second one is how an interpolated field should look. The first one is too aliased for some reason. Now here are the same 2 frames coming out of TDeint:

https://i.ibb.co/3Fxdqk9/3-good.png
https://i.ibb.co/YD2ZBPx/4-good.png

Now that looks better, just interpolation as I would expect to see. So perhaps bwdif is only good for 1:1 content? When I think about it, 3:2 is kind of a strange beast to deinterlace what with the pattern of 2 combed frames followed by 3 progressive frames. I'm sure that is somehow making things confusing for the deinterlacer.

edit: seems to be an issue with LWLibavVideoSource, as the issue goes away when using FFmpegsource2 with the same clip. Hard telecined clips works fine with both though.

edit2: false alarm. The issue appears to be with the clipping software that I used to generate the short clips from the full 22 minute episode (XMediaRecode, which uses ffmpeg). It must have mucked up the repeat field flags during the clipping process, because everything is working fine with the full 22 minute clip and both LWLibav & FFMpegSource2, except for the latter with rffmode=1, which causes the issue to occur again. Also the full episode shows more consistent frame rates and timecodes, in relation to:

Originally Posted by hello_hello

LWLibavVideoSource("D:\soft telecine.lwi", repeat=true)
29.97fps, 317 frames, 10.577 sec

LWLibavVideoSource("D:\soft telecine.lwi")
29.97fps, 256 frames, 8.541 sec

FFVideoSource("E:\soft telecine.mkv", cachefile="D:\soft telecine.ffindex", threads=1, RFFMode=1)
29.97fps, 318 frames, 10.610 sec

FFVideoSource("E:\soft telecine.mkv", cachefile="D:\soft telecine.ffindex", threads=1)
24.3789fps, 256 frames, 10.5 sec

The frame rate for the second ffms2 example is usually the indicator there's repeat flags being ignored, because there's hard telecined or video sections too, making the average frame rate non standard. It'll also mess with the audio sync. I don't know why lsmash didn't output the same frame rate.

Sorry about that, I guess we can't use short clips to debug this kind of issue, which is really annoying.

edit3: forgot to mention, must use the latest version of BWDIF which contains some fixes, and use it like this:

propDelete("_FieldBased") # latest version bwdif requires this
BWDIF(field=-2) #-2 = auto detect field order, which is necessary to avoid above aliasing problem

[flossy_cake]

Originally Posted by SaurusX

I've mentioned it before, but a feature of TFM I'd love to see is a "pattern bias" feature. It would attempt to keep to a defined telecine pattern as it compares the combing metrics for each field combination.

Are you sure it's not TDecimate causing that? TDecimate has [mode2, rate, maxndl] params which should do what you are after. With some of my videos I need to use this configuration to get a consistent decimation pattern.

[SaurusX]

Originally Posted by flossy_cake

Are you sure it's not TDecimate causing that? TDecimate has [mode2, rate, maxndl] params which should do what you are after. With some of my videos I need to use this configuration to get a consistent decimation pattern.

Pretty sure. TFM is the one combining the fields into a full frame and it's the one giving a string of frames for TDecimate to choose from. That combined frame is the problem since two essentially identical stacked fields will of course have a lower combing metric.