Properly deinterlace mostly telecined content

[pcroland]

Hi!

I have videos where most of the content (like 99%) are telecined but have some animations and transitions that are regular interlaced frames. If I do a TFM().TDecimate() I'll have nice progressive frames but sometimes get interlaced frames. If I use QTGMC for the whole thing then I lose some resolution. Is there a way apply IVTC and then detect frames that are still interlaced and only deinterlace those frames?

Sample:
https://x0.at/hAf.mkv
https://mega.nz/file/DB5jCYiY#J89Tsm4E93dkHI6mZ4J2nCJec0whmH7EXL-d8nyNEhM

Thanks in advance.

[johnmeyer]

The PP settings in TFM are your friend.

[poisondeathray]

TFM will only return 29.97 max

A mix of 23.976p, 30p, 59.94p can be done with

TDeint(mode=1,tryweave=true, edeint=QTGMC())
#dedup 1st pass
#dedup 2nd pass

It will try to weave progressive content, but deinterlace interlaced content with QTGMC , all in a 59.94p base (23.976p sections will have 3:2 duplicates, 29.97p sections will have duplicates, if you had 12p sections they would have 5x duplicates...you get the idea)

Run 2pass dedup afterwards, this will decimate duplicates and output timecodes to make unique frames, and VFR timing to keep sync

[pcroland]

TFM will only return 29.97 max

A mix of 23.976p, 30p, 59.94p can be done with

TDeint(mode=1,tryweave=true, edeint=QTGMC())
#dedup 1st pass
#dedup 2nd pass

It will try to weave progressive content, but deinterlace interlaced content with QTGMC , all in a 59.94p base (23.976p sections will have 3:2 duplicates, 29.97p sections will have duplicates, if you had 12p sections they would have 5x duplicates...you get the idea)

Run 2pass dedup afterwards, this will decimate duplicates and output timecodes to make unique frames, and VFR timing to keep sync
Thank you very much! I thought it will be much harder. Now I need to decide whether I want encode it as 23.976 or 59.94fps or VFR. I really don't like VFR but since I want to upscale the whole series with waifu2x-ncnn-vulkan it would be much faster. Thanks again.

[Overdrive80]

As poison proposes, maybe after TDeint you can try Srestore instead of dedup.

[pcroland]

I would lose frames at 59.94fps parts with Srestore but I would get alternating frame lengths at 23.976fps parts with DeDup (33.3 / 50 ms). From my tests the alternating frame lengths are not really noticable.

[hello_hello]

According to TIVTC, just under 11% of that episode is video. Those interlaced transitions must add up. I encoded the series myself about 2 years ago, and tried picking out the interlaced sections manually while creating a timecodes file from scratch, but if memory serves I eventually realised there was too much I missed, so for most episodes, the second time around I just let TIVTC do it's thing for a variable frame rate. Those encodes have been sitting on my hard drive waiting for me to check them for 6 months now, so I had a look at the episode from which your sample came. TIVTC seems to have done a good job (using QTGMC to de-interlace). I'll have to encode it again though because there's banding I would've fixed if I'd noticed it. It's in the source so I'm not sure why I didn't. Anyway....

TFM glitched in one spot, for which I can probably blame myself because I was aware of the problem but apparently forgot to give TFM a hand before encoding. To save you a "gotcha", look out for frame #3434. It has a field that doesn't belong. The text hasn't started scrolling. Three frames later you can see it has.

https://i.postimg.cc/B86Q3WT7/01.jpg (https://postimg.cc/B86Q3WT7)

https://i.postimg.cc/FYGswFLJ/02.jpg (https://postimg.cc/FYGswFLJ)

I upscaled a little with nnedi3 myself, although some episodes contain such a plethora of encoding artefacts I almost didn't bother (the first episode was so bad I spent time replacing bad frames with good duplicates), but partly because I'm not overly crazy about more clever upscaling, at least not yet. I'd be interested to learn if the result is better, and if you've a clever way of dealing with the encoding artefacts.

Later versions of TDecimate have an argument called orgOut that seems to be designed to help with the creation of a timecodes file with the video sections at 59.94fps, but I've not played with it myself. I'll have to encode my effort again to fix the problems I mentioned, but here's a small VFR sample upscaled to 720p for anyone who cares (it's just the first minute or so). Personally, I prefer VFR encoding for hybrid sources.

Drawn.zip (https://www.udrop.com/53do/Drawn.zip) (25.6 MB)

First pass:

DClip = QTGMC(FPSDivisor=2)
TFM(Output="0215TFM.txt", Clip2=DClip)
TDecimate(Output="0215TD.txt", Mode=4, Hybrid=2, MKVOut="0215TimeCodes.txt")

2nd pass:

DClip = QTGMC(FPSDivisor=2)
TFM(Input="0215TFM.txt", Clip2=DClip)
TDecimate(Input="0215TD.txt", Mode=5, Hybrid=2, MKVOut="0215TimeCodes.txt", TFMIn="0215TFM.txt")
# Other filtering here.

[poisondeathray]

I would get alternating frame lengths at 23.976fps parts with DeDup (33.3 / 50 ms). From my tests the alternating frame lengths are not really noticable.

This is a form of "judder."

If you watched 23.976 CFR on a 60Hz display you'd get similar frames with different display times

I didn't test to verify this - but if you use ChangeFPS(120000,1001) before dedup, it will repeat frames in a "base" of 120000/1001. This should even out the timestamp deltas, and all typical content framerates are evenly divisible into it (integer result) . But if you watch that on a 60Hz display , you still will get judder from the display

[pcroland]

TFM glitched in one spot, for which I can probably blame myself because I was aware of the problem but apparently forgot to give TFM a hand before encoding. To save you a "gotcha", look out for frame #3434. It has a field that doesn't belong. The text hasn't started scrolling. Three frames later you can see it has.
TDeint also has that small glitch. It doesn't really bother me. I guess it can't detect it because there are just a few pixels that the interlacing is visible on compared to the whole image.
1st pass:
DClip = QTGMC(FPSDivisor=2)
TFM(Output="0215TFM.txt", Clip2=DClip)
TDecimate(Output="0215TD.txt", Mode=4, Hybrid=2, MKVOut="0215TimeCodes.txt")

2nd pass:

DClip = QTGMC(FPSDivisor=2)
TFM(Input="0215TFM.txt", Clip2=DClip)
TDecimate(Input="0215TD.txt", Mode=5, Hybrid=2, MKVOut="0215TimeCodes.txt", TFMIn="0215TFM.txt")
# Other filtering here.
I got this error for this:
https://i.kek.sh/lFFVunHDGOY.png


With this script however:
FFMS2("source.mkv")
TDeint(mode=1, tryweave=true, edeint=QTGMC())
#DupMC(log="dupmc.txt")
DeDup(threshold=1.2, maxcopies=20, maxdrops=10, log="dupmc.txt", times="dedup.txt")
I extracted the frames with ffmpeg, upscaled them with waifu2x-ncnn-vulkan (-s 2 -n 3) and encoded them with x264 tMod:
https://0x0.st/-PXh.mkv
https://mega.nz/file/2R5yhJxa#QyZvBPA5UEGzRKklsad5uRfMyhnac9PmFgoor7vivE4
It was just a quick test to see if everything works and it looks great in my opinion. I'll encode both the original and the waifu2x upscaled version (I know not everyone is a fan of this smoothed out upscaled image)
And I have another question:
If I encode with an mkv output both mediainfo and ffmpeg reports 59.94fps CFR and after a remux with mkvmerge they properly report VFR. What would be the correct workflow? Should I pass "--tcfile-in" for x264 and also specify the timecodes file when I remux it or should I only specify the timecodes file at encoding? Both will result in proper playback but I don't know if there is any difference at all.

[poisondeathray]

If I encode with an mkv output both mediainfo and ffmpeg reports 59.94fps CFR and after a remux with mkvmerge they properly report VFR. What would be the correct workflow? Should I pass "--tcfile-in" for x264 and also specify the timecodes file when I remux it or should I only specify the timecodes file at encoding? Both will result in proper playback but I don't know if there is any difference at all.


Either works. The difference is x264 is frame rate aware. If you use CRF encoding, it will adjust the quantizers according the the framerate with tcfile-in. Higher quantizers for higher fps sections. If you encoded it as CRF as CFR, it will be slightly different allocation

[pcroland]

Hmmm. Progressive frames that have interlaced fading will make the image jump up and down:
https://i.kek.sh/VxYHxiCv5km.gif
Can I combat this somehow? Or should I separate the fields and match the brightness before using TDeint so it doesn't detect the frame as interlaced in the first place??

[poisondeathray]

Not sure, maybe use ovr , overrides

[pcroland]

What do you mean by ovr?

[poisondeathray]

What do you mean by ovr?

ovr is the overrides file path

Look in the documentation , there are examples

There might be other settings you could play with; ovr is usually "last resort" - if you do too many ovr, you might as well do it manually anyways (just specify sections that are ivtc'ed , double rate deinterlaced, and write the timestamp ranges in v2 format)

[pcroland]

I found it meanwhile. It would take many hours since this show has 36 episodes. I guess it's fine like this, I can live with this at least, already much better than what I was starting with. Thanks for all the help :-)

[hello_hello]

There's no need to use the timecodes file when muxing if x264 is writing it's output directly to MKV and you've used --tcfile-in, although if the output is a raw h264 stream, I'm not sure.

I'm short on internet time today, so a quick post to say I thought the quality of your upscaled encode looked quite good. The artefacts in the source I never found a way to remove completely without sacrificing picture detail.... did the upscaling take care of that in the process, or was there other filtering involved?

Sometime later today I'll try a 2 pass encode with TIVTC to see if I have the same problem and report back. I'm not sure I've bumped into that exact error myself. I did have a quick look at the old metrics files, and TFM seemed to be in agreement with TDecimate.

The last few lines of the TDecimate metrics file:


38065 0 0
38066 0 0
38067 0 0
38068 0 0
38069 0 0
38070 0 0
38071 0 0
38072 512 0

From TFM:

38068 c - [0]
38069 c - [0]
38070 c - [0]
38071 c - [0]
38072 c - [0]
#
#
# OVR HELP INFORMATION:
#
# [COMBED FRAMES]
#

One other quick thought. FFMS2 runs in VFR mode by default. Repeat flags are ignored and it decodes at the average frame rate. If there's no soft telecine it should be okay, but to tell FFMS2 to behave as DGIndex/DGDecode does and honour repeat flags....
FFMS2("source.mkv", rffmode=1)
Just in case you weren't aware.

##### int rffmode = 0
Controls how RFF flags in the video stream are treated; in other words it's equivalent to the "field operation" mode switch in DVD2AVI/DGIndex.
Valid modes are:

- **0**: Ignore all flags (the default mode).
- **1**: Honor all pulldown flags.
- **2**: Equivalent to DVD2AVI's "force film" mode.

Note that using modes 1 or 2 will make `FFVideoSource` throw an error if the video stream has no RFF flags at all.
When using either of those modes, it will also make the output be assumed as CFR, disallow vertical scaling and disallow setting the output colorspace.
`FFPICT_TYPE` will also not be set as the output is a combination of several frames.
Other subtle behavior changes may also exist.

[pcroland]

"did the upscaling take care of that in the process, or was there other filtering involved?"
Yes, I didn't do anything yet besides TDeint and Dedup.

[pcroland]

These are my scripts for the whole process if anyone is interested:

upscale.sh
#!/bin/bash

temp='temp'
output='output'
path_ffmpeg='ffmpeg.exe'
path_x264='x264.exe'
path_waifu2x_ncnn_vulkan='waifu2x-ncnn-vulkan.exe'
tg_token='asd'
tg_chat_id='asd'

current_frames() {
sleep 3
while true; do
current_frames=$(ls "$temp"/frames_upscaled | wc -l)
percent=$(bc <<< "scale=2; $current_frames*100/$frames")
delta_upscale=$(($SECONDS - start_time))
fps=$(bc <<< "scale=2; $current_frames/$delta_upscale")
printf '\rupscaling frames \e[93m%s/%s [%s%%]\e[0m, FPS=\e[93m%s\e[0m' "$current_frames" "$frames" "$percent" "$fps"
sleep 0.5
done
}

print_separator() {
printf '%.0s─' $(seq 1 "$(tput cols)")
}

notification() {
printf 'sending telegram notification...\n'
curl -s -X POST https://api.telegram.org/bot"$tg_token"/sendMessage -d chat_id="$tg_chat_id" -d parse_mode=html -d text="\
<code>${i}</code>%0A\
running dupmc with avsmeter took <code>${delta_dupmc}</code> seconds.%0A\
extracting frames took <code>${delta_extract}</code> seconds. (${size_extract})%0A\
upscaling <code>${frames}</code> frames with <code>${fps}</code> FPS took <code>${delta_upscale}</code> seconds. (${size_upscale})%0A\
merging upscaled frames took <code>${delta_merge}</code> seconds. (${size_merged}/${size_merged_all})" > /dev/null
}

for i in "$@"; do
b=$(basename "$i")
printf '\e[92m%s\e[0m\n' "$b"
rm -rf "$temp"/frames_extracted "$temp"/frames_upscaled
mkdir -p "$temp"/frames_extracted "$temp"/frames_upscaled "$output"

start_time=$SECONDS
printf 'running dupmc with avsmeter...'
AVSMeter.exe -o ${i%.*}_dupmc.avs &> /dev/null
end_time=$SECONDS
delta_dupmc=$((end_time - start_time))
printf ' took \e[93m%s\e[0m seconds.\n' "$delta_dupmc"

start_time=$SECONDS
printf 'extracting frames...'
"$path_ffmpeg" -v quiet -i "$i" -pix_fmt rgb24 "$temp"/frames_extracted/%06d.png
end_time=$SECONDS
delta_extract=$((end_time - start_time))
size_extract=$(du -s --block-size=M "$temp"/frames_extracted | sed -e 's/\s.*$/iB/')
printf ' took \e[93m%s\e[0m seconds. (%s)\n' "$delta_extract" "$size_extract"

start_time=$SECONDS
frames=$(ls "$temp"/frames_extracted | wc -l)
printf 'upscaling frames'
current_frames &
pid=$!
"$path_waifu2x_ncnn_vulkan" -n 3 -i "$temp"/frames_extracted -o "$temp"/frames_upscaled &> /dev/null
kill -PIPE "$pid"
end_time=$SECONDS
delta_upscale=$((end_time - start_time))
size_upscale=$(du -s --block-size=M "$temp"/frames_upscaled | sed -e 's/\s.*$/iB/')
fps=$(bc <<< "scale=2; $frames/$delta_upscale")
printf '\rupscaling frames \e[93m%s/%s [100.00%%]\e[0m, FPS=\e[93m%s\e[0m took \e[93m%s\e[0m seconds. (%s)\n' "$frames" "$frames" "$fps" "$delta_upscale" "$size_upscale"

start_time=$SECONDS
printf 'merging upscaled frames...'
"$path_x264" --qp 0 --preset veryfast --tcfile-in ${i%.*}.txt --output "$output"/${b%.*}.mkv source.avs &> /dev/null
end_time=$SECONDS
delta_merge=$((end_time - start_time))
size_merged=$(du -s --block-size=M "$output"/${b%.*}.mkv | sed -e 's/\s.*$/iB/')
size_merged_all=$(du -s --block-size=M "$output" | sed -e 's/\s.*$/iB/')
printf ' took \e[93m%s\e[0m seconds. (%s/%s)\n' "$delta_merge" "$size_merged" "$size_merged_all"

notification
print_separator
done

xy_dupmc.avs:
FFMS2("path\upscale\source\01.mkv")
TDeint(mode=1, tryweave=true, edeint=QTGMC())
DupMC(log="01_dupmc.txt")

xy.avs:
FFMS2("path\source\01.mkv")
TDeint(mode=1, tryweave=true, edeint=QTGMC())
DeDup(threshold=1.2, maxcopies=20, maxdrops=10, log="01_dupmc.txt", times="01.txt")

source.avs:
RT_WriteFileList("path\temp\frames_upscaled\*.png", "temp.txt")
a=RT_FileQueryLines("temp.txt")
ImageSource("path\temp\frames_upscaled\%06d.png", start=1, end=Int(a))
ConvertToYV12()

For example if I run
./upscale.sh scripts/01.avs
It runs 01_dupmc.avs with AVSMeter, then extracts the frames from 01.avs with ffmpeg. Then frames inside temp/extracted_frames gets upscaled with waifu2x-ncnn-vulkan to temp/upscaled_frames.
Then x264 converts the frames to a video at the output folder with --qp 0 from source.avs. The source.avs file is static because the temp folders gets removed on every iteration.
(I use --qp 0 because I'll replace live action parts in the upscaled video with the original.)

It looks something like this in shell:
https://i.kek.sh/y2UTzDXJqXa.png

[hello_hello]

Hmmm. Progressive frames that have interlaced fading will make the image jump up and down:
Can I combat this somehow? Or should I separate the fields and match the brightness before using TDeint so it doesn't detect the frame as interlaced in the first place??

I'd forgotten about that. I mentioned hacking into episode one the other day, but I forgot about the eradication of fading whites.

I've got to stay in the real world for most of today so it'll be tonight or sometime tomorrow, but I can dig out the old script so you don't have to hunt down frame ranges from scratch. The first minute of the episode is below. If you can live with the frames I dumped/duplicated and the whites that no longer fade etc, I'll post the Trims from the script, assuming you want them. If you don't like what I did for the first minute I won't be offended.

Drawn 1.zip (https://www.udrop.com/53qk/Drawn_1.zip)