syncing IVTC and frameblending... (neuron2 / tritical)

[scharfis_brain]

Maybe this is a weird question...

I almost finished a script, that reliably distinguishes between 24p, 30p, and 60i contents on NTSC stuff in order to do a fully automated NTSC->PAL conversion, that cares for those three types of video and lets the user decide, which kind of conversion is done with each type of video (changefps/convertfps or mvcompensated).

But I stumbled over a little surprise: Switching back and forth between IVTCed and fieldblended introduces weird stutter, if the IVTCed video is not in phase with the fieldblended video.

here some pseudocode (only 24p and 60i handling) for a NTSC -> PAL speedup conversion:


i=xxxsource("blah.xxx").assumeXff()

bob=i.somebob()

is24p=bob.check24p()
# this creates a full white 64x64 pixels clip, if 24p is detected,
# and a black one, if 24p patter recognition fails.

is24p=is24p.changefps(23.976*2)
#reduce this flag-clip to double FILM-rate

video=bob.convertfps(23.976*2)
film=i.fieldmatcher().decimator().selectevery(2,0,0)
# create 47.952 fps clips

conditionalfilter(is24p, film, video, "averageluma()",">","200")
# select the correct mode according to the flag-clip is24p

assumfps(50)
resize()
reinterlace()


the problem that occures here is now, that the fieldmatching generates 30 fps and the decimator reduces it to 24 fps. Those 24 fps need to doubled to 48 fps to be able to switch between the blended and the IVTCed video.
But this halving/doubling does throw away the temporal phase information of the 3:2 Pattern of the source video.

I already tried various decimators on this (pseudo code, again):
smartfieldmatchingbob().decimate(fps=47.952)
but it throws out a pattern like this:
3131313131313 or if I have some luck I get my wished 222222222222 pattern.

can a decimator be modified in a way, that it smoothes out patterns at its best while maintaining the sync to methods like convertfps?

[tritical]

Don't have an answer for your decimation stuff atm, but what method are you using for distinguishing between 24p and 60i? Is it a per frame method or per cycle method? Are you looking for 1 repeated field in every 5 of the same parity? How do you plan to find 30p parts? Are you using a combed frame detection method? I've looked at some other blind source detection routines (mainly dscaler code), but haven't seen any that would reliably/correctly identify on a per frame basis all three types. Usually they would either get 24p/60i right but detect 30p as interlaced, or they would get 24p/30p right but sometimes miss 60i.

[scharfis_brain]

here is my detection routine:
loadplugin("d:\x\DGDecode.dll")
loadplugin("D:\x\tivtc.dll")
loadplugin("D:\x\masktools.dll")

#initializincg soma vars
th=8
global fac=1.5
global off=0.0
global max=2.0

global la5=0
global la4=0
global la3=0
global la2=0
global la1=0
global la=0

global lb5=0
global lb4=0
global lb3=0
global lb2=0
global lb1=0
global lb=0



function check24p(float l0, float l1, float l2, float l3, float l4, float l5)
{
x0=((((l0+off)*fac) < l2) && (((l0+off)*fac) < l4) && \
(((l1+off)*fac) < l2) && (((l1+off)*fac) < l4) && \
(((l3+off)*fac) < l2) && (((l3+off)*fac) < l4) && \
(((l5+off)*fac) < l2) && (((l5+off)*fac) < l4))&& \
((l0<max) && (l1<max) && (l3<max) && (l5<max)) ? true : false

x1=((((l0+off)*fac) < l1) && (((l0+off)*fac) < l3) && \
(((l2+off)*fac) < l1) && (((l2+off)*fac) < l3) && \
(((l4+off)*fac) < l1) && (((l4+off)*fac) < l3) && \
(((l5+off)*fac) < l1) && (((l5+off)*fac) < l3))&& \
((l0<max) && (l2<max) && (l4<max) && (l5<max)) ? true : false

x2=((((l1+off)*fac) < l0) && (((l1+off)*fac) < l2) && (((l1+off)*fac) < l5) && \
(((l3+off)*fac) < l0) && (((l3+off)*fac) < l2) && (((l3+off)*fac) < l5) && \
(((l4+off)*fac) < l0) && (((l4+off)*fac) < l2) && (((l4+off)*fac) < l5))&& \
((l1<max) && (l3<max) && (l4<max)) ? true : false

x3=((((l0+off)*fac) < l1) && (((l0+off)*fac) < l4) && \
(((l2+off)*fac) < l1) && (((l2+off)*fac) < l4) && \
(((l3+off)*fac) < l1) && (((l3+off)*fac) < l4) && \
(((l5+off)*fac) < l1) && (((l5+off)*fac) < l4))&& \
((l0<max) && (l2<max) && (l3<max) && (l5<max)) ? true : false

x4=((((l1+off)*fac) < l0) && (((l1+off)*fac) < l3) && (((l1+off)*fac) < l5) && \
(((l2+off)*fac) < l0) && (((l2+off)*fac) < l3) && (((l2+off)*fac) < l5) && \
(((l4+off)*fac) < l0) && (((l4+off)*fac) < l3) && (((l4+off)*fac) < l5))&& \
((l1<max) && (l2<max) && (l4<max)) ? true : false

x = (x0 || x1 || x2 || x3 || x4) ? true : false
return x

}

function check30p(float l0, float l1, float l2, float l3, float l4, float l5)
{

x2=((((l0+off)*fac) < l1) && (((l0+off)*fac) < l3) && (((l0+off)*fac) < l5) && \
(((l2+off)*fac) < l1) && (((l2+off)*fac) < l3) && (((l2+off)*fac) < l5) && \
(((l4+off)*fac) < l1) && (((l4+off)*fac) < l3) && (((l4+off)*fac) < l5))&& \
((l0<max) && (l2<max) && (l4<max)) ? true : false

x4=((((l1+off)*fac) < l0) && (((l1+off)*fac) < l2) && (((l1+off)*fac) < l4) && \
(((l3+off)*fac) < l0) && (((l3+off)*fac) < l2) && (((l3+off)*fac) < l4) && \
(((l5+off)*fac) < l0) && (((l5+off)*fac) < l2) && (((l5+off)*fac) < l4))&& \
((l1<max) && (l3<max) && (l5<max)) ? true : false

x = (x2 || x4) ? true : false
return x

}

mpeg2source("video.d2v") #,cpu=6)

i=last.crop(0,60,0,-60,align=true)

a=i.bob() #or a smartbobbed clip for better recognition in almost static scenes

# prepare the flag
black=a.blankclip(width=64,height=64)
white=black.invert()

# I use a motionmask for pattern recognition
global ba=a.motionmask(thy1=th,thy2=255,thSD=255,u=0,v=0).undot()
# temporally shift that clip to be able to do a backwards pattern search, too (bad edit recognition)
global bb=ba.trim(5,0)+ba

c=scriptclip(stackvertical(stackhorizontal(black,black),stackhorizontal(black,black)), \
"stackvertical(stackhorizontal(check24p(la,la1,la2,la3,la4,la5) ? white : black, \
check30p(la,la1,la2,la3,la4,la5) ? white : black),\
stackhorizontal(check24p(lb,lb1,lb2,lb3,lb4,lb5) ? white : black, \
check30p(lb,lb1,lb2,lb3,lb4,lb5) ? white : black)).subtitle(string(la))")

c0=c.frameevaluate("")
c1=c0.frameevaluate("")
c2=c1.frameevaluate("")
c3=c2.frameevaluate("")
c4=c3.frameevaluate("")
c5=c4.frameevaluate("la=averageluma(ba)")
c6=c5.frameevaluate("la1=la")
c7=c6.frameevaluate("la2=la1")
c8=c7.frameevaluate("la3=la2")
c9=c8.frameevaluate("la4=la3")
c10=c9.frameevaluate("la5=la4")
c11=c10.frameevaluate("lb=averageluma(bb)")
c12=c11.frameevaluate("lb1=lb")
c13=c12.frameevaluate("lb2=lb1")
c14=c13.frameevaluate("lb3=lb2")
c15=c14.frameevaluate("lb4=lb3")
c16=c15.frameevaluate("lb5=lb4")
c17=c16.frameevaluate("")
c18=c17.frameevaluate("")
c19=c18.frameevaluate("")
cx=c19.frameevaluate("")

overlay(a,cx,opacity=0.8,y=200)


this script overlays a dark square over the video.
it is divided into four quandrants:

left top: past 24p check
left bottom future 24p check

right top: past 30p check
right bottom: future 30p check.

for final decisions top and bottom are logically linked with OR.
if all quadrants are black, the video is considered 60i.

of course there needs to be applied some jitter and collision protection of these quadrants (flags) to be implemented.

my strategy is: "favourize 60i, when unsure."

[Mug Funky]

"favourize"... lol, i like this word :) perhaps "favour" would work better... but i'm splitting hairs...

i've encountered the phase problem before (on a much, much less sophisticated version of the same thing you're doing), and, yes, it's a brain twister.

consider a scene where there's a 60i graphic on a 3:2 scene. how do you handle that one, even with perfect 3:2/30p/60i detection?

i notice that even on Snell and Wilcox whitepapers, they say "garbage in - garbage out", and that's when referring to a simple 3:2 pattern.

the problem could be sidestepped by favouring 60i conversion more than film conversion - ie if there's the slightest doubt, treat as 60i. but that will of course give suboptimal results on almost everything (except 60i).

however, the problem is partly solved by treating the video in distinct chunks. i have no idea how to script this, but if you were to look ahead several frames, and only do IVTC if there's a good run of telecine, then the switching between conversion modes will happen seamlessly, and if not it'll only mismatch on static areas (which are preserved well no matter what technique is used).

[mg262]

consider a scene where there's a 60i graphic on a 3:2 scene. how do you handle that one, even with perfect 3:2/30p/60i detection?Masks?

i have no idea how to script this, but if you were to look ahead several frames, and only do IVTC if there's a good run of telecine...
I have no idea how to script it either ;) but for what it's worth there is a programming technique called dynamic programming which is a) the one really clever idea in algorithmics and b) solves this issue perfectly. So it could be added if this thing were pluginized.

[Mug Funky]

Masks?

solves some of the problem, but a hard mask would look bad and a soft mask would display some weirdnesses due to the blended 3:2 pattern being merged with the sped-up IVTC pattern. it probably wont be noticable on playback (or at least not annoying), but i believe it's the problem that scharfi was posting about.

a programming technique called dynamic programming which is a) the one really clever idea in algorithmics and b) solves this issue perfectly.

i find your opinions interesting and wish to subscribe to your newsletter :). i'm not quite sure what you have in mind (i've only ever been a hobby programmer, and that was in high school, so there's a lot of theory missing in my knowledgebase), but it sounds like you have a plan...

[mg262]

Heh... kind of. DP per se is essentially two-pass, and very few people here put up with that. I need to run tests with building something similar that is one pass (for field blend reversal), but I'm being pretty slow about getting round to doing it, probably because I don't actually have a source that I want reversed ATM.

DP is in algorithms textbooks, but I don't know of any good online treatments; I might try to write about it properly someday but it would take a bit of work. I think I have a IRC log from discussing it with an mplayer dev, which I could send if you wanted it... but it's very long and he had a strong maths background, so I'm not sure how useful it would be.