easy manual constant pattern deblending

[scharfis_brain]

two days ago, I found a really simple way to deblend fieldblended movies (cinematic productions!) without any jerkyness and much better rendering speed.

The approach is quite simple:

Example for PAL-Film blended into NTSC 59.94i:

AVISource("video.avi")
assumetff()
bob() # bob-deinterlace
changefps(25*4,linear=false) # oversample the framerate to four times of the original framerate (25fps)
selectevery(4,x) # x= 0...3 / try those values until the output is smooth.

Example for NTSC-Film blended into PAL 50i:

AVISource("video.avi")
assumetff()
bob() # bob-deinterlace
changefps(23.976 * 10,linear=false) # oversample the framerate to TEN times of the original framerate (23.976 fps)
selectevery(10,x) # x= 0...9 / try those values until the output is smooth.

Example for NTSC-progressive (29.97p) blended into PAL 50i:

AVISource("video.avi")
assumetff()
bob() # bob-deinterlace
changefps(29.97 * 10,linear=false) # oversample the framerate to TEN times of the original framerate (29.97 fps)
selectevery(10,x) # x= 0...9 / try those values until the output is smooth.



the more close the original framerate is to the containers framerate the more oversampling needs to be applied eg.:
PAL 23.976 fps in 50 fps 10x oversampling
versus
NTSC 25 fps in 59.94 fps 4x oversampling


Enhanced video quality can for sure be achieved by replacing the rather simple bob() by
leakkernelbob(order=?,threshold=7) or even better
tdeint(mode=1,full=false)

[Mug Funky]

that's a pretty cool idea. it should be real easy to hack pattern-matching into there - as soon as a blend is detected (using a relatively simple metric?) the pattern shifts by 1.

[scharfis_brain]

also tried automatic pattern recognition.
(choosing the pattern offset with the highest edgemasked averageluma)

on some clips, it works like a charm, but on others, it fails big time.

Altogether, One could do it semi-automatic:
1st step) pattern finder: show which pattern has highest averageluma
2nd step) let the script process the entire movie (or selecting of it where the pattern is constant) with a fixed offset.

For PAL-Film to NTSC conversions, it should work without problems, becaus PAL-Film (commonly) never changes parity...
even after edits...

[Mug Funky]

that's true - PAL encodes from NTSC sources are usually done all in 1 go, so the pattern never changes.

the only exceptions are when edits are done after encoding. this is very rare and generally only happens if there's gratuitous sex that has to be cut post-encoding (Mezzo Forte for instance... they sent us the uncut dub and it's quite eye-popping, and would be refused classification without editing). people can be so prudish :)

[MOmonster]

@scharfis_brain
Very nice idea. Don´t thought that it would be still so fast.
There are two points because I won´t your function that way. First I think it isn´t that easy to decide what is the right value for selectevery. You have to look very close to the source. But more important, not one of my sources show a constant pattern.
But your idea is still very useful for me. Maybe I could combine this function with another function (smoothmotion) I just just work on, to avoid the use of a decimater. Don´t know if it will work.
also tried automatic pattern recognition.
(choosing the pattern offset with the highest edgemasked averageluma)

on some clips, it works like a charm, but on others, it fails big time.
I know two other methods to choose the pattern offset, that work better for me, but there would be still much problems with some sources, because they don´t seems to have any regular pattern.

@Mug Funky
that's a pretty cool idea. it should be real easy to hack pattern-matching into there - as soon as a blend is detected (using a relatively simple metric?) the pattern shifts by 1.
No, it´s really not that simple. For many sources I have, the pattern shift by 1 won´t be enough and if you only simple trust the blenddetection, you won´t get the smooth motion you want.

[mg262]

No, it´s really not that simple. For many sources I have, the pattern shift by 1 won´t be enough and if you only simple trust the blenddetection, you won´t get the smooth motion you want.But if 1 isn't enough it will detect another blend and shift by another 1... and so on until it stops finding blends?

[MOmonster]

@mg262
Yes, of course theoretical this is possible, but the first point is: In what direction would you shift the pattern offset? It depends on the editing, what direction is the right. But more problematic is the blenddetection. In many cases if you use x, or x+1 or x-1 it is possible that you won´t detect any blends, because the blendlevel is just to small, but you will see the small difference in the motion.

Edit: It is not hard to make my Cdeint avoid all clear blends, but than getting a smooth motion is just the bigger problem.

[scharfis_brain]

take a look at this quickly hacked code (no opimisations done, just proof of concept)
Also I was too lazy to translate my comments... Sorry

loadplugin("d:\x\dgdecode.dll")
loadplugin("d:\x\masktools.dll")
loadplugin("d:\x\tdeint.dll")

#framecache fürs smooth-scrolling
import("D:\x\fc.avs")

#simple edge-mask, aus R24 geklaut
function edgemask(clip i)
{ logic(i.DEdgemask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5",setdivisor=true,divisor=6),
\ i.DEdgemask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5",setdivisor=true,divisor=6),"max",u=1,v=1)

}

# liefert 0...9 zurück
function max10index(float i0, float i1, float i2, float i3, float i4, float i5, float i6, float i7, float i8, float i9)
{ x= (i0>=i1) && (i0>=i2) && (i0>=i3) && (i0>=i4) && (i0>=i5) && (i0>=i6) && (i0>=i7) && (i0>=i8) && (i0>=i9) ? 0 : \
(i1>=i0) && (i1>=i2) && (i1>=i3) && (i1>=i4) && (i1>=i5) && (i1>=i6) && (i1>=i7) && (i1>=i8) && (i1>=i9) ? 1: \
(i2>=i0) && (i2>=i1) && (i2>=i3) && (i2>=i4) && (i2>=i5) && (i2>=i6) && (i2>=i7) && (i2>=i8) && (i2>=i9) ? 2: \
(i3>=i0) && (i3>=i1) && (i3>=i2) && (i3>=i4) && (i3>=i5) && (i3>=i6) && (i3>=i7) && (i3>=i8) && (i3>=i9) ? 3: \
(i4>=i0) && (i4>=i1) && (i4>=i2) && (i4>=i3) && (i4>=i5) && (i4>=i6) && (i4>=i7) && (i4>=i8) && (i4>=i9) ? 4: \
(i5>=i0) && (i5>=i1) && (i5>=i2) && (i5>=i3) && (i5>=i4) && (i5>=i6) && (i5>=i7) && (i5>=i8) && (i5>=i9) ? 5: \
(i6>=i0) && (i6>=i1) && (i6>=i2) && (i6>=i3) && (i6>=i4) && (i6>=i5) && (i6>=i7) && (i6>=i8) && (i6>=i9) ? 6: \
(i7>=i0) && (i7>=i1) && (i7>=i2) && (i7>=i3) && (i7>=i4) && (i7>=i5) && (i7>=i6) && (i7>=i8) && (i7>=i9) ? 7: \
(i8>=i0) && (i8>=i1) && (i8>=i2) && (i8>=i3) && (i8>=i4) && (i8>=i5) && (i8>=i6) && (i8>=i7) && (i8>=i9) ? 8: 9
return x }


mpeg2source("the-wall.d2v",cpu=0)
crop(0,80,0,-80,align=true)
i=assumetff()

# framerate auf 10 zielframerate aufblasen
global ix=i.tdeint(mode=1,type=2,full=false)
global ix=ix.changefps(23.976 * 10,linear=false)

# wieder auf zielframerate mit den Offsets 0..9 reduzieren
ix0=ix.selectevery(10,0)
ix1=ix.selectevery(10,1)
ix2=ix.selectevery(10,2)
ix3=ix.selectevery(10,3)
ix4=ix.selectevery(10,4)
ix5=ix.selectevery(10,5)
ix6=ix.selectevery(10,6)
ix7=ix.selectevery(10,7)
ix8=ix.selectevery(10,8)
ix9=ix.selectevery(10,9)



# edgemask bauen, 10*zielframerate
iy=i.bob(0,0.5).reduceby2().edgemask().reduceby2().changefps(23.976*10,linear=false)

bl=8 # temporaler blur radius
#offsettisieren, edgemasks temporal smoothen... (wie sauber bei pattern offsets/szenenwechseln ist das?)
global i0=iy.selectevery(10,0).temporalsoften(bl,255,255)
global i1=iy.selectevery(10,1).temporalsoften(bl,255,255)
global i2=iy.selectevery(10,2).temporalsoften(bl,255,255)
global i3=iy.selectevery(10,3).temporalsoften(bl,255,255)
global i4=iy.selectevery(10,4).temporalsoften(bl,255,255)
global i5=iy.selectevery(10,5).temporalsoften(bl,255,255)
global i6=iy.selectevery(10,6).temporalsoften(bl,255,255)
global i7=iy.selectevery(10,7).temporalsoften(bl,255,255)
global i8=iy.selectevery(10,8).temporalsoften(bl,255,255)
global i9=iy.selectevery(10,9).temporalsoften(bl,255,255)

#averagelumas der Offsets 0..9 ins Video kritzeln
ann0=scriptclip(i.changefps(23.976),"""ix.selectevery(10,max10index(l0,l1,l2,l3,l4,l5,l6,l7,l8,l9)).subtitle("0: "+string(l0),y=32) \
.subtitle("1: "+string(l1),y=64) \
.subtitle("2: "+string(l2),y=96) \
.subtitle("3: "+string(l3),y=128) \
.subtitle("4: "+string(l4),y=160) \
.subtitle("5: "+string(l5),y=192) \
.subtitle("6: "+string(l6),y=224) \
.subtitle("7: "+string(l7),y=256) \
.subtitle("8: "+string(l8),y=288) \
.subtitle("9: "+string(l9),y=320) \
.subtitle("Max: "+string(max10index(l0,l1,l2,l3,l4,l5,l6,l7,l8,l9)),y=352)""")
ann1=ann0.frameevaluate("l0=averageluma(i0)")
ann2=ann1.frameevaluate("l1=averageluma(i1)")
ann3=ann2.frameevaluate("l2=averageluma(i2)")
ann4=ann3.frameevaluate("l3=averageluma(i3)")
ann5=ann4.frameevaluate("l4=averageluma(i4)")
ann6=ann5.frameevaluate("l5=averageluma(i5)")
ann7=ann6.frameevaluate("l6=averageluma(i6)")
ann8=ann7.frameevaluate("l7=averageluma(i7)")
ann9=ann8.frameevaluate("l8=averageluma(i8)")
anneliese=ann9.frameevaluate("l9=averageluma(i9)")


# anneliese ausgeben, und gucken welcher wert am GRÖSSTEN ist. dieses offset ist dann das richtige
#anneliese

# video mit richtigem offset ausgeben
# die auswahl sollte natürlich im cond. env. geschehen!
# und zwar mit min./max. prüfung der 10 avglumas (min zu max abstand muss 4 bis 6 offset-werte betragen! [symmetrie])
# ausserdem muss der min-max unterschied groß genug sein, wenn nicht, pattern blind weiterlaufen lassen!

#ix8

#siehe oben. Framecache. Witschtisch!
fc(100)

the clip anneliese (sounds similar to Analyse in German, hehe) returns a clip showing the average averageluma of the edgemasks of each pattern offset overlaid over the offset-clip with the highest averageluma.
when scrolling through the clip, one will see that some movies have a constant pattern, like Pink Floyd-The Wall. (its pattern only breaks two times!)
upon this detected value, on could use one of the variables
ix0 ... ix9 for final non-analysed output.

with automated blend / pattern detection I see one major problem here:

the offset 9 shows the highest avgl, but in the next frame it is offset 0 (measuring error!)
what happens? A jerk occurs! because offset 0 is too far away from 9!
instead of 0 the value 10 had to be choosen. but how should one implement this?

[scharfis_brain]

because the blendlevel is just to small, but you will see the small difference in the motion.

could one use motionmask on the outputted frames to check for fluid motion?
jerky motion will give a more uneven temporal avgluma-curve....

[MOmonster]

@scharfis_brain
Your function looks really interesting. Is the method with egdemask accurate enough or does it fails often?
If I have a little bit more time I´ll do a check for fluid motion. So or so a nice idea. :D

[scharfis_brain]

the edgemask is just for testing. I stripped it out of R24.
in r24 there is a lot more code for the edgemask. Eg. for supressing static areas etc...

also the output of max10index needs to be smoothed in order to avoid huge orphan deviations.

also one could also do a reverse check: min10index
and compare it with max10index (they have to be in a distance of 5 offset values)

further there also could be a value-check, if the maximum and miniumum value aren't different enough: just create functions called max10value & min10value
if the difference between min & mex is too low, just let the pattern proceed blindly until the min-max-difference resumes above a certain threshold.

Maybe it is also useful to extend to pattern range:
(bold == current range == one pattern cycle)

-9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

the extension may become necessary to avoid jerkyness while jumping from 9 to 0 (will become 9 to 10) or from 0 to 9 (will become 0 to -1)

but again the probleme here: how to ensure that the offsets are ALWAYS centered in 0...9 an only rarely leave the center?!?

[MOmonster]

And so fast the function gets more and more complex. :D
But it´s worth a try I think. We´ll see.

[mg262]

@mg262
Yes, of course theoretical this is possible, but the first point is: In what direction would you shift the pattern offset? It depends on the editing, what direction is the right. But more problematic is the blenddetection. In many cases if you use x, or x+1 or x-1 it is possible that you won´t detect any blends, because the blendlevel is just to small, but you will see the small difference in the motion.@MOmonster,

Point taken on direction. What kind of blend level (as a rough percentage) do you reckon is too small to detect? I was finding that automated detection was better than my eye at finding blends... although my eye is not very good at this ;) . It is of course possible that this is true and that the residual jerk in motion is visible by eye.

@scharfis_brain

Would you like minimum and maximum index functions (taking variable numbers of arguments)? Together with anything similar that you can think of that messes with floats/integers, they should be quick to build.
__________

Reading the last few posts reminds me of one thing that's been troubling me for a while, namely that detecting blends -- or rather the blend pattern -- is not a local problem. What I mean is that a decision taken early on in the clip as to say whether frame 50 is a blend or not can have ramifications much further down the line, because it can potentially affect the number of frames in the output. (I.e. affect whether the first 250 blended frames should yield 239 or 240 frames of output... and of course that decision affect whether all subsequent flames are 'slid' forward by one or not.) So in order to obtain e.g. frame 10,000 of the output, you will ideally have to analyse about 10,000 frames of input. So the problem is not well-suited to AVISynth type processing as opposed to linear processing. Of course there are various hacks and rules of thumb which we can/could use to get around this, but nothing really satisfactory.

I'm not writing very clearly at present -- sorry.

[MOmonster]

Point taken on direction. What kind of blend level (as a rough percentage) do you reckon is too small to detect? I was finding that automated detection was better than my eye at finding blends... although my eye is not very good at this. It is of course possible that this is true and that the residual jerk in motion is visible by eye.
I have seen many blends, that just looks a little bit more soften than the next clear field. Especially for low motion scenes, where the differences between the fields are so small, it is hard to recognize the blends. Of course normally the blends around the pattern offset (from clear field first to blend field first) are good visible (because of the position in the pattern). But what, if we have just in this small part no motion? I just mean that this task isn´t that easy.

@scharfis_brain
I tested your method just a little bit. I was surprised, how good it work for one of my sources, but yes it fails a lot for many other sources (I don´t use mvtools, just my eyes). I still think comparing the motion on the small parts where more then one field (the blend) is dropped, could give sometimes better results.
I still play with the idea to use your function in a way combined with my Cdeint. But still the function doesn´t look flexible enough for much really strange conversions. So or so, thanks for your good ideas Scharfis_Brain :D

Edit: Why use the oversampling of ten times and not higher (up till 25), what should be more accurate. Is it to hard to get the small difference between the offset, or doesn´t it work?

[scharfis_brain]

Edit: Why use the oversampling of ten times and not higher (up till 25), what should be more accurate. Is it to hard to get the small difference between the offset, or doesn´t it work?

Because I didn't want to make it slower and 10x already seemed to give a high enough precision.

but feel free to implement it yourself

[MOmonster]

Because I didn't want to make it slower and 10x already seemed to give a high enough precision.
I tested it with 25x and doesn´t see any speed or memory differences. My next Crestore will maybe work with this function, but it is not tht easy to realize. :sly:
Cu

[MOmonster]

@scharfis_brain
I failed to implement it in my function, but I found another solution for it.
Can you please explain me, what the linear parameter in changefps does. For me it doesn´t make any difference if I set it true or false. Are there any little speed differences? :(

[mg262]

Hope you don't mind me butting in...

ChangeFPS can end up skipping frames; so for example it could return frames 0, 1, 2, 3, 5, 6, 7 of the input (skipping frame 4). Setting the linear parameter to true will make it calculate frame 4 (and then throw it away). Don't ask me what the point of this is (unless it helps with cache effects)... that's not to say that it doesn't have a point, by the way, just that I don't understand it.

If you are only applying spatial operations before ChangeFPS, linear =false should be faster because it has to calculate fewer frames. HTH,

M.

[MOmonster]

@mg262
thanks for the answer. So I will use linear=false for my function. ;)