blend cancellation - would this work?

[Mug Funky]

hi all.

i've got a bit of a problem with field-blended anime that restore24 hasn't been able to solve: 2 or even 3 consecutive blended fields.

it seems to me that it would be possible to treat these blends somewhat by subtracting the previous or next field at the right opacity. sure there may be a little posterization, but most of us use temporal filters that would probably eliminate any small artefacts left over.

my problem is this: i'm not sure how to go about determining the right opacity to use. there are a lot of different blend-boxes out there, and they don't all work the same. also, regular patterns would be hard to determine, and are likely to change easily (or drift out of sync, with awful results).

at this point i'm just after throwing some ideas around, but it seems to me a practical solution is possible, and hopefully it would run at a decent speed.

btw, i'm not expecting to be able to pull 60i stuff out of field-blended PAL, but 23.976 or 25 good frames per second would be enough for me.

[edit]

perhaps after detecting that a frame is a blend, averageluma for the blend, the previous and the next could be determined and scaled? hopefully this would correspond to the amount of blending, and could be used usefully.

i'm currently having a look to see if i can mod restore24 a tad, but to be honest, that code's hard to get into from where i'm sitting :scared:

[Didée]

Mug, don't break your head too much over Restore24: it is not suited for that kind of multiple-field blending. It's supposed to work on "clean" (clean - HEHE,HARHARR!) blend sequences where each field has (should have) a clean neighbor on both sides.

The kind of cancellation you're thinking of theoretically is possible. But only if there is a stable pattern of blends occurances. If there's no clear pattern, the chances of doing an automated blend reckognition are more than poor.

[Mug Funky]

okay... after a lot of messing around, i seem to have a way of DETECTING blends, but what i actually wanted was to cancel them...

but i'm heartened by the results here, because i'm sure it was just a simple typo i have to track down before i get blend cancellation working.

i'm not completely sure my logic is working, but for what it's worth here's the current script in a VERY alpha stage...


### attempt at blend cancellation below...
global a=1
global b=1
global c=1
global blend=1
global expr="x y + 2 /"

function blendcancel (clip clip)
{

global gl_c = clip

clip = clip.frameevaluate("""b=averageluma(gl_c)"+chr(13)+
\ "a=averageluma(gl_c.duplicateframe(0))"+chr(13)+
\ "c=averageluma(gl_c.deleteframe(0))"+chr(13)+
\ "blend=(c-b)/(c-a)"+chr(13)+
#\ "blend=(blend > -1)&&(blend < 1)? 1-blend : 0 "+chr(13)+
\ "expr="x "+string(blend)+" * y "+string(blend)+" 1 - * +"""",after_frame=true)

#clip.scriptclip("subtitle( string( blend ) )")

clip.scriptclip("(blend<-1)||(blend>1)? gl_c :
\ (blend>0)?
\ yv12lutxy(gl_c,gl_c.invert().duplicateframe(0),yexpr=expr,uexpr=expr,vexpr=expr,Y=3,U=3,V=3) :
\ yv12lutxy(gl_c,gl_c.invert().deleteframe(0),yexpr=expr,uexpr=expr,vexpr=expr,Y=3,U=3,V=3)")

}



[edit]

okay... looks like the expression for YV12LUT isn't actually being evaluated. bugger.

[edit 2]

looks like the best result i can get out of this is a NOP... anybody not baffled by the conditional filtering environment is welcome to pick at this script... i'm rapidly getting too tired to think.

[edit 3]

fixed syntax error :)... not doing what i'd like yet, but getting better.

[scharfis_brain]

do you mean this way of detecting blends?
http://forum.doom9.org/showthread.php?s=&threadid=75772&highlight=unblend

[Mug Funky]

Scharfi: i'm not trying to detect them, i'm trying to cancel them out by blending them in the opposite direction so to speak.

[edit]

it does seem to use the same kind of concepts though... i'll read it more thoroughly and try it out later :)

[edit 2]

this works, but the goddamn thing is running 1 frame behind, and adding any deleteframe/duplicateframe stuff seems to turn it into a NOP.

### attempt at blend cancellation below...
global a=1
global b=1
global c=1
global blend=1
global expr="x y + 2 /"

function blendcancel (clip clip)
{

global gl_c = clip.deleteframe(0)

gl_c = gl_c.frameevaluate("""b=averageluma(gl_c)"+chr(13)+
\ "a=averageluma(gl_c.duplicateframe(0))"+chr(13)+
\ "c=averageluma(gl_c.deleteframe(0))"+chr(13)+
\ "blend=(c-b)/(c-a)"+chr(13)+
\ "expr="x "+string(blend)+" * y "+string(blend)+" 1 - * +"""",after_frame=true)

gl_c.scriptclip("(blend<-1)||(blend>1)? gl_c :
\ (blend>0)?
\ yv12lutxy(gl_c.invert(),gl_c.duplicateframe(0),yexpr=expr,uexpr=expr,vexpr=expr,Y=3,U=3,V=3) :
\ yv12lutxy(gl_c.invert(),gl_c.deleteframe(0),yexpr=expr,uexpr=expr,vexpr=expr,Y=3,U=3,V=3)"+chr(13)+"
\ subtract(gl_c,last)")

}


[edit 3]

hmmm... maybe there isn't a lag. it's possible my maths is off, but i can't quite tell at the moment.

it doesn't seem to be catching "forward blends" but is doubling up on "backward blends"

[edit 4]

actually, it might be both :) on some bits my maths looks okay but behind by 1 frame, and on other bits it looks like the maths is right but i'm not :confused:

well, anyhoo, i'm going to bed. it's WAY too late here, so i think i'll pass the baton to anyone willing to crunch away at this idea (plugin writers? hellooooo? :D )

[Mug Funky]

hmm... bumping my own post. i'm so l33t...

this script is a proof-of-concept. thanks to didee, i sorted the frameevaluate problems (at least to a level that allows me to continue), and i seem to be getting de-blended output with this.

however, the averageluma method of determining the blend amount is a little simplistic, and will fail on some scenes (the upward pans of a cloudy blue sky in the intro of "battle doll angelic layer" R4 PAL, more specifically, but basically any situation where 2 frames and their blend have nearly identical average luma).

try it on some blended content... i'm definitely open to ideas on how to refine the blend-amount method, but i'm pleased that this works as well as it does.


### attempt at blend cancellation below...
global a=1
global b=1
global c=1
global blend=1
global expr="x y + 2 /"

function blendcancel (clip clip)
{
global c10=clip
global c99=clip

#c99 = scriptclip(c10,"subtitle(string(blend))")
c99 = scriptclip(c10,"
\ (blend<=0)||(blend>=1)? c99 :
\ (blend>=0.5)?
\ yv12lutxy(c10,c10.deleteframe(0),yexpr=expr,uexpr=expr,vexpr=expr,Y=3,U=3,V=3) :
\ yv12lutxy(c10,c10.duplicateframe(0),yexpr=expr,uexpr=expr,vexpr=expr,Y=3,U=3,V=3)
\ "+chr(13)+"subtitle(string(blend))")

c50 = frameevaluate(c99,"""expr="x "+string(blend)+" * y 1 "+string(blend)+" - * +"""",after_frame=true)
c40 = frameevaluate(c50,"global blend = (a-b)/(a-c)")
c30 = frameevaluate(c40,"global c = averageluma(c10.deleteframe(0))")
c20 = frameevaluate(c30,"global b = averageluma(c10)")
c10 = frameevaluate(c20,"global a = averageluma(c10.duplicateframe(0))")

return c10
}


[edit]

by the way, just looking at the output, this will need some kind of tolerance parameter, as it'll detect a small amount of blending occasionally and deblend it the wrong way, leading to a posterized image (for 1 frame). i'll try to track this one down ASAP.

hmmm... maybe if i test edge-detected clips, i can find a way to determine blending amount?

[edit 2]

added some globals... i always seem to miss them with ctrl+c when they lie above the actual function.

[Didée]

Too much post editing, Mug ;)

c50 = frameevaluate(c99,"""expr="x "+string(blend)+" * y 1 "+string(blend)+" - * +"""",after_frame=true)Once this line has grown up, it might become a yv12LUTxy expression. Or a pilot, or a doctor ... who knows :D


Oh sh... Before criticizing any syntax, I should read the code correctly :o

But nonetheless above function reports

"Evaluate: unary minus can only be used with numbers ([ScriptClip], line 1)"

And that message is lying on top of another error message. This 2nd error message disappears if I define more globals:

global a=0.0
global b=0.0
global c=0.0
global blend=0.0
global expr=""

But the "unary minus" message persists ...

[Mug Funky]

edited once again, just for didee :)

i forgot to copy the globals.

the unirary minus thing is something i haven't seen before, but try the script again now and it should work.

i might add that it only seems to cancel blends properly in a _very_ limited set of circumstances, so i implore the doom9 faithful to throw some alternate metrics at me, because i'm not ready to abandon this line of speculation just yet - i haven't mined it completely.

oh, and there's still the problem of ye olde two-or-three-blends-in-a-row to deal with. i suspect a more intelligent script will help choose which frame to blend from in a better way, and for the 3-frames blended situations, well, they're usually not very noticable on at least 1 of those, and besides, we could always run restore24 after this...

my aim here is to come up with something intriguing enough to get one of our plugin-writers thinking. i can't stand a slow encode, but haven't learnt C or C++ (yet...).

[Didée]

Still the same error message - but wait.

Could be that everything is fine - I'm here @ a rather suspicious machine ATM. I'll try at home later, and report back tomorrow.


Originally posted by Mug Funky
my aim here is to come up with something intriguing enough to get one of our plugin-writers thinking.
I hold thumbs.
Actually, Restore24's (present) design isn't very complicated. It just looks so because of the script's length and so, but that are the needs of Avisnth-scripting all needed stuff. The *core* itself is pretty simple, it's almost ridiculous. However, nobody picked it up.

(And, you're currently de-blending only. Try also decimating & getting a fluid output :D)

[Mug Funky]

hehe.. i'm not holding my breath on the plugin front (indeed, if i were really desperate, i'd just get off my arse and learn C++).

IMO fluid motion after de-blending is impossible unless we somehow restore the original framerate (maybe possible in some instances). even restore24 gives unavoidable jerks (my filter gives way more, as it'll catch some blends but not others, causing a similar effect to wrong-field order on some bits).

the whole reason the blending is there is to give smooth motion, so obviously removing it but keeping the destination framerate will produce jerks.

after i tweak this a little, i might hack it into restore24 and come up with some kind of "restore24mod" that only uses blend-cancellation when there's no good fields to grab onto.

2 or 3 consecutive blends happens more often than you might think, at least with my stuff. i concede that ALL my field-blended stuff comes from the same authoring house (madman), and most of it from the same frame-rate converter box (they upgraded to a new box last year, actually, just after FLCL disc 3 R4 was released, but i haven't noticed a difference).

i can make samples available if it'll help things.

[edit]

in my opinion, blend-removal is becoming less of a problem, because the encoders are getting better and we can leave things interlaced. right now i'm encoding a fansub to blend-converted-interlaced SVCD, with an average bitrate of 1300 kbps, and it's actually looking good, and keeping to ~ Q3 (using peter cheat modded ffmpeg/QuEnc)

[Didée]

Okay, I got the script technically working, after cleaning the little mess with variable names. ;)
(Could it be that Avisynth 2.55 has some racing problems if global and local variables of the same name appear?)

- c10 & c99 were present as globals (=input for conditional fuctions) as well as function privates (=members of the 'conditional cascade'). Renaming the globals helped a lot.

- my input clips were named "a" or "b" (Restore24+Blendbob script framework...) Although these (private) clip variables should not interfere with the globals a,b,c used in the function, they did nonetheless. Renaming the input helped a lot.

Having made these changes, the script works for me without reporting errors. Went on to testing on some natural sources - no comic or animee.

Well ... Mug, did >you< test it with natural input? My results are ... funky. I get irregular frame blendings, some forward/backward jumpings, and such stuff. Success rate is rather negative than zero.

Small test:
BlendCancel (http://www.spselectronic.com/english/~temp/BC.zip) (2.3 MB, 50fps/bobbed)
Restore24 (http://www.spselectronic.com/english/~temp/R24.zip) (1.4 MB, 23.976fps restored)

used script:
global a=1
global b=1
global c=1
global blend=1
global expr="x y + 2 /"

function blendcancel (clip clp)
{
global in=clp
global in=clp

c99 = scriptclip(in,"
\ (blend<=0)||(blend>=1)? in :
\ (blend>=0.5)?
\ yv12lutxy(in,in.deleteframe(0),yexpr=expr,uexpr=expr,vexpr=expr,Y=3,U=3,V=3) :
\ yv12lutxy(in,in.duplicateframe(0),yexpr=expr,uexpr=expr,vexpr=expr,Y=3,U=3,V=3)
\ ")

c50 = frameevaluate(c99,"""expr="x "+string(blend)+" * y 1 "+string(blend)+" - * +"""",after_frame=true)
c40 = frameevaluate(c50,"global blend = (a-b)/(a-c)")
c30 = frameevaluate(c40,"global c = averageluma(in.deleteframe(0))")
c20 = frameevaluate(c30,"global b = averageluma(in)")
c10 = frameevaluate(c20,"global a = averageluma(in.duplicateframe(0))")

return c10
}

mpeg2source("test.d2v")
assumeTFF()
crop(0,80,-0,-0)

bbb=KernelBob(5)

blendcancel(bbb)
#Restore24(bbb,bbb)

return last # .trim( range )

[Mug Funky]

that's okay, didee. i got the same kind of results with anime. it doesn't seem to catch the right blends.

that's all down to the metrics though. if there were some good way of not only detecting if something is a blend or not (like restore24's very cool edge masking idea), but to determine exactly how blended it is, this idea would have some legs. i can't think of anything at the moment using the things available within conditionalfilter, but maybe some kind of windowed scanning? (i'm clutching at straws here).

i don't think this idea is ready to be abandoned just yet. maybe if i re-learn some of the maths i've forgotten over the years, i might find a good function for getting blend-amount out of edge masks.

[Didée]

Ah, okay.

I've also thought about somehow determining blend weights and then try to undo the blending - but that's tricky. Fixed major weightings, like 25% 50% 75% steps or so, perhaps could be squeezed out of the edgemasks, too.

Also, the idea of windowed searching held me busy for some time. But that seems impossible to do inside the conditional environment of an avisynth script: I tried to do it with a recursive script, but never managed to get beyond "access violation" :/

[Mug Funky]

didee, sometimes i worry about you :eek:

i think this is one to put on hold until either i learn C++ or somebody decides to try it out in a plugin. for now it's an idea...

an idea to get around the skipping back-and-forth in time - how about doubling the framerate, then doing both previous and next "unblending", then reducing the framerate down to the original 59.94 using the best of the interpolations.

i still haven't thought of a good way around the 3 consecutive blends problem... however, i think when i see a 3rd blend, it's actually an artefact of the mpeg-2 encoding - when interlace patterns ring, they tend to affect both fields, giving a blended look. with that in mind, we only have to worry about 2 fields in a row, which is a much easier problem to deal with (pick the best field to use in the unblending).

[edit]

here's a per-pixel blend detection that is much more accurate. i haven't done anything with it yet, but with the help of some histogram functions (like getting the higest significant spike in the histogram) this will give much, much better blend detection.

it's pretty to watch, too :)

CminusB = yv12lutxy(last.deleteframe(0),last,yexpr="x y - 128 +")
CminusA = yv12lutxy(last.deleteframe(0),last.duplicateframe(0),yexpr="x y - 128 +")
result = yv12lutxy(cminusb,cminusa,yexpr="x 128 - y 128 - / 256 *").yv12lut(yexpr="x 128 < x 0 ?").greyscale()
result