Field unblending without bobbing (works, but help needed to improve)

[thetoof]

Well, the thread title says it... here are some results to show that the concept can work:
http://img252.imageshack.us/img252/6420/working0005co2.th.jpg (http://img252.imageshack.us/my.php?image=working0005co2.jpg)http://img252.imageshack.us/images/thpix.gif (http://g.imageshack.us/thpix.php)http://img373.imageshack.us/img373/4545/working0008gl6.th.jpg (http://img373.imageshack.us/my.php?image=working0008gl6.jpg)http://img373.imageshack.us/images/thpix.gif (http://g.imageshack.us/thpix.php)

However, it doesn't work in a few cases, one being if the same field is blended for 2 successive frames:
http://img518.imageshack.us/img518/5404/not0000nb6.th.jpg (http://img518.imageshack.us/my.php?image=not0000nb6.jpg)http://img518.imageshack.us/images/thpix.gif (http://g.imageshack.us/thpix.php)http://img147.imageshack.us/img147/4082/not0001eu9.th.jpg (http://img147.imageshack.us/my.php?image=not0001eu9.jpg)

This is what it currently does (based on this (http://forum.doom9.org/showpost.php?p=1192032&postcount=253) script, mod in the next post):
- separatefields + selecteven/odd
- blend detection on the even field clip
- blend detection on the odd field clip
- if a frame in the even field clip is declared blended, drop that field in the source clip and use the odd field to do spatial interpolation of the missing field
- opposite happens if odd field is blended
- if none are blended, use source frame

Problem:
The even/odd fields are compared together, so if two successive top (or bottom) fields are blended, it'll screw up blend detection and the function won't work as expected

Solution:
Compare the even and odd fields of the same frame to declare them as clear or blended

Concept (and this is where I need help :p):
- assumetff().separatefields() to make blend calculation possible on the fields (that "became" frames)
- compare the fields of the same frame for blend detection (after previous operation, separatefields' frame 0 & 1 (top and bottom fields of source's frame 0), frame 2 & 3 (top and bottom fields of source's frame 1) and so on.
Since both fields are very similar if the frame is clear, the comparison could be about "are they almost duplicates or not"
- compare even frame to next frame (let's say it's frame 32 & 33, corresponding to even and odd field of source's frame 16)
- if even (32) is considered as a blend, source's frame 16's top field is blended
- discard top field of source's frame 16 and use interpolation to replace it (like nnedi(0))
- compare odd frame to the previous frame (to compare the even (top) field of the same frame)
- if odd (33) is considered as a blend, source's frame 16's bottom field is blended
- discard bottom field of source's frame 16 and use interpolation to replace it (like nnedi(1))
- If none are blended, use source frame
- If the difference with previous or next frame is very small (duplicate), use that other frame instead of the one created with interpolation (interpolated frame will most likely contain less detail than the duplicate)

Stream to play with (http://www.mediafire.com/download.php?jgi2morjvvy)
Script to play with (or totally rewrite :p) is right below

Improvements to consider if the concept ever works fully:
- Using cheap (faster) interpolation before checking if neighbor frame is a duplicate of the current one & then replace current // if not, use good interpolation
- Use mo-comp interpolation instead of spatial-only (nnedi for now)
- Speed improvements with optimal conditional environment usage

Have fun! ^_^

[thetoof]

Here goes:
function bcalcmod(clip source, float "speed", float "blocks", float "thresh", bool "chroma", int "cache", clip "dclip")
{
###### parameters & other necessary vars ######
rad = isfloat(speed) && speed>=1 ? sqrt(speed)*16 : 48
cache = default(cache,-1)
bsize = isfloat(blocks) && blocks>=1 ? sqrt(blocks)*4 : 16
sourcetop = source.assumetff().separatefields().selecteven()
sourcebottom= source.assumetff().separatefields().selectodd()
global thrx = abs(default(thresh,22))*0.1
global bfacx = sqrt(bsize*0.25)*0.54


###### lut & more ######
#top preparation
detxt = sourcetop.killaudio().converttoyv12()
detxt = detxt.pointresize(int(width(detxt)/rad+1)*16, int(height(detxt)/rad+1)*16)
detxt = default(chroma,false) ? stackvertical(stackhorizontal(detxt.utoy(),detxt.vtoy()),detxt) : detxt

code = "x 128 == y 128 == | x y + 256 == | 0 x 128 - abs y 128 - abs < x 128 - abs x y + 256 - abs < & x 128 - abs 1 - " +
\ "y 128 - abs x y + 256 - abs < y 128 - abs 1 - x y + 256 - abs 1 - ? ? 2.4 ^ x 128 - y 128 - * 0 > -0.6 0.6 ? * ? 128 +"
difft = mt_makediff(detxt,detxt.trim(1,0),y=3,u=1,v=1)
global bclpxt = mt_lutxy(difft,difft.trim(1,0),expr=code,y=3,u=1,v=1).bilinearresize(round(width(detxt)/bsize+4)*4,round(height(detxt)/bsize+4)*4)

#bottom preparation
detxb = sourcebottom.killaudio().converttoyv12()
detxb = detxb.pointresize(int(width(detxb)/rad+1)*16, int(height(detxb)/rad+1)*16)
detxb = default(chroma,false) ? stackvertical(stackhorizontal(detxb.utoy(),detxb.vtoy()),detxb) : detxb

diffb = mt_makediff(detxb.trim(1,0),detxb,y=3,u=1,v=1)
global bclpxb = mt_lutxy(diffb.trim(1,0),diffb,expr=code,y=3,u=1,v=1).bilinearresize(round(width(detxb)/bsize+4)*4,round(height(detxb)/bsize+4)*4)

###### output preparation ######
global outbt = source.nnedi(0) #keep bottom #do your filtering here, it will be faster and more stable
global outbb = source.nnedi(1) #keep top #if you do it outside the conditional enviroment
global outc = source


###### initialise variables ######
global lfrx = -100


###### evaluation call & output calculation ######
scriptclip( source, """

### preparation ###
cfrx = current_frame
jmpx = lfrx+1==cfrx
lfrx = cfrx

#top preparation
bx_vt = 128-yplanemin(bclpxt)
bx_vt = bx_vt<0 ? 0 : bx_vt * bfacx
cx_vt = yplanemax(bclpxt)-128
cx_vt = cx_vt<0 ? 0 : cx_vt * bfacx

## blend value shifting ##
bxpt = jmpx ? bxct : bx_vt
bxct = jmpx ? bxnt : bx_vt
bxnt = bx_vt

## clear value shifting ##
cxpt = jmpx ? cxct : cx_vt
cxct = jmpx ? cxnt : cx_vt
cxnt = cx_vt

#bottom preparation
bx_vb = 128-yplanemin(bclpxb)
bx_vb = bx_vb<0 ? 0 : bx_vb * bfacx
cx_vb = yplanemax(bclpxb)-128
cx_vb = cx_vb<0 ? 0 : cx_vb * bfacx

## blend value shifting ##
bxpb = jmpx ? bxcb : bx_vb
bxcb = jmpx ? bxnb : bx_vb
bxnb = bx_vb

## clear value shifting ##
cxpb = jmpx ? cxcb : cx_vb
cxcb = jmpx ? cxnb : cx_vb
cxnb = cx_vb

#top
isblendt = bxct+cxpt>bxpt+cxct+thrx && bxct+cxnt>bxnt+cxct+thrx #&& bxct+cxpt>bxcb+cxpb && bxct+cxnt>bxcb+cxnb #detects nearly every blend but also frames with very low motion
isblendsavet = isblendt && bxct+cxpt>bxpt+(1+.2*thrx)*cxct && bxct+cxnt>bxnt+(1+.2*thrx)*cxct && bxct>(1+.2*thrx)*cxct #saver blenddetection

#bottom
isblendb = bxcb+cxpb>bxpb+cxcb+thrx && bxcb+cxnb>bxnb+cxcb+thrx #&& bxcb+cxpb>bxct+cxpt && bxcb+cxnb>bxct+cxnt #detects nearly every blend but also frames with very low motion
isblendsaveb = isblendb && bxcb+cxpb>bxpb+(1+.2*thrx)*cxcb && bxcb+cxnb>bxnb+(1+.2*thrx)*cxcb && bxcb>(1+.2*thrx)*cxcb #saver blenddetection

### output clip ###
isblendt ? outbt : isblendb ? outbb : outc
#outc.subtitle(string(isblendt)).subtitle(string(isblendsavet),y=16).subtitle(string(isblendb),y=32).subtitle(string(isblendsaveb),y=48) #to compare isblend and isblendsave

""")


###### final decimation & caching ######
cache<0 ? last : last.RequestLinear(8, cache, 5, false, false)

return last
}

[MOmonster]

It won't work this way.

Example: c_b c_b c_d b_c b_c d_c

even: c c c b b d
odd: b b d c c c

There is no function that can detect these double-blends easily. For the blenddetection you need the neigboured fields.

-> Use seperatefields or for a better detection bob.reduceby2.

In the last frame there is no blend, but not touching this frame means that it will stay interlaced. Is this wanted, or do you want to match the right fields?

I think you want to avoid interpolation if it is not necessary. But this is nothing else than matching if possible.

[MOmonster]

You have to be a bit more clear about the usage of this wanted function. It's not a problem to write such a script, but what would be the usage.

Your idea sounds pretty similar to my very old Cdeint function. This function compared the two fields of one frame and interpolated the one with the lower blend-possibility. The output was used as postprocessor for tfm, so if possible the fields were weaved else deblended.
Sounds like a nice concept, but it fails because of three reasons:

1. Blenddetection wasn't very save (not the big problem anymore)
2. Decimation didn't work well with the conditional function (also not such a big problem anymore)
3. This problem is still present:
example: ab b bc c c d de e e ef #fields
possibel output:
1. b c c e e #frames
2. b c d e e #frames
3. b b c c c d e e e e #fields

The output of Cdeint was the first one.
The third output would be also no problem. If there is no blend the frame is not touched.
I think the second output is the one you want. But with all the necessary calculations it will have the same speed (or slower) as srestore. And the problem with the right decimation is still left.

[thetoof]

You have to be a bit more clear about the usage of this wanted function.
You're right, here is what I want to do with the concept:
-field matching preprocessor (to avoid unnecessary bobbing)
-bobbing preprocessor (to avoid feeding blended fields to the bobber (unless I'm mistaken, it's not very good for mo-comp))

The optimal script would look like this:
1-Detect the blends on the bob.reduceby2 clip
2-Compare the clear field to the previous and next ones
3-If they are identical (meaning the frame is a duplicate (it will mainly be used on anime, so it's frequent)), replace the blended field with it's corresponding clear neighbour
4-If the neighbour frames are different, discard the blended field and replace it by an interpolated one

5-Apply subsequent processing, including external decimation (iirc srestore compares the frames to choose the "best" duplicate... but that's something I'd like to get into after seeing if the above concept can be applied).

The first problem I had when I tried writing the script was that I don't know how to analyse a (cheap) bobbed clip (or separate field) clip and apply filters on the source clip (half framerate) in scriptclip.

Thanks a lot for your time.