Experimental Fieldblend Reversal/Removal (prev. "RestoreFPS")

[mg262]

Those of you reading this thread for the first time might want to start at this post, which is where the attack on fieldblending starts for real.
____________________________________________

RestoreFPS

Brief description:
Reverses the kind of blending generated by ConvertFPS, restoring original framerate.

Search keywords: ConvertFPS, FPS, framerate, restore, reverse, like unblend, deblend, restore24.

Full description:
RestoreFPS(clip, float fps, float phase) - takes a YV12 clip which has an 'underlying' frame rate of fps but a higher actual frame rate due to blending, and reverses the blending to restore the original frame rate. phase is a number between 0 and 1 which specifies the relative displacement of old and new clips (see example below).

The framerate to restore should be less than the current framerate, and more than half of it. (So restoring from 24 back to 25 and from 24 back to 11 are both illegal.)

The method used is described here.

Examples:
In order to test the filter we need to generate a clip using ConvertFPS. A script like this:

bicubicresize(4*60,3*60)
selectevery(200,0)
converttoyuy2
assumeFPS (24/1.001)
convertFPS (25)
converttoyv12
overlay(crop(0,0,-0, 26). showframenumber)

will do the trick. Some frames from this:



Now running

RestoreFPS(24/1.001, 0.00)

produces a stream like this (do look at the messed up 'frame numbers'):




In this case, the start (in time) of the input clip corresponds exactly to the start (in time) of a frame of the original clip. In other words the situation looks like this:



That won't always be true -- if we take the 'convertFPS' clip above and trim it (say add Trim(10,0)), then we have something more like this:



Dealing with this requires us to guess the amount by which the converted clip is 'slid' with respect to the original, i.e. to guess the phase parameter. I used this script to help with this:

Function phase(clip c, float phi)
{
c
RestoreFPS(24/1.001, phi) #set appropriate frame rate

#trim here if desired

a=selectevery(6,0)
b=selectevery(6,1)
c=selectevery(6,2)
d=selectevery(6,3)
e=selectevery(6,4)
f=selectevery(6,5)

stackhorizontal(a,b,c,d,e,f)
Reduceby2()
stackvertical(last, trim(1,0), trim(2,0), trim(3,0), trim(4,0), trim(5,0))
Trim(0,-1).Loop(0,1000)
subtitle(String(phi, "%f"))
}

#source clip here
animate (0, 100,"phase", 0.0, 1.0)
trim(0, 100)

Save the output of this, open it up in VirtualDub, and drag the slider around to find a phase that looks correct. (You may get an unusual effect in the first few frames of the clip, because of an edge effect.)

Here's a real example to finish up with:

Source (from VHS):


Restored:


(Actually, this example doesn't seem to work exactly like ConvertFPS on a wider scale, but leave that aside for now.)

Notes:
- If you're dealing with a clip in which fields (not full frames) are blended, separate them and process them separately; I think the phase parameters for the top and bottom fields will differ by 0.5 (though I'm not at all sure).

- This isn't optimised; there are several ways in which I could speed it up, and I will if anyone finds it useful.

- If you get the frame rate slightly wrong, this filter will work in a small region and get worse and worse as you drift away from it.

- It will not deal with cuts. You will need to take each cut section and find a phase for it separately. (If the sections are too short, this method may not be of any use.) On the other hand, there are some ways to estimate the phase automatically (esp. autocorrelation), and cuts could be detected by the change in phase.

- Although I've only set it up to reverse ConvertFPS type blends, it will work perfectly well on any regular blend pattern, including the types mentioned by scharfis_brain in this thread. The hard part is analysing the blend type -- I have some scattered thoughts but this post is too long already!

By the way, I'm really sorry I trailed off/disappeared at the end of that last thread... real life got in the way. I'll try and do better with this filter (assuming anyone finds a clip it works on!)

[Didée]

Hey hey, this looks very promising!

Despite the restriction "The framerate to restore should be less than the current framerate, and more than half of it" -- do we here finally have a method to tackle 29.97i -->[fieldblending]--> 25i conversions? Restoring 29.97i probably is out of reach, but perhaps we can get 29.97p without blending?
(I'd have two sources to test that, but they're on the other machine I won't get to the next few days.)

[scharfis_brain]

WOW!

restoring 50fps out of a 50i->60i conversion worked really well.

but the sample I had was interfered by a blending type, I call "negative blending"
(didee knows what I mean...)

restoring 23.976 fps out of a 23.976p->59.94i->50i conversion fails.

this is due to the weird blending pattern and the limitation, that the restored framerate must not drop under the half of the input framerate.

but this type of restoration IMO doesn't need such a kind of deblending.
selecting non-blended fields, like Restore24 deos is fully sufficient, if restoring-framerate is smaller than half of the input framerate.

[mg262]

Proper replies soon, but the half framerate part of the restriction can certainly be removed.

[Mug Funky]

wowee! i'll have to try this out.

i eagerly look forward to an optimised version

[edit]

question: i assume this is correct, but tell me if it's not.

i'd like to recover clean 24fps (NOT 23.976) from old "spinning prism" transfers from film to NTSC. these are the kind that give "peculiarblend" effects. i work with a lot of old, crappy looking anime, and this would be a good treatment to give some of it, if i can spare the time. it's certainly worth experimenting with.

[mg262]

First up, I should say I know much much less about what blend patterns arise than any of you. I'm also in PAL territory so don't know much about IVTC, etc. I need to go and read up on them before I can say anything useful. But here's a quick attempt at replies -- apologies if some of it is confused.

>The framerate to restore should more than half of the current framerate

The reason for this was just that (as @scharfis_brain notes) if you upsample the frame rate by more than a factor of two, all your original frames are preserved unblended, (together with lots of blended frames). So this case is actually much easier to deal with than the other -- you just find the frame(s) in the upsampled version corresponding to the one you want (and you average them if there is more than one). I just didn't think this case would be very useful -- but I will put it in when I have a moment if anyone wants it. You can do something similar as follows:

To get from 25fps back to 11fps:
RestoreFPS(22.0, ?)
And then average every pair of frames (possibly skipping one at the beginning).

In the field-based cases, as I understand it, top fields are only ever made up of original top fields, and bottom fields are only ever made up of original bottom fields. (Please correct me if I'm wrong.) So you can separate them and deal with them individually before interleave/weaving. To be absolutely clear: in the analysis (including finding the phases) it is useful to consider all fields together, but in the actual reversal it's not.

Telecine could probably also be dealt with, but not with the code as it stands -- it assumes regular 'spacing' of blended frames(or fields), which doesn't apply to the ABCDD pattern (the D lasts twice as long as the others).

@Mug Funky
I should also have said that its currently only to reverse ConvertFPS in blend mode -- I forgot that switch mode existed -- sorry. It might well be possible to deal with the other though.

@scharfis_brain
Sounds like you can't reverse the 59.94i->50i step because of telecine?

@everyone
I will keep reading and come back; but if there are a few common blend patterns that it would be really useful to reverse, would you mind posting scripts that generate them?

[kassandro]

I haven't yet tested this filter, but I see a problem with the phase parameter, if the filter is applied in the real world: The phase parameter has to be reset to a new value, if for some reason a frame or an entire mpeg2 GOP is dropped. If a GOP is dropped because of transmission problems the transport stream demuxer will recognise it, but if single frame or a single field is dropped by the broadcaster, then there is no way to realise it.

[mg262]

So, I get the impression that a useful thing to do would be to reverse

23.976 ---telecine--> 29.97 ---ConvertFPS-type-blend--> 25

Just to check before I try to implement anything, looking at the top field only (for now), does that correspond to

[23.976 fps input]
SelectEvery(4,0,0,1,2,3)
ConvertFPS(25)

OR

SelectEvery(4,0,1,1,2,3)
ConvertFPS(25)

OR one of the 2 similar patterns? In other words, is it useful to reverse the effect of the above script(s)?

@kassandro
Yes, that could be a problem, depending on the frequency of the drops. (Do you know how common similar drops are in DVDs as opposed to broadcast material?) Similarly it will mess up the choice from the four telecine patterns mentioned above. The problem is much the same as with cuts (except that when only one frame/field is dropped I know the exact effect on the phase parameter). So I may well need to implement automatic guessing of parameters -- telecine pattern and phase and possibly others, depending on what kind of blend is being reversed. I do think that will be possible, but I'd rather leave it until I have a good sense of what range of types of blend the filter should be trying to reverse.

So for the moment I'm concentrating on the above type of IVTC+unblending and then on speeding things up. (Unless prodded in other directions...)


Edit:
I forgot about the pattern
0,1,2,3,4,4,5,6,7,8,8, ... (which needs a Trim as well as SelectEvery).

[Mug Funky]

hmm... as far as blend patterns go, it gets a little difficult. usually there's no fixed "pattern", and phase is different every time, because realtime units are always running (even if they're standards converting zero input, and outputting PAL green noise, they'll be converting).

if you like, i could run some test signals through the converter at work so you can take a look at how it does things. i'm not sure what kind of test signals you'd prefer, so maybe you could send something (an m2v?) and i can run it through if/when there's some spare time.

if you want to know more, here's it's write-up on the Leitch site:

http://www2.leitch.com/__85256A16005...donotpublishme

(hehe... looks like it's discontinued now. wasn't when i checked 3 weeks ago)

[mg262]

So here's a new version. It should be substantially faster, and it removes the factor of two limitation. I still don't particularly recommended it for that case, as the output is very sensitive to the phase parameter -- i.e. having that even slightly out will cause a few blends to drop through. (So it really requires automated guessing of the blend parameter.) By the way, the filter speed seems to fluctuate substantially -- running it through VDub, closing that down, and then opening VDub again makes it run much faster -- I assume this is some sort of caching effect.

It also incorporates a function that @scharfis_brain thought of:

convertfps(newfps,blend-threshold)
newfps -> this value should be known, hehe
threshold -> only blend two frames together, when the weight of the lower weighted frame is higher than the threshold.

example:
the new frame is between two frames of the input stream and their weights are:
20% and 80%, if I set the threshold to 30%, only the 2nd frame will be in the output.
If I choose a threshold of zero, the output-frame will be a 20:80 - blend of its neighbors.

If you set this imaginary threshold to 45%, only those frames are getting blended, whose temporal position is close to the middle of its neighbors, all other frames are just copies of their nearest neighbors (like changefps does)

Except that I've named the function
ThresholdIncreaseFPS (float fps, float threshold)

(@scharfis_brain: I tried to PM you to find out what you wanted done in the decrease fps case where you can end up blending three or more frames together -- but your inbox was full. So I left it out. it's easily added.)

Because of the structure of the filter adding that function automatically added the reverse function (in the sense that RestoreFPS is the reverse of ConvertFPS). So if anyone has a use for it, I just have to give it a name.

The new function is YV12-only like the others, although at some point I'll add YUY2 (to all of them together).

Anyone want any more weird blends? (Actually, I think the filter probably needs renaming to ReverseBlend or Blend or something like that.)

@Mug Funky
Thank you for the offer of test data -- I may take you up on that at some point later. Part of what I was asking wasn't about the blend pattern, just the telecine pattern (told you I was new to this)... I just wanted to check that the forward telecine script (23.976 --> 29.97) looked something like this

separatefields
e = selecteven.selectevery(4,0,1,2,3,3)
o = selectodd.selectevery(4,0,1,1,2,3)
interleave( e,o )
weave

But the variety of blend patterns may prove a real problem. The 'Motion adaptive' mentions in the converter specifications are particularly awkward. the manual says that 'spatial and temporal apertures... are selected on a pixel by pixel basis' -- there's no sane way to reverse that. For that matter, I've no idea what "the full processing power of IXION’s 10-bit motion adaptive conversion core" does when you feed it with blended (telecined) source.

Edit: threshold is a fraction (e.g. 0.45) not a percentage.

[Wilbert]

Is there any chance you will release the source ?

[mg262]

@Wilbert -- I PMed you a couple of days ago...
_______________

I wrote a version that reversed this kind of thing:

[23.976]
SelectEvery(4,0,1,1,2,3)
ConvertFPS(25)

It works fine on that script [i.e. on SelectEvery(4,0,1,1,2,3).ConvertFPS(25) ] but is useless on a blend sample from another thread. I think it's a red herring, but FWIW, the syntax is RestoreFPSIVTC(29.97, phase=0.0...1.25, telecine=0,1,2,3,4). There's an associated script if anyone wants it.

Slightly more usefully, I realised using a fixed blend pattern (1-lambda 0 lambda) on frames (n-1 n n+1) and subtracting the results from the original (i.e. frame n) was a good way of finding blend weights where frames are duplicated (esp. in animation). The function is Reblend (clip, float weight)

Test with the above 'phase' script, replacing:

RestoreFPS(24/1.001, phi) #set appropriate frame rate

with

cn = overlay(crop(0,0,-0, 26). showframenumber)
subtract(c, Reblend(c, phi))
stackvertical (cn, last)

(Should have said before: Reduceby2 on the source is a good idea before any of these scripts... or if you can Reduceby2.Reduceby2 and then remove the Reduceby2 at the end of the phase function.)

Example:

Only the foreground (the man here) is useful as the background pans every frame. You can see that frame 4616 is approx .47 * 4615 + .53 * 4617.

I need to speed the filter up before this is really useful. SSE2 is coming; requests for SSE/MMX? Actually, I should really build a VDub filter so that you can scroll through the clip and use a separate slider for the weight...

[Inc]

First of all mg262 ... I really do apreciate your developings on such a kind of filter-dll ! Thanks a lot!

In case of simulating a source input: "If I understood right" at Scharfis_Brain's Interlacing-Explanation-Reference, in PAL-Land TV Stations mostly use conversations in two ways (well I know these from here in Germany) based on IF the source comes to the PAL TV stations as 24p or 29,97 (60i telecined):

Simulating in Avisynth:


24p > IVTC > 50i

AVISource("24fpsFilm.avi")
SelectEvery(2,0,0,0,1,1).SeparateFields().SelectEvery(4,1,2).Weave()
DGBob(order=1).ConvertFPS(50).SeparateFields.SelectEvery(4,0,3).Weave()

24p > 50i

AVISource("24fpsFilm.avi")
ConvertFPS(50).SeparateFields().SelectEvery(4,0,3).Weave()

Reference:
http://home.arcor.de/scharfis_brain/...esInterlacing/

Im looking forward to try out your dll tonight
I hope I'll catch a typical broadcasting to capture tonight for testing.

[Alain2]

Hi,

Interesting work indeed Hope it will become something usable on pal-land japanimation at a better speed than restore24..

Is there anywhere an english (or french ^^ ) translation of scharfis_brain post http://home.arcor.de/scharfis_brain/...esInterlacing/ ?

[Didée]

Originally posted by Alain2
Hope it will become something usable on pal-land japanimation at a better speed than restore24..

The keyword here is:

"serves a different purpose"

Means: up to now, I have not seen a ConvertFPS-style framerate conversion in any broadcasted material. At least not in non-animee.

[Alain2]

I agree Didée, that's why initially i didn't find restorefps useful for zone 2 pal animation, but from incredible's post I thought maybe it could be adapted to reverse convertions like 24p > IVTC > 50i or 24p > 50i as described...
I tested the convertion described by incredible on a film material and after it really looks like what we have on some japanimation dvds (I compared with a rip from the ghost in the shell - stand alone complex dvd). That's also why I would like to understand better scharfis_brain post in german.

But so far restore24 is really the best i know

[Leak]

Originally posted by Alain2
I tested the convertion described by incredible on a film material and after it really looks like what we have on some japanimation dvds (I compared with a rip from the ghost in the shell - stand alone complex dvd).

Which version of GITS:SAC would that be? Definitely not the US version, and I don't think the UK has any problems in that area as well...

[mg262]

Proper replies soon, but one thing now:

The way the filter works is by generating a pattern like

New frame 0 = 1.00 * Old frame 0
New frame 1 = 0.33 * Old frame 0 + 0.66 * old frame 1
New frame 2 = 0.66 * Old frame 1 + 0.33 * old frame 2
New frame 3 = 1.00 * Old frame 2

And then using some maths to reverse the pattern and so reverse the blend.

Internally, there is a 'ConvertFPS' which is used to generate the pattern, but it can be replaced by anything else -- e.g. (forward) telecine, or (forward) telecine + ConvertFPS, or really anything else as long as the blend weights are known. So the hard part is figuring out the blend weights (= forward blending pattern). I can make it read patterns from a text file if that is useful.

[Mug Funky]

@ Alain2:

the region 4 GitS SAC discs are veeery nice. i encoded some some of it myself

they're progressive PAL speedups, and the UK discs should be the same (though they'll have DTS audio and thus a lower max bitrate, but i think they also come in 2-disc sets).

@ mg262:

i think it may be possible (with a lot of head-busting and hair-pulling) to somehow mathematically determine the pattern. maybe check the source for such things as restore24 and TFM/tDeint to get some ideas. you've already got a functioning de-blend algo, which in my opinion is probably most of the work. all you need is a nice way to determine blend-weights reliably and without insane artefacting.

i made a yv12LUTxy thing that grabbed blend weights, but it was incomplete and generated useless results. the maths was dead simple and probably wrong, but i have faith that you can do better

because different standards-converters generate different patterns, some form of auto-adaptation would be the only way to make this filter one of the holy-grails of video processing.

a treatment for "peculiarblend" style conversions also seems like it'd be possible with relatively simple maths (i hope).

[scharfis_brain]

The most simple blending pattern might be this one:

New 1 = Old 1
New 2 = ( Old 1 + Old 2 ) / 2
New 3 = ( Old 2 + Old 3 ) / 2
New 4 = Old 3
New 5 = Old 4

(three clean and two blended frames)

this pattern is created, if someone is blend-deinterlacing or just downsizing 3:2 hard telecined film.

Some tiiiimeee ago, it tried to write a AVS-function that is able to revert this stuff (even with jumping phase).
But on scenechanges or lumiosity changes, it failed to detect the pattern...