Experimental Fieldblend Reversal/Removal (prev. "RestoreFPS")

[mg262]

Those of you reading this thread for the first time might want to start at this post (http://forum.doom9.org/showthread.php?p=691286#post691286), which is where the attack on fieldblending starts for real.
____________________________________________

RestoreFPS (http://people.pwf.cam.ac.uk/mg262/posts/RestoreFPS.dll)

Brief description:
Reverses the kind of blending generated by ConvertFPS, restoring original framerate.

Search keywords: ConvertFPS, FPS, framerate, restore, reverse, like unblend, deblend, restore24.

Full description:
RestoreFPS(clip, float fps, float phase) - takes a YV12 clip which has an 'underlying' frame rate of fps but a higher actual frame rate due to blending, and reverses the blending to restore the original frame rate. phase is a number between 0 and 1 which specifies the relative displacement of old and new clips (see example below).

The framerate to restore should be less than the current framerate, and more than half of it. (So restoring from 24 back to 25 and from 24 back to 11 are both illegal.)

The method used is described here (http://forum.doom9.org/showthread.php?s=&threadid=89266).

Examples:
In order to test the filter we need to generate a clip using ConvertFPS. A script like this:

bicubicresize(4*60,3*60)
selectevery(200,0)
converttoyuy2
assumeFPS (24/1.001)
convertFPS (25)
converttoyv12
overlay(crop(0,0,-0, 26). showframenumber)

will do the trick. Some frames from this:

http://people.pwf.cam.ac.uk/mg262/posts/convertfps.jpg

Now running

RestoreFPS(24/1.001, 0.00)

produces a stream like this (do look at the messed up 'frame numbers'):

http://people.pwf.cam.ac.uk/mg262/posts/convertfps restored.jpg


In this case, the start (in time) of the input clip corresponds exactly to the start (in time) of a frame of the original clip. In other words the situation looks like this:

http://people.pwf.cam.ac.uk/mg262/posts/fpsconvert.jpg

That won't always be true -- if we take the 'convertFPS' clip above and trim it (say add Trim(10,0)), then we have something more like this:

http://people.pwf.cam.ac.uk/mg262/fpsconvert2.jpg

Dealing with this requires us to guess the amount by which the converted clip is 'slid' with respect to the original, i.e. to guess the phase parameter. I used this script to help with this:

Function phase(clip c, float phi)
{
c
RestoreFPS(24/1.001, phi) #set appropriate frame rate

#trim here if desired

a=selectevery(6,0)
b=selectevery(6,1)
c=selectevery(6,2)
d=selectevery(6,3)
e=selectevery(6,4)
f=selectevery(6,5)

stackhorizontal(a,b,c,d,e,f)
Reduceby2()
stackvertical(last, trim(1,0), trim(2,0), trim(3,0), trim(4,0), trim(5,0))
Trim(0,-1).Loop(0,1000)
subtitle(String(phi, "%f"))
}

#source clip here
animate (0, 100,"phase", 0.0, 1.0)
trim(0, 100)

Save the output of this, open it up in VirtualDub, and drag the slider around to find a phase that looks correct. (You may get an unusual effect in the first few frames of the clip, because of an edge effect.)

Here's a real example to finish up with:

Source (from VHS):
http://people.pwf.cam.ac.uk/mg262/posts/blends.jpg

Restored:
http://people.pwf.cam.ac.uk/mg262/posts/blends removed.jpg

(Actually, this example doesn't seem to work exactly like ConvertFPS on a wider scale, but leave that aside for now.)

Notes:
- If you're dealing with a clip in which fields (not full frames) are blended, separate them and process them separately; I think the phase parameters for the top and bottom fields will differ by 0.5 (though I'm not at all sure).

- This isn't optimised; there are several ways in which I could speed it up, and I will if anyone finds it useful.

- If you get the frame rate slightly wrong, this filter will work in a small region and get worse and worse as you drift away from it.

- It will not deal with cuts. You will need to take each cut section and find a phase for it separately. (If the sections are too short, this method may not be of any use.) On the other hand, there are some ways to estimate the phase automatically (esp. autocorrelation), and cuts could be detected by the change in phase.

- Although I've only set it up to reverse ConvertFPS type blends, it will work perfectly well on any regular blend pattern, including the types (http://home.arcor.de/scharfis_brain/ExotischesInterlacing/#2.3) mentioned by scharfis_brain in this thread (http://forum.doom9.org/showthread.php?s=&threadid=89266). The hard part is analysing the blend type -- I have some scattered thoughts but this post is too long already!

By the way, I'm really sorry I trailed off/disappeared at the end of that last thread... real life got in the way. I'll try and do better with this filter (assuming anyone finds a clip it works on!)

[Didée]

Hey hey, this looks very promising!

Despite the restriction "The framerate to restore should be less than the current framerate, and more than half of it" -- do we here finally have a method to tackle 29.97i -->[fieldblending]--> 25i conversions? Restoring 29.97i probably is out of reach, but perhaps we can get 29.97p without blending?
(I'd have two sources to test that, but they're on the other machine I won't get to the next few days.)

[scharfis_brain]

WOW!

restoring 50fps out of a 50i->60i conversion worked really well.

but the sample I had was interfered by a blending type, I call "negative blending"
(didee knows what I mean...)

restoring 23.976 fps out of a 23.976p->59.94i->50i conversion fails.

this is due to the weird blending pattern and the limitation, that the restored framerate must not drop under the half of the input framerate.

but this type of restoration IMO doesn't need such a kind of deblending.
selecting non-blended fields, like Restore24 deos is fully sufficient, if restoring-framerate is smaller than half of the input framerate.

[mg262]

Proper replies soon, but the half framerate part of the restriction can certainly be removed.

[Mug Funky]

wowee! i'll have to try this out.

i eagerly look forward to an optimised version :)

[edit]

question: i assume this is correct, but tell me if it's not.

i'd like to recover clean 24fps (NOT 23.976) from old "spinning prism" transfers from film to NTSC. these are the kind that give "peculiarblend" effects. i work with a lot of old, crappy looking anime, and this would be a good treatment to give some of it, if i can spare the time. it's certainly worth experimenting with.

[mg262]

First up, I should say I know much much less about what blend patterns arise than any of you. I'm also in PAL territory so don't know much about IVTC, etc. I need to go and read up on them before I can say anything useful. But here's a quick attempt at replies -- apologies if some of it is confused.

>The framerate to restore should more than half of the current framerate

The reason for this was just that (as @scharfis_brain notes) if you upsample the frame rate by more than a factor of two, all your original frames are preserved unblended, (together with lots of blended frames). So this case is actually much easier to deal with than the other -- you just find the frame(s) in the upsampled version corresponding to the one you want (and you average them if there is more than one). I just didn't think this case would be very useful -- but I will put it in when I have a moment if anyone wants it. You can do something similar as follows:

To get from 25fps back to 11fps:
RestoreFPS(22.0, ?)
And then average every pair of frames (possibly skipping one at the beginning).

In the field-based cases, as I understand it, top fields are only ever made up of original top fields, and bottom fields are only ever made up of original bottom fields. (Please correct me if I'm wrong.) So you can separate them and deal with them individually before interleave/weaving. To be absolutely clear: in the analysis (including finding the phases) it is useful to consider all fields together, but in the actual reversal it's not.

Telecine could probably also be dealt with, but not with the code as it stands -- it assumes regular 'spacing' of blended frames(or fields), which doesn't apply to the ABCDD pattern (the D lasts twice as long as the others).

@Mug Funky
I should also have said that its currently only to reverse ConvertFPS in blend mode -- I forgot that switch mode existed -- sorry. It might well be possible to deal with the other though.

@scharfis_brain
Sounds like you can't reverse the 59.94i->50i step because of telecine?

@everyone
I will keep reading and come back; but if there are a few common blend patterns that it would be really useful to reverse, would you mind posting scripts that generate them?

[kassandro]

I haven't yet tested this filter, but I see a problem with the phase parameter, if the filter is applied in the real world: The phase parameter has to be reset to a new value, if for some reason a frame or an entire mpeg2 GOP is dropped. If a GOP is dropped because of transmission problems the transport stream demuxer will recognise it, but if single frame or a single field is dropped by the broadcaster, then there is no way to realise it.

[mg262]

So, I get the impression that a useful thing to do would be to reverse

23.976 ---telecine--> 29.97 ---ConvertFPS-type-blend--> 25

Just to check before I try to implement anything, looking at the top field only (for now), does that correspond to

[23.976 fps input]
SelectEvery(4,0,0,1,2,3)
ConvertFPS(25)

OR

SelectEvery(4,0,1,1,2,3)
ConvertFPS(25)

OR one of the 2 similar patterns? In other words, is it useful to reverse the effect of the above script(s)?

@kassandro
Yes, that could be a problem, depending on the frequency of the drops. (Do you know how common similar drops are in DVDs as opposed to broadcast material?) Similarly it will mess up the choice from the four telecine patterns mentioned above. The problem is much the same as with cuts (except that when only one frame/field is dropped I know the exact effect on the phase parameter). So I may well need to implement automatic guessing of parameters -- telecine pattern and phase and possibly others, depending on what kind of blend is being reversed. I do think that will be possible, but I'd rather leave it until I have a good sense of what range of types of blend the filter should be trying to reverse.

So for the moment I'm concentrating on the above type of IVTC+unblending and then on speeding things up. (Unless prodded in other directions...)


Edit:
I forgot about the pattern
0,1,2,3,4,4,5,6,7,8,8, ... (which needs a Trim as well as SelectEvery).

[Mug Funky]

hmm... as far as blend patterns go, it gets a little difficult. usually there's no fixed "pattern", and phase is different every time, because realtime units are always running (even if they're standards converting zero input, and outputting PAL green noise, they'll be converting).

if you like, i could run some test signals through the converter at work so you can take a look at how it does things. i'm not sure what kind of test signals you'd prefer, so maybe you could send something (an m2v?) and i can run it through if/when there's some spare time.

if you want to know more, here's it's write-up on the Leitch site:

http://www2.leitch.com/__85256A160051AE1A.nsf/0/77264FD232F903FE85256B480053875E?Open&Highlight=2,ixion,SubDocument,donotpublishme

(hehe... looks like it's discontinued now. wasn't when i checked 3 weeks ago)

[mg262]

So here's a new version (http://people.pwf.cam.ac.uk/mg262/posts/RestoreFPS_100505.dll). It should be substantially faster, and it removes the factor of two limitation. I still don't particularly recommended it for that case, as the output is very sensitive to the phase parameter -- i.e. having that even slightly out will cause a few blends to drop through. (So it really requires automated guessing of the blend parameter.) By the way, the filter speed seems to fluctuate substantially -- running it through VDub, closing that down, and then opening VDub again makes it run much faster -- I assume this is some sort of caching effect.

It also incorporates a function that @scharfis_brain thought of: convertfps(newfps,blend-threshold)
newfps -> this value should be known, hehe
threshold -> only blend two frames together, when the weight of the lower weighted frame is higher than the threshold.

example:
the new frame is between two frames of the input stream and their weights are:
20% and 80%, if I set the threshold to 30%, only the 2nd frame will be in the output.
If I choose a threshold of zero, the output-frame will be a 20:80 - blend of its neighbors.

If you set this imaginary threshold to 45%, only those frames are getting blended, whose temporal position is close to the middle of its neighbors, all other frames are just copies of their nearest neighbors (like changefps does)
Except that I've named the function
ThresholdIncreaseFPS (float fps, float threshold)

(@scharfis_brain: I tried to PM you to find out what you wanted done in the decrease fps case where you can end up blending three or more frames together -- but your inbox was full. So I left it out. it's easily added.)

Because of the structure of the filter adding that function automatically added the reverse function (in the sense that RestoreFPS is the reverse of ConvertFPS). So if anyone has a use for it, I just have to give it a name.

The new function is YV12-only like the others, although at some point I'll add YUY2 (to all of them together).

Anyone want any more weird blends? :sly: (Actually, I think the filter probably needs renaming to ReverseBlend or Blend or something like that.)

@Mug Funky
Thank you for the offer of test data -- I may take you up on that at some point later. Part of what I was asking wasn't about the blend pattern, just the telecine pattern (told you I was new to this)... I just wanted to check that the forward telecine script (23.976 --> 29.97) looked something like this

separatefields
e = selecteven.selectevery(4,0,1,2,3,3)
o = selectodd.selectevery(4,0,1,1,2,3)
interleave( e,o )
weave

But the variety of blend patterns may prove a real problem. The 'Motion adaptive' mentions in the converter specifications are particularly awkward. the manual says that 'spatial and temporal apertures... are selected on a pixel by pixel basis' -- there's no sane way to reverse that. For that matter, I've no idea what "the full processing power of IXION’s 10-bit motion adaptive conversion core" does when you feed it with blended (telecined) source.

Edit: threshold is a fraction (e.g. 0.45) not a percentage.

[Wilbert]

Is there any chance you will release the source ? :)

[mg262]

@Wilbert -- I PMed you a couple of days ago...
_______________

I wrote a version (http://people.pwf.cam.ac.uk/mg262/posts/RestoreFPS_130505.dll) that reversed this kind of thing:

[23.976]
SelectEvery(4,0,1,1,2,3)
ConvertFPS(25)

It works fine on that script SelectEvery(4,0,1,1,2,3).ConvertFPS(25) ] but is useless on a blend sample from another thread. I think it's a red herring, but FWIW, the syntax is RestoreFPSIVTC(29.97, phase=0.0...1.25, telecine=0,1,2,3,4). There's an associated script if anyone wants it.

Slightly more usefully, I realised using a fixed blend pattern (1-lambda 0 lambda) on frames (n-1 n n+1) and subtracting the results from the original (i.e. frame n) was a good way of finding blend weights where frames are duplicated (esp. in animation). The function is Reblend (clip, float weight)

Test with the above 'phase' script, replacing:

RestoreFPS(24/1.001, phi) #set appropriate frame rate

with

cn = overlay(crop(0,0,-0, 26). showframenumber)
subtract(c, Reblend(c, phi))
stackvertical (cn, last)

(Should have said before: Reduceby2 on the source is a good idea before any of these scripts... or if you can Reduceby2.Reduceby2 and then remove the Reduceby2 at the end of the phase function.)

Example:
http://people.pwf.cam.ac.uk/mg262/posts/reblend.jpg
Only the foreground (the man here) is useful as the background pans every frame. You can see that frame 4616 is approx .47 * 4615 + .53 * 4617.

I need to speed the filter up before this is really useful. SSE2 is coming; requests for SSE/MMX? Actually, I should really build a VDub filter so that you can scroll through the clip and use a separate slider for the weight...

[Inc]

First of all mg262 ... I really do apreciate your developings on such a kind of filter-dll ! Thanks a lot!

In case of simulating a source input: "If I understood right" at Scharfis_Brain's Interlacing-Explanation-Reference, in PAL-Land TV Stations mostly use conversations in two ways (well I know these from here in Germany) based on IF the source comes to the PAL TV stations as 24p or 29,97 (60i telecined):

Simulating in Avisynth:


24p > IVTC > 50i AVISource("24fpsFilm.avi")
SelectEvery(2,0,0,0,1,1).SeparateFields().SelectEvery(4,1,2).Weave()
DGBob(order=1).ConvertFPS(50).SeparateFields.SelectEvery(4,0,3).Weave()

24p > 50iAVISource("24fpsFilm.avi")
ConvertFPS(50).SeparateFields().SelectEvery(4,0,3).Weave()

Reference:
http://home.arcor.de/scharfis_brain/ExotischesInterlacing/

Im looking forward to try out your dll tonight :)
I hope I'll catch a typical broadcasting to capture tonight for testing.

[Alain2]

Hi,

Interesting work indeed :) Hope it will become something usable on pal-land japanimation at a better speed than restore24..

Is there anywhere an english (or french ^^ ) translation of scharfis_brain post http://home.arcor.de/scharfis_brain/ExotischesInterlacing/ ?

[Didée]

Originally posted by Alain2
Hope it will become something usable on pal-land japanimation at a better speed than restore24..
The keyword here is:

"serves a different purpose"

Means: up to now, I have not seen a ConvertFPS-style framerate conversion in any broadcasted material. At least not in non-animee.

[Alain2]

I agree Didée, that's why initially i didn't find restorefps useful for zone 2 pal animation, but from incredible's post I thought maybe it could be adapted to reverse convertions like 24p > IVTC > 50i or 24p > 50i as described...
I tested the convertion described by incredible on a film material and after it really looks like what we have on some japanimation dvds (I compared with a rip from the ghost in the shell - stand alone complex dvd). That's also why I would like to understand better scharfis_brain post in german.

But so far restore24 is really the best i know :)

[Leak]

Originally posted by Alain2
I tested the convertion described by incredible on a film material and after it really looks like what we have on some japanimation dvds (I compared with a rip from the ghost in the shell - stand alone complex dvd).

Which version of GITS:SAC would that be? Definitely not the US version, and I don't think the UK has any problems in that area as well...

[mg262]

Proper replies soon, but one thing now:

The way the filter works is by generating a pattern like

New frame 0 = 1.00 * Old frame 0
New frame 1 = 0.33 * Old frame 0 + 0.66 * old frame 1
New frame 2 = 0.66 * Old frame 1 + 0.33 * old frame 2
New frame 3 = 1.00 * Old frame 2

And then using some maths to reverse the pattern and so reverse the blend.

Internally, there is a 'ConvertFPS' which is used to generate the pattern, but it can be replaced by anything else -- e.g. (forward) telecine, or (forward) telecine + ConvertFPS, or really anything else as long as the blend weights are known. So the hard part is figuring out the blend weights (= forward blending pattern). I can make it read patterns from a text file if that is useful.

[Mug Funky]

@ Alain2:

the region 4 GitS SAC discs are veeery nice. i encoded some some of it myself :)

they're progressive PAL speedups, and the UK discs should be the same (though they'll have DTS audio and thus a lower max bitrate, but i think they also come in 2-disc sets).

@ mg262:

i think it may be possible (with a lot of head-busting and hair-pulling) to somehow mathematically determine the pattern. maybe check the source for such things as restore24 and TFM/tDeint to get some ideas. you've already got a functioning de-blend algo, which in my opinion is probably most of the work. all you need is a nice way to determine blend-weights reliably and without insane artefacting.

i made a yv12LUTxy thing that grabbed blend weights, but it was incomplete and generated useless results. the maths was dead simple and probably wrong, but i have faith that you can do better :)

because different standards-converters generate different patterns, some form of auto-adaptation would be the only way to make this filter one of the holy-grails of video processing.

a treatment for "peculiarblend" style conversions also seems like it'd be possible with relatively simple maths (i hope).

[scharfis_brain]

The most simple blending pattern might be this one:

New 1 = Old 1
New 2 = ( Old 1 + Old 2 ) / 2
New 3 = ( Old 2 + Old 3 ) / 2
New 4 = Old 3
New 5 = Old 4

(three clean and two blended frames)

this pattern is created, if someone is blend-deinterlacing or just downsizing 3:2 hard telecined film.

Some tiiiimeee ago, it tried to write a AVS-function that is able to revert this stuff (even with jumping phase).
But on scenechanges or lumiosity changes, it failed to detect the pattern...

[Alain2]

@ Leak
Yes you never have blend problems in the us with japanimation i think because they use the ntsc standard in Japan.. The problem is that when here in Europe (zone 2) editors convert the master to pal 25fps, they don't do it properly, meaning they don't ivtc, and they apply some sort of convertion that creates blended fields, as can be simulated by incredible's quotes.

@ Mug Funky
Yes i know there are some nice dvds somewhere ^^ , but not in France.. I admit I often take fansubs because they are better video quality than the dvd sold here. I think the uk version is the same than the french version, usually the only difference is that they remove the french translation and subtitles and only leave english subs, but video is the same..
I did a small sample of our version of gits-sac to demonstrate to some friends the benefit of using restore24, you can find it here (homepages.nildram.co.uk/~cestial/gits_comp_r24.rmvb) (the top video is just a separateFiefds().SelectEven().biCubicResize(...), the bottom video is using restore24)

[Leak]

Originally posted by Alain2
The problem is that when here in Europe (zone 2) editors convert the master to pal 25fps, they don't do it properly, meaning they don't ivtc, and they apply some sort of convertion that creates blended fields, as can be simulated by incredible's quotes.

Eh... I know that, actually - I'm living in Europe myself... ;)

Still, I mostly but R1 DVDs because importing them is just much cheaper than buying German releases, the dubs are better and everything is released earlier - but this is getting a bit off-topic... :)

np: Markus Guentner - Track 3 (In Moll)

[Mug Funky]

I admit I often take fansubs because they are better video quality than the dvd sold here.

hehe... email that to Manga Entertainment. they might get the hint that people want them to send good masters to the poor authoring house. though i haven't seen the UK discs, i'd be very surprised if they are standards-converted - Manga AFAIK is UK based after all... maybe it's England-France rivalry again? ;)

IIRC, the US GitS SAC discs are pretty bad too - oversharpened and lacking detail.

[mg262]

SSE2-enabled version (http://people.pwf.cam.ac.uk/mg262/posts/RestoreFPS_180505.dll) (about 3x faster).

@Mug Funky
>i have faith that you can do better
I'm flattered -- & getting nervous now! Although my maths is okay, I have a lot less intuition about video than any of you... so please don't raise your expectations too high. I will keep trying, though. I think you're right in that I need to put down this filter and look at a) finding the blend pattern and b) dealing with cuts.

One problem I have is that I actually only have two examples of source blends of my own, both animation, one of which is from VHS and so very noisy. So I see patterns -- e.g. if top field is blended, the bottom field usually isn't (& vice versa) -- but I can't tell whether they're general enough to rely on. So I would be very grateful for clips (and advice!) from anyone reading this thread. I set up a GMail account at blendarchive@gmail.com; everyone on this thread is welcome to the account password -- just PM me.

If anyone does send anything, please include a best guess of source process (e.g '23.976p -> 25p'), and also some idea of how important/useful reversing that kind of pattern would be. The GMail attachment limit is 10MB; clip length is much more important than resolution (but please crop and/or ReduceBy2 or ReduceBy2.ReduceBy2 rather than resizing).

@scharfis_brain
The 'simple blend pattern' you mention seems a v. good place to start dealing with cuts (i.e. pattern changes) -- that would be particularly useful if you wouldn't mind sending it.

Last thing -- I can't analyse that many examples by hand, so please don't anybody put themselves out of their way too much to send them. Plus I take a loooong time analysing things, so be warned in advance. Oh, and the RestoreFPS code is fairly close to a releasable state so I will tidy it up if anyone wants to look at it soon...?

[Mug Funky]

i'll dig around and find some samples that'll be good.

i'm thinking the intro sequence to Azumanga Daioh - it's 30p in NTSC land, and i have examples of blends from 2 different standards converter (ADV will usually do their own PAL conversion, but on the last disc the PAL mpeg-2 they gave us was total crap and we got them to send us their digibetas so we could do our own conversion). it might be interesting to see the same footage fed through different converters.

if you can think of test signals you want converted, the offer is still open (if you like i could go NTSC to PAL, and PAL to NTSC if that helps). i think something helpful might be the video equivalent of an impulse - a full field (or frame) of white against black video, repeated several times to get slightly different blends on it. i can also write down the settings used in the converter if they're relevant (AFAIK everything's on "medium", and motion threshold is on auto, which floats around the 1.2 mark)

[mg262]

@Mug Funky,

:eek: I didn't realise you did this stuff for real -- I'm impressed!

I definitely want to take you up on that offer (thank you!); reason I haven't already is that I think (software) matching up of frames from the input and output clips could be messy because there will be

1) Horizontal translation of the picture
2) Horizontal zooming of the picture (dependent on capture window of capture device)
3) Change in levels

-- so I have been thinking about what kind of input would get round this. Best I have thought of is something like this (white bar scrolling down the screen):

Function bar(clip ignore, int i)
{
w=720
h=576
l= 24

black = BlankClip(width = w, height =h, length = l)
white = BlankClip(width = w, height =h/l, length = l, color=$ffffff)
overlay( black, white, y=i*(h/l) )
}

animate( BlankClip(), 0, 23, "bar", 0, 23)
#loop this however many times...

What do you think? BTW, do you normally leave the motion stuff on for anime?

_____________________________________

I sifted through a bundle of DVDs I borrowed and found two more with blends, so started some analysis. (Both are animated as of the dozen or so non-animated DVDs I looked at, not one had blends). I had a method that estimated blend weight for a field, and I modified it to give a 'confidence' level, i.e. how likely it was that that frame really was a blend (lower valuue = more likely blend). Results for two cases:

Jayce
http://people.pwf.cam.ac.uk/mg262/posts/blend dist jayce.jpg

X-Men animated
http://people.pwf.cam.ac.uk/mg262/posts/blend dist xmen.jpg

Edit: If anyone happens to wants to try this on their own sources I'll modify the relevant stuff to be more user-friendly + release it...

[ADLANCAS]

Could this filter be used to restore 24fps from Super8(24fps) -> Camcorder (29.97) :confused:

[mg262]

@Adlancas:

Afraid I don't know anything about Super8 -- but are you sure it's not just a normal telecine conversion? Step through the clip in VDub + see if you get a mix of interlaced + progressive frames, in a pattern repeating every 5 frames.

______________________________

Alain2 has kindly sent me a very clean source on which the blending pattern stands out very clearly. The analysis is quite lengthy so I don't want to make everyone wade through it (I'll put it up on a webpage later) but the weird fact that stands out is that every 25 fields appear to come from 23 fields worth of source. (Actually approx 22.98.) Any guesses as to why?

Quick Edit: scratch all that... penny just dropped.

[Mug Funky]

What do you think? BTW, do you normally leave the motion stuff on for anime?

"motion adaptive" isn't the same as "motion compensated"... a mocomped converter would cost a LOT more than the boss dude here is willing to part with :). an order of magnitude more...

the Ixion's method is very similar to Kernelbob with an automatic (and low) threshold (and some kind of artefact protection that i've yet to figure out, because i don't see as many ghosty outlines).

interlaced vertical pans would be an interesting thing to test - i'm not sure, but i think it has failed on those before (it could have been like that on the master tape though).

i'll have a look at that test file you posted. i wont have a chance to run it through the converter until there's a bit more free time (it's messed up here ATM...).

i'm very interested in the progress of this filter though - reversing PAL to NTSC conversions would be good, as believe it or not, we've been given stuff that's originally PAL, been converted to NTSC, which we then have no choice but to convert back to PAL again... some dubsters are absolute morons, but there's nothing we can do but bear it in most cases (i'm also getting mighty sick of seeing digibeta masters that are quite obviously dubs from composite sources... only to find the US DVD is the same crappy source).

[mg262]

@Mug Funky

Don't worry about the conversion -- it's a fair bit further down the line that that will be really useful. But let me tap your brain on something... in another thread (which I can't now find) IIRC you mention that sometimes animation is telecined to 30fps and THEN CG effects are added. would you happen to know if this applies to alain2's gits-sac case?

Edit:
Sorry, head switched off; as its a DVD from France that's not a terribly fair question!

[ADLANCAS]

@mg262

I´ll make a cap from Super8 and return info.

[mg262]

Alain2 has kindly provided me with a very clean clip exhibiting blends, and I'm going to use it to illustrate different options for how the filter might work.

IMO, the right way to do this is in three steps:

1. Calculate which input fields are blends and the blend weights (in terms of other input fields).
2. Look at the overall blend pattern, and produce a list of the relation of input fields to putative 'original' fields.
3. Use this list to reconstruct the original fields.

1. I started by taking an earlier plug-in I had written, and modifying it so it produced an output a bit like this:

...
Field 3 is not a blend.
Field 4 is approximately 20% * field 3 + 80% * field 5
Field 5 is not a blend.
Field 6 is approximately 41% * field 5 + 59% * field 7
Field 7 is not a blend.
...

Here and below, for simplicitly, I have ignored the fact that every other field is shifted down by half a scanline -- so I'm considering the top and bottom fields of each frame to be identical.

On Alain2's sample, the pattern roughly repeated itself every 25 frames, so I filled the blend weights into a table, putting 25 entries on each row.

http://people.pwf.cam.ac.uk/mg262/posts/GTS pattern 25 grey.jpg

So e.g. the weight for field 80 is in the row labelled 75, Column labelled 5. A value of 0.8 means 20% of the previous field + 80% of the the following field. Adjacent cells have been coloured yellow when they are duplicates.


2. This is where the options arise. I'll start by analysing the pattern above, then giving an okay-ish method based on it, then talking about better method(s).

You can take a line (25 fields) of the pattern above and divide it into the original fields; it's easiest to understand from an example:

http://people.pwf.cam.ac.uk/mg262/posts/GTS pattern 25 halfcut.jpg
All I've done is put a division in the 'middle' of every blended frame, and also between adjacent unblended (nonduplicate) frames. We have 12 fields (A-L) worth of source material, which is what we would expect from an original 24p progressive source.

[Actually this is slightly misleading, as e.g. a 0.1 blended frame is very like the frame before it, and so should be cut well past the 50% point (probably at the 90% point). Like this:

http://people.pwf.cam.ac.uk/mg262/posts/GTS pattern 25 cut.jpg
Although I've done this by hand, the short/long/short/long alternation is hopefully visible. But leave that aside for now.]

What I've just done is totally mechanical & can be done by the computer on each set of 25 fields. It can increase/decrease the blend sensitivity (and duplicate sensitivity) to make sure 12 original fields are reconstructed.

So that would be one complete method. Example output. (http://people.pwf.cam.ac.uk/mg262/posts/exout.txt) But there are reasons I'm not not going with it immediately. The important one is that we're throwing away a huge amount of information. E.g. you can see massive similarities between adjacent lines in the table that are thrown away by this method. There are various ad hoc ways to get round this, but IMO the right thing to do is to try and understand the pattern and make a global fit to the data. This would also

- allow direct 23.976 fps restoration (needs extra messiness with above method)
- detect telecine changes + behave sensibly
- detect cuts so that they can be dealt with sensibly (esp. ensuring audio sync.).
- allow use of a limited set of blend weights [0.2, 0.4, 0.6, 0.8 in the Jayce example above (http://forum.doom9.org/showthread.php?p=658815#post658815)]

So at the moment, I'm trying to understand the pattern, and to break it into a 'telecine' and 'conversion' part. (I know different converters will produce different patterns, so at some point the software will have to guess the pattern.) But this is proving rather hard :(

3. This is the part already implemented in RestoreFPS.
__________

It might actually be worth making 1-3 three separate filters (with the data flowing between them encoded in clips). Then it would be easy to change step 2, which is the one that requires intelligence...

Thoughts on any of this v. welcome.

[Alain2]

Well I don't fully follow your ideas but I am glad you can make use of the sample I sent you ;) You can try your algorythm on the GTO sample as well ; I think it is not as easy to see the pattern in the GTO sample because there is not a long travelling movement in the animation, but i think it follows the same pattern than in gits-sac.

[mg262]

All right... something coming soon. I'm really sorry this has taken so long -- bits of it have required a lot of concentration.

@Alain2... is it all right if I post links to a filtered version of the clip you sent me side-by-side with the original?

Re-edit (obsolete notes deleted as this post is too long already):

A reverser based on the (suboptimal) method discussed above is here (http://people.pwf.cam.ac.uk/mg262/posts/ReverseBlend_23Jul05A.dll) (updated 0723GMT 23 Jul). It only works on source that has been fieldblended to 25fps. (It takes a little while to scan the source so test on short clips.) Use this script:

separatefields
ExperimentalReversalAlpha(Bob) #bobbed clip only for finding blends, not used in reversal
weave

It certainly won't work on everything; it depends on the type of conversion. Of the two clips @Alain2 sent me, it works (IMO) pretty nicely on one and produces total junk on the other. If you want to see whether your source is appropriate, you need to look at the 'signature' of the conversion, using this script:

function strip5(clip i)
{ stackhorizontal(i.selectevery(5,0), i.selectevery(5,1), i.selectevery(5,2), i.selectevery(5,3), i.selectevery(5,4)) }

function doublev(clip i)
{ stackvertical(i.selecteven, i.selectodd) }

#source
a=bob
mini = a.BicubicResize(32,20)
chequered = interleave( blankclip(mini), blankclip(mini, color=$202020) )

#Show12Breaks(a, mini) #pattern superimposed on original fields
Show12Breaks(a, chequered) #pattern superimposed on chequered background
strip5.strip5
doublev.doublev.doublev.doublev

Again the script takes a while to load. It displays the fields from your sample in a grid (25 to a row), and superimposes white vertical lines to indicate a 'picture change', i.e. the divisions between original fields that I was talking about in my last post. Where a white line goes through the middle of a field, it means that that field is a blend. The reverser is currently working reasonably well on this kind of pattern...
http://people.pwf.cam.ac.uk/mg262/posts/blendpattern1.jpg

In this case you can see that the 'breaks' are slowly drifting right every 25 frames. (If we were dealing with exactly 24FPS source, they would stay in the same place; the reason for the drift is because its 23.976.)

In order to build a general-purpose reverser, we need to know what kind of patterns arise... i.e., need to run this script on lots of different field blended sources and see what we get.

I think I may be being a bit unclear about what the white lines here (grey vertical lines in the last post) mean, so I will try and explain that with the help of a few pictures in a bit...

[mg262]

All right... I want to step back, summarise what I have, and ask for advice.
____________________________________________________

Results on a sample @alain2 kindly supplied:

Input (http://people.pwf.cam.ac.uk/mg262/posts/blend/G.I.T.S_Sample.vob) (VOB)
Output (http://people.pwf.cam.ac.uk/mg262/posts/blend/gts_upload.avi) (XVid Quant 1) #from latest version (http://people.pwf.cam.ac.uk/mg262/posts/blend/ReverseBlend_26Jul05.dll) as of 26 Jul

Decoding from DVD and saving to HuffYUV, this runs at ~22FPS on my 2.4GHz SSE2 P4 with a script:

SeparateFields.ExperimentalReversalAlpha(LeakKernelBob(order=?)).Weave

If you want to try playing the clip, add the argument buffer = true.

I don't think it's that meaningful to post stills, but I also know that no one will reply if I don't ;). Because input and output frames don't match up exactly, I've posted two input stills which correspond to one output still:

Input 1
http://people.pwf.cam.ac.uk/mg262/posts/blend/gts in80.jpg

Input 2
http://people.pwf.cam.ac.uk/mg262/posts/blend/gts in81.jpg

Output
http://people.pwf.cam.ac.uk/mg262/posts/blend/gts out 78_8081.jpg

Out of mercy to those of you one dial-ups those are quality 10 JPEGs.
PNGs as follows... Input 1 (http://people.pwf.cam.ac.uk/mg262/posts/blend/gts in80.png) ... Input 2 (http://people.pwf.cam.ac.uk/mg262/posts/blend/gts in81.png) ... Output (http://people.pwf.cam.ac.uk/mg262/posts/blend/gts out 78_8081.png).

There are still some glitches, especially at the beginning and end of the clip, but you can begin to see how it might look.
___________________________________________

There are two ways to go now:

1. Dropping seeking
If we allow the filter to pre-analyse* the whole clip (at ~32FPS), this allows:
Global blend pattern specification, giving much better results
+Automatic local pattern fitting (i.e. finding offsets)
+Automatic cut (pattern break) detection
Noise level specification giving better blend/duplicate detection
Output to HuffYUV should be at ~30 FPS (frames) regardless of any of the above being implemented
*The analysed data can be saved to a file, so this only has to be done once on a given source. The time splits as 53% decoding, 35% bobbing and only 12% analysing, so I can't speed it up much more.

2. Retaining seeking
This is more restrictive:
Limited local pattern guidance, improving still scene behaviour
Noise level specification giving better blend/duplicate detection
Output to HuffYUV should be at ~22 FPS (frames) regardless of any of the above being implemented

So... this is where I ask for help. All the extension from reversing ConvertFPS to the above version was based on your suggestions above. I still don't know much about fieldblending, and I don't know which way (if any) to go now...

The reason I dealt with 24p->3:2 pulldown->field blending->PAL was that the two samples I was sent were of that type. @scharfis_brain has convinced me that restoring 50i->60i is a more important task -- indeed he said so above, but at the time I didn't know enough about field blending to understand the comment. But I have no samples of that type... . At the moment I am taking a breather and then analysing the second sample @alain2 sent, on which the filter doesn't work.

Footnote: I wrote most of this message (in the test forum) before recent discussions, so to those who are concerned, please don't think I'm ignoring you -- I just can't face rewriting this again! Please note that the output version above is newer than anything I have shown anyone.

[Inc]

mg262, thanks a lot for your developing on this! :)

Even if I dont have that much skills in fieldblended sources like others do have in here, I do follow this thread on every new post.

[Leak]

Input (http://people.pwf.cam.ac.uk/mg262/posts/blend/G.I.T.S_Sample.vob) (VOB)

[OT] Makes me kinda glad I went for the R1 DVDs... :eek:

np: F.S. Blumm - Blick (Zweite Meer)

[Mug Funky]

leak: the r4 ones are better :)

[Leak]

leak: the r4 ones are better :)
AUGH! Don't tornment me like this, for $DEITY's sake... :eek:

Then again - would you mind putting up a short VOB-clip from the first episode somewhere so I can compare? (And, of course, for extra tornment on my side... :D)

Oh, and how many episodes of it are out right now in R4?

[Mug Funky]

7th disc (eps 24-26) was encoded last night. authoring will take a week or so, then over a month before it's in stores. all the others are out, i think (maybe vol 6 isn't?).

i've got a sample capped directly off the tape in 10-bit 4:2:2 uncompressed, but it's a tad too big to post casually.

i might post a vob sample in a while - i'll have to rip it off my copy.

[edit]

i will definitely have to check this script out when i find a field-blended NTSC source that needs to come out PAL - i don't mind blending because it's an occupational hazard of working in a PAL country with NTSC sources. but double blending is something i totally don't want to do if it can be avoided...

[MOmonster]

@mg262
thanks a lot for your developing on this function, from me too.
I read of your restore function some time ago, but it doesn´t help me for my fieldblended pal material. So I didn´t follow the development, but the new ideas you realized are really interesting.
I like the method of your blenddetection. It is very similar to the method I use for my Crestore function :cool: .
Because I don´t know how your function exactly get the blendinformations (I think really similar like my function), I can´t help you so much, but I´m really interested in the developing of your function (the output of my function looks fine on my sources, but the conditional enviroment and the combination with decimate I really don´t like).
If you could explain a little bit more how your function works it would be really helpful.

Some questions I have:
1. Why your ExperimentalReversalAlpha function produces blends and interlacing artefacts at the beginning and the end of the clip? At the beginning maybe it doesn´t has analysed enough to get the pattern, but at the end?
2. Your function has just the simple fields as input. A smart bobbed input should give a better blenddetection. How get your function the blendlevels for this input?
3. My function detects the blends much better, if I soften the blendclip a bit to avoid the strong influence of artefacts. That makes the blendleveldetection much more accurate. Does your function internal something similar?
4. Is for the first pass a bobber necessary (theoretical) and can I use different bobbers for the passes?

And an idea for the future development:
I don´t like two passes. It should be just ok, if your function just analyse the next one or two cycles. For fieldblended pal source for example it would first analyse the frame 1 till 50 and if I step to frame 2 it would analyse frame 51 and so on. Don´t know if this is possible. It would use more ram and don´t be really faster than the two pass method, but I would prefer a one pass method.
My function only analyse the next two frames, but is still able to find the pattern offset (for natural, low motion scenes not that accurate, but still ok), so to analyse the whole clip before shouldn´t be necessary.

Tip: For sources that shows something like a pattern (also if it change really oftern), you should work on a really save blenddetection.

Hope I can use your plugin instead of my function in the near future. ;)

[Backwoods]

Would it be possible to use this filter on this source:

http://rapidshare.de/files/3419982/VC.m2v.html

Trying out a script posted results in a very odd behavior. Alot of oversharpened frames and some green flashing once in a while.

It's FILM - 29.97 with maybe some PAL tossed in (I have no idea, it's a real bad transfer).

Scharfis_brain made a script that would work on the samples I provided but would not work on the over all movie.

[mg262]

@incredible,

The scripts you posted above were very much appreciated, and the moral support is even more so!

@backwoods,

The reverser as it stands is meant for field blended PAL... I'm sorry I didn't make that clear above. I'm definitely willing to look at the sample but at the moment I can't make head or tail of it... I'm not very used to NTSC, so I indexed it on the default settings and using either tff or bff I get the video moving backwards at some frames. Could you point me at @scharfis_brain's analysis?

@all (before I start on the long part)

Can I beg for more sample data? Before implementing pattern guidance I need to examine many more patterns by hand... at present I only have 2 clips from @alain2 (which were invaluable -- without them the reverser simply would not exist), and DVDs of two old cartoons. As I may have said, the GitS pattern is very simple; and I broke another pattern last night and found that both my old 'Jayce' sample and @alain2's (presumably quite new) GTO sample conformed to it... so there is some hope that there are a relatively small number of (24p->field blend->PAL) patterns out there. But I need a lot of sample data to check that...

At the moment I'm focusing on 24p->field blend->PAL, but I will happily look at 50i->60i if sent any samples. Please, no hybrids containing 30->PAL or 60i->50i for now... I may extend the filter to deal with these, but not yet. I would prefer 'typical' samples rather than isolated pathological cases... what constitutes 'typical' you will know better than me. Sections containing fast pans are ideal.

@MOmonster

Thank you both for the kind comments and for the feedback... lots of ideas! 1. Why your ExperimentalReversalAlpha function produces blends and interlacing artefacts at the beginning and the end of the clip?At the beginning, there is a fade... and the problem with that is that the fields are not (detected as) duplicates, and every field looks like a blend of the previous and next one. (Pattern guidance will fix this.) Actually, the current version will probably have trouble on most too-still scenes, though I think I know how to fix that.

The end case is simpler... the filter divides the clip up into blocks of 25 fields for analysis, and at the end there aren't enough fields left to give a complete block... so I pass them through unchanged.
2. ... A smart bobbed input should give a better blenddetection.It's hidden in there, but there is a bobber...

SeparateFields.ExperimentalReversalAlpha(LeakKernelBob(order=?)).Weave

That bobber is not used in blend removal/reversal. In fact, you can save the analysis data to a small file:

LeakKernelBob(order=?)
WriteBlendData("blend data.txt")

And then (in the same script or a separate one) use

SeparateFields.ExperimentalReversalAlpha("blend data.txt").Weave

3. My function detects the blends much better, if I soften the blendclip a bit to avoid the strong influence of artefacts. That makes the blendleveldetection much more accurate. The more I think about it, the more that seems like a really good idea.Does your function internal something similar?No... but you can try something like

SeparateFields.ExperimentalReversalAlpha(LeakKernelBob(order=?).mysofteningfilter).Weave

... and I know people here will come up with better variants in scripts than I could in code.
4. Is for the first pass a bobber necessary (theoretical) and can I use different bobbers for the passes?You can try without a bobber for the first pass, but bobbing improves the analysis. For now, the reversal (second pass) does not use bobbed input ... top fields are reconstructed from top fields (only) and bottom fields are reconstructed from bottom fields (only). You could replace all the bottom fields in the original clip with black frames or noise, and top fields would still be reconstructed exactly as now.
____________

For fieldblended pal source for example it would first analyse the frame 1 till 50 and if I step to frame 2 it would analyse frame 51 and so on.
At present output fields 0...23 depend on input fields 0...24 (only), output 24...47 on input 25...49, etc.* The buffer=true option makes it work exactly as you describe, spreading the load to prevent 'freezing' every 24 output fields. (But it looks ahead 25 fields not 50.) *This is a slight simplification for clarity... something more intelligent happens at the 'edges' of blocks.but I would prefer a one pass method.I was originally leaning towards the 'pre-scanning' method myself, but you have pushed me the other way... I think what I'm going to do is this. The analyser will break the pattern into blocks of a specifiable size (25 above) and analyse them... but when analysing a block it will take a 'window' or 'context' around that block, also of specifiable size, possibly of larger size than the block, and try and fit the pattern in that window. So to make that clear, blocks won't overlap, windows will. So you will be able to choose between speed/fast scanning and quality yourselves.

I wrote down part of the method in an old post (http://forum.doom9.org/showthread.php?p=603808#post603808), and there is a related discussion in this thread (http://videoprocessing.11.forumer.com/viewtopic.php?t=13) on @kassandro's forum (http://videoprocessing.11.forumer.com/index.php)... but it will take me a bit to write out a proper explanation* in words and pictures rather than maths (to make sure anyone reading this can follow). If you prefer the maths I can give that too. *with an answer to the 'blend levels' q. I will also answer about patterns then.
Tip: For sources that shows something like a pattern (also if it change really often), you should work on a really save blenddetection.I'm really sorry, I'm being thick, I don't understand what you mean...

And your final comment was very kind, but I don't ever envisage replacing scripts. In my experience, the people who understand video best are for the most part scripters rather than coders... so I see my remit as providing building blocks for scripters to use in making something well beyond my ability. 3. above is a good example of what I mean.

[Mug Funky]

i've got a good one for you: 24p -> 60i, "peculiar" blended.

i'm wanting to unblend this so i can speed it up and convert it to PAL. it looks to be a good source for you to play with. i tried scharfi's technique on it, but it changed every 300 or so frames (and some i couldn't get a good match on at all).

it's the japanese intro to "kimba the white lion".

will post tomorrow maybe? i'm kinda busy and want to get home now (still at work at 9:47 PM...).

[mg262]

@Mug Funky,

That would be great -- thank you. There's no hurry at all - tomorrow or whenever you have time.

Please be warned that it will take probably quite a while before I will even be able to say anything intelligent about it ... I've never seen peculiar blends before and I doubt I'll be able to adapt this solution or use my current analysis tools, so it will be something to work on from scratch... I don't know whether I'll be able to come up with anything but I'll try my best.

Edit: Still at work at 9:47 PM...) You know, I used to think you were lucky to be able to work with this stuff all day...

[Mug Funky]

hehe.

well, the thing about this kind of blending is that it approximately follows the same kind of patterns you'd be seeing with standards-converters, except there's no 3:2 jerky pattern like with 23.976->PAL.

i believe this pattern comes from oldschool telecine machines in the days before 3:2 patterns were considered important. try look up "polygonal prism telecine".

[MOmonster]

Thanks for the reply mg262

I'm really sorry, I'm being thick, I don't understand what you mean...

If I have a little bit more time (next monday I think), I will explain you what I mean, how my function works and what ideas I have to make it better (I´m not able to realize it with an avisynth script till now, hm, I should learn coding :sly: )
I though the bobber is there to recreate the blended field. I should known it, because this is the way restorefps does it.

@Backwoods
No, not at this stage of development and I also think not in the future, therefore your sample is just too seldom. This film is "Brothers" right? A friend has a pal version of the first film. Maybe I´ll have a look at this pal version. Could you give us the link to the thread, you offered this problem? I want to see scharfis_brain solution for your source. You say the script would not work on the overall movie. Is it a simple pattern function for that you has to set the offset parameter or doesn´t it work, because of different conversations?
I could create you a automated offset finder for your special pattern, but for real pattern changes there won´t be any simple solution. I like to help you, but in your thread, not here ;)

[Backwoods]

@Backwoods
No, not at this stage of development and I also think not in the future, therefore your sample is just too seldom. This film is "Brothers" right? A friend has a pal version of the first film. Maybe I´ll have a look at this pal version. Could you give us the link to the thread, you offered this problem? I want to see scharfis_brain solution for your source. You say the script would not work on the overall movie. Is it a simple pattern function for that you has to set the offset parameter or doesn´t it work, because of different conversations?
I could create you a automated offset finder for your special pattern, but for real pattern changes there won´t be any simple solution. I like to help you, but in your thread, not here ;)

Hey sure, no problem. Actually it's "Violent Cop", Beat's first film. This release and "Boiling Point" by the same company has this problem. Unfortunately no other US company has put this movie out, very frustrating.

http://forum.doom9.org/showthread.php?t=75441 Here is the thread. I'll try his method out again and see. It's been over a year and I've learned alot since then (but not close to alot of the people on here already know).

[insanedesio]

Can I beg for more sample data?
How does the Kenshin NTSC DVD source sound? I haven't touched it in a very, VERY long time, so I'm not sure if the blending is patterned or just screwed-up random, but if you want (and give me some ftp info or something like that) I'll upload part of its vob. If yeah, let me know your preference for ep 1 vs. ep 3 (ep 1 has blending in the intro mainly, ep 3 is all over the ep, shouldn't really matter which one though).

[mg262]

@insanedesio,

Thank you for the offer! But before I trouble you, let's see if anyone knows what kind of blending this is... as Japan is (IIRC) NTSC, it strange that there's blending in an NTSC version of a Japanese animation; and I don't want you to go to the trouble of ripping and sending it if it's something this method can't handle...

You wouldn't happen to know what's going on with this one, Mug Funky?