Restore24: Script to recover FILM from field-blended PAL

[BBQmyNUTZ]

Originally posted by scharfis_brain
@BBQmyNUTZ: I don't think that this will work...

23.976fps are progressive and this script handles 25fps interlaced video.

in an 24fps progressive video there are only 24 states per second and 24 full frames/sec and according to your description about 20 to 50% of those frames are blends. but those blends don't have an neighbor that represents their unblended state.

in 25fps interlaced video there are 50states per second but only 24frames are put in this stream. here about 20 to 30% of the video are blends. so it es "easy" to select the non-blended frames and use them for the new output...

I hope my german-english was not that confusing...

Man, that sucks... well, it was worth asking. ^_^ Thanks for the explanation!

Kai

[scharfis_brain]

can you provide a sample-clip?

[HomiE FR]

Sorry but the link for MaskTools 1.4.1 is dead and I can't use Restore24... Kurosu seems to be debugging it for now, then it would be nice if someone could put MaskTools 1.4.1 online (or send it to me via mail -> homie.fr at wanadoo.fr.

Thanks in advance.

[scharfis_brain]

use this link! http://kurosu.inforezo.org/avs/MaskT...skTools1.4.zip

[Bogalvator]

I would like to test out this script, but I can't figure out how to use it

Assuming I have got as far as "MPEG2Source=(...)" in my script, what do I do next in order to implement this method?

[scharfis_brain]

just make an *.avs -file with the following content:

loadplugin("x:\path\TomsMoComp.dll")
loadplugin("x:\path\SmartDecimate.dll")
loadplugin("x:\path\masktools.dll")
loadplugin("x:\path\mpegdec.dll")

paste here the toms-bob-functions that didee posted

paste next the restore24-function

mpegsource("x:\path\yourvideo.m2v")
restore24()

[HomiE FR]

Thanks for the tip scharfis_brain. I had tried .../MaskTools1.4.0.zip and .../MaskTools1.4.1.zip but not MaskTools1.4.zip. Dammit.

[Didée]

Very funny. I typed all I wanted to, and five seconds before submitting this machine BSOD'ed.

Now, ultra-short:
- script speed-up by ~20-25% (without scientific benchmark)
- slightly adjusted a levels command for better measuring
- modified TomsBobSpecial a little. For me, does the best rendering job here

I'm very busy with actually using Restore24, not with developping: 100 GB are to process, additional 40 GB are coming this weekend - my HDs are full to the brim.

Until I start implementing even more precision, it would be helpful to get some feedback of your milage. Different sources, different results (?).
I suggest to use the script that I just updated

Good night

- Didée

[Didée]

Erhm, version 0.4b had a little bug: the comparison for the actual frame was done on wrong points in the timeline ... n+1 instead of n-1
V0.4c on page 1 is working correctly. The funny thing is, I actually encoded one episode with the buggy version, and it had almost no impact at all - hihi.

- Didée

[bilu]

Replacing a blend is quite easy, as soon as we have detected one. The bigger question is: replace with what?
The obvious answer seems to be: replace it with the neighbor that is more similar. However, in practice, this is not necessarily true. My first attempts did use the most similar neighbor, but the end result was quite jerky in places. After fiddling a long time without noticeable improvement, I lost nerves and simply replaced with the preceeding frame *always*. To my surprise, the result was much, much smoother - but still not satisfying.

Since your work is an AVS script and not a filter, you probably have no way to match the next field instead of the previous

I think skipping could look better than stuttering as side effect.
This is just theory of mine, I haven't tried your tool.


Cheers,
Bilu

[Didée]

I'm still on my mission.

bilu: as stated on the other place, that doesn't take place here.

Things keep getting better ... and complicater. A much stronger pattern guidance is almost ready, with other stuff I am struggling. The last step, decimation, is still a little of a black-box with little possibilities of controling.
With another concept, I get limited by Avisynth's poor conditional functions ... [THAT WAS A JOKE!]

But I keep on.
Is anyone else still interested, or am I fiddling for my own fun only?

- Didée

[Sigma9]

@Didee:

Oh yes I am VERY interested in these things getting improved.
I am constantly recording that stuff. However, because I'm short im time at the moment, I've just got the raw PVAs so far. I temporarily halted SVCDing right before the end of season one.

Thus, as long as my HD is not full, you have time to improve the script... - no honestly: as soon as possible I am going to apply your script and report problems if there are any.

Tx anyway for your efforts
/Sigma9

[cooli]

Tried your script, and think this could be the
start for a good solution, since for now only thing
i found helpful for this problem was speeding up to 50fps,
but files too big, encoding time extended and much decoding
power needs....

Restore24 makes this Clip Jumping ?

http://www.futurvision.ch/vob/newtest.mpg

it's a little file if you can try and tell me if you have the same problem ?

Thanks for your help !

Cooli is sorry for his English

[Didée]

cooli:
The script is made for TopFirst material. More exactly, this is what is hard-coded in the bob-routines I use.
For your test clip, you have to apply "ComplementParity()" right before Restore24. Then, v0.4c on page 1 renders your clip fine - except for the last few moments. My current (private) version renders that clip 100% perfect.

@ all
Please remember that the script assumes TFF material. If your source is BFF, you have to use "ComplementParity()" beforehand!
[edit: please perform one of the two procedures that siwalters offers two posts below!]

Okay the current status:

I'm still fighting with the implementation of an advanced pattern guidance. In general, that already works very well - except for the few situations where bad comes to bad, and everything goes crazy
The main problem is the final decimation step. Alas, SmartDecimate uses a too limited search range to find the next dominant field to use. Therefore, everything works smooth as silk as long as the 3-dups in the de-blended stream are distributed absolut evenly. That means, they must have a distance in the range of 20~30 frames at most. From the blending pattern, this *should* be the case in the end. But if Restore24 misguesses at one place and produces an out-of-order-triple, then SmartDecimate will produce either an interlaced frame in the output stream, or a dup frame + a skip frame (by the nature of the whole process, an outputted dup frame is alwas accompanished with a skip frame in the neighborhood).

For exactly these reasons, the advanced pattern guidance should solve the problems. But here, I have another problem:
Only working at the actual position in the stream, it is almost impossible to distuingish between a "right hit" and a "false hit" when trying to lock to the pattern. After all, the pattern is always shifting around a little (theoretically at latest after each 1000 frames, but it does shift much more frequently).
So, I tried to simultaneously read ahead in the stream to look if the next "hit" appears at the right position, to certify the first "hit". Now, that works ... but AviSynth claims tremendous amounts of RAM when I do that, speed drops to ridiculous amounts, and some very odd errors occur (sometimes frames from a completely other position appear - memory managment??).
So it seems I am reaching the limits of AviSynth scripting. I am not out of ideas by far, but it is questionable if realization through AviSynth scripting is possible.

Help on pattern guidance - neuron2, please ?

Help on decimation - Kevin, Donald, please ?


I'll try to make the "simple" advanced pattern guidance as stable as it can get, and will then make it public again.

With a headache

- Didée

[HomiE FR]

Thanks for your hard work Didée.

I would be too bad if you stopped your work, now that you know how to do the job. I know how sad it is when you find out that Avisynth has reached its limits (I had the same problem with FMF).

I really hope that someone will be able to help you in your task (I don't know anything about coding an Avisynth plugin / pattern guidance).

HomiE FR

[Si]

Please remember that the script assumes TFF material. If your source is BFF, you have to use "ComplementParity()" beforehand!

Normally that isn't sufficient.

BFF people should do a full field dominance reversal before using any TFF only filter on BFF material (and vice-versa) such as

AVISource(...)
ReverseFieldDominance() #available via my sig or
# use the SepFields Trim(1,0) Weave or another method....
AssumeTFF()

regards
Simon

[Didée]

Simon, thanks for correcting this.

With all the field-related stuff, I'm really not that familiar (speak: too dumb). Luckily, in the script I'm only dealing with full frames.

However, a simple ComplementParity() worked out on cooli's clip, so ...
And that's how I'm handling fieldorder and that things mostly: fiddling until it fits. :ashamed:

[edit]

HomiFR:
No, I don't surrender yet. Too much ideas still floating, and some nifty ideas about completely different topics keep floating also ...

[/edit]

If they only had never invented the idea of interlacing.

- Didée

[Sigma9]

@Didee: nice to hear you are going on...

@all (esp. programmers...)

Now if it is true what Didee says that scripting reaches a limit here, would it be obvious (?) to hard-code it, i.e. write a filter?

I am not able to do that, nor to answer:
- is it possible?
- is it worth the effort?
- is it improving things?

Maybe programmers can comment on that.

The reason why I think to follow this project is so important is that I have a strong feeling that we are just at the beginning of the "destroy quality before airing" behaviour of the TV stations (cost pressure, DVD competition, ...). IMHO much more of that to come and thus Restore24 dramatically needed.

/Sigma9

[Dermir]

[solved]The link for masktools is unfortunately dead (really dead now). And the dll on warpenterprises site is an old one (1.02) Can anyone make masktools1.4 available for download somewhere/(email it to me) so I can host it? [/solved]

edit: Sorry, after doing a quick search I found it here:Thread

[LiquidGhost]

Hi,
Has anyone tried restore24 with those Madman released PAL anime DVDs? Also..how do I get this to work. I just get various errors like mostly { expected at collum zxy row xyc. Also, will this work on Avisynth 2.5x?