14.999fps to DVD: which way to go?

[Chainmax]

Having experienced the goodness of recent block removal utilities, I decided to dust off my Gabriel Knight 1 CDs (I have two) and try to make a miniDVD out of the "making of". Problem is, the source is exactly 14.999fps, so I figure these are the options:

1) Alter framerate to 23.976fps, encode and then pulldown.

2) Alter framerate to 25fps.


I'd like to hear your opinions on which should be better and why. Also, in order to change the framerate, should I use interpolated/dupe frames or speed up the video?

[AI]

Original Progressive?
NTSC
abc=>aabbccInterleave(Last,Last)
AssumeFPS(60000, 1001,True)
ResampleAudio(48000)PAL
abc-\aabbc
abc-/abbccAssumeFPS(15,True)
DoubleWeave
SelectEvery(6,0,1,2,3,4)
ResampleAudio(48000)PAV v2
abc-\abbcc
abc-/abbcaInterleave(SelectEvery(3,0),SelectEvery(3,1),SelectEvery(3,1),SelectEvery(3,2),DoubleWeave.SelectEvery(6,5))
AssumeFPS(25,True)
ResampleAudio(48000)

[scharfis_brain]

if you want to force progressive encoding without introducing an unsmooth pattern of duplicated frames for PAL, you may speedup the 14.999fps to 16.667 fps (16 2/3) and then pulldown to 25fps.

16.6667 fps ist the lowest framerate achievable with PAL and pulldown.

[foxyshadis]

Motion-interpolation! :p Okay, with a cinepak source you may as well pray for sierra to send you their master copy, but at least it might help. (The GK videos have been on my "list of stuff to process later" for a while now.)

[Chainmax]

Everyone: thanks for the advices :).

AI: yes, the original is progressive.


if you want to force progressive encoding without introducing an unsmooth pattern of duplicated frames for PAL, you may speedup the 14.999fps to 16.667 fps (16 2/3) and then pulldown to 25fps.

16.6667 fps ist the lowest framerate achievable with PAL and pulldown.
I read once that any other pulldown than 23.976-->29.97 using DGPulldown will cause some jittering (wobbling, right?). Does that not apply to 14.999--->16.667? If not, how would you go about speeding up to 16.667?


foxyshadis: it's not Cinepak, it's Microsoft Video 1 and it's so crappy that I couldn't use AVISource right away since the decoder couldn't handle YUV or RGB. I ended up loading the video itself, making an uncompressed RGB encode and loading that into AVISource.


Everyone, this is the current filterchain:
Levels(0,1,245,16,235)
ConvertToYV12()
FFT3DFilter(sigma=6,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16)
hqdn3d(0,0,3,3)
AddBorders(0,4,0,4)
Deblock_QED(quant1=30,quant2=51)
Crop(0,4,160,120,align=true)
GaussResize(192,144)
dull=last
sharp=dull.LimitedSharpenFaster(SMode=4,LMode=3,Strength=1000,wide=true,soft=75)
Soothe(sharp,dull,30)
EEDI2().TurnRight().EEDI2().TurnLeft()
FFT3DFilter(sigma=1,bw=32,bh=32,bt=3,ow=16,oh=16,sharpen=0.7)
GaussResize(3xx,2xx,p=100) <--- depends on wether I go PAL or NTSC
Tweak(sat=1.2)
AddGrain(10,0,0)
AddBorders(16,16,16,16)
ConvertToYUY2() <--- for CCE

(by the way foxyshadis, we should include a check within BlockTerminator that detects when source width and height are not MOD16 so that it adds suitable borders before deblocking and removes them after, like I did here)

and this is a comparison with both source and result blown up to 640x480 for a better view:
http://img82.imageshack.us/img82/8593/ba7oh.png (http://imageshack.us)

Any kind of feedback on how to make it look better is welcome.

[AI]

Chainmax
What you think about my pulldown with out chenge speed?

Note: "a","b","c" is fields as manual built-in pulldoun.

[foxyshadis]

Oh, I knew it was one of those codecs that made me install the stupid "legacy codec pack", that turn every 2x2 pixel block into a single huge pixel. Since I dealt with some similar cinepak videos a few months ago I was going to show you what I did, which was similar to an immature BlockAhnold+immature mcnr_simple, but that already looks better anyway. Dang, I guess I need to redo Jesse Cook now. EEDI2 is so effective at repairing those old junkers!

[IanB]

16.6667 fps is the lowest framerate achievable with PAL and pulldown.Why does it have to be a 3:3 pulldown from an 11% speed up. What is wrong with a 4:3:3 pulldown pattern from 15 fps. The 5Hz jitter won't be detectable over the 15hz judder.

[scharfis_brain]

because then interlace encoding is required.
with an 3:3 pulldown you can stick with progressive encoding.

[IanB]

Ahah, I see your point, if you do it in AviSynth, yes.

But if you use DGPulldown this is not a problem.

So I guess I would have to recomend PAL....
AssumeFPS(15, True)
ResampleAudio(48000)Encode as 15FPS progressive
Use DGPulldown to set to 25 FPS

or NTSC....
AssumeFPS(15000, 1001, True) # 14.985
ResampleAudio(48000)Encode as 14.985FPS progressive
Use DGPulldown to set to 29.97FPS

[scharfis_brain]

Ianb: you didn't read my previous posts carefully enough.

1) I am not suggesting to do anything in AVISynth
2) DGPulldown only allows 3:3 pulldowns at max.
this means:
with PAL (25fps native) you can only go down to 16 2/3 fps. not lower
with NTSC (29.97fps naive) you can only go down to 19.98 fps

this effectively means: for PAL one needs to speedup 15fps to 16.667 fps in order to be ABLE to do the 3:3 Pulldown.

3) framerates like 15 or 14.985 aren't supported by mpeg2.
you need to abuse (speed change) one of the valid framerates like
23.976, 24, or 25.
then use dgpulldown and the desired ratio of 16.6667 to 25.0 .
dgpulldown will overwrite the previously assumed framerate of the encoded mpeg2.

[Chainmax]

foxyshadis: 2x2 pixel block? Is that a real "feature" of those codecs or just a sarcastic coment on the crapyness of these old codecs? By the way, the thing that impresses me the most about this filterchain is that the source doesn't even look like real life footage but hand drawn since the textures (or whatever they are called) are so screwed up, and even though the result is much too blurry, at least it looks like live footage again :).

AI: your methods sound great, but I'm waiting for more opinions on wether I should do everything in Avisynth or use DGPulldown at some point.

scharfis_brain: why does 15fps---25fps needs interlace encoding?

[scharfis_brain]

25p -> 25 points on the time axis per second possible
25i -> 50 point on the time axis per second possible

this brings us to the following patterns (Large Letter: Top field on TV, small letter: Bottom Field on TV):

15p to 25p: AaAaBbBbCcDdDdEeEeFf, which is an 2:2:1 pattern. This means it has a VERY noticeable jerkyness

15p to 25i: AaAbBbCcCcDdDeEeFfFf, which is an 3:3:4 pattern, which only has a slight mostly unnoticeable remaining jerkyness.

My method applies the speedup, to 16,667 fps and thus realizes a 3:3:3 pattern, which is completely jerk-free (exept of the 15fps stutter) and allows progressive encoding!
So it is the best of both methods: progressive encoding AND jerkyness suppression.

Chainmax, can you provide a sample video?
I am just curious, how motion compensated temporal super resolution approaches might help here...

[IanB]

@schafis,

Yes you are correct. I should learn to check these things before I open my big mouth.

So the question remains how do you Force/Trick/Deceive whatever into doing a 4:3:3 pulldown pattern of a progressive mpeg2 encode without an 11% speedup.

i.e. I want AI's pattern with a progessive encode
abc-\aabbc
abc-/abbcc

[scharfis_brain]

it is possible, but it could be VERY hard to set up!

1st step: creat a progressive mpeg file with this pattern:
AaBbCcCcDdEeFfFf (2:2:4 Pattern)

then Pulldown this MPEG to
AaAbBbCcCcDdDeEeFfFf (3:3:4 Pattern)

but it is quite difficult, because the frames containing Aa, Bb, Dd, and Ee needs to become pulldown flagged, while the already doubled Frames containing Cc and Ff must remain untouched.

I have no clue, how to do this with dgpulldown!

maybe a completely new pulldown application could be useful here. I think about an application that adaptively insert Null-P-frames into the video stream, where more than 3 is needed in the pulldown pattern...

[Chainmax]

Here's (http://rapidshare.de/files/16946446/Sample.avi.html) a sample.

[scharfis_brain]

processed file: http://rapidshare.de/files/16951339/sample-25p.avi.html

there are still some blotchy artifacts, for whose I was too lazy to compensate them.
It is just a test of my approach.
also the video has been interpolated up to 25 fps. So no need for any pulldown...

[Chainmax]

The motion seems quite fluid for me, but I'd like to see what others think about it. Is this much different than using 16.667 speedup+pulldown? About the filtering, it doesn't convince me: it's far too smooth and there is some kind of weird residual motion on the backgrounds.

[scharfis_brain]

made another sample: http://rapidshare.de/files/16986312/sample-25p1.avi.html

btw.: please show me the results of your approach as a
video from the same snipplet I got from you.
IMHO the result of your approach according to the image
above has too much staistepping left and thus is only
virtually sharper. stairstepping is a kind of sharpness,
but not one that pleases my eyes.
That's why I prefer some kind of smoothness in that case.

my 2nd sample is a little bit sharper and has more stable backgrounds.
http://home.arcor.de/scharfis_brain/samples/schock.png

of course the approach is also very different from the pulldown
approach, because no pulldown is needed anymore.
It is true 25fps now (upwards interpolated the framrate from 14.999 to 25fps).


EDIT: my script just uses two external plugins: mvtools and eedi2.
the rest is a 3x redundant chain of pointresize, temporalsoften,
eedi2 and mvflow calls to merge 5 frames temporally while
effectively raising resolution and image detail.

[Mug Funky]

@ scharfi:

that's an amazing result right there.

btw, any reason not to bump fps to 30000/1001 and encode progressive? there's redundant frames but that's the least of this sample's problems :)

[Chainmax]

That looks freaking awesome, although I'd use something like AddGrain(2,0,0) at the end to give an illusion of detail. The reason there's stairstepping is probaby because of the p=100 switch I used in GaussResize. I'll encode the sample with my filterchain to ffdshow's huffyuv YV12 and upload it for you soon.

[Chainmax]

You can download my encode here ( http://rapidshare.de/files/17113602/Sample.avi.html). I think we should stick to 352x240 by the way, 704x576 amplifies the errors too much.


[edit]Could you try your filterchain on this (http://www.31012.com/~azulftp/MOV01825.MPG)? I'd like to see what results it yields.

[scharfis_brain]

You should decide.

either 352x240@29.97 fps
or 352x288@25fps

but since the source is able to give the amount of informataion, I'd stick with 640x480 in a letterboxed PAL or in Full NTSC format.

this means: 584x480 with added borders into 704x576

btw.: I know your second clip. It is pretty old right now ;)
1) it doesn't seem to violate the nquist theorem in spatial domian (no staistepping)
2) it has lots of shaky and differential (concurring) movements in it
3) it is block like hell

these conditions make it hard to upsample.

the other clip (the one which is this thread about) is os 'easy' to upsample, because it violates the nyquist theoreme (it shows HEAVY aliasing!): http://en.wikipedia.org/wiki/Super-resolution

[Chainmax]

I see. Please upload an encode made with your filterchain at 704x576 so that I can see how it looks in motion.

[scharfis_brain]

here http://forum.doom9.org/showpost.php?p=808013&postcount=19 I already posted a link to a processed 640x480@25fps video.

just scale that to:
-either 704x576 and recompress as progressive PAL MPEG2
- or 584x480 and addborders (60,48,60,48) afterwards an compress to progressive PAL MPEG2 as well

I'll post my script later.

[Chainmax]

Oh, I didn't see it. It looks pretty good, but I'd still use AddGrain(10,0,0) as a final filter because it looks too "empty", plus the grain should help concealing the slight remaining background instability.
By the way, I've been thinking: since you chose to use interpolation to the final framerate, wouldn't it be better to interpolate to 23.976fps? That way we can use NTSC FILM framerate and framesize for encoding, which would mean a higher bits/pixel*frame ratio and therefore a better overall picture, right?

[scharfis_brain]

By the way, I've been thinking: since you chose to use interpolation to the final framerate, wouldn't it be better to interpolate to 23.976fps? That way we can use NTSC FILM framerate and framesize for encoding, which would mean a higher bits/pixel*frame ratio and therefore a better overall picture, right?
No. The best would be an interpolation to 29.97 fps, because then every even frame is an original and every odd frame is an interpolated image.
Also 23.976 is - at least on TV Sets - much more jerky than 29.97fps. And only every 5th frame is an orignial frame. the other four are interpolated ones.

I won't mind about compressibility and compression artifacts here, because the video itself has not much complicated information in it (except if you are adding grain).
But anyways with 4mbps at 704x480@29.97fps progressive you should end with a very decent compression result.

[Chainmax]

29.97fps it is then.

[Chainmax]

By the way, here's your result + FFT3DFilter on chroma + Tweak + AddGrain:

http://img214.imageshack.us/img214/4618/schfilt0pi.png (http://imageshack.us)

[scharfis_brain]

I'll cleanup my script and post it during the evening.

[2Bdecided]

Any chance of seeing that script please scharfis_brain?

I have some video footage shot on a digital still camera that might respond well to this.

Cheers,
David.

[scharfis_brain]

Sorry. Totally forgot that:

f=4

avisource("Sample1.avi")
a=last
a.converttoyuy2()
eedi2().turnleft().eedi2().turnright()
pointresize(a.width*f,a.height*f)
vf=last.mvanalyse(isb=false,blksize=16, truemotion=true)
vb=last.mvanalyse(isb=true,blksize=16, truemotion=true)
vf1=last.mvanalyse(isb=false,delta=2,blksize=16, truemotion=true)
vb1=last.mvanalyse(isb=true,delta=2,blksize=16, truemotion=true)

xf1=last.mvflow(vf1)
xf=last.mvflow(vf)
xb=last.mvflow(vb)
xb1=last.mvflow(vb1)

interleave(xf1,xf,last,xb,xb1)
temporalsoften(2,40,50,mode=2)
x=selectevery(5,2)

x.converttorgb()
xa=pointresize(a.width*2,a.height*2).converttoyuy2().eedi2().turnleft().eedi2().turnright()
xb=crop(1,1,0,0).addborders(0,0,1,1).converttoyuy2().pointresize(a.width*2,a.height*2).eedi2().turnleft().eedi2().turnright()
x=merge(xa,xb)


xvf=x.mvanalyse(isb=false,blksize=16, truemotion=true)
xvb=x.mvanalyse(isb=true,blksize=16, truemotion=true)
xvf1=x.mvanalyse(isb=false,delta=2,blksize=16, truemotion=true)
xvb1=x.mvanalyse(isb=true,delta=2,blksize=16, truemotion=true)

xxf1=x.mvflow(xvf1)
xxf=x.mvflow(xvf)
xxb=X.mvflow(xvb)
xxb1=x.mvflow(xvb1)

interleave(xf1,xxf,x,xxb,xxb1)
temporalsoften(2,255,255,mode=2)
selectevery(5,2)




#last.mvflowfps(vb,vf,num=25,den=1)
#bicubicresize(704,576)

[Chainmax]

So, it needs MVTools (is v1.1.1 recent enough?) and EEDI2, right?

[scharfis_brain]

yeah.

[Chainmax]

It has MVFlowFPS set to 25 but the line is commented out, would the final result be 29.97fps? If not, what should I change to make it so?

[scharfis_brain]

uncomment it and set:
num=30000, den=1001

and then resize to 704x480

afterwards you may add grain etc.

[2Bdecided]

Quite a different task from the problem originally posed, but a useful result so I'll report it here...

I tried this script on the file I'm currently working with: a 30fps 320x240 M-JPEG from a 6M-pixel digital still camera. The original is highly aliased (the 320x240 seems to be just spot sampled from the 6M sensor with no aliasing or filtering) and then lots of mosquito noise is added on top.

The results from the script were near miraculous. The amount of detail recovered from the (quite poor) original was great. Not always perfect, but so much better than just bicubic resizing the original! I resampled down to 768x576 at the end to watch on the PC, and didn't change the frame rate.

There were some "interesting" artefacts - fingers or even hands turning into alien like appendages when they waved or moved - but looking at the original clip, it seems there just isn't enough information at these moment to do anything better. I wonder if a fall-back mode which avoided this with fast motion of small objects (i.e. ones where there was nothing to be recovered from this technique) would be possible? But it may not help much.

However, over all I am very impressed with the result, and will be using this.

I might tack a frame rate change to 50fps on the end and interlace for a DVD, and see what happens. I wonder if there's any way of doing the frame rate conversion within the original process to save time and quality? I'll try and get my head around the script.

Only one (obvious) request: what can I do for a speed up? Parsing through AVS2AVI (no recompression) managed 0.05fps. Going straight to TMPG was even worse, and both started eating memory - I think TMPG even hit virtual memory at some point, which may explain the speed problem. Any suggestions?

Many thanks for sharing this scharfis_brain (and everyone who wrote the functions within) - it's really appreciated!

Cheers,
David.

[Chainmax]

Great, thanks :).

[scharfis_brain]

THis script is just a proof of concept. I WILL fail terribly with lots of concurring motion of small objects in it.

It is working like a motion compensated denoiser. But it is merging everything together without any regard to possible artifacts.

It can be improved for sure but I dunno how.
Maybe Didee can tell something about it.

[Chainmax]

I get an error at line 19:

vf=last.mvanalyse(isb=false,blksize=16, truemotion=true)

saying it needs YV12 data. I suspect it comes from line 16:

a.converttoyuy2()

Can I change that to ConvertToYV12() or will it have a negative effect on the script's functionality?

[Chainmax]

Changing all converttoyuy2() calls to converttoyv12() made it work for me. Hopefully I didn't break anything by doing so :).

[Chainmax]

After checking on different spots in the source, I can confirm that this SuperResolution script works only on sources with very little movement, like a person talking. When there is motion, it suffers from the "oil painting" effect. Here's a comparison:

Source + Resize + Borders:
http://img138.imageshack.us/img138/9015/origresizeborders2zz.png (http://imageshack.us)

SuperResolution:
http://img231.imageshack.us/img231/8879/schsr1kb.png (http://imageshack.us)

BlockTerminator:
http://img138.imageshack.us/img138/4860/blockterminator9eg.png (http://imageshack.us)

Now, the BlockTerminator script doesn't have framerate conversion, and all the times I used something other than assume/changefps for framerate conversion I also got oil paiting/motion issues, so I'll try to include the fps conversion from the SuperResolution script (only mvflowfps(num=30000,den=1001), right?) and post the corresponding image.

[scharfis_brain]

something is going wrong there.

please post that scene.

[Chainmax]

Here are two scenes that show the problem:

http://rapidshare.de/files/17857860/Problem_Scenes.avi.html

[Chainmax]

Were you able to download it?

[Chainmax]

Well?

[Chainmax]

bUmP.

[2Bdecided]

I can't make rapid share work, so can't access you file - but either you're very unlucky with that clip, or something is very wrong.

On pans, for example, it's working fine for me.

The only weird effects are where things move against a background; the mo comp fails, and the averaging gives very strange results.

If there's lots of random movement in your clip, I suppose it would look like the stills you posted, even if the software was working "correctly".

Sorry I can't be any more help, but I thought I'd offer this information since no one else stepped in!

Cheers,
David.

[scharfis_brain]

Sorry Chainmax, I was away for 10 days.
I'll be away next week, too.

I won't have time to look at your samples until next week.

[Chainmax]

No problem at all. After all you can't rush a masterpiece :). By the way, YGPM.