Closest AVS scripts to Alchemist results

[Trixter]

I assumed that lower blocksizes meant better motion end results.

Last night I did a quick test using smaller blocksizes and it actually made the bad frames even more worse. I'll post pictures when I get home (I'm at work now).

[Chainmax]

...
I am using MVFlowFPS2 for everything and I put that on track B; I then do a conversion using ConvertFPS(59.94, zone=80) and put that on track A. Anywhere MVFlowFPS2 makes a mistake, I cut that frame out and use the frame from the ConvertFPS conversion. Tedious, but it's pretty much a flawless conversion when I'm done.

I am trying ConvertFPS(29.97, zone=80) on a 15-to-29.97 conversion and I have to say that ConvertFPS does an awful job. When stepping through the ConvertFPS script, it sometimes a given frame is exactly like the previous one but with the bottom half shifted to the left or right. It's very nasty :(.

I really wish MVTools could be altered so that a new MVFlowFPS parameter would be created. This new parameter would be linked to the artifacting (no idea how) and if in a given frame it exceeds or falls below a user-defined threshold then the frame in question would be replaced by an interpolation of the previous and next frame. How feasible would that be?

[actionman133]

Last night I did a quick test using smaller blocksizes and it actually made the bad frames even more worse. I'll post pictures when I get home (I'm at work now).

I agree with Trixter on this. I've found that lower blksize = 4 produces poorer results than blksize=8. I attributed it to the fact that a 4x4 block doesn't have much detail in it to provide reasonable vectors, whereas an 8x8 block can show better edges, and thus get a better vector. I also think the radius defined by searchparam is directly related to the block size. So blksize=4 has half the search radius of blksize=8....

I might be wrong on that last point, so feel free to correct me if I am.

[Revgen]

I wish there was a way to adaptively use multiple block sizes kinda like what X264 does.

[actionman133]

Perhaps an algorithm could be written where it initially tests 16x16 blocks. If the amount of detail (determined by some algorithm that searches for texture) in a block exceeds one threshold, it reduces that block to 4 blocks of 8x8. If it exceeds a higher threshold, it could reduce it to 16 blocks of 4x4.

Would that be possible?

Actually, should this go to the MVTools thread?

[Trixter]

I am trying ConvertFPS(29.97, zone=80) on a 15-to-29.97 conversion and I have to say that ConvertFPS does an awful job. When stepping through the ConvertFPS script, it sometimes a given frame is exactly like the previous one but with the bottom half shifted to the left or right. It's very nasty :(.

Well, you have three options without motion compensation:


ChangeFPS: Add or drop frames to meet target. Good: No blended frames. Bad: Jerky motion on pans.
ConvertFPS: Blend frames across time to meet target. Good: Softer (not smoother) motion. Bad: Blended frames.
ConvertFPS(zone): Blend frames, but only at the precise moment in a CRT's scanrate. Good: Precise moment-in-time switches. Bad: Individual "half" frames.


Yes, with the third option you can see a frame blend across 80 lines into the new image... but when you're watching on a television that scans top to bottom 60 times a second, 99% of the population won't notice it. It hurts horizontal panning/crawls, but little else. And remember, most people aren't going to be pausing to stare at a bad image that is 1/60th of a second long :-)

Since my DVD is mastered specifically for television, the zone=80 method is what I'm using. In motion, it's less jerky than without zone=80. Freeze-framed, it has those "inbetween" frames. So you have to pick what you like best.


I really wish MVTools could be altered so that a new MVFlowFPS parameter would be created. This new parameter would be linked to the artifacting (no idea how) and if in a given frame it exceeds or falls below a user-defined threshold then the frame in question would be replaced by an interpolation of the previous and next frame. How feasible would that be?

Not at all, since what metric would you have that the estimation was wrong? How could you tell? Remember, you're creating frames where none exist. Our eyes can tell us when something doesn't look right, but to the mocomp algorithm, it's making it's best guess.

That being said, it might be possible to "fall through" to a ChangeFPS/ConvertFPS frame if the certainty of the mocomp vector analysis falls below a certain threshhold, but I have no idea how to script that. Scharfis might have already done something like that somewhere in mvbob() but I'm not brave enough to comment every line to find out.

[Trixter]

I agree with Trixter on this. I've found that lower blksize = 4 produces poorer results than blksize=8.

Thanks, you saved me from making new shots ;-)

[actionman133]

Your welcome, Trixter... always glad to help where I can. :D

But do you reckon the possibility of variable block size analysis would be feasible for application? They do it for motion compensation in video compression... Would it work with MVFlowFPS?

[foxyshadis]

x264 has some of the most advanced ME algorithms available in the open source world, if they could be cut-and-pasted into mvtools I'm sure it would improve output (and cut speed), though it'd take quite a bit of work to port their interfaces. It's certainly possible. On the other hand, x264 gets to rely on texture bits to make up for any deficiencies, and it's designed for qpel (although it doesn't necessarily have to use it).

It'd certainly be a cool summer project by someone interested. ;)

[actionman133]

Should this recommendation be forwarded over to the MVTools thread?

[Mug Funky]

FWIW, here's a _severely_ bodge-hacked version of motionprotectedfps that provides an extra (important) layer of protection.

i'm currently using it for NTSC to PAL conversions that include pure interlaced, progressive, telecine and blendy video.

basically it blends when it can't compensate, and uses a very conservative mask with soft edges.

i'm sure it can be greatly improved, at the very least by code cleanup...

function salFPS(
\ clip input, float FPS,
\ int "mode", int "protection", int "protection2",
\ int "iterate", int "reset", int "initialise")
{
mode = default(mode, warpfast)
protection = default(protection, 80)
protection2 = default(protection2, 32)
iterate = default(iterate, 2)
reset = default(reset, 50)
initialise = default(initialise, 6)

originalFPS = input.framerate()
input

mp = FindMotion(iterate = iterate, reset = reset,
\ initialise = initialise)
p = MotionFPS(FPS,mode,Motion = mp)
maskp = mp.SumStretchFPSMask(FPS, protection)

mn = FindMotion(iterate = iterate, reset = reset,
\ initialise = initialise, from = next)
n = MotionFPS(FPS,mode,Motion = mn, source = next)
maskn = mn.SumStretchFPSMask(FPS, protection, source = next)

maskp
assumeFPS(originalFPS)

tehmask=last.CombineFPS(FPS, maskp, maskn)
last.CombineFPS(FPS, maskp, maskn.Invert())
bilinearresize(width*8, height*8)
crop(0, 0, input.width(), input.height())
fity2uv()
mpro=last

tehmask
mt_binarize(protection2,false)
mt_expand()
mt_inflate()
bicubicresize(width*8, height*8,1,0)
crop(0, 0, input.width(), input.height())
fity2uv()
tehmask=last
mt_merge(p, n, mpro, u=3, v=3)

mt_merge(last,input.blendfps(FPS),tehmask, u=3,v=3)
#merge(last,tehmask.greyscale(),.5)
}

if you uncomment the last line it'll show you which parts are being blended (white) and which compensated (black).

"protection2" is the one to vary - it's a crude threshold.

enjoy :)

[edit]

did i mention that this produces almost no visible artefacts with default settings on "generic" video?

[Chainmax]

I can't wait to try this once it's optimized. Thanks for a new toy to play with :).

[Mug Funky]

you can try it now... speed isn't much different from the original motionprotectedfps.

the code is just a little uglier because i was in a hurry.

i think i might need to think a little more about how i go about choosing which parts to blend and which to compensate, as there's a bit of "noise around the edges" so to speak, where parts of the same object get different treatments when motion gets higher. it looks better than regular blendfps, but could be better still.

also, it sucks at making slow-motion. use the original script for that...

[Didée]

Small suggestion to Clouded's part of the script: Make the following mod:
crop(0,0,width+8,height+8,true)
bicubicresize(width*8, height*8,1,0)
crop(0, 0, input.width(), input.height(),true)
It's not a big deal ... but upsizing a original-resolution clip to 800% * 800% of the original resolution, just to immediately crop & keep an 1.5% area of the huge 6400% frame, is not exactly "efficient". ;)


About getting a more on-spot error correction ... well, it would be doable, but rather complicated, and very slow. For sure, if you try to do it with an Avisynth script.
The current methods of MVFlowFPS and Motion(Protected)FPS are solely based on vector probabilities and vector SADs. And these measures, albeit giving usable indications, are simply not significant enough to make reliable decisions. They didn't do for full compensations, and the same way they'll not do for intermediate interpolations. For full compensation, meanwhile I've a method that works out clearly better ... but it's almost impossible (at least extremely hard & cumbersome) to translate that method to the interpolation domain. As already said, the plugins don't offer enough grips to ply with the vectors.
For the limited case of interpolating to 2*fps, something could be hacked together, probably. If someone could supply me with an 8th day per week, that is.

[Backwoods]

For the limited case of interpolating to 2*fps, something could be hacked together, probably. If someone could supply me with an 8th day per week, that is.

I have 3 days off every other week, I could give you one of those once in a while.

[Didée]

Yippieh, thank you. Send'em by postal, please :D

---

Okay, here is something to try. It's not at all what I had in mind, but another approach. A trick, better said.

The most annoying parts are those where the warping creates those "funny structures", aren't they. By try-and-error, I found that compairing

edges(source)-->MVConvertFPS

with

edges(source-->MVConvertFPS)

seems to give a halfway usable indication of those typical artefacts. The following script builds a mask out of this comparison, and as a replacement copies a frameblend onto the affected areas.
It's far from perfect, and surely some spots can be found where it looks worse than before. But overall, it looks like an improvement.

(Needs mt_masktools.dll) fps_num = 50 # Numerator & Denominator for
fps_den = 1 # target framerate, as usual
edgemode = "prewitt" # or "min/max"
sensitivity = 9 # the lower, the more blending

clp = last
ox = clp.width()
oy = clp.height()
SNS = string(sensitivity)

#blend = interleave(clp,merge(clp,clp.deleteframe(0),0.5))
blend = o.converttoyuy2().convertfps(float(fps_num)/fps_den).converttoyv12()
edge_orig = clp.mt_edge(mode=edgemode,thY1=0,thY2=255)

bw1_vec1 = clp.MVAnalyse(isb = true, truemotion=true, lambda=200, searchparam=8, pel=2, idx=1)
fw1_vec1 = clp.MVAnalyse(isb = false, truemotion=true, lambda=200, searchparam=8, pel=2, idx=1)
cropped = clp.crop(4,4,-4,-4,true)
bw1_vec2 = cropped.MVAnalyse(isb = true, truemotion=true, lambda=4000, searchparam=8, pel=2, idx=2)
fw1_vec2 = cropped.MVAnalyse(isb = false, truemotion=true, lambda=4000, searchparam=8, pel=2, idx=2)

FPSconverted = clp.MVFlowFps2(bw1_vec1,fw1_vec1,bw1_vec2,fw1_vec2,num=fps_num,den=fps_den,idx=1,idx2=2)
FPSconverted_edge = FPSconverted.mt_edge(mode=edgemode,thY1=0,thY2=255)
edge_FPSconverted = edge_orig.MVFlowFps2(bw1_vec1,fw1_vec1,bw1_vec2,fw1_vec2,num=fps_num,den=fps_den,idx=3,idx2=4)

edgediff = mt_lutxy(edge_FPSconverted,FPSconverted_edge,yexpr="y x - 1 - x 1 + / 300 * x 2 ^ x 2 ^ "+SNS+" + / *",U=1,V=1)
repmask = edgediff.mt_expand().mt_expand().bicubicresize(ox/3/4*4,oy/3/4*4)
\ .mt_inflate().blur(1).bicubicresize(ox,oy,1,0)

mt_merge(FPSconverted,blend,repmask,U=3,V=3,luma=true)

Result on a short sequence:

http://img256.imageshack.us/img256/4751/motiontestzb7.th.jpg (http://img256.imageshack.us/my.php?image=motiontestzb7.jpg)


But now, please, somebody help out the clueless n00b that I am ...
I've tried it only on 25-->50 conversion. In the script, you can see that I used a manual blending for the replacement. This is okay for integral FPS conversion, but obviously not for free ones.
Now, just a minute ago, I discovered that the "ConvertFPS" blending looks quite different than my manual one ... it doesn't even look like an even merge. Why is this? Shouldn't they be identical in the case of 25->50 fps? What is ConvertFPS actually doing?

[foxyshadis]

Perhaps instead of plain blending, replacing with a blend of back & forward MVDepan (you already have the analysis, after all), but I don't think MVDepan can actually give you that sort of interframe interpolation. (Perhaps for v1.5.0.) Object-based depan would be even cooler as a fallback, but I'm sure it's an area fraught with dragons.

Any "trick function" of yours usually becomes a widely recommended , cpu-abusive filter, so make it a good one. ;) btw, it might've been nice to add a pure convertfps row on your example, to find out if it looks much better than both or leans too heavily on the convertfps.

[Didée]

Yeah, right ... of above 5 frames, here are the middle 3 frames for comparison, with a plain blend conversion:

http://img69.imageshack.us/img69/3094/motiontest2ako7.th.jpg (http://img69.imageshack.us/my.php?image=motiontest2ako7.jpg)

In comparison with the previous 5-frame shot, one can see that the correctly interpolated areas are hardly affected ... from another point of view:

These are the masks that come up from the edgemask-comparison -- the raw errormask on top, on bottom the broadened mask for covering the suspicious areas:

http://img107.imageshack.us/img107/7148/motiontest2bmx0.th.jpg (http://img107.imageshack.us/my.php?image=motiontest2bmx0.jpg)


However, I fear there isn't too much headroom to further improve this thing. The sensitivity can be adjusted, and perhaps the raw errormask could get a little more hotpixel removed, before broadening. But in general, I don't see how to make it more intelligent ... I'm surprised enough that it works as nice as it actually does. ;)

***

Object-based depan, eh? Well, spelled it is easily. :D

[foxyshadis]

Ah, yes, that really does seem to be the best of both worlds (as far as a quick and dirty function goes). I'll point fizick toward it, if he hasn't seen it already, and perhaps a variation on it can be included in the next revision as a better mask.

I realized just now that "object-based depan" could be approximated simply with a big block size. 48 or 64 or more. I wonder if playing around with ml would force it to be more like a block-based compensation that would presumably look better for fallback, given that it's flow's artifacts we're trying to avoid here. ^^;

[Chainmax]

http://img204.imageshack.us/img204/5669/lovethreadxs3.jpg (http://imageshack.us)

:)

[Fizick]

Didйe,
Yes, ConvertFPS has a bug.

int nsrc = int( n * fa / fb );
int frac = int( (((n*fa) % fb) << resolution) / fb );

if( zone < 0 ) {

// Mode 1: Blend full frames
int mix_ratio = one - min( int( (fb * (one - frac)) / fa ), one);

// Don't bother if the blend ratio is small
if( mix_ratio < threshold )
return child->GetFrame(nsrc, env);
....

a_data[x] += ((b_data[x] - a_data[x]) * mix_ratio + half) >> resolution;



IMHO, must be simple :


int mix_ratio = frac;

I can change v2.5.7 source in CVS, while IanB is busy :)

Thanks for interesting discusion.
Some comments to rate conversion.
MVFlowFPS currently produces sharp output.
Probably I will add some (motion ?) blur to "funny" regions in next version (and some parameter to control it).
May be by weigthed mix of original, shifted, and half-way shifted pixels.

[Chainmax]

...
(Needs mt_masktools.dll) fps_num = 50 # Numerator & Denominator for
fps_den = 1 # target framerate, as usual
edgemode = "prewitt" # or "min/max"
sensitivity = 9 # the lower, the more blending

clp = last
ox = clp.width()
oy = clp.height()
SNS = string(sensitivity)

#blend = interleave(clp,merge(clp,clp.deleteframe(0),0.5))
blend = o.converttoyuy2().convertfps(float(fps_num)/fps_den).converttoyv12()
edge_orig = clp.mt_edge(mode=edgemode,thY1=0,thY2=255)

bw1_vec1 = clp.MVAnalyse(isb = true, truemotion=true, lambda=200, searchparam=8, pel=2, idx=1)
fw1_vec1 = clp.MVAnalyse(isb = false, truemotion=true, lambda=200, searchparam=8, pel=2, idx=1)
cropped = clp.crop(4,4,-4,-4,true)
bw1_vec2 = cropped.MVAnalyse(isb = true, truemotion=true, lambda=4000, searchparam=8, pel=2, idx=2)
fw1_vec2 = cropped.MVAnalyse(isb = false, truemotion=true, lambda=4000, searchparam=8, pel=2, idx=2)

FPSconverted = clp.MVFlowFps2(bw1_vec1,fw1_vec1,bw1_vec2,fw1_vec2,num=fps_num,den=fps_den,idx=1,idx2=2)
FPSconverted_edge = FPSconverted.mt_edge(mode=edgemode,thY1=0,thY2=255)
edge_FPSconverted = edge_orig.MVFlowFps2(bw1_vec1,fw1_vec1,bw1_vec2,fw1_vec2,num=fps_num,den=fps_den,idx=3,idx2=4)

edgediff = mt_lutxy(edge_FPSconverted,FPSconverted_edge,yexpr="y x - 1 - x 1 + / 300 * x 2 ^ x 2 ^ "+SNS+" + / *",U=1,V=1)
repmask = edgediff.mt_expand().mt_expand().bicubicresize(ox/3/4*4,oy/3/4*4)
\ .mt_inflate().blur(1).bicubicresize(ox,oy,1,0)

mt_merge(FPSconverted,blend,repmask,U=3,V=3,luma=true)
...


So, this needs MT_MaskTools and MVTools, right? Can someone transform it into a standalone function?

[Boulder]

Would it simply be
function WhatEver(clip clp, int "fps_num", int "fps_den", string "edgemode", int "sensitivity")
{
fps_num = default(fps_num, 50) # Numerator & Denominator for
fps_den = default(fps_den, 1) # target framerate, as usual
edgemode = default(edgemode, "prewitt") # or "min/max"
sensitivity = default(sensitivity, 9) # the lower, the more blending

ox = clp.width()
oy = clp.height()
SNS = string(sensitivity)

#blend = interleave(clp,merge(clp,clp.deleteframe(0),0.5))
blend = o.converttoyuy2().convertfps(float(fps_num)/fps_den).converttoyv12()
edge_orig = clp.mt_edge(mode=edgemode,thY1=0,thY2=255)

bw1_vec1 = clp.MVAnalyse(isb = true, truemotion=true, lambda=200, searchparam=8, pel=2, idx=1)
fw1_vec1 = clp.MVAnalyse(isb = false, truemotion=true, lambda=200, searchparam=8, pel=2, idx=1)
cropped = clp.crop(4,4,-4,-4,true)
bw1_vec2 = cropped.MVAnalyse(isb = true, truemotion=true, lambda=4000, searchparam=8, pel=2, idx=2)
fw1_vec2 = cropped.MVAnalyse(isb = false, truemotion=true, lambda=4000, searchparam=8, pel=2, idx=2)

FPSconverted = clp.MVFlowFps2(bw1_vec1,fw1_vec1,bw1_vec2,fw1_vec2,num=fps_num,den=fps_den,idx=1,idx2=2)
FPSconverted_edge = FPSconverted.mt_edge(mode=edgemode,thY1=0,thY2=255)
edge_FPSconverted = edge_orig.MVFlowFps2(bw1_vec1,fw1_vec1,bw1_vec2,fw1_vec2,num=fps_num,den=fps_den,idx=3,idx2=4)

edgediff = mt_lutxy(edge_FPSconverted,FPSconverted_edge,yexpr="y x - 1 - x 1 + / 300 * x 2 ^ x 2 ^ "+SNS+" + / *",U=1,V=1)
repmask = edgediff.mt_expand().mt_expand().bicubicresize(ox/3/4*4,oy/3/4*4)
\ .mt_inflate().blur(1).bicubicresize(ox,oy,1,0)

return mt_merge(FPSconverted,blend,repmask,U=3,V=3,luma=true)
}

Probably not :D

[Didée]

Yep, that's all. Adding a function header line, one { on top, one } at bottom, and do the "default()" thingy for the parameters. Easy, isn't it? ;)

Next exercise: porting for motion.dll filters. :)

[Chainmax]

Thanks, Boulder :). So, the dependencies are MVTools and MT_Masktools only? Also, it doesn't have anything to do with salFPS (what Mug Funky posted), right?

[Pookie]

Boulder - Shouldn't the line

blend = o.converttoyuy2()

be modified to

blend = clp.converttoyuy2()

Maybe I missed the "o" variable declaration earlier in the thread.

[Didée]

Yes, it should be like that.

Oh-oh, my fault. I missed that one during replacing all o's with clp's, in order to place a linear script in front of Chainmax, so that he'll feel the urgent need to make a function out of it.
... The important thing is, the fish has bitten. :D

[Chainmax]

At the risk of quoting my custom title: huh?

[Backwoods]

German Didée English to American English translation: Something slipped through while changing calls over, whoops! Well at least people are interested and caught the mistake.

I haven't tried it yet (nothing on my machine to play with), have you Chainmax?

[Chainmax]

Will do ASAP, I have a 15fps video that shows very noticeable artifacts on rapid camera movements.

[Chainmax]

Brom a very brief gander at one trouble scene, there's still quite a bit of artifacting. IIRC, it's not as messy as plain MVFlowFPS/2, but it's still there.

I just had an idea: remember how I was talking about trying to come up with a metric that would be tied to the artifacting and would be used as a threshold? Well, MVAnalyse can estimate motion, right? What if Didée's function was altered so that when motion exceeds a (user definable) threshold then the extra frames would be created via duplication or interpolation of the source's frames?

[PeppeontheNet]

Hello,
taking pieces of code here and there....

PAL to NTSC conversion

loadplugin("C:\Programmi\AviSynth 2.5\plugins\dgdecode.dll")
loadplugin("C:\Programmi\AviSynth 2.5\plugins\tdeint.dll")
loadplugin("C:\Programmi\AviSynth 2.5\plugins\tivtc.dll")
loadplugin("C:\Programmi\AviSynth 2.5\plugins\depan.dll")
video=MPEG2Source("C:\...\bbb.d2v").LanczosResize(352,576)
audio=WAVsource("C:\...\bbb T01 2_0ch 256Kbps 48KHz.wav")
audiodub(video,audio)
delayaudio(0.0)
blindpp(cpu=4, iPP=true)
assumetff()
TDeint(mode=1, tryweave=true, full=false)
Bicubicresize(width, 480)
pal2ntsc()
separatefields().selectevery(4,0,3).weave()

NTSC to PAL conversion

loadplugin("C:\Programmi\AviSynth 2.5\plugins\dgdecode.dll")
loadplugin("C:\Programmi\AviSynth 2.5\plugins\tdeint.dll")
loadplugin("C:\Programmi\AviSynth 2.5\plugins\tivtc.dll")
loadplugin("C:\Programmi\AviSynth 2.5\plugins\depan.dll")
video=MPEG2Source("C:\...\aaa.d2v").Bicubicresize(352, 480)
audio=WAVsource("C:\...\aaa T01 2_0ch 256Kbps 48KHz.wav")
audiodub(video,audio)
delayaudio(0.0)
blindpp(cpu=4, iPP=true)
assumetff()
TDeint(mode=1, tryweave=true, full=false)
ntsc2pal()
LanczosResize(352,576)
separatefields().selectevery(4,0,3).weave()

where pal2ntsc() and ntsc2pal() are functions written in an .avsi file placed in the avisynth plugins' directory.
Here's that .avsi file:

function PAL2NTSC(clip video)
{
backward_vec = video.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vec = video.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
cropped = video.crop(4,4,-4,-4) # by half of block size 8
backward_vec2 = cropped.MVAnalyse(isb = true, truemotion=true, pel=2, idx=2)
forward_vec2 = cropped.MVAnalyse(isb = false, truemotion=true, pel=2, idx=2)
return video.MVFlowFps2(backward_vec,forward_vec,backward_vec2,forward_vec2,num=60000,den=1001,idx=1,idx2=2)
}

function NTSC2PAL(clip video)
{
backward_vec = video.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vec = video.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
cropped = video.crop(4,4,-4,-4) # by half of block size 8
backward_vec2 = cropped.MVAnalyse(isb = true, truemotion=true, pel=2, idx=2)
forward_vec2 = cropped.MVAnalyse(isb = false, truemotion=true, pel=2, idx=2)
return video.MVFlowFps2(backward_vec,forward_vec,backward_vec2,forward_vec2,num=50,den=1,idx=1,idx2=2)
}

These are the outcomes:
PAL to NTSC sample (http://www.savefile.com/files/206253)
NTSC to PAL sample (http://www.savefile.com/files/206377)

I cannot judge the quality of the PAL to NTSC conversion, because I don't have an NTSC tv screen.... but the outcome of the NTSC to PAL conversion is, by far, the best I have ever experienced.

Suggestions to improve?

[Mug Funky]

here's my latest attempt:

function salFPS3(
\ clip input, float FPS,
\ int "mode", int "protection", int "protection2",
\ int "iterate", int "reset", int "initialise")
{
mode = default(mode, warp)#fast)
protection = default(protection, 80)
protection2= default(protection2,20)
iterate = default(iterate, 2)
reset = default(reset, 50)
initialise = default(initialise, 6)

originalFPS = input.framerate()
input

mp = FindMotion(iterate = iterate, reset = reset,
\ initialise = initialise)
p = MotionFPS(FPS,mode,Motion = mp)
p_edge = mt_edge(mode="prewitt",thy1=protection2,thy2=0).MotionFPS(FPS,mode,Motion = mp)
maskp = mp.SumStretchFPSMask(FPS, protection)

mn = FindMotion(iterate = iterate, reset = reset,
\ initialise = initialise, from = next)
n = MotionFPS(FPS,mode,Motion = mn, source = next)
n_edge = mt_edge(mode="prewitt",thy1=protection2,thy2=0).MotionFPS(FPS,mode,Motion = mn, source=next)
maskn = mn.SumStretchFPSMask(FPS, protection, source = next)

maskp
assumeFPS(originalFPS)
CombineFPS(FPS, maskp, maskn.Invert())
bilinearresize(width*8, height*8)
crop(0, 0, input.width(), input.height())
fity2uv()
mpro=last

mt_merge(p_edge, n_edge, last, u=3, v=3)

mt_lut(expr="128 x 128 - abs - 2 *")
mt_inpand().greyscale().converttorgb32()
bicubicresize(width/8,height/8,1/3.,1/3.)
mod16().converttoyv12(matrix="pc.601")
mt_binarize(protection2,false)
temporalsoften(1,255,255,255)
mt_binarize(80,false)
mt_expand(mode=mt_circle(3))
mt_inflate()
bicubicresize(width*8, height*8,1,0)
crop(0, 0, 720, 480)
fity2uv()
tehmask=last

mt_merge(p, n, mpro, u=3, v=3)

mt_merge(last,input.blendfps(FPS,aperture=FPS/float(input.framerate()))
\,tehmask, u=3,v=3)
}


basically, it builds on didee's edgemasking idea.

the same motion data is used to motion compensate both the original clip and a binary edgemask of it.

my assumption is that when the vectors are poor, the converted binary mask will be less binary :) run a LUT on it that makes the middle tones white and the darks and lights both black, and you have your mask.

this appears to work pretty well on some 30p animation i'm playing with (intro to Clone High), but i haven't tried it on natural video yet. i'll do that on monday when i'm at work and have some video to play with...

observations so far:

it seems to work the way i thought it would, however that doesn't play out visually - the edges of objects go very wobbly sometimes.

i think we need some more tools in our mask-making endeavors. namely something that can detect objects, or at least follow a likely edge around objects. this way we can decide on an object-by-object basis whether blending or compensation is the way to go. however, with that said this kind of data can also very much improve motion compensation, and possibly remove much of the need for blending - if we can detect objects we can define their motion and interpolate it, possibly taking occlusion into account. i think we're a fair way off that.

so anyway, have a look at this script and see if it gives anyone some ideas...

[krieger2005]

What is "mod16()"?

[Mug Funky]

ah crap, sry i forgot about that. it's a little function to force a clip to mod16 size, so it can be worked with more easily. it's got a counterpart "unmod16()", but that's not used in there.

function Mod16 (clip c, bool "mirror")
{
mirror = default(mirror,false)

modwidth = ceil(c.width/16 + 0.5)*16
modheight = ceil(c.height/16 + 0.5)*16

out = c.addborders(0,0,modwidth - c.width,modheight - c.height)
out2 = stackvertical(stackhorizontal(c,c.fliphorizontal()),stackhorizontal(c.flipvertical(),c.turn180())).crop(0,0,modwidth,modheight,align=true)
global gl_diff_width = int(out.width - c.width)
global gl_diff_height = int(out.height - c.height)
mirror==false? out : out2
}

function unMod16 (clip c)
{
c.crop(0,0,-gl_diff_width,-gl_diff_height)
}

just copypasta that somewhere and it should work.

[Chainmax]

Mug Funky, do you want me to upload a short clip of the trouble scene I used to test some of the early methods posted in here?

[Chainmax]

By the way, do you think it would be possible to include some sort of motion metric (DeShaker uses one on the first pass IIRC) and on scenes with motion higher than a threshold the frames that have to be created would be a blend of its two neighbors?

[Mug Funky]

there probably wouldn't be much to be gained from that, speed wise.

consider that motion.dll tools by default need 50 frames previous to the current frame to initialise... this means a conditional filter that's decided 3 frames in a talking head video have not enough motion to need compensation will actually be performing a motion search on an extra 50 frames once the low motion frames are done with, and 47 of these frames will have been searched before already... that's a fair bit of double-handling.

i know because the current incarnation of "autoPAL" does similar to this, and lemme tell you it's a big and unnecessary speed hit. of course you can lower the "reset" parm in the appropriate functions, but this will have a (very slightly) negative effect on smooth video.

btw, when the above function is working the way i'd like, it could probably be posted in this thread as an alternative to the alchemist - already it's kicking quite a bit of arse, it's just got a couple of sticking points (like 3:2 pattern changes getting blended unnecessarily, and very subtle but visible interlaced motion being picked up as film because the metrics say yes while the heart says no).

[foxyshadis]

That sounds like a very interesting alternative to the hclip I suggested for TDecimate. It's rather more widely applicable too. Based on your experience with it, would you say it'd be fairly easy to take any such mess and generate a single progressive stream in a somewhat arbitrary framerate? For instance, if this was to be used in NTSC hybrid->cfr conversions, or pc-based applications with integral framerates.

[Mug Funky]

with a good enough framerate converter (or an interlaced output), it's perfectly possible to go to an arbitrary frame rate. i've implemented a very simple function that takes the "data" clip (8x4 red, green or blue clip that's the output of the analysis function) and outputs whatever framerate you ask for. it's not optimal yet because i've been focusing on the NTSC-to-PAL part of things for the last couple of months.

certainly if tfm could output a similar data clip, it'd be very useful - it'd be faster and support more than the vanilla 3:2 with no edits which my scripted version detects. there'd also be less double-handling as it could be used as hints for the IVTC that follows (right now there's tons of double-handling in autopal mainly due to my own laziness - expanding the format of the data clip from more than just 3 colours to something that includes the position in the 3:2 cycle and stuff like that would be good).

[Chainmax]

there probably wouldn't be much to be gained from that, speed wise.

consider that motion.dll tools by default need 50 frames previous to the current frame to initialise... this means a conditional filter that's decided 3 frames in a talking head video have not enough motion to need compensation will actually be performing a motion search on an extra 50 frames once the low motion frames are done with, and 47 of these frames will have been searched before already... that's a fair bit of double-handling.

i know because the current incarnation of "autoPAL" does similar to this, and lemme tell you it's a big and unnecessary speed hit. of course you can lower the "reset" parm in the appropriate functions, but this will have a (very slightly) negative effect on smooth video.
...

It can always be made optional.

[Mug Funky]

it could, but to what end? blending frames with low enough motion would give similar-but-probably-worse results than motion-compensation, and take longer unless the video being processed is very still indeed. i don't see a reason to use it to be honest, though i think there's room for the mocomped converting function to take low motion into account and work faster to save time - perhaps regular motionprotectedfps (very fast) gets used if the motion-masks used are below a certain peak luma or something? it'd have to be something that gives a net gain in speed though, so metrics have to be very simple and fast to calculate.

[Chainmax]

Actually, what I was referring to was to use blending in scenes with high motion, which is usually when warping occurs. In any case, do you want me to upload the clip I talked about for testing purposes?

[Mug Funky]

aah. okay.

uploading wont be necessary - i haven't really got the time to check out samples :( (there's enough stuff here to deal with already :)).

one possible problem is that pans can be very high motion and yet still be easily compensated. what's needed is a good way to detect warping. didee's edge-detection idea is the strongest contender so far (i had a shot at one, but it's severely lacking in some cases, like sudden luma changes where no motion is detected. whole frames can get very odd looking). perhaps if edge-based warp detection could be combined with the block-warping masks made in motionprotectedfps something nice could be done. i'm not sure how fast such a solution would be though.

i've read a little on what akupenguin's wanting to do with snow - mesh-based warping falling back on OBMC on edges. perhaps the switching decision does something we'd be interested in seeing here?

[Didée]

Some time ago I had the following thought about (perhaps) improving on the warping-artefacts problem of motion interpolation. However it's impossible to do by a script - the needed steps can only be done by the motion engine itself.

Basic problem:
A motion vector in itself might appear reasonably good (low enough SAD-or-whatever) when looking its result at the 'discrete' (frame->frame) target point. But then, although the result of the "full" vector seems good, its result is visually bad when only a *part* of the vector is used. ("part" of a vector -> e.g. the "time" parameter of interpolating commands). This is because in fact the vector is not aligned with the "true" motion.
Now, the machine won't really reckognize what the "true" motion really is, because the machine is dumb. Also, there is no reference frame available that we could check the created intermediate frame against. (Frame->frame compensations, as e.g. in denoising, can be checked nicely that way ... but not fractional interpolations.)

Okay, here the idea:
A filter like MVFlowFPS creates a new frame, at some certain "time" between to existing frames. To create this inbetween frame, of each motion vector's length only a certain fraction is used.

Now, do the following:
Take the created inbetween frame (which is possibly containing artefacts), and do "the remaining" compensation. Means: for each block/pixel, make an interpolation by using those parts of the vectors that have not been used in the former interpolation step. Then, compare this 2nd-step interpolation to the destination frame. If the difference (TwoStepInterpolation - Destination) is noticeably bigger than the difference (OneStepCompensation - Destination), then the related vectors are marked as bad. Finally, the one-step interpolation is recreated, now with the additional information of which vectors are bad.

Visually (kind of):
We have frames A and B. Needed is creation of an intermediate i(A,B), to get A i(A,B) B.
Currently, vectors v(A,B) are searched. Then by means of v(A,B), full compensations A' and B' are created. (Ignoring the checks of vector disrupture etc.), it is checked A against A' and B against B' to decide if vectors are good or not (SAD measure). If yes, from v(A,B) the needed part v'(A,B) is used to create i(A,B).
My suggestion is to not only check the result of A -> A' against B, but to do
A -> i(A,B) -> A'', and to check A'' against B. The artefacts appearing in i(A,B) will also distort A'', so that checking A'' against B should give a reasonable measure to better judge the vectors.


Overall, this sounds more dreadful than it is. Basically there's only one point which I can't judge if it's easy, difficult or impossible to do: for each pixel, it must be memorized and/or figured which parts of which vectors have been used to create the "normal" inbetween interpolation, in order to later on use the not-used "remaining parts" of the vectors.
That's the cruicial point. Can this be done, or can it not? The obvious foottrap is the interpolation-of-vectors that's used by the warping technique.

IF this is possible, then I suspect that the warping artefacts appearing with the normal method should map "nicely" in the 2nd-step interpolation, and could be reckognized thereby.

[Blue_MiSfit]

My brain hurts, but I think I grasp the idea. Now if only I could write C code! :)

[Chainmax]

Mug Funky: what does the switching decision when using warping w/fallback on OBMC on edges do?

Didée: that sounds like an excellent way to reduce warping. Do you expect it to reduce filtering speed by much?

[Fizick]

Didйe,
I remember some motion compensated deinterlace method (patent ?) description, with very long and complex pixels checking and replacing (temporal, spatial, linear, etc), with final part :"if the resulted pixel is considered as bad, use other method". Of course, other method description is omitted. :)

IMO, the main problem is not how to "judge" bad vector, but what to do if we found that it is bad.

we can easy create Ai as partially forward compensated A (at intermediate position).
we can easy create Bi as partially backward compensated B (at the same intermediate position).
We can compare Ai with Bi and declare all non-almost-equal pixels (or block and corresponded vectors) as "BAD".
(we will got very many such pixels).
O.K. What we will use as a result pixels at these places? Fallback to blended? .
(Similar method uses MVBob, and it often loses most of useful motion compensated information).

I do not understand your proposal :)
but IMO it is similar to part of idea of bidirectional vector interpolation - it may be interesting becourse of possibility to improve the of motion vectors (I considered to implement it some time ago, but ... it is not so easy and vector disrupture is not a small problem anyway.

but if only I could write C code of all my rejected projects!
:)
I know one man who try wrote phase based motion interloalation (but for deinterlacing).
But we need more .

[Didée]

I remember some motion compensated deinterlace method (patent ?) description, with very long and complex pixels checking and replacing (temporal, spatial, linear, etc), with final part :"if the resulted pixel is considered as bad, use other method". Of course, other method description is omitted. :)
Ya, I like that solution. Encourages me to start writing patents. :D


IMO, the main problem is not how to "judge" bad vector, but what to do if we found that it is bad.
"What to do else" is a problem too, yes ... but step by step. Firstly, it must be made the distinction between
- vectors that are really bad
- vectors that seem bad but are usable
- vectors that seem good but in fact are bad

Once there is a better certainty of which are the bad vectors, then it is time to think about better handling of the problematic areas. ;)


I do not understand your proposal :)
but IMO it is similar to part of idea of bidirectional vector interpolation - .
Quite possible. I don't claim that all things I think of are worldbreaking news ... so much things have already been thought of, and so much has been written. If I would try to read everything that already has been written, there would be no more time to do thinking-by-myself, and to have own ideas. But I like having "my own" ideas, even if they should be not new. :)


- it may be interesting becourse of possibility to improve the of motion vectors (I considered to implement it some time ago, but ... it is not so easy and vector disrupture is not a small problem anyway
Parallelization of forward & backward motion search? I.e. when dealing with different candidates for a vector, not just take that candidate with little better SAD / a little better predictor / whatever ... but also look at the vector field of the other direction's interpolation, and try using those of the possible candidates that have the same (just reverse) direction?

Sounds easy, but I bet that's a pain to implement. (?) :D

But it ~could~ solve this problem, at least partly:
we can easy create Ai as partially forward compensated A (at intermediate position).
we can easy create Bi as partially backward compensated B (at the same intermediate position).
We can compare Ai with Bi and declare all non-almost-equal pixels (or block and corresponded vectors) as "BAD".
(we will got very many such pixels).
Since doing ^so^, each vector field (forward/backward) has been computed completely independant of the other one, and some vectors might have been "drifted" in an unlucky direction, because SAD & Co. just happend to make it so. A combined motion search might have delivered better results ... choosing slightly worse candidates instead of the better ones, because the slightly worse resulted in better forward/backward correlation.


O.K. What we will use as a result pixels at these places? Fallback to blended? .
(Similar method uses MVBob, and it often loses most of useful motion compensated information).
Blending wouldn't be all that bad as a preliminary solution. :)

A possible way: do as described with bigger (16) blocks; in areas being problematic, try again with smaller (8) blocks ... and once more with (4).


Okay ... just gimme a :angry::angry: if I'm doing too much rambling here ... it's just that if I have some ideas that cannot be scripted, I tend to post them & ask about feasability. :)


or ... would it be preferable if I made some serious-looking PDF documents out of such ideas, starting with "In this document we propose a superior method to do blablabla...", blown up with overcompressed before-after pictures that look both the same, throw in lots of greek letters and integral signs, plus putting a long list of references at the end, which noone ever reads?
If that is the problem, it can be solved. :D

[Chainmax]

Mug Funky: I intend to use salFPS3 on a DVD I'm creating. The source is a bad NTSC-to-PAL conversion which I'll bob then filter. Would this:

Bobbing
filtering
salfps3(fps=59.94)
separatefields().selectevery(4,0,3).weave()

be a good call or would a better (albeit slower) call be possible? Also, what are salFPS3's dependencies?