MVTools2 artifacts, but only at high resolution

[johnmeyer]

Problem: MFlowFPS interpolated frames look awful at 1600x1080 resolution, but look great when that same video is first scaled to 720x480, and then interpolated.

Here's the test file (16 MB):

16 fps 1600x1080 film capture (http://dl.dropbox.com/u/1561578/Reel1_1_Snip_for_eval.avi)

Here's a test script (using the "latest" MVTools2, version 2.5.11.3):

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")
film=Avisource("E:\Documents\Dnload\UNPACK\Reel1_1_Snip_for_eval.avi")

#film = film.Lanczos4Resize(720,480)

superfps= MSuper(film,pel=2)
backward_vec2 = MAnalyse(superfps, isb = true,blksize=16,overlap=2,search=3)
forward_vec2 = MAnalyse(superfps, isb = false,blksize=16,overlap=2,search=3)
output = MFlowFps(film,superfps, backward_vec2, forward_vec2, num=32, den=1, ml=100,thSCD1=400)
return output

If you feed the video into this script, you will see that every other frame (the interpolated frames) look awful. Here's a sample frame:

https://i.imgur.com/nXOCSTL.jpg

However, if you un-comment the "film = film.Lanczos4Resize(720,480)" line, the interpolation works correctly. Here's what that looks like:

https://i.imgur.com/kVwGpaX.jpg

I have tried changing just about every parameter in MSuper, MAnalyze, and MFlowFPS and at this point have concluded that there is some sort of bug in MVTools2 when dealing with higher resolutions. Having said that, I would have thought lots of people would have reported something. However, this is a very low framerate source, and consequently there are unusually large differences between each frame, so perhaps that is part of the problem.

Any ideas of suggestions would be welcome. Thanks!

[wonkey_monkey]

Hard to say since you've tried changing parameters already, but I would guess that


backward_vec2 = MAnalyse(superfps, isb = true,blksize=32,overlap=4,search=3)
forward_vec2 = MAnalyse(superfps, isb = false,blksize=32,overlap=4,search=3)


would do better. blksize and overlap were the only params I had to change on a certain clip when I went to HD (the defaults worked fine on SD but I got break-up on HD), but of course that's just one clip.

David

[johnmeyer]

Hard to say since you've tried changing parameters already, but I would guess that

<snip>

would do better. blksize and overlap were the only params I had to change on a certain clip when I went to HD (the defaults worked fine on SD but I got break-up on HD), but of course that's just one clip.Yeah, I should have made my initial post a little longer. That was the first thing I tried: since the number of pixels was doubled in each direction (roughly speaking) I figured that doubling the block size and overlap would give MVTools roughly the same "structure" to deal with. It did make things a little better, but not much.

[chainik_svp]

since the number of pixels was doubled in each direction (roughly speaking) I figured that doubling the block size and overlap would give MVTools roughly the same "structure" to deal with

This's a mistake. For the same output block sizes / overlaps should be the same but you need to adjust motion vectors field coherence parameters - "lambda" at the first place.
OR you can just set "plevel"=0 in both cases.

[johnmeyer]

For the same output block sizes / overlaps should be the same but you need to adjust motion vectors field coherence parameters - "lambda" at the first place.
OR you can just set "plevel"=0 in both cases.I've tried dozens of different lambda settings, and none of them make much improvement. Also, at your suggestion, I have now tried plevel=0, and that also makes no difference.

Perhaps I need to do something with a mask. Or, perhaps, there is a way I can use the 720x480 version to create the motion vectors ... no, that won't work.

I'm stumped. Fortunately, I can just use gold old-fashioned pulldown and create the final 29.97 output that way.

I guess there is no solution using MVTools on this "high-res" 16 fps progressive source.

[poisondeathray]

not sure of the answer to your original question, but you can try other mvtools based functions

smoothfps2 seems to do better at original resolution on that clip (of course, still far from "perfect")



function SmoothFPS2(clip source, int num, int den) {
super = MSuper(source, pel=2, hpad=0, vpad=0, rfilter=4)
backward_1 = MAnalyse(super, chroma=false, isb=true, blksize=16, searchparam=3, plevel=0, search=3, badrange=(-24))
forward_1 = MAnalyse(super, chroma=false, isb=false, blksize=16, searchparam=3, plevel=0, search=3, badrange=(-24))
backward_2 = MRecalculate(super, chroma=false, backward_1, blksize=8, searchparam=1, search=3)
forward_2 = MRecalculate(super, chroma=false, forward_1, blksize=8, searchparam=1, search=3)
backward_3 = MRecalculate(super, chroma=false, backward_2, blksize=4, searchparam=0, search=3)
forward_3 = MRecalculate(super, chroma=false, forward_2, blksize=4, searchparam=0, search=3)
MBlockFps(source, super, backward_3, forward_3, num, den, mode=0)
}

[johnmeyer]

smoothfps2 seems to do better at original resolution on that clip (of course, still far from "perfect")Ah, that's much better. Thanks!

I hadn't thought of using MBlockFPS. I'll tweak some settings and see what happens.

[Didée]

In case of integral framerate conversion (here: 16-to-32), you could also do something like:

- perform conversion at small resolution
- upsize
- calculate all-new vectors, make MCompensate, then interleave original/MCompensated frames

film = Avisource(...

film2 = film.BicubicResize(720,480,-.5,.25)

super = MSuper(film2,pel=2)
bvec = MAnalyse(super, isb = true, blksize=16,overlap=2,search=3)
fvec = MAnalyse(super, isb = false,blksize=16,overlap=2,search=3)
out1 = MFlowFps(film2,super, bvec, fvec, num=0, den=0, ml=100,thSCD1=400)
\.spline36resize(film.width(),film.height())

super2 = MSuper(out1,pel=2)
bvec2 = MAnalyse(super2, isb = true, blksize=16,overlap=8,search=3)
fvec2 = MAnalyse(super2, isb = false,blksize=16,overlap=8,search=3)
bcmp = out1.MCompensate(super2,bvec2)
fcmp = out1.MCompensate(super2,fvec2)
mix = Merge(bcmp,fcmp)

output = Interleave( film, mix.selectodd() )

return( output )

[johnmeyer]

In case of integral framerate conversion (here: 16-to-32), you could also do something like:Thanks for that Didée. I probably should have emphasized, in my OP, that this script was a test so I could easily compare the original (even frames) with the MC frames (even frames). The actual script used 30000/1001 as the num/den ratio.

I'll see if I can adapt some of your ideas to the actual final output.

Thanks!

[Didée]

For non-integral framerate conversion it wil get quite tricky.

However, the general idea ist just this: instead of being done with either-method's artifacts (MFlowFPS: swirling / MBlockFPS: blocking [due to "overlap" not being available]), first create an interpolation with whatever-method, then use another motion-instance to "MCompensate" the original frames onto the 1st-stage-interpolations.

As said: this is pretty easy to do for integral framerate ratios, but will get tricky for odd ratios. (The usual trick would be to use the method to first create a 4-times, 5-times, ... or 10-times framerate increase, and then SelectEvery() or ChangeFPS() to get the desired framerate. This implies a small jitter in the result, but usually it isn't noticeable.)

[chainik_svp]

There're two issues here.
1. Motion vectors. The simpliest way to get almost identical vectors in both cases (hi-res and low-res) is "plevel"=0. You can check the actual vectors with MShow ;)
2. Masking. It's called "occlusion mask" in MVTools docs and it can be really confusing in some cases (and by "confusing" I mean "wrong"). Try to disable masking with "ml"=<some_big_number>.

I believe that with plevel=0 and ml=99999 you should get almost the same output in hi- and low- resolutions. I'm not saying it'll be good but the same.
Anyway SVPflow is always better ;)

[johnmeyer]

That's a great help Didée. When I get back to the project on Monday, I'll see if I can incorporate your ideas into a modified script.

[johnmeyer]

I believe that with plevel=0 and ml=99999 you should get almost the same output in hi- and low- resolutions. I'm not saying it'll be good but the same.
Anyway SVPflow is always better ;)I had played with both plevel and ml, but not to those extremes (I was using ml=100 through ml=200), and not in those combinations.

As to SVPflow, I didn't realize that it used different algorithms. I thought most of changes were made in order to cut out much of the "un-used" stuff from MVTools2 and then optimize the remaining code (but using the same algorithms) so you could get real-time frame rate conversion for home theater, etc., especially with MT versions of AVISynth.

I do remember one post in that thread where Didée mentioned improved block detection (something to do with contrast, I think), but I didn't realize that a major change had been made.

I guess I need to download and try SVPflow in my little test script.

I am well aware of the limitations of ME for synthsizing frames (even Twixtor breaks down on picket fences, people walking in front of the camera, etc.), but the problems with this test clip should not be happening, so I'm sure that between the four ideas presented so far (MBlockFps; separate interpolation and compensation into separate steps; continue to tweak MVTools2 settings; use SVPflow) I will end up with a much, much better result.

[-Vit-]

The problem you're seeing is mostly due to a limitation in MVTools. For MFlow, MFlowInter, MFlowFPS and MFlowBlur the motion vector x & y values cannot exceed 127 or the results will be (increasingly) incorrect. It's because those functions use a single byte to hold the vector values rather than the 4-byte integer they are normally stored in. Using a Pel of 2 that's 127 half-pixels - faster motion will not interpolate properly in those functions. In fact MFlow is completely broken in the last "official" release, but cretindesalpes seems to have fixed that in the most recent dither-tools MVTools build (which you should be using). But the 127 limitation remains. It wouldn't be complex to fix, but it would require quite a lot of changes to those functions and a new resizer.

The reason Didée's suggestion works is because MCompensate is unaffected. I don't think MBlockFPS is affected even though it uses some of the same code.

I wrote a little used plugin for vector scaling. You can find the functions in the latest dither-tools MVTools build. You can analyze the vectors on a half-sized clip then double the scale of the vectors for use on the full size clip. Doing that removes a few artefacts, but the main problem I outlined above remains because after scaling the vector values exceed 127. But were MVTools fixed then this would be a good approach - it's faster, uses less memory and can be more accurate for low detail material like this. Comment in one of the output lines at the end to see either the vectors or the small or large resolution flowfps output:

clip = AviSource("...")
half = clip.BicubicResize(clip.Width()/2,clip.Height()/2)

superHalf = half.MSuper(pel=2, hpad=8,vpad=8)
super = clip.MSuper(pel=2, levels=1, hpad=16,vpad=16) # Require double padding to use with double sized vectors

bVec1h = superHalf.MAnalyse(isb=true, blksize=16,overlap=4, search=3) # Calculate vectors at half-size
fVec1h = superHalf.MAnalyse(isb=false, blksize=16,overlap=4, search=3)
bVec1 = bVec1h.MScaleVect(2) # Scale vectors to full size, need MVTools from dither tools
fVec1 = fVec1h.MScaleVect(2)

#output = MShow(superHalf,bVec1h) # Show half-size vectors
#output = MShow(super,bVec1) # Show scaled full-size vectors - showing that they're just the same as above
#output = half.MFlowFps(superHalf,bVec1h,fVec1h,den=0, ml=100,thSCD1=400) # Show flowfps on half-size clip - OK
output = clip.MFlowFps(super,bVec1,fVec1,den=0,ml=100,thSCD1=400) # FlowFPS on full-size clip with scaled vectors - 127 problems :(

return output


lambda, plevel and ml are only relevant when things are working properly, they will have little impact on this problem.

[chainik_svp]

-Vit- - yeah, it's a good point :) if 63 pixels vector is the real problem in this case.

BTW MVs are clipped to +-1027 in SVPflow (3 extra bits comparing to MVTools) :)

[-Vit-]

if 63 pixels vector is the real problem in this case.
I'm certain the 127 pel limit is a major factor here, this video has high resolution, fairly rapid panning and low frame-rate. It's easy to test for, try upscaling the source to ever increasing resolutions and you see two images diverge as the forward and backward compensations hit their 127pel limit. There may be some lesser issues underneath... but the analysis seems OK since Didée's solution works fairly well.

[johnmeyer]

The problem you're seeing is mostly due to a limitation in MVTools. For MFlow, MFlowInter, MFlowFPS and MFlowBlur the motion vector x & y values cannot exceed 127 or the results will be (increasingly) incorrect.Bingo! This fits exactly with what I was seeing, especially your description that "the results will be increasingly incorrect." In particular, when I artificially increased the resolution of the clip (using a simple resize), I observed an increasingly large "shadow" of the main objects. This was in addition to the more obvious and perverse structures that started to appear, and which you can see in the first still image in my initial post. My first thought is that the SAD threshhold was set wrong, but after a few tests it was obvious that this wasn't the case.

The question back to chainik_svp is whether any of these limitations were altered, reduced, or eliminated in SVPflow?

As for the version to use, I am using version 2.5.11.3. I cannot find any reference to an MVTools2 "dither tools" release, but when I searched for MVTools and "cretindesalpes," I found a version 2.5.11.1 beta, but that was released almost a year before the version that I am using, and it is labeled "beta."

Is that the version you think I should try using?

[edit] I see -vit- posted while I was writing my post. His post seems to confirm what I wrote above.

[Reel.Deel]

The latest Dither package can be found here (http://forum.doom9.org/showthread.php?p=1386559#post1386559).

[johnmeyer]

The latest Dither package can be found here (http://forum.doom9.org/showthread.php?p=1386559#post1386559).Thanks!!

[chainik_svp]

johnmeyer[/b
The question back to chainik_svp is whether any of these limitations were altered, reduced, or eliminated in SVPflow?

Right now vector's components are clipped to +-1023 and technically there're no problems to deal with unclipped vectors [b]in gpu mode but I don't know why :) may be it'll be useful for 4k video...

[johnmeyer]

I tried SVpflow (1.0.5), but the SVSmoothFps appears broken. (Perhaps it was I who broke it!).

I used, more or less, the sample script:

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\svpflow1.dll")
loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\svpflow2.dll")

threads=6

SetMTMode(3,threads)
Avisource("E:\Documents\Dnload\UNPACK\Reel1_1_Snip_for_eval.avi")
AssumeFPS(16)
ConvertToYV12()
SetMTMode(2)

#Lanczos4Resize(720,480)

super_params="{pel:2,gpu:1}"

analyse_params="""{block:{w:32,h:32},
main:{search:{coarse:{distance:-10}}},
refine:[{thsad:200}]}"""

smoothfps_params="{num:2,den:1,algo:21,cubic:1}"

super = SVSuper(super_params)
vectors = SVAnalyse(super, analyse_params)
smooth_video = SVSmoothFps(super, vectors, smoothfps_params, url="www.svp-team.com", mt=threads)

return smooth_video

The problem is that no matter what values I use for num and den, I always get a 2x increase in frame rate. I tried using {abs:true}, and also used gpu:0 (in case there was something strange with my GPU), but I still couldn't get anything except 2x. Also, in this 2x default mode, it appears that the synthesized frames are not at the temporal midpoint between the adjacent non-synthesized frames.

The good news is that the motion interpolation appears to operate correctly, even on the "high res" versions of the source clip. Also the performance is quite good compared to MVTools2. So, I'd like to continue to use this function if I can get it to create the correct frame rate.

My test clip is still available in the link in the first post in this thread.

[Bloax]

Ah yes, I definitely know the "2x Framerate" issue.

AFAIR it started around the 1.0.1 versions. (Probably later.)

[johnmeyer]

Ah yes, I definitely know the "2x Framerate" issue.

AFAIR it started around the 1.0.1 versions. (Probably later.)OK, so I'm not alone. However, others are not reporting it (I've looked at the SmoothVideo forum, the AVS forum, and this forum, and haven't seen others complaining, although a lot of the focus has been on framedoubling, so perhaps not many have tried anything except for 2:1).

I'm using AVISynth 2.58 tsp MT version 5(mod seraphy).

[Reel.Deel]

I have not used the SVP plugins like johnmeyer has, but I use the Interframe (http://forum.doom9.org/showthread.php?t=160226) script which uses the SVP plugins and have not had a problem going from 23.976 to 59.94.

[johnmeyer]

OK, I took a coffee break and found the problem:

The sample script is wrong, and needs to be changed.

If chainik_svp doesn't reply here in the next few days, I'll send him a PM so he can make the change.

The "sample.avs" script included in the download includes this line for the parameter definitions for SVSmoothFPS:

smoothfps_params="{num:5,den:2,algo:21,cubic:1,light:{aspect:1.33}}"

This needs to be changed to:
smoothfps_params="{rate:{num:5,den:2},algo:21,cubic:1,light:{aspect:1.33}}"

Interpolation of frames still appears to not be in the middle of the adjacent frames when doing 2:1 (framedoubling). It seems to yield motion that is closer to preceding frame (in time) than frame after.

[chainik_svp]

Have you read the manual (http://www.svp-team.com/wiki/Plugins:_SVPflow)? :)

the sample script in the archive is wrong, right...

Interpolation of frames still appears to not be in the middle of the adjacent frames when doing 2:1

o_O you're the first one who thinks so. any evidence?

[johnmeyer]

Have you read the manual (http://www.svp-team.com/wiki/Plugins:_SVPflow)? :)

I have read that page so many times that I almost have it memorized. Also, as I indicated, I have read messages on your forum; I have read the messages in your SVP thread here in doom9; and I have read the last part of the framedoubling thread on the AVS forum.

the sample script in the archive is wrong, right...Yes, that is the problem that needs to be fixed in the next release. I was able to figure out the correct syntax because ... I read the manual!

Interpolation of frames still appears to not be in the middle of the adjacent frames when doing 2:1o_O you're the first one who thinks so. any evidence?Yes.

Download the video (see my initial post). Then, run that video through this script: loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\svpflow1.dll")
loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\svpflow2.dll")

threads=6

SetMTMode(3,threads)
Avisource("E:\Documents\Dnload\UNPACK\Reel1_1_Snip_for_eval.avi")
AssumeFPS(16)
ConvertToYV12()
SetMTMode(2)

#Lanczos4Resize(720,480)

super_params="{pel:2,gpu:1}"

analyse_params="{block:{w:32,h:32},
main:{search:{coarse:{distance:-10}}},
refine:[{thsad:200}]
}"

smoothfps_params="{rate:{num:2,den:1,abs:false},algo:21,cubic:1}"

super = SVSuper(super_params)
vectors = SVAnalyse(super, analyse_params)
smooth_video = SVSmoothFps(super, vectors, smoothfps_params, url="www.svp-team.com", mt=threads)

return smooth_video

First, note that there is no motion between frames 1 & 2. Second, note that motion between 2&3 is small, followed by a big jump between 3&4. This small/large jump continues in the next several pairs of frames, but gradually improves. It is as though the interpolation algorithm has to "get up to speed." By frame 12 (or thereabouts), the motion seems reasonably uniform.

Since this is true 16 fps (from 8mm film), the spatial difference between frames is quite large. Because of this, perhaps there is some setting I need to change (scene change??).

[chainik_svp]

I promise to run your video and script tomorrow but I'm afraid we're talking about different things.

You said "interpolation of frames still appears to not be in the middle of the adjacent frames".
IF the motion vector is correct then the intermediate object position WILL be exactly in the middle. This is simple math. Otherwise it will be blurred and twisted (or just NOT in the middle in case of 1st and 2nd shaders).
Maximum vector length = 1023 doesn't mean the algorithm really can find such "long" motion. And in your sample there's opposite movement of objects and background which is the real evil.

I was able to figure out the correct syntax because ... I read the manual!
yep, the manual is the key :D

[johnmeyer]

... I'm afraid we're talking about different things.

You said "interpolation of frames still appears to not be in the middle of the adjacent frames".

IF the motion vector is correct then the intermediate object position WILL be exactly in the middle. This is simple math. Otherwise it will be blurred and twisted (or just NOT in the middle in case of 1st and 2nd shaders). You are correct that we are talking about two different things. Let me try to be more clear.

The FlowFPS algorithms creates new pixel blocks in new x/y locations (spatial estimates). In doing so, these algorithms attempt to estimate where those pixels would be if an actual, real frame of video had existed existed at an intermediate moment in time exactly halfway in time between the two frames of video being used for the estimates.

So for example, if everything works correctly, and if there is only horizontal motion (e.g., a side-to-side camera pan over a stationary scene), then if a block is at x=10 in the first frame and x=30 in the second frame, and we double the frame rate, I would expect that the estimated block for the intermediate, synthesized frame would be created at x=20.

What I am instead seeing, mostly at the beginning of my test clip, but also occasionally elsewhere (in scripts other than the reduced test script that I posted) is that the estimated block is at x=13 (or something like that), resulting in a small change between the first original frame and the estimated frame, followed by a much larger jump between the estimated frame and the following frame.

And in your sample there's opposite movement of objects and background which is the real evil.Yes, this is the "broken legs" issue. I am fully aware of this problem and have seen it since I first used this technology using MotionPerfect back in the 1990s. The very expensive Twixtor also has not figured out a solution to this.

Correct settings can reduce, but not eliminate, this problem.

I am also aware of the "morphing" problems shared by all motion estimation algorithms, where certain blocks end up in strange locations, making tiny features, like those on people's faces, appear to "morph" rather than move.

These things are not what I am talking about.


Thanks for creating this superior technology!

[chainik_svp]

there is no motion between frames 1 & 2

That's incorrecty detected scene change.

motion between 2&3 is small, followed by a big jump between 3&4

I see now, it's an "adaptive" interpolation, set scene.mode=0 to get rid of this.
I agree that may be it's a bug to use "adaptive" for 2:1 interpolation, thanks.

[johnmeyer]

That's incorrecty detected scene change.I always forget to check the scene threshold. It needs to be above 7000 for this clip. I set it to 8500. That forced the interpolation and eliminated the dup frame.

I see now, it's an "adaptive" interpolation, set scene.mode=0 to get rid of this.Ah, I should have seen that in the documentation. The word "adaptive" should have given me the hint.

Now that I understand a little, this does look like a "safer" choice for a default.

Thanks!

P.S. Here's the revised smoothfps_params code:
smoothfps_params="{info:true,rate:{num:30,den:16,abs:false},scene:{mode:0,limits:{scene:8500}},algo:21,cubic:1}"

[johnmeyer]

Speaking of scene detection, is there any way to get information on screen or to debugview that can show internals so I can make intelligent choices for various settings?

[chainik_svp]

Sure ;)

debug.qmap=true - paints transparent color over each block showing its "quality":
- no overlay: MV's quality is below "scene.limits.zero" value and it's excluded from estimation
- blue: scene.limits.zero <= MV's quality < scene.limits.m1
- green: scene.limits.m1 <= MV's quality < scene.limits.m2
- yellow: scene.limits.m2 <= MV's quality < scene.limits.scene
- red: MV's quality >= scene.limits.scene

debug.qmode=true - paints color rectangle in the top left corner showing overall interpolation quality. colors are the same as for qmap, so "red" is a scene change detected.

[johnmeyer]

Thanks for that.

Just for the sake of clarity, for those reading this thread, the debug statement is inserted into the smoothfps_params variable (assuming you follow the syntax structure in the examples) and the actual syntax is as follows:

debug:{qmap:true}