SVSmoothFps, params_string, scene question

[markfilipak]

Bounty offered in my 2nd post.

I don't read 'C' so even if I found the source code (which I think is closed source) I wouldn't be able to answer my questions.

My question is about the mode setting in the scene_string in the SVSmoothFps function.
My frame diagrams and questions are in the notes.

I would run some tests using a cooked up source having frame numbers, but I'm a novice and as of now, the only access I have to SVSmoothFps is via InterFrame and InterFrame doesn't support changes to 'scene:'. In that regard, I'd be grateful for any suggestions.

Regards,
Mark Filipak.


SVSmoothFps(source,super,vectors,params_string,[sar]:float,[mt]:integer)

params_string = "{"+rate_string+","+algo_string+","+mask_string+","+scene_string+","+light_string+"}"
:
Extended "scene change" controls.

scene_string = "scene:{mode:3,blend:false,limits:{m1:1600,m2:2800,scene:4000,zero:200,blocks:20},luma:1.5}"
:
0 ...uniform interpolation for maximum smoothness. For example for 24->60 conversion output will be: "1mmmm1mmmm..." [note 1]
1 ..."1m" mode that gives "1mm1m1mm1m..." [note 2]
2 ..."2m" mode: "1m11m11m11..." [note 3]
3 ...adaptive mode that switches between modes 0,1,2 based on overall vector field quality.

[note 1] 'mode:0' "1mmmm1mmmm..." (2 of 4 frames to 10 frames).
Is 'mode:0' the following? (which excludes frame [B] from the interpolation)
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[AAAAA_][AAAAC_][AAACC_][AACCC_][ACCCC_][CCCCC_][CCCCA_][CCCAA_][CCAAA_][CAAAA_] - output (via interpolated motion vectors)
---><--------------- 1/12 s ---------------><--------------- 1/12 s ------------ - picture centers & picture judder: none
Or is 'mode:0' the following? (which includes frame [B] in the interpolation)
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[AAAAA_][AAABB_][ABBBB_][BBBBC_][BBCCC_][CCCCC_][CCCDD_][CDDDD_][DDDDA_][DDAAA_] - output (via interpolated motion vectors)
---><----- 1/24 s -----><----- 1/24 s -----><----- 1/24 s -----><----- 1/24 s -- - picture centers & picture judder: none

[note 2] 'mode:1' "1mm1m1mm1m..." (4 frames to 10 frames).
Is the following what "1mm1m1mm1m..." means?
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[AAAAAA][AAAABB][AABBBB][BBBBBB][BBBCCC][CCCCCC][CCCCDD][CCDDDD][DDDDDD][DDDAAA] - output (via interpolated motion vectors)
------><--- 7/180 s ----><------ 2/45 s ------><--- 7/180 s ----><------ 2/45 s - picture centers & picture judder: 7:8 cadence @ 12 judders/s.

[note 3] 'mode:2' "1m11m11m11..." (7 frames to 10 frames).
Is the following what "1m11m11m11..." means?
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[AA____][AB____][BB____][BB____][BC____][CC____][CC____][CD____][DD____][DD____] - output (via interpolated motion vectors)
s ----><-- 13/360 s ---><------ 17/360 s ------><--- 161/1440 s ----><---- 8/225 - picture centers & picture judder: 516:1145 cadence @ 6 judders/s.

[Alexkral]

Hi, check this out:

https://www.svp-team.com/forum/viewtopic.php?pid=40315#p40315

[markfilipak]

Hi, check this out:

https://www.svp-team.com/forum/viewtopic.php?pid=40315#p40315

Thanks, Alexkral, but there are things that indicate that the diagram is not authoritative.

Just below the diagram, vanden asks, "Is that correct ?"

What is shown as 3:2 pulldown is actually 6:4 pulldown -- a quibble, I know.

If, in the SVSmoothFps documentation, "smoothness" means judder, than "1mmmm1mmmm...maximum smoothness", "1mm1m1mm1m...less smoothness", and "1m11m11m11...much less smoothness" is not consistent with my findings (below) in which "2m" mode appears to have much less judder than does "1m" mode.

1m@60fps (1mm1m1mm1m...): From vanden's diagrams, 1m@60fps (as a frame timing diagram, with my calculations) is:
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[AA____][AB____][AB____][BB____][BC____][CC____][CD____][CD____][DD____][DA____] - output (via interpolated motion vectors)
------><--- 11/300 s ---><----- 7/150 s ------><--- 11/300 s ---><----- 7/150 s - picture centers & picture judder: 24%, 11:14 cadence, 12 judders/s.
A center (from left edge) = (1*(-1*8+4)+2*(0*8+4)+1*(1*8+4)+1*(2*8+4))/5 = 7.2 characters.
B center (from left edge) = (1*(1*8+4)+1*(2*8+4)+2*(3*8+4)+1*(4*8+4))/5 = 24.8 characters.
C center (from left edge) = (1*(4*8+4)+2*(5*8+4)+1*(6*8+4)+1*(7*8+4))/5 = 47.2 characters.
D center (from left edge) = (1*(6*8+4)+1*(7*8+4)+2*(8*8+4)+1*(9*8+4))/5 = 64.8 characters.
A to B (center-to-center) = 24.8-7.2 = 17.6 characters = 11/300 s.
B to C (center-to-center) = 47.2-24.8 = 22.4 characters = 7/150 s.
C to D (center-to-center) = 64.8-47.2 = 17.6 characters = 11/300 s.
D to A (center-to-center) = 80-64.8+7.2 = 22.4 characters = 7/150 s.
judder% = 100*(frame_width_difference)/(average_frame_width)% = 100*(22.4-17.6)/((17.6+22.4)/2)% = 24%.
judder cadence = 17.6:22.4 = 11:14.

2m@60fps (1m11m11m11...): From vanden's diagrams, 2m@60fps (as a frame timing diagram) is:
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[AA____][AA____][AB____][BB____][BB____][CC____][CC____][CD____][DD____][DD____] - output (via interpolated motion vectors)
300 s ---><----- 1/25 s -----><---- 13/300 s ----><----- 1/25 s -----><----- 13/ - picture centers & picture judder: 8%, 12:13 cadence, 12 judders/s.

Uniform@60fps (1mmmm1mmmm...): From vanden's diagrams, Uniform@60fps (as a frame timing diagram) is:
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[AA____][AB____][AB____][BC____][BC____][CC____][CD____][CD____][DA____][DA____] - output (via interpolated motion vectors)
---><----- 1/24 s -----><----- 1/24 s -----><----- 1/24 s -----><----- 1/24 s -- - picture centers & picture judder: none

There's 2 related issues.

Background: Vanden's diagrams suggest that all existing motion vector interpolation is solely between a preceeding frame and a following frame. To give that a name, I would call it a "span-of-3 interpolation".
For example, this (1m@60fps):
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[A_____][AB____][AB____][B_____][BC____][C_____][CD____][CD____][D_____][DA____] - output (via interpolated motion vectors)
shows 4 span-of-3 interpolations:
[AB] interpolated from [A] & ,
[BC] interpolated from [B] & [C],
[CD] interpolated from [C] & [D], and
[DA] interpolated from [D] & [A],
plus 2 frame repeats:
[AB] and
[CD].
Comment: Even though the repeated frames are interpolated frames, frame repeats are not interpolation. The frame repeats produce judder.

Issue 1:
The SVSmoothFps documentation for 'algo:11' refers to a "dynamic median" without defining what that means.
Does "dynamic median" mean 4 span-of-3 interpolations followed by 8 span-of-3 interpolations between them followed by 6 discards as follows?
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[A_______][AB______][B_______][BC______][C_______][CD______][D_______][DA______] - 48fps (via span-of-3 interpolations)
For example, [AB] interpolated from [A] & [B],
followed by
[A_______][AB______][B_______][BC______][C_______][CD______][D_______][DA______] - 48fps (via span-of-3 interpolations)
[A__][AAB][AB_][ABB][B__][BBC][BC_][BCC][C__][CCD][CD_][CDD][D__][DDA][DA_][DAA] - 72fps (via span-of-3 interpolations)
For example, [AAB] interpolated from [A] & [AB],
followed by
[A__][AAB][AB_][ABB][B__][BBC][BC_][BCC][C__][CCD][CD_][CDD][D__][DDA][DA_][DAA] - 72fps (via span-of-3 interpolations)
[A_____][AB____][B_____][BBC___][BC____][C_____][CD____][D_____][DDA___][DA____] - 60fps output (via 6 discards)
-><---- 149/3360 s ----><-- 131/3360 s --><---- 149/3360 s ----><-- 131/3360 s - - picture centers & picture judder: 13%, 131:149 cadence, 12 judders/s.
which discards 6 frames to achieve 60fps.
Comment: As you can see, "dynamic median" results in judder that's more than "2m" but less than "1m". The judder is produced by the discards.
Comment: When attempting to convert 2 frames to 5 frames, any process that tries to do it via nested span-of-3 interpolations is going to have to repeat (or discard) some frames. Judder is caused by the repeats (or discards).

Issue 2:
Because I'm a novice and didn't know better, in my original post I had this:
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[AAAAA_][AAABB_][ABBBB_][BBBBC_][BBCCC_][CCCCC_][CCCDD_][CDDDD_][DDDDA_][DDAAA_] - output (via interpolated motion vectors)
The above has 4, weighted span-of-4 interpolations:
[AAABB_][ABBBB_] interpolated from [A] & [B],
[BBBBC_][BBCCC_] interpolated from [B] & [C],
[CCCDD_][CDDDD_] interpolated from [C] & [D], and
[DDDDA_][DDAAA_] interpolated from [D] & [A].
There is no dynamic median. There are no frame repeats or discards. There is no judder.
Comment: Weighted span-of-4 interpolations is the motion-vector equivalent of weighted span-of-4 spacial pixel interpolation (which I have done via ffmpeg's 'mix' filter).
It appears that weighted span-of-4 interpolation is not available.

[B]Bounty
Since professional programmers need to make a living, even for FOSS software, I will pay a bounty for the creation of weighted span-of-4 interpolation.

[Alexkral]

Just below the diagram, vanden asks, "Is that correct ?"

There are more diagrams below, and a post from the dev saying that they are all "generally" correct. How accurate that is, I don't know, but the dev himself has used that same diagram in other posts (like this one (https://www.svp-team.com/forum/viewtopic.php?pid=49164#p49164)) to explain this, and the Avatar images from the first post on the next page seem to confirm it.

There's 2 related issues.

Background: Vanden's diagrams suggest that all existing motion vector interpolation is solely between a preceeding frame and a following frame. To give that a name, I would call it a "span-of-3 interpolation".
For example, this (1m@60fps):
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[A_____][AB____][AB____][B_____][BC____][C_____][CD____][CD____][D_____][DA____] - output (via interpolated motion vectors)
shows 4 span-of-3 interpolations:
[AB] interpolated from [A] & [B],
[BC] interpolated from [B] & [C],
[CD] interpolated from [C] & [D], and
[DA] interpolated from [D] & [A],
plus 2 frame repeats:
[AB] and
[CD].
Comment: Even though the repeated frames are interpolated frames, frame repeats are not interpolation. The frame repeats produce judder.

They are not repeated frames, they are interpolated at different fractional positions in a time scale.

It appears that weighted span-of-4 interpolation is not available.

I think it's possible to achieve similar things by mixing interpolation and blending, as in the SVP x2 1m + madVR Smooth Motion example from the Avatar images.

[markfilipak]

There are more diagrams below, and a post from the dev saying that they are all "generally" correct. How accurate that is, I don't know, but the dev himself has used that same diagram in other posts (like this one (https://www.svp-team.com/forum/viewtopic.php?pid=49164#p49164)) to explain this, and the Avatar images from the first post on the next page seem to confirm it.

There's 2 related issues.

Background: Vanden's diagrams suggest that all existing motion vector interpolation is solely between a preceeding frame and a following frame. To give that a name, I would call it a "span-of-3 interpolation".
For example, this (1m@60fps):
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] - input
[A_____][AB____][AB____][B_____][BC____][C_____][CD____][CD____][D_____][DA____] - output (via interpolated motion vectors)

shows 4 span-of-3 interpolations:
[AB] interpolated from [A] & [B],
[BC] interpolated from [B] & [C],
[CD] interpolated from [C] & [D], and
[DA] interpolated from [D] & [A],
plus 2 frame repeats:
[AB] and
[CD].
Comment: Even though the repeated frames are interpolated frames, frame repeats are not interpolation. The frame repeats produce judder.
They are not repeated frames, they are interpolated at different fractional positions in a time scale.
Are you sure? I read all the stuff you pointed to -- thank you for that -- and I'd like to point out 2 things:
1, The discussion was about new PTSs. Well, of course there would be new PTSs. That doesn't mean that the motion vector interpolation goes further than 1 interpolated frame between a pair of original frames (i.e. what I'm calling a "span-of-3 interpolation").
And 2, The output of mode 0 does have judder. It's very small, much much smaller than 3:2 pull-down (or 6:4 pull-down), but it is judder. If there was no repeated frames, there would be no judder. Do you agree?
I think it's possible to achieve similar things by mixing interpolation and blending, as in the SVP x2 1m + madVR Smooth Motion example from the Avatar images.
Not blending. Never blending. Not unless it's absolutely the only way to get a 'pleasant' image (which is almost always not the case).

Though we disagree a bit, you've been very helpful. I'm so grateful. Thanks again.

--Mark.

[markfilipak]

I've reworked what I posted. I reworked it to expose the judder in the output pictures instead of the input pictures. That actually makes more sense because it's the output pictures that are seen.

1m@60fps contains 8 pictures, so 1m@60fps is 60fps that carries 48pps (pictures-per-second).
2m@60fps contains 7 pictures, so 2m@60fps is 60fps that carries 42pps.
Uniform@60fps contains 6 pictures, so Uniform@60fps is 60fps that carries 36pps.
The details follow.

1m@60fps -- 1mm1m1mm1m -- converts pictures-per-second from 24pps to 48pps and conveys them via 60fps.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AB______][AB______][B_______][BC______][C_______][CD______][CD______][D_______][DA______] output, 60fps(48pps) via interpolated motion vectors
1 m m 1 m 1 m m 1 m
60 -><--- 1/40 ----><--- 1/40 ----><- 1/60 -><- 1/60 -><--- 1/40 ----><--- 1/40 ----><- 1/60 -><- 1/ center-to-center picture timing, seconds
picture judder: 40%, 2:2:3:3 cadence, 12 judders/s.

2m@60fps -- 1m11m11m11 -- converts pictures-per-second from 24pps to 42pps and conveys them via 60fps.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AB______][B_______][B_______][BC______][C_______][C_______][CD______][D_______][D_______] output, 60fps(42pps) via interpolated motion vectors
1 m 1 1 m 1 1 m 1 1
----><- 1/60 -><--- 1/40 ----><--- 1/40 ----><--- 1/40 ----><--- 1/40 ----><--- 1/40 ----><-- 1/40 - center-to-center picture timing, seconds
picture judder: 40%, 2:3:3:3:3:3:3 cadence, 6 judders/s.

Uniform@60fps -- 1mmmm1mmmm -- converts pictures-per-second from 24pps to 36pps and conveys them via 60fps.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AB______][AB______][BC______][BC______][C_______][CD______][CD______][DA______][DA______] output, 60fps(36pps) via interpolated motion vectors
1 m m m m 1 m m m m
----><--- 1/40 ----><------ 1/30 ------><--- 1/40 ----><--- 1/40 ----><------ 1/30 ------><---- 1/40 center-to-center picture timing, seconds
picture judder: 29%, 3:3:4 cadence, 12 judders/s.
What I want is not available.

(not available) -- converts pictures-per-second from 24pps to 60pps and conveys them via 60fps.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AAABB___][ABBBB___][BBBBC___][BBCCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDAAA___] output, 60fps(60pps) via interpolated motion vectors
1 m m m m 1 m m m m
----><-------- 1/24 ---------><-------- 1/24 ---------><-------- 1/24 ---------><-------- 1/24 ----- center-to-center picture timing, seconds
picture judder: none

[poisondeathray]

Uniform@60fps -- 1mmmm1mmmm -- converts pictures-per-second from 24pps to 36pps and conveys them via 60fps.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AB______][AB______][BC______][BC______][C_______][CD______][CD______][DA______][DA______] output, 60fps(36pps) via interpolated motion vectors
1 m m m m 1 m m m m
----><--- 1/40 ----><------ 1/30 ------><--- 1/40 ----><--- 1/40 ----><------ 1/30 ------><---- 1/40 center-to-center picture timing, seconds
picture judder: 29%, 3:3:4 cadence, 12 judders/s.



This one should not be happening with mode:0, but I can confirm it does despite fiddling with the settings. Those duplicates shouldn't be there. It affects avs version too

It might be fixed in newer versions with svpflow manager running, but I'm running older "free" version.


mvtools2 works ok, both avs and vpy

[markfilipak]

Uniform@60fps -- 1mmmm1mmmm -- converts pictures-per-second from 24pps to 36pps and conveys them via 60fps.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AB______][AB______][BC______][BC______][C_______][CD______][CD______][DA______][DA______] output, 60fps(36pps) via interpolated motion vectors
1 m m m m 1 m m m m
----><--- 1/40 ----><------ 1/30 ------><--- 1/40 ----><--- 1/40 ----><------ 1/30 ------><---- 1/40 center-to-center picture timing, seconds
picture judder: 29%, 3:3:4 cadence, 12 judders/s.

This one should not be happening with mode:0, but I can confirm it does despite fiddling with the settings. Those duplicates shouldn't be there. It affects avs version too
poisondeathray! Hello my friend. Thanks for replying. And thanks for confirming my frame diagram.

Those duplicates shouldn't be there? If not, what do you think should be there?
It might be fixed in newer versions with svpflow manager running, but I'm running older "free" version.
I don't know anything about svpflow manager.

mvtools2 works ok, both avs and vpy
Well, I don't know what you mean. Do you mean that it behaves differently in mvtools2? (I know nothing about mvtools2.)

Do you know of anyone who'd like to earn the bounty by writing this:
What I want is not available.

(not available) -- converts pictures-per-second from 24pps to 60pps and conveys them via 60fps.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AAABB___][ABBBB___][BBBBC___][BBCCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDAAA___] output, 60fps(60pps) via interpolated motion vectors
1 m m m m 1 m m m m
----><-------- 1/24 ---------><-------- 1/24 ---------><-------- 1/24 ---------><-------- 1/24 ----- center-to-center picture timing, seconds
picture judder: none

[poisondeathray]

Those duplicates shouldn't be there? If not, what do you think should be there?


Not according to what they are selling or describing



I don't know anything about svpflow manager.


Newer versions require another program to run in the background. It's not free either



Well, I don't know what you mean. Do you mean that it behaves differently in mvtools2? (I know nothing about mvtools2.)


ie. Yes, no duplicates

vapoursynth version too

Here is an example of one variation. You would change num to 60000, den to 1001

https://forum.doom9.org/showthread.php?p=1814002#post1814002



Do you know of anyone who'd like to earn the bounty by writing this:
If you mean specifically for svpflow, you'd have to determine if it's not already fixed first in the newer versions. Older versions were seriously overrated, and GPU produced worse artifacting than CPU in my experience

[Alexkral]

I just realized that what you are looking for is exactly the same as SVP 2.5x mode:0. I can't test it now but I can't see any judder when using it at home with the SVP 4 Pro Manager, so it must be a problem with the SVPflow libs you are using.

[markfilipak]

Not according to what they are selling or describing

Did you look at this? https://www.svp-team.com/forum/viewtopic.php?pid=49164#p49164

My diagrams are basically those but layed on their sides. You see, I'm a hardware design engineer and I prefer diagrams that look like timing diagrams. Mine have a lot more detail, though.

Newer versions require another program to run in the background. It's not free either

Oh, is that the Russians at Smooth Video Project? I sensed that they weren't actually developers but were just trying to make money from other peoples' work. Should I send them the $20?

ie. Yes, no duplicates

vapoursynth version too

No duplicates? The post from MAG79 shows duplicates.

Here is an example of one variation. You would change num to 60000, den to 1001

https://forum.doom9.org/showthread.php?p=1814002#post1814002



If you mean specifically for svpflow, you'd have to determine if it's not already fixed first in the newer versions. Older versions were seriously overrated, and GPU produced worse artifacting than CPU in my experience

[poisondeathray]

Oh, is that the Russians at Smooth Video Project? I sensed that they weren't actually developers but were just trying to make money from other peoples' work. Should I send them the $20?

I would wait for Alexkral to test if it does what you want it



No duplicates? The post from MAG79 shows duplicates.
I didn't really look at the diagram closely. I read the description and used the settings


0 - uniform interpolation for maximum smoothness. For example for 24->60 conversion output will be: "1mmmm1mmmm...", where "1" stands for original frame and "m" for interpolated one.


Maybe I'm misunderstanding his description of mode:0

"maximum smoothest" does not really "jive" with "duplicates" in my book

None of the modes (0,1,2,3) were smooth, all had duplicates.

[markfilipak]

I just realized that what you are looking for is exactly the same as SVP 2.5x mode:0. I can't test it now but I can't see any judder when using it at home with the SVP 4 Pro Manager, so it must be a problem with the SVPflow libs you are using.
My diagrams are based on theory. I'm sorry to say that I don't know what SVP 2.5x is? It has a 'mode:0' also? Oh, wait, is SVP "Smooth Video Project"?

If the diagrams by MAG79 that you showed me are correct, then my diagrams are correct. There is judder (which actually becomes very evident when SVSmoothFps.mask.area is greater then 300), but at lower 'area'values and at 60fps, the judder is difficult to see. What I do is view my test videos over a 60Hz TV via DisplayPort. The TV is 50-inch and is about a foot and a half away, so critcal analysis is fairly easy.

[markfilipak]

"maximum smoothest" does not really "jive" with "duplicates" in my book
That's my point. All the modes must have judder, more or less.

You should examine my diagrams. I spent a lot of time developing them and I think there are no errors.

Did you look at MAG79's diagrams (https://www.svp-team.com/forum/viewtopic.php?pid=49164#p49164)?

[markfilipak]

Hey, is VapourSynth written entirely in python? No 'C'?

Never mind. The DLLs have to be in 'C', so I'm not going to develop my own mode:0 that doesn't have judder.

[poisondeathray]

That's my point. All the modes must have judder, more or less.


You should examine my diagrams. I spent a lot of time developing them and I think there are no errors.

Did you look at MAG79's diagrams (https://www.svp-team.com/forum/viewtopic.php?pid=49164#p49164)?

Where are the duplicates for uniform mode:0 in the diagram ?

Maybe I'm misreading it?

For example MI(0,1)a and MI(0,1)b are different. One is at 0.33 , the other at 0.66



Looking at it more closely, I might be getting something slightly different than you;

You're describing for mode:0
[A_______][AB______][AB______][BC______][BC______][C_______][CD______][CD______][DA______][DA______]

That would imply that A and AB are different.

But I'm actually getting no interpolation on the duplicates. There is no telltale artifacts, or slight warping of edges, or blending, or mix

I'm getting A,A,B,B,C,C,C,D,D,E

But my point is duplicates are "not smooth". And we're both getting them, even if the pattern description is slightly different

[Alexkral]

Oh, wait, is SVP "Smooth Video Project"?

Yes. Look at the SVP 2.5x Uniform mode diagram here (https://www.svp-team.com/forum/viewtopic.php?pid=40344#p40344), there are no duplicates, the frames are interpolated at different positions in the time scale.

The last SVPflow versions that work correctly without the SVP Manager running are posted here:

https://forum.doom9.org/showthread.php?p=1914245#post1914245

[poisondeathray]

Yes. Look at the SVP 2.5x Uniform mode diagram here (https://www.svp-team.com/forum/viewtopic.php?pid=40344#p40344), there are no duplicates, the frames are interpolated at different positions in the time scale.

The last SVPflow versions that work correctly without the SVP Manager running are posted here:

https://forum.doom9.org/showthread.php?p=1914245#post1914245


Those fix the duplicates for me, for the vpy version. It works as expected now

EDIT: spoke too soon, still some problems with duplicates...checking

[poisondeathray]

Just so everyone is on the same page -

Here is a test clip, "test_6frames.mp4", 6 frames 23.976p 640x640 . It's a synthetic move, 20px horizontal per frame, so you can overlay a grid and calculate how precise or what ratio the interpolated frame is. (I know you're looking at 4 frame source cycles, but I added extra, because some algorithms have a "look ahead" and examine n+1,n+2 frames)

The pattern 1mmmm1mmmm indicates which original frames positions are kept (frame zero, frame six), and that is consistent with the avisynth mvtools2 way of interpolation. There are no duplicates, but there are pro/cons to either resampling all frames , or keeping some original frames.

Original frames have higher quality, synthesized frames have lower quality, more blur. Pros/cons. The quality fluctuation by keeping some original frames can be percieved as a sort of flicker. It's a more drastic effect on film grain, or noisy content. "1mmmm1mmmm" can be slightly less smooth than resampling all frame completely, but resampling all frames is generally prone to more artifacts

https://www.mediafire.com/file/2tdxaklxfj1yn3y/motion_interpolation_tests.zip/file

test_6frames.mp4 is the test source
test_frc.mp4 is mvtools2 avisynth based using default settings
test_mvflowfps.mp4 is the mvtools2 vpy script above
test_svpflow_mode0.mp4 is the avs version of svpflow, I think 2.x something. vpy version was the same, I didn't bother including

Maybe Alexkral can test if newest SVP 4.x with manager exhibits the issue

[markfilipak]

Where are the duplicates for uniform mode:0 in the diagram ?

[A_______][AB______][AB______][BC______][BC______][C_______][CD______][CD______][DA______][DA______]
:........::..................::..................::........::..................::..................:
picture 1 picture 2 picture 3 picture 4 picture 5 picture 6

Maybe I'm misreading it?

For example MI(0,1)a and MI(0,1)b are different. One is at 0.33 , the other at 0.66

That's not how I interpret what MAG79 writes (based on vanden's diagrams). I interpret the diagrams thus: MI(0,1)a and MI(0,1)b are the same but have differing time stamps. Here: https://www.svp-team.com/forum/viewtopic.php?pid=40354#p40354, MAG79 says, "So any source frame will be placed at integer values in time scale. But any interpolated frame is situated between two source frames so it has fractional position." I think he refers to PTSs, not temporal interpolation. I may be wrong because I'm interpreting what MAG79 writes and it seems that MAG79 is not a native English speaker.

I'm going to resolve this issue. I'm going to dump raw frames and compare their data. If I need help with that, will you help, poisondeathray?

Looking at it more closely, I might be getting something slightly different than you;

You're describing for mode:0
[A_______][AB______][AB______][BC______][BC______][C_______][CD______][CD______][DA______][DA______]

That would imply that A and AB are different.

Correct. That's 6 pictures contained in 10 frames.

But I'm actually getting no interpolation on the duplicates. There is no telltale artifacts, or slight warping of edges, or blending, or mix

I'm getting A,A,B,B,C,C,C,D,D,E

Huh? You're getting only repeated frames? And how could you be getting [E] as the 10th frame? I don't get it.

But my point is duplicates are "not smooth". And we're both getting them, even if the pattern description is slightly different

I don't know what you're 'saying'. Does "them" refer to duplicates or "not smooth" frames? And to what "pattern description" do you refer?

[poisondeathray]

That's not how I interpret what MAG79 writes (based on vanden's diagrams). I interpret the diagrams thus: MI(0,1)a and MI(0,1)b are the same but have differing time stamps. Here: https://www.svp-team.com/forum/viewtopic.php?pid=40354#p40354, MAG79 says, "So any source frame will be placed at integer values in time scale. But any interpolated frame is situated between two source frames so it has fractional position." I think he refers to PTSs, not temporal interpolation. I may be wrong because I'm interpreting what MAG79 writes and it seems that MAG79 is not a native English speaker.

The whole point of optical flow is temporal and spatial interpolation . Why would the "smoothest" mode:0 have duplicates? What would the point be of using optical flow at all? It would make the whole exercise useless. Ok, the "non smooth" modes balance off smoothness vs. artifacts, those might make sense


I'm going to resolve this issue. I'm going to dump raw frames and compare their data. If I need help with that, will you help, poisondeathray?


Sure, but it might be already resolved in the newer versions. Wait for Alexkral to check. I uploaded a simple test source video




I don't know what you're 'saying'. Does "them" refer to duplicates or "not smooth" frames? And to what "pattern description" do you refer?

Duplicates are not smooth . We're both getting duplicates for SVP. Therefore we're both experiencing "not smooth" results for SVP

Your pattern description for SVP differs from the actual results I get for SVP. I uploaded my video for SVP

mvtools2 results have no duplicates. The cadence in test_mvflowfps.mp4 or test_frc.mp4 is what you would expect for svpflow mode:0 . You can actually measure the interpolated distance of each frame with the vapoursynth grid plugin. The source animation was 20 pixels per frame, linear movement.

[markfilipak]

Just so everyone is on the same page -

Here is a test clip, "test_6frames.mp4", 6 frames 23.976p 640x640...
test_6frames.mp4 is the test source
test_frc.mp4 is mvtools2 avisynth based using default settings
test_mvflowfps.mp4 is the mvtools2 vpy script above
test_svpflow_mode0.mp4 is the avs version of svpflow, I think 2.x something. vpy version was the same, I didn't bother including

Maybe Alexkral can test if newest SVP 4.x with manager exhibits the issue

Here's my analysis.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________][E______________________][F______________________] test_6frames.mp4
[A_______][AAABB___][ABBBB___][BBBBC___][BBBCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDEEE___][E_______][EEEFF___][EFFFF___][EFFFF___][EFFFF___] test_frc.mp4
[A_______][AAABB___][ABBBB___][BBBBC___][BBBCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDEEE___][E_______][EEEFF___][EFFFF___] test_mvflowfps.mp4
[A_______][A_______][BBBBC___][BBBBC___][C_______][C_______][C_______][DDDDA___][DDDDA___][E_______][E_______][E_______][EFFFF___] test_svpflow_mode0.mp4

THANK YOU! What a relief. I'm very glad I was wrong.

Now, how do I make the best one for myself?

[poisondeathray]

Now, how do I make the best one for myself?

Which one do you want, is it the pattern at the end of post 6?



What I want is not available.

(not available) -- converts pictures-per-second from 24pps to 60pps and conveys them via 60fps.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AAABB___][ABBBB___][BBBBC___][BBCCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDAAA___] output, 60fps(60pps) via interpolated motion vectors
1 m m m m 1 m m m m
----><-------- 1/24 ---------><-------- 1/24 ---------><-------- 1/24 ---------><-------- 1/24 ----- center-to-center picture timing, seconds
picture judder: none


What I don't understand about that pattern is the last 2 frames, [DDDDA___][DDAAA___] . Those look "backwards" to me. You're combining frames or data from the past.

[markfilipak]

Which one do you want, is it the pattern at the end of post 6?

I'd prefer the tool that made test_frc.mp4... Is that mvtools2? I think so. I prefer it because it does fairly clean mv interpolation and because it maintains the running time by adding duplicate frames at the end (which will usually be a dark, change scene section).

What I don't understand about that pattern is the last 2 frames, [DDDDA___][DDAAA___] . Those look "backwards" to me. You're combining frames or data from the past.

<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________] input, 24fps(24pps)
[A_______][AAABB___][ABBBB___][BBBBC___][BBCCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDAAA___] output, 60fps(60pps) via interpolated motion vectors
1 m m m m 1 m m m m
----><-------- 1/24 ---------><-------- 1/24 ---------><-------- 1/24 ---------><-------- 1/24 ----- center-to-center picture timing, seconds
picture judder: none
Ah, I understand. :-)
You have to imagine that the sequence repeats modulo-10, thusly:

[A______________________][B______________________][C______________________][D______________________][A______________________][B___ ...
[A_______][AAABB___][ABBBB___][BBBBC___][BBCCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDAAA___][A_______][AAABB___][ABBBB___] ...

That's the sort of thing you have to imagine when you mentally visualize 2:3 pull-down, eh?

[poisondeathray]

I'd prefer the tool that made test_frc.mp4... Is that mvtools2? I think so. I prefer it because it does fairly clean mv interpolation and because it maintains the running time by adding duplicate frames at the end (which will usually be a dark, change scene section).


All the open source tools discussed in this thread are based on mvtools2, even svpflow

frc is framerateconverter from avisynth , it hasn't been directly ported to vapoursynth yet .


Ah, I understand. :-)
You have to imagine that the sequence repeats modulo-10, thusly:


ahhh...now I understand :)

I'd actually prefer "E" frame

To clarify in this prior post , the DDDDA for frame 8 (on all 3 interpolated versions) , is really DDDDE ?

Here's my analysis.
<-------------------------------- 1/6 s (scale = 480 characters/s) -------------------------------->
[A______________________][B______________________][C______________________][D______________________][E______________________][F______________________] test_6frames.mp4
[A_______][AAABB___][ABBBB___][BBBBC___][BBBCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDEEE___][E_______][EEEFF___][EFFFF___][EFFFF___][EFFFF___] test_frc.mp4
[A_______][AAABB___][ABBBB___][BBBBC___][BBBCC___][C_______][CCCDD___][CDDDD___][DDDDA___][DDEEE___][E_______][EEEFF___][EFFFF___] test_mvflowfps.mp4
[A_______][A_______][BBBBC___][BBBBC___][C_______][C_______][C_______][DDDDA___][DDDDA___][E_______][E_______][E_______][EFFFF___] test_svpflow_mode0.mp4

[markfilipak]

ahhh...now I understand :)

I'd actually prefer "E" frame

If that works for you, it works for me. I'll adjust all my documentation.

To clarify in this prior post , the DDDDA for frame 8 (on all 3 interpolated versions) , is really DDDDE ?

That's correct. You see, doing a timing diagram identifies PTSs (i.e. [-time), running times, judder, all sorts of things.

I like pictures better than words. ;)

[markfilipak]

@poisondeathray

How did test_svpflow_mode0.mp4 (i.e. svpflow, though I have been using InterFrame) screw up so badly? InterFrame works pretty well for me, but I am having some complicated difficulties with it.

[poisondeathray]

@poisondeathray

How did test_svpflow_mode0.mp4 (i.e. svpflow, though I have been using InterFrame) screw up so badly? InterFrame works pretty well for me, but I am having some complicated difficulties with it.

I don't know - it's unexpected.

But both avisynth and vapoursynth versions of svpflow exhibited the issue for me.

I hope it's just because of old/outdated plugin...or some local config issue...

If I run interframe on that test clip, I get blends for all interpolated frames

[Alexkral]

So both the version I linked (4.3.0) and the latest (4.5.0) work as expected and in fact the results seem to be identical. The problem seems to only happen with clips that are too short, so I used a 30 second test clip. I have not analyzed the sequence, but you can clearly see that there are no repeated frames:

https://drive.google.com/file/d/127vns90LEYKjQeq1ZAQJiO1VWRM8AMdb/view?usp=sharing

[poisondeathray]

Yes, svpflow only seems to fail on short test clip.

mode:0 works ok as expected on longer tests

It's not duration related, because if you loop that short test, svpflow still fails . That's potentially problematic if you have short scenecuts

[Alexkral]

It's not duration related, because if you loop that short test, svpflow still fails . That's potentially problematic if you have short scenecuts

Yeah, for your test clip it is solved with block.width=8 in SVAnalyse.

[poisondeathray]

Yeah, for your test clip it is solved with block.width=8 in SVAnalyse.

This worked for the avs version, but not for the vapoursynth version for some reason..trying to figure out why

[Alexkral]

@poisondeathray

This works for me:

import vapoursynth as vs
core = vs.get_core(threads=11)

core.std.LoadPlugin(r"F:\System\Documents\test\SVPflow_LastGoodVersions\x64_vs\svpflow1_vs.dll")
core.std.LoadPlugin(r"F:\System\Documents\test\SVPflow_LastGoodVersions\x64_vs\svpflow2_vs.dll")

clip = core.ffms2.Source(r"F:\System\Documents\test\motion_interpolation_tests\test_6frames.mp4")
clip = clip.resize.Bicubic(format=vs.YUV420P8)

super = core.svp1.Super(clip,"{scale:{up:0},gpu:1,rc:true}")
vectors= core.svp1.Analyse(super["clip"],super["data"],clip,"{block:{w:8},main:{search:{coarse:{distance:-12,bad:{sad:2000}},type:2}},refine:[{thsad:250}]}")
smooth = core.svp2.SmoothFps(clip,super["clip"],super["data"],vectors["clip"],vectors["data"],"{gpuid:11,gpu_qn:2,rate:{num:5,den:2},algo:23,mask:{cover:80},scene:{mode:0}}")
smooth = core.std.AssumeFPS(smooth,fpsnum=smooth.fps_num,fpsden=smooth.fps_den)

smooth.set_output()

Original frames have higher quality, synthesized frames have lower quality, more blur. Pros/cons. The quality fluctuation by keeping some original frames can be percieved as a sort of flicker. It's a more drastic effect on film grain, or noisy content. "1mmmm1mmmm" can be slightly less smooth than resampling all frame completely, but resampling all frames is generally prone to more artifacts

I don't perceive any type of flicker, but I do notice sometimes some very annoying artifacts around moving objects. What I've been doing for some time now is interpolating to 120 fps and blending the odd frames with the even ones. I use mode 1 for this, but it seems that for 5x mode 0 and mode 1 are the same. The result of this is equally smooth and I think it helps to cut down on the artifacts I was talking about. I had never thought about the inferior quality of the blended frames, now that I've done some comparisons I'd say that while it's true, most of the time it is barely noticeable.

[markfilipak]

@poisondeathray

This works for me:

import vapoursynth as vs
...
smooth.set_output()
May I contribute?
Thanks for the vpy. I'm an informed novice and I need a way to bypass InterFrame to do things with SVSuper(), SVAnalyse(), SVConvert(), and SVSmoothFps() that InterFrame doesn't support. Your vpy provides some valuable clues.
I don't perceive any type of flicker, but I do notice sometimes some very annoying artifacts around moving objects. What I've been doing for some time now is interpolating to 120 fps and blending the odd frames with the even ones. I use mode 1 for this, but it seems that for 5x mode 0 and mode 1 are the same. The result of this is equally smooth and I think it helps to cut down on the artifacts I was talking about. I had never thought about the inferior quality of the blended frames, now that I've done some comparisons I'd say that while it's true, most of the time it is barely noticeable.
I've done a massive amount of testing. I'm sure that for algo:11 and higher, some blending is being done.

I've found 4 types of svpflow2 distortions:
1, Apparent 'dragging' of patterned backgrounds -- 'halos' -- that surround tracking objects in panning shots, and
2, Large, flickering, blocky 'phantom' areas -- like afterimages -- that overlay vertical patterns (e.g. "picket fence") during stationary or slowly moving scenes that are masked by setting SVSmoothFps.mask.area greater than 200, and
3, Judder for SVSmoothFps.mask.area greater than 200 and becoming intolerable at SVSmoothFps.mask.area greater than 400, and
4, Some pretty brutal blending for SVSmoothFps.algo greater than 2.

I'll supply more info in a day or two. For now, I'd like to share the revised diagrams. If you care to comment, I'd appreciate it.

-- Mark.


scene_string = "scene:{mode:3,blend:false,limits:{m1:1600,m2:2800,scene:4000,zero:200,blocks:20},luma:1.5}"
: : : : : : : :
: : : : : : : ????? ...additional correction parameter for "average luma" value.
: : : : : : ????? ...threshold which sets how many blocks in percents have to change [note 1].
: : : : : ????? ...vectors with "adjusted SAD" less than this value are excluded from consideration [note 1].
: : : : ????? ...limit for scene change detection.
: : : ????? ...limit for changing from 'mode:1' to 'mode:2' when 'mode:3' is specified.
: : ????? ...limit for changing from 'mode:0' to 'mode:1' when 'mode:3' is specified.
: true ...blend frames at scene change like ConvertFps.
: false ...repeat last frame at scene change like ConvertFps.
0 ...uniform interpolation for maximum smoothness. For example for 24->60 conversion output will be: "1mmmm1mmmm...", where "1" stands for original frame and "m" for interpolated one [note 2].
1 ..."1m" mode that gives "1mm1m1mm1m..." output in the above example => less artifacts at the cost of less smoothness [note 3].
2 ..."2m" mode: "1m11m11m11..." => much less artifacts and much less smoothness [note 4].
3 ...adaptive mode that switches between modes 0,1,2 based on overall vector field quality.

[note 1] Limits for vector field quality / scene change detection. For example scene change will be detected if number of blocks with "adjusted SAD" > "limits.scene" will be more than "limits.blocks" percents of all blocks, that has "adjusted SAD" value > "limits.zero", where "adjusted SAD" is "block SAD"/"block average luma". Comment: What the preceeding word salad means is a mystery.

[note 2] 'mode:0' -- 1mmmm1mmmm -- converts 4 pictures (in 4 frames) to 10 pictures (2 original + 8 new, in 10 frames), and thereby converts pictures-per-second from 24pps @ 24fps to 60pps @ 60fps.
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] input: 24fps(24pps)
[A_____][AAABB_][ABBBB_][BBBBC_][BBCCC_][C_____][CCCDD_][CDDDD_][DDDDE_][DDEEE_] output: 60fps(60pps) via interpolated motion vectors
1 m m m m 1 m m m m
---><----- 1/6 s ------><----- 1/6 s ------><----- 1/6 s ------><----- 1/6 s --- center-to-center output picture durations, seconds
picture judder: none

[note 3] 'mode:1' -- 1mm1m1mm1m -- converts 4 pictures (in 4 frames) to 10 pictures (4 original + 6 new, in 10 frames), and thereby converts pictures-per-second from 24pps @ 24fps to 60pps @ 60fps.
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] input: 24fps(24pps)
[A_____][AAB___][ABB___][B_____][BC____][C_____][CCD___][CDD___][D_____][DE____] output: 60fps(60pps) via interpolated motion vectors
1 m m 1 m 1 m m 1 m
------><--- 7/180 s ----><------ 2/45 s ------><--- 7/180 s ----><------ 2/45 s center-to-center output picture durations, seconds
picture judder: 13%, 7:8 cadence, 12 judders/s.

[note 4] 'mode:2' -- 1m11m11m11 -- converts 4 pictures (in 4 frames) to 7 pictures (4 original + 3 new, in 10 frames), and thereby converts pictures-per-second from 24pps @ 24fps to 42pps @ 60fps.
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] input: 24fps(24pps)
[A_____][AB____][B_____][B_____][BC____][C_____][C_____][CD____][D_____][D_____] output: 60fps(42pps) via interpolated motion vectors
1 m 1 1 m 1 1 m 1 1
s ---><-- 13/360 s ---><------ 17/360 s ------><----- 43/900 s -----><--- 8/225 center-to-center output picture timing, seconds
picture judder: 29%, 70:85:86:64 cadence, 6 judders/s.

[Alexkral]

I've found 4 types of svpflow2 distortions:
1, Apparent 'dragging' of patterned backgrounds -- 'halos' -- that surround tracking objects in panning shots, and
2, Large, flickering, blocky 'phantom' areas -- like afterimages -- that overlay vertical patterns (e.g. "picket fence") during stationary or slowly moving scenes that are masked by setting SVSmoothFps.mask.area greater than 200, and
3, Judder for SVSmoothFps.mask.area greater than 200 and becoming intolerable at SVSmoothFps.mask.area greater than 400, and
4, Some pretty brutal blending for SVSmoothFps.algo greater than 2.

Yes, I have seen all that, also the same as 1 on the edges of the screen in all kinds of camera movements. And I don't use mask.area at all because of the judder as you say.

I'm sure that for algo:11 and higher, some blending is being done.

Yes indeed, according to the reference it's a "time weighted blend of forward and backward partial motion compensations". (By the way, here (https://www.svp-team.com/wiki/Manual:SVPflow)'s a pretty comprehensive SVPflow reference, I'm not sure if you knew about it).

What I see with the poisondeathray test clip is this:

mode 0:

[A_____][AAABB_][ABBBB_][BBBBC_][BBCCC_][C_____][CCCDD_][CDDDD_][DDDDE_][DDEEE_] (same as yours)

mode 1:

[A_____][AB____][B_____][BBC___][BCC___][C_____][CD____][D_____][DDE___][DEE___]

mode 2:

[A_____][A_____][B_____][B_____][BC____][C_____][C_____][D_____][D_____][DE____]

[poisondeathray]

@poisondeathray

This works for me:

import vapoursynth as vs
core = vs.get_core(threads=11)

core.std.LoadPlugin(r"F:\System\Documents\test\SVPflow_LastGoodVersions\x64_vs\svpflow1_vs.dll")
core.std.LoadPlugin(r"F:\System\Documents\test\SVPflow_LastGoodVersions\x64_vs\svpflow2_vs.dll")

clip = core.ffms2.Source(r"F:\System\Documents\test\motion_interpolation_tests\test_6frames.mp4")
clip = clip.resize.Bicubic(format=vs.YUV420P8)

super = core.svp1.Super(clip,"{scale:{up:0},gpu:1,rc:true}")
vectors= core.svp1.Analyse(super["clip"],super["data"],clip,"{block:{w:8},main:{search:{coarse:{distance:-12,bad:{sad:2000}},type:2}},refine:[{thsad:250}]}")
smooth = core.svp2.SmoothFps(clip,super["clip"],super["data"],vectors["clip"],vectors["data"],"{gpuid:11,gpu_qn:2,rate:{num:5,den:2},algo:23,mask:{cover:80},scene:{mode:0}}")
smooth = core.std.AssumeFPS(smooth,fpsnum=smooth.fps_num,fpsden=smooth.fps_den)

smooth.set_output()


vpy version still doesn't work for me with those settings (same duplicates pattern)



I don't perceive any type of flicker, but I do notice sometimes some very annoying artifacts around moving objects.

It's common for all optical flow approaches - the object boundaries cannot be distinguished accurately . It's impossible to track in some situations - e.g. objects move behind other objects , or object deformation and shape changes. The "solve" is usually motion tracking and rotoscoping


I had never thought about the inferior quality of the blended frames, now that I've done some comparisons I'd say that while it's true, most of the time it is barely noticeable.
It's more noticable on high quality sources, masters, or grainy sources. The fluctuation can be quite noticable and disturbing. There is another category of optical flow that only synthesize all frames (IIIIIIII...) - most Hollywood VFX work use this approach instead, but workflow is typically degrain, FX, then regrain

[markfilipak]

You have:
What I see with the poisondeathray test clip is this:
[A_____][AB____][B_____][BBC___][BCC___][C_____][CD____][D_____][DDE___][DEE___] mode 1
1 m 1 m m 1 m 1 m m <== mode 1 pattern you got
1 m m 1 m 1 m m 1 m <== mode 1 pattern should be
[A_____][A_____][B_____][B_____][BC____][C_____][C_____][D_____][D_____][DE____] mode 2
1 1 1 1 m 1 1 1 1 m <== mode 1 pattern you got
1 m 1 1 m 1 1 m 1 1 <== mode 2 pattern should be
Edited by Mark.

Hahaha, it would be really easy for us to get confused between what video was mv interpolated in what mode, wouldn't it?
Let's tread carfully, eh?

You took poisondeathray's test_6frames.mp4 and did your own transcodes, right? One in mode:0, one in mode:1 and one in mode:2, right?

mode 1's pattern does match... Look.

1 m 1 m m 1 m 1 m m <== mode 1 pattern you got
1 m m 1 m 1 m m 1 m <== mode 1 pattern should be
shift by 3 frames...

1 m 1 m m 1 m 1 m m <== mode 1 pattern you got
1 m m 1 m 1 m m 1 m <== mode 1 pattern should be
They do match.

Your mode 2 doesn't match what it should be.

1 1 1 1 m 1 1 1 1 m <== mode 2 pattern you got
1 m 1 1 m 1 1 m 1 1 <== mode 2 pattern should be

Your mode 2 pattern is awfully improbable.

[Alexkral]

You took poisondeathray's test_6frames.mp4 and did your own transcodes, right? One in mode:0, one in mode:1 and one in mode:2, right?

So I have to assume that you can't check for yourself with the vpy? Do you need an avs version?

mode 1's pattern does match... Look.

Sure, I know.

Your mode 2 doesn't match what it should be.

According to what? The reference could be outdated or just wrong. I have seen a lot of diagrams that match what I see (in most of the links that I have posted), only the starting point of the sequence is different, as with mode 1. Here you have one more:

https://www.svp-team.com/forum/viewtopic.php?pid=25281#p25281

"2m algo is sequence (2m2) like 3:2 pulldown, but third repeat of frame is MI-frame".

So 2m22m22m...

Your mode 2 pattern is awfully improbable.

That could be useful to define some things that we are used to seeing every day. ;)

[markfilipak]

So I have to assume that you can't check for yourself with the vpy? Do you need an avs version?
I'm sorry. Your post (http://forum.doom9.org/showthread.php?p=1941407#post1941407) didn't show your vpy. I thought the diagrams at the bottom were referring to the 60fps transcodes, not the 24fps source.
https://www.svp-team.com/forum/viewtopic.php?pid=25281#p25281

Ah! Now I see the pattern. I guess I didn't know what "2" in, e.g., "2m22m22m..." meant. Now I do. Thanks!

( I still like my pictures better ...:o )

[markfilipak]

Now I see what lead me astray:
https://www.svp-team.com/wiki/Manual:SVPflow
shows this:
"2m" mode: "1m11m11m11..."
which, using the '2-frame' form, would be this:
"1m2m2m2..."
which, if extended from 4 input frames to 8 input frames, would be this:
1m2m2m21m2m2m2
which has an obvious problem going from frame 9 back to frame 0.

But I guess "2m" mode is really this:
"11m1111m11..."
which, using the '2-frame' form, is this:
2m22m2
which, if extended from 4 input frames to 8 input frames, is this:
2m22m22m22m2
and has no problem going from frame 9 back to frame 0.

I guess '2m' mode has changed, eh? And now https://www.svp-team.com/wiki/Manual:SVPflow is stale.

I'll revise my documentation.

Thanks again.

Regards,
Mark.

PS: I don't know what's wrong, but every time I do anything, I have to log back in. Is this forum the same for you?

[markfilipak]

:thanks: Thanks to both of you, I think I'm about to make some real progress...

1 - I'm using InterFrame, but InterFrame uses the default 'mode:'. That means it's using Adaptive (mode:3), not Uniform (mode:0). I think that explains some of the troubles I've encountered. I will try to write my own version of InterFrame to use only mode:0. I hope I can figure out how to integrate it into VS.

2 - Regarding the correction in "2m" mode, that made quite a difference in judder metrics. Here are my revised diagrams
'mode:0' -- 1mmmm1mmmm -- converts 4 pictures (in 4 frames) to 10 pictures (2 original + 8 new, in 10 frames),
and thereby converts pictures-per-second from 24pps @ 24fps to 60pps @ 60fps.
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] input: 24fps(24pps)
[A_____][AAABB_][ABBBB_][BBBBC_][BBCCC_][C_____][CCCDD_][CDDDD_][DDDDE_][DDEEE_] output: 60fps(60pps) via interpolated motion vectors
1 m m m m 1 m m m m
---><----- 1/6 s ------><----- 1/6 s ------><----- 1/6 s ------><----- 1/6 s --- center-to-center output picture durations, seconds
picture judder: none

'mode:1' -- 1mm1m1mm1m -- converts 4 pictures (in 4 frames) to 10 pictures (4 original + 6 new, in 10 frames),
and thereby converts pictures-per-second from 24pps @ 24fps to 60pps @ 60fps.
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] input: 24fps(24pps)
[A_____][AAB___][ABB___][B_____][BC____][C_____][CCD___][CDD___][D_____][DE____] output: 60fps(60pps) via interpolated motion vectors
1 m m 1 m 1 m m 1 m
------><--- 7/180 s ----><------ 2/45 s ------><--- 7/180 s ----><------ 2/45 s center-to-center output picture durations, seconds
picture judder: 13%, 7:8 cadence, 12 judders/s.

'mode:2' -- 2m22m2 -- converts 4 pictures (in 4 frames) to 6 pictures (4 original + 2 new, in 10 frames),
and thereby converts pictures-per-second from 24pps @ 24fps to 36pps @ 60fps.
<---------------------- 1/6 s (scale = 480 characters/s) ---------------------->
[A_________________][B_________________][C_________________][D_________________] input: 24fps(24pps)
[A_____][A_____][AB____][B_____][B_____][C_____][C_____][CD____][D_____][D_____] output: 60fps(42pps) via interpolated motion vectors
1 1 m 1 1 1 1 m 1 1
00 s ----><----- 1/25 s -----><---- 13/300 s ----><----- 1/25 s -----><---- 13/3 center-to-center output picture durations, seconds
picture judder: 8%, 12:13 cadence, 12 judders/s.

[poisondeathray]

1 - I'm using InterFrame, but InterFrame uses the default 'mode:'. That means it's using Adaptive (mode:3), not Uniform (mode:0). I think that explains some of the troubles I've encountered. I will try to write my own version of InterFrame to use only mode:0. I hope I can figure out how to integrate it into VS.


It looks like Interframe uses mode:0 for both avs and vpy versions. These lines are the only mention of "mode:" . If/else condition for tuning=weak - and both conditions are set to mode:0

In havsfunc.py line 3919



if Tuning == 'weak':
SmoothString += ',area_sharp:1.2},scene:{blend:true,mode:0,limits:{blocks:50}}}'
else:
SmoothString += ',area_sharp:1.2},scene:{blend:true,mode:0}}'



avs version , line 145


Tuning == "Weak" ? Eval("""
SmoothString = SmoothString + ",area_sharp:1.2},scene:{blend:true, mode:0, limits: {blocks: 50}}}"
""") : Eval("""
SmoothString = SmoothString + ",area_sharp:1.2},scene:{blend:true, mode:0}}"
""")



Yes, you could modify the InterFrame function in the .py script, or .avs to add additional settings or presets - but you could just use svpflow directly too.

It helps to debug scripts when all parameters are listed out in the open. It's also easier to apply different settings to different sections. e.g. maybe you need a larger blocksize for the "picket fence" sections .

[markfilipak]

It looks like Interframe uses mode:0 for both avs and vpy versions. These lines are the only mention of "mode:" . If/else condition for tuning=weak - and both conditions are set to mode:0

In havsfunc.py line 3919

Yes, that's the text I was refering to.

avs version , line 145

I hope this doesn't raise too many chuckles, but what is avs?

Yes, you could modify the InterFrame function in the .py script, or .avs to add additional settings or presets - but you could just use svpflow directly too.

You mean directly calling SVSuper(), SVAnalyse(), SVConvert(), and SVSmoothFps(), right? Simply bypassing InterFrame, right?

It helps to debug scripts when all parameters are listed out in the open.

That's what I'm doing right now (and for the last 2 days). It's a bit of a chore made worse because it appears that the authors of the documentation use "smooth" and "smoothing", sometimes referring to judder, and sometimes referring to anti-aliasing. That makes it difficult to know what a particular json directive is all about.

It's also easier to apply different settings to different sections. e.g. maybe you need a larger blocksize for the "picket fence" sections .

That's a wonderful suggestion. Thanks!

[poisondeathray]

I hope this doesn't raise too many chuckles, but what is avs?

avs is avisynth script ; analogous to what vpy is to vapoursynth . The vpy version is a direct translation of the avs version of interframe.



You mean directly calling SVSuper(), SVAnalyse(), SVConvert(), and SVSmoothFps(), right? Simply bypassing InterFrame, right?


Yes, similar to the example Alexkral posted above. You can include other switches, adjust the settings. The documentation link has all the settings you can adjust


That's what I'm doing right now (and for the last 2 days). It's a bit of a chore made worse because it appears that the authors of the documentation use "smooth" and "smoothing", sometimes referring to judder, and sometimes referring to anti-aliasing. That makes it difficult to know what a particular json directive is all about.


Some of the descriptions might be lost in "translation"; but it's really not that important - because what happens with one source might be slightly different when applied to another source with slightly different characteristics

What is important is the actual effect or result. Just preview it in an editor like vsedit . Change a value , preview it again (f5). Near instant feedback. You can apply different settings for different sections.

But it's pretty futile trying to combat the "picket fence" issues. This is one of the prototypical "fail" scenarios for all optical flow methods, even newer "AI" or neural net ones. You can reduce the artifacts slightly by tweaking the settings, but you'll almost never get decently clean results without some manual work.

[Selur]

The vpy version is a direct translation of the avs version of interframe.
Not sure whether it's helpful, but there's also a slightly modified version (https://github.com/Selur/VapoursynthScriptsInHybrid/blob/master/interframe.py), which allows to overwrite overwrite Super/Vectors/Smooth options, using for example:
clip = interframe.InterFrame(clip, NewNum=50, NewDen=1, overwriteSuper="{scale:{up:0},gpu:1,rc:true}", overwriteVectors="{main:{search:{coarse:{distance:-8},type:2}},refine:[{thsad:250}]}", overwriteSmooth="{rate:{num:2,den:1,abs:false},algo:2,mask:{area:100},scene:{mode:0}}") # new fps: 50
Here's an AviSynth example where the parameters are overwritten and SVSuper/SVAnalyse/SVSmoothFps are called individually:
input = last
super=SVSuper(input, "{scale:{up:0},gpu:1,rc:true}")
vectors=SVAnalyse(super, "{main:{search:{coarse:{distance:-8},type:2}},refine:[{thsad:250}]}")
SVSmoothFps(input, super, vectors, "{rate:{num:2,den:1},algo:2,mask:{area:100},scene:{mode:0}}", mt=32)


Cu Selur