How to blend video frames and place them between mainframes

[Mitra]

I have an interlaced video stream and need blend video frames and place them between mainframes like this :

Input frames: [1] [2] [3] .....> FPS=25
| \ / | \ / |
| \ / | \ / |
Output frames: [1] [Bl(1,2)] [2] [Bl(2,3)] [3] .....> FPS=50

Please tell me how to do this?

[DTL]

If you create blends of interlaced field-based frames it will blend 4 phases of motion into single frame. Do you want to keep output FPS=50 interlaced ?

With 2 input frames (1st frame = field1+field2, 2nd frame = field3+field4) simple 2D full-height blending will create interlaced frame with field1+3, field2+4 fields. May be you need first to make deinterlacing to make 25fps frame-based stream ?

If you need simply more smooth video - it is better to make motion-compensated interpolation on a sequence of separated fields and pack back to 50fps interlaced frames. The more complex and possibly better quality is to make
1. 25i->50p double rate deinterlacing (QTGMC like complex plugin)
2. 50p->100p double rate mc-interpolation (mvtools2 like mc-engine)
3. 100p->50i interlaced packing/compression (script-based transform with built-in AVS commands)

[Dogway]

Is this ok?
blend = Merge(selectevery(1, 0),selectevery(1, 1))
interleave(last,blend)

EDIT: oops sorry, you meant interlaced!

[Mitra]

.... Do you want to keep output FPS=50 interlaced ?

.... May be you need first to make deinterlacing to make 25fps frame-based stream ?


Thank you for your reply.

I need to get real-time smooth slow motion from i50 sources. the scan type of output does not matter, but the higher frame rate is better (I think).

I tried different frame interpolation methods (like SVP , Ai ,...), but to be honest, today I realized that none of them are as smooth as vMix (https://www.vmix.com/software/download.aspx)real-time slowmotion output.
I think vMix uses a simple blending method, but I'm not an expert , if possible an expert please try vMix and tell me what method it uses?

[Mitra]

Is this ok?
blend = Merge(selectevery(1, 0),selectevery(1, 1))
interleave(last,blend)

EDIT: oops sorry, you meant interlaced!

Thank you for your reply.

Do you have any ideas about interlaced or vMix that i mentioned above?

[DTL]

May be vMix simply uses NVIDIA Optical Flow HWacc mc-interpolation so it outperform in quality and speed the old SVP ? NVIDIA promises good progress in Optical Flow in latest chips.

"from i50 sources. "

If it is 50 fields per second - to get even smoother you need 100 fields/frames per second or more. Not very easy to make in realtime even with latest end-user CPUs for big enough frame size.

[Dogway]

So you want it deinterlaced I guess, you should run first QTGMC as DTL suggested and then my code snippet. But yes, probably you want frame interpolation so you can try MysteryX's FrameRateConverter, there are better AI interpolators in VapourSynth though but haven't used them.

[StainlessS]

Whats this look like to you ? [output 100p]

# Simple realtime
AssumeTFF() # or BFF
Bob() # Or alternative bob like realtime, out doublerate 50p
ConvertFPS(100.0) # Blended (alternate) frames(originally fields), or some fast doublerate interpolator

EDIT:

# Alternative Simple realtime
AssumeTFF() # or BFF
NED=NNedi3(Field=-2, nns=2) # Field -2 = double rate, internal parity value to start
YadifMod2(Order=-1,Mode=1,edeint=NED) # Order -1 = Internal Parity. Mode 1=double rate do spactial check (3=no spatial check).
ConvertFPS(100.0) # Blended (alternate) frames(originally fields), or some fast doublerate interpolator


EDIT:

# Alternative Simple realtime
AssumeTFF() # or BFF
QTGMC(Preset="fast",NoisePreset="Fast",SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1, tr2=0) # Boulder
ConvertFPS(100.0) # Blended (alternate) frames(originally fields), or some fast doublerate interpolator

[Mitra]

May be vMix simply uses NVIDIA Optical Flow HWacc mc-interpolation so it outperform in quality and speed the old SVP ? NVIDIA promises good progress in Optical Flow in latest chips.

"from i50 sources. "

If it is 50 fields per second - to get even smoother you need 100 fields/frames per second or more. Not very easy to make in realtime even with latest end-user CPUs for big enough frame size.

I think it does not use the GPU, but how to check it?

[poisondeathray]

I think it does not use the GPU, but how to check it?

You use a software based hardware monitor to measure GPU % usage like GPU-z, while performing the task

[StainlessS]

May be vMix simply uses NVIDIA Optical Flow HWacc mc-interpolation
If so, then would be a requirement of vMix [no idea if it is a requirement, vmix ad should say if so].

[Mitra]

Thanks everyone, I will try now and let you know.

[Mitra]

Thanks to all again.

Now I'm sure vMix does not use NVIDIA Optical Flow.

@StainlessS, Thank you so much!

QTGMC seems to be what I need, your script result is great and so better than vMix output, but so slow for real-time.

I tried the "Ultra Fast" preset, but it was not fast enough yet.

Do you have an idea to speed it up or improve the quality of the other your scripts?

[poisondeathray]

Are you using multithreading ? Prefetch(x) where "x" is some value .

[DTL]

If the simple double-rate deinterlace 25i to 50p is enough smooth for you and you need more speed - you can again look into NVIDIA (may be AMD too) motion-adaptive hardware deinterlace. May be already having some way to download 50p frames back to host memory and feed into your application. For simple realtime playback you can just run any player with hardware double-rate deinterlace. I typically use MPC-HC player.

There are many double-rate deinterlace onCPU plugins. I sometime use yadif (yadifmod2) http://avisynth.nl/index.php/Yadifmod2 to get double-rate deinterlaced inside AVS environment because QTGMC too complex for me to setup infinite number of required plugins.

[StainlessS]

QTGMC is inherently slow,
Those settings were provided by Boulder as producing little processing/denoising, and so reasonably fast,
maybe not fast enough for your machine.

What machine are you playing with ? [just so the boys/and girls, have some idea]

Maybe try the YadifMod2 deinterlacer thingy instead, perhaps with a HQDN3D afterwards, its fast.

ConvertFPS would be about as fast as you can get for uprate 50p to 100p,
if Yadifmod2 with HQDN3D is fast enough, perhaps SvpFlow might be better quality doublerater than ConvertFPS,
is supposed to be better for realtime, but I dont really use it.

You could always forget about 50p -> 100p, and look for better denoise. [FFT3DFilter(Sigma=1.8,Plane=0) is pretty fast, increase sigma for more denoise]

The boys & girls always like a sample to play with.

EDIT: And yes of course Prefetch(4) or therabouts would be a good idea at end of script [before Return Last]

[Mitra]

@poisondeathray, Thanks, I had forgotten MT, I enabled Prefetch(4) now it's faster but still not enough for real time.

@DTL, Thanks for your recommendations, I will try them.

[Mitra]

@StainlessS,

I use an i7 10700k + Nvidia 1660Ti and think it is fast enough.

Thanks for your recommendations,I will try them.

[kedautinh12]

Fast don't mean that quality. Too fast can detroy your details in video

[johnmeyer]

You can do this in any NLE (i.e., video editor). It's a lot easier. AVISynth is wonderful, but is a lousy tool for this. It takes me less than 1 second to do this in my NLE (Vegas), and I can instantly preview the effect.

[poisondeathray]

@poisondeathray, Thanks, I had forgotten MT, I enabled Prefetch(4) now it's faster but still not enough for real time.

I use an i7 10700k + Nvidia 1660Ti and think it is fast enough.


You should be able to get faster than realtime on that CPU . 8C/16T . Try Prefetch(16)

Don't go faster than preset="super fast" for QTGMC, otherwise yadif or bob is used for the repair mode . In that case you might as use yadif(1,1), or bwdif(field=3) .

[Mitra]

@poisondeathray, Thanks, Prefetch(16) increased the speed by about 20%.
I will try yadif(1,1) and bwdif(field=3)

[StainlessS]

If you are not gonna supply sample, what is rez [EDIT: and colordepth] of your source ?

Maybe other could give better args to QTGMC for HD source. [blocksize, subpel, and others, I usually just use a preset, if I use QTGMC at all]

EDIT:
@poisondeathray, Thanks, Prefetch(16) increased the speed by about 20%.
your CPU has 8 cores, 16 threads with hyperthreading,
suggest at least try Prefetch(8), and maybe 12. [it varies but physical cores often recommended]

[Mitra]

If you are not gonna supply sample, what is rez [EDIT: and colordepth] of your source ?

Maybe other could give better args to QTGMC for HD source. [blocksize, subpel, and others, I usually just use a preset, if I use QTGMC at all]

EDIT:

your CPU has 8 cores, 16 threads with hyperthreading,
suggest at least try Prefetch(8), and maybe 12. [it varies but physical cores often recommended]

Sorry, 1080i50 , YUV422 8bits