Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th October 2021, 20:23   #1  |  Link
markfilipak
Registered User
 
markfilipak's Avatar
 
Join Date: Jul 2016
Location: Mansfield, Ohio (formerly San Jose, California)
Posts: 265
MV interpolation/interframes explained -- Correctly?

Frankly, I find the documentation of Super(), Analyse(), and SmoothFps() somewhat vague. It's undoubtedly my shortcoming. How do these illustrations and text look to you? Of course they are consistent and seem to make sense, but that doesn't mean they aren't wrong. Kindly don't say "Good, Mark" unless you absolutely know they are correct. I suppose that means you're familiar with the source code.

Principles: Show, don't tell. Short, simple declarative sentences. Don't be pedantic.

I eagerly await your thoughts. Thanks, folks!
--Mark.
Code:
                              __________   [diagram 8A] Image fragments and motion.
                              |    frag C|
                              |          |   This texipix diagram illustrates how an image fragment can appear to move frame to frame:
              __________    |          |   Imagine 3 source video frames, 'N-1' 'N' & 'N+1'; imagine time flows 'N-1'->'N'->'N+1'.
              |    frag B|    |in frame N+1  Imagine that 'N-1' 'N' & 'N+1' are conveniently stacked and that we can 'see' through them.
              |          |    |__________|   Imagine that 'B' is an image fragment of 'N', and likewise for 'A' (of 'N-1') & 'C' (of 'N+1').
    _________|         |                   In the diagram, 'A'->'B'->'C' shows how the image fragment 'moves' during 'N-1'->'N'->'N+1'.
    |    frag A|     in frame N              Current: Imagine that 'N' is the 'current' frame and that, consequentially, 'B' is the 'current' fragment.
    |          |_________|
    |          |
    |      in frame N-1       __________   [diagram 8B] Interframes with linear positioning.
    |__________|         .....|......   C|
                    .....:....|.    :    |   This texipix diagram illustrates an example of the processing objective.
              _____:____:    |:    :    |   Objective: Synthesize interframes between source frames to increase frame rate.
           ...|........ B|    |:    :    |   In this example, 2 interframes are synthesized to triple frame rate.
       ....:..|....   :  |    |:____:____|   Interframes are 'filled in' by copying the fragments, 'A' 'B' 'C', from the frames to the interframes.
    __:___:__|  :   :  |.....:....:        The copied fragments are positioned linearly, by proportional parts, along straight paths.
    |         A|  :   :  |.....:             The directions, 'A'->'B' & 'B'->'C', point the way; the pointers are the vectors!
    |          |__:___:__|
    |          |..:...:
    |          |..:           __________   [diagram 8C] Interframes with spline positioning.
    |__________|        ......|.....    C|
                   .....:.....|    :     |   This texipix diagram illustrates spline positioning based on 'ballistics'.
              .____:_____.    :    :     |   Ballistics implies expanding the scope from 'N-1'->'N'->'N+1' to 'N-2'->'N-1'->'N'->'N+1'->'N+2'.
          ....|.......  B|    :    :     |   Spline positioning would produce 'smoother' vectors along curved paths (not straight paths).
       ...:...|....  :   |    :____:_____|   It would produce slightly better results.
    .__:__:___|.  :  :   |....:....:         However, in the quest for simplicity, spline positioning is not done.
    |         A|  :  :   |....:              Overlap (i.e. Analyse.block.overlap) is meant to compensate for less precision (but does it, really?).
    |          |__:__:___|
    |          |..:..:
    |          |..:
    |__________|
   _____________frame_N________________     [diagram 8D] Grid blocks in 'current' frame.
   |                              
   |                        _________   This texipix diagram illustrates the block grid in frame 'N' (the 'current' frame).
   |                        |        C|   All frames have identical grid layouts and block sizes, but only the 'current' frame's grid is active.
   |                        |         |   This diagram reveals how image fragment 'B' is chosen; it's simply the image in its particular grid block.
   |     __________  |     |   Each grid block contains its own fragment, its own matches in 'N-1' & 'N+1', and its own motion vector.
   |          |         B|    |         |   Note that the locations of fragments 'A' or 'C' will rarely align with the grid.
   |          |          |    |_________|   Also note that frames 'N-1' 'N' & 'N+1' (and their images) are never altered.
   |_________|         |          
   ||         A|         |                  During 'B's processing, 'A' (and its location in 'N-1') and 'C' (in 'N+1') are found by searching for them.
   ||     |_________|           The Analyse() function -- configured by configAnalyse -- does the searching & matching.
   ||         |                           Except for completely static video, exact matches are not possible.
   ||         |                           All the image fragments in all the grid blocks of 'N' are searched for and matched in 'N-1' & 'N+1'.
   ||_________|                           'B' is just an example.
   |                              
   _____________frame_N________________     [diagram 8E] Backward interpolation.
   |                              
   |                                      Focusing on the 'current' fragment, 'B', this texipix diagram illustrates 'backward' interpolation.
   |                                      'Backward' means backward in time, from the 'current' frame, 'N', to the previous frame, 'N-1'.
   |                                      'B-1' & 'B-2', are the 'backward' copies of 'B' along 'B'->'A' (and are identical to 'B').
   |     __________           The positions of 'B-1' & 'B-2' are interpolated from the 'B'-to-'A' vector length & direction.
   |       ...|........ B|          
   |   ....:..|....B-1:  |          
   |   :   :  |B-2:   :  |          
   |   :   :  |   :   :  |          
   | : : |___:___:__|       
   |   :   :.....:...:            
   |   :.........:                
   |                              
   |                              
   _____________frame_N________________     [diagram 8F] Bidirectional interpolation.
   |                              
   |                                      Focusing on fragment 'B', this texipix diagram illustrates bidirectional interpolation.
   |                                      'Bidirectional' means 'backward', 'N' to 'N-1', plus 'forward', 'N' to 'N+1'.
   |               ...........            'B-1' is the 'backward' copy of 'B' (along 'B'->'A'); 'B+1' is the 'forward' copy (along 'B'->'C').
   |     _____:____ B+1:        The position of 'B-1' is interpolated from the 'B'-to-'A' vector length & direction.
   |       ...|........ B|     :            The position of 'B+1' is interpolated from the 'B'-to-'C' vector length & direction.
   |       :  |    B-1:  |     :    
   |       :  |       :  |     :    
   |       :  |       :  |.....:    
   |   : |_______:__|       
   |       :.........:            
   |                              
   |                              
   |                              
                              __________   [diagram 8G] Backward interpolation and image fragment transitions.
                         .....|......   C|
                    .....:....|. C-1:    |   This texipix diagram illustrates the 'A'->'B'->'C' transition for backward interpolation.
              _____:____:  C-2:    :    |   'A' & 'B-2' (copied from 'B') abut (in time) -- the transition is 'A'->'B-2'.
           ...|........ B|    |:    :    |   'B' & 'C-2' (copied from 'C') abut (in time) -- the transition is 'B'->'C-2'.
       ....:..|....B-1:  |    |:____:____|
    __:___:__|B-2:   :  |.....:....:
    |         A|  :   :  |.....:
    |          |__:___:__|
    |          |..:...:
    |          |..:           __________   [diagram 8H] Bidirectional interpolation and image fragment transitions.
    |__________|         .....|......   C|
                    .....:....|. C-1:    |   This texipix diagram illustrates the 'A'->'B'->'C' transition for bidirectional interpolation.
              _____:____:  B+1:    :    |   'A+1' (copied from 'A') & 'B-1' (copied from 'B') abut (in time) -- the transition is 'A+1'->'B-1'.
           ...|........ B|    |:    :    |   'B+1' (copied from 'B') & 'C-1' (copied from 'C') abut (in time) -- the transition is 'B+1'->'C-1'.
       ....:..|....B-1:  |    |:____:____|
    __:___:__|A+1:   :  |.....:....:        It's inferred that bidirectional interpolation is superior to backward interpolation.
    |         A|  :   :  |.....:             The difference cannot be the image fragments because they're the same in both cases.
    |          |__:___:__|                   The difference must be the vectors: backward is longer, bidirectional is half the length.
    |          |..:...:                      It's possible that when all fragments are written to a particular interframe, gaps may be left.
    |          |..:                          In aggregate across an entire interframe, shorter vectors produce fewer such gaps, and smaller.
    |__________|                             Fewer, smaller gaps in the abuting interframes means better overall images needing less 'blending'.

The goodness of interframe images depends on how well Super.pel, Analyse.block.w, Analyse.block.h, Analyse.block.overlap, and SmoothFps.algo harmonize.

Last edited by markfilipak; 11th October 2021 at 00:00. Reason: Update, improvements
markfilipak is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:54.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.