abolibibelot
28th December 2019, 23:07
I gave yet another shot at improving this 4 years old video of my grandma (already mentioned here (https://forum.doom9.org/showthread.php?t=175151) and here (https://forum.doom9.org/showthread.php?t=175918)) for what is hoped to be the final version of an insanely long running project. For this I had to make intensive use of Avisynth frame interpolation functions to fix an almost constant jerkiness / blurriness defect introduced in the raw footage by a defective image stabilizer : mainly FrameSurgeon, seconded by Morpheus and Morph for some tricky frames which were not satisfyingly processed by FrameSurgeon.
– As a general comment on the overall aspect of the interpolated frames, those produced by Morpheus tend to be slightly less precise while producing less egregious artifacts, compared with those produced by FrameSurgeon, which may or may not be preferred depending on each particular frame. When the adjacent frames are not too dissimilar and when there's little motion involved, both produce excellent and virtually indistinguishable result. Morph produces a markedly different output in that it tends to blend adjacent frames rather than attempting to interpolate the missing one ; in somes instances it may look cleaner on a still frame but most of the time it's more disturbing when seen in motion than an interpolated frame with moderate motion artifacting (especially when Morph is used to repair several frames in a row, or several individual frames located in close succession, especially if a large object is moving at the forefront, for instance if a person is walking right in front of the camera, and several frames are processed by Morph, it gives the impression that the person is teleporting rather than materially moving... or if the frame itself is moving rapidly it can look like a crossfade between two distinct clips) – although it may still be preferred when the interpolation artifacts are very conspicuous (in some cases as a last resort I pasted together several versions of a particularly tricky frame with GIMP and inserted the PNG output back into the script with RemapFrames – which is definitely overkill as noone but me will would have noticed and complained, or even cared). It also has a stability drawback : it can choke when more than a few dozen Morph commands are called within a script, and can cause a crash of some kind (Avisynth crash, not a system wide crash). I've extensively tested quite a few others interpolation functions before sticking to those three, which are very efficient in their own right and complimentary.
– According to "StainleSS", Morpheus was conceived to handle thousands of interpolation commands, yet in my experience it isn't as slick as FrameSurgeon in that regard. In particular, it generates a LOT of temporary files, and these ~_MORPHEUS_X_YYYYMMDD_HHMMSS_XXX.DB files are quite huge, 1344KB for most of them (it may not sound like much but when there are thousands it begins to add up to an awful lot), while a new bunch seems to be generated each time the script is reloaded in AVSPMod or any other utility, currently 132 new files for a total of 173MB (with 125-130 Morpheus commands in the script), to the point that it repeatedly filled up the working partition (a 100GB partition on a SSD), and, not knowing if I could delete them altogether, I resorted to apply NTFS compression on them, which reduced their size to 8KB each (those files are actually almost empty). When I noticed that issue in September I noted the total number of files and corresponding sizes, for each day :
2019/09/22 : 431 files, total size = 520MB, 3.23MB with NTFS compression
2019/09/07 : 2140 files, total size = 2.73GB, 16.6MB with NTFS compression
2019/09/10 : 59 files, total size = 76.3MB, 468KB with NTFS compression
2019/09/16 : 470 files, total size = 614MB, 3.66MB with NTFS compression
2019/09/17 : 1298 files, total size = 1.66GB, 10.1MB with NTFS compression
2019/09/18 : 813 files, total size = 1.04GB, 6.35MB with NTFS compression
2019/09/22 : 4571 files, total size = 5.85GB, 35.7MB with NTFS compression
2019/09/23 : 4873 files, total size = 6.24GB, 38.0MB with NTFS compression
2019/09/24 : 3844 files, total size = 4.92GB, 30.0MB with NTFS compression
2019/09/25 : 4426 files, total size = 5.67GB, 34.5MB with NTFS compression
And now I put all those files in a subfolder which amounts to a total of 70874 files for a whopping 90.4GB (574MB with NTFS compression).
There should be a way to reuse them the next time around when the same commands are used (or if a single one is added for instance), if at all possible with no significant programming or performance issue, or at least Morpheus should apply the NTFS compression automatically on those files, and put them in a subfolder right away, so as to avoid clogging the main directory which contains the scripts and other related files which need to be easily accessible.
By contrast, the temporary files generated by FrameSurgeon / ClipClop are much smaller, 35KB at most, and only a few are created during each session (besides they only seem to contain a copy of the list of commands).
– When running the script in VirtualDub2 x64 through Avisynth+ 0.1 r2772 x64, I noticed that some frames processed by Morph (not all of them) {=> actually all of them : see below} were wonky – it looks like the n-1 frame (for instance frame 12345 with the command Morph(12345,12347) which is supposed to re-create frame 12346) but dimmer and with weird patterns in some of the lighter areas. Is there any obvious explanation ? According to member "StainleSS" in this other thread, "Jenyok functions sometimes have 'undocumented features', and tend to be a bit big and cumbersome" ; could it be related with this issue ?
Illustrations :
17099
17100
The first screenshot is taken from VirtualDub2 x86, the frame is correctly processed (in this case two frames are processed in a row with the command Morph(23302,23305)) ; the second screenshot is taken from VirtualDub2 x64 (using the PullInterpFrames analysis function which allows to review only the interpolated frames),
– A different issue, not directly related with interpolation functions, although it did cause me trouble when applying them : using LWLibavVideoSource to load M2TS video files, I found out that it would display a few frames wrong, typically as a duplicate of the next one. With FFVideoSource the same frame appears normal. And if one of those frames (they are few and far between, with no obvious pattern) is interpolated with FrameSurgeon, it can produce that kind of wonky output :
17101
17102
Here the first screenshot shows frame #239 after a FrameSurgeon interpolation of 3 consecutive frames with the command I3 237 ; the second case is even weirder as it shows frame #5410 after the interpolation of the next one, #5411, with the command I1 5411. If I remember correctly, disabling the FrameSurgeon command, reloading the script, then enabling it again and reloading again would correct the display.
So, is LWLibavVideoSource known to be unreliable ? If so, is it a general problem, or does it affect some file types in particular ? Is there a list of recommended source plugins for each video format, or a list of the known behaviour of each one with its advantages and drawbacks depending on the situation ? (If not, there should !)
Thanks for any insight (if hindsight is 20/20, how much is insight ?... and oversight ?...).
EDIT : Actually, I verified again, all the Morph commands produce an abnormal output through Avisynth+ 64 bits.
{*} As a side note, I don't understand why those PNG screenshots have to be recompressed to JPG, as their size is only about 800KB, which is not that much by nowadays standards.
– As a general comment on the overall aspect of the interpolated frames, those produced by Morpheus tend to be slightly less precise while producing less egregious artifacts, compared with those produced by FrameSurgeon, which may or may not be preferred depending on each particular frame. When the adjacent frames are not too dissimilar and when there's little motion involved, both produce excellent and virtually indistinguishable result. Morph produces a markedly different output in that it tends to blend adjacent frames rather than attempting to interpolate the missing one ; in somes instances it may look cleaner on a still frame but most of the time it's more disturbing when seen in motion than an interpolated frame with moderate motion artifacting (especially when Morph is used to repair several frames in a row, or several individual frames located in close succession, especially if a large object is moving at the forefront, for instance if a person is walking right in front of the camera, and several frames are processed by Morph, it gives the impression that the person is teleporting rather than materially moving... or if the frame itself is moving rapidly it can look like a crossfade between two distinct clips) – although it may still be preferred when the interpolation artifacts are very conspicuous (in some cases as a last resort I pasted together several versions of a particularly tricky frame with GIMP and inserted the PNG output back into the script with RemapFrames – which is definitely overkill as noone but me will would have noticed and complained, or even cared). It also has a stability drawback : it can choke when more than a few dozen Morph commands are called within a script, and can cause a crash of some kind (Avisynth crash, not a system wide crash). I've extensively tested quite a few others interpolation functions before sticking to those three, which are very efficient in their own right and complimentary.
– According to "StainleSS", Morpheus was conceived to handle thousands of interpolation commands, yet in my experience it isn't as slick as FrameSurgeon in that regard. In particular, it generates a LOT of temporary files, and these ~_MORPHEUS_X_YYYYMMDD_HHMMSS_XXX.DB files are quite huge, 1344KB for most of them (it may not sound like much but when there are thousands it begins to add up to an awful lot), while a new bunch seems to be generated each time the script is reloaded in AVSPMod or any other utility, currently 132 new files for a total of 173MB (with 125-130 Morpheus commands in the script), to the point that it repeatedly filled up the working partition (a 100GB partition on a SSD), and, not knowing if I could delete them altogether, I resorted to apply NTFS compression on them, which reduced their size to 8KB each (those files are actually almost empty). When I noticed that issue in September I noted the total number of files and corresponding sizes, for each day :
2019/09/22 : 431 files, total size = 520MB, 3.23MB with NTFS compression
2019/09/07 : 2140 files, total size = 2.73GB, 16.6MB with NTFS compression
2019/09/10 : 59 files, total size = 76.3MB, 468KB with NTFS compression
2019/09/16 : 470 files, total size = 614MB, 3.66MB with NTFS compression
2019/09/17 : 1298 files, total size = 1.66GB, 10.1MB with NTFS compression
2019/09/18 : 813 files, total size = 1.04GB, 6.35MB with NTFS compression
2019/09/22 : 4571 files, total size = 5.85GB, 35.7MB with NTFS compression
2019/09/23 : 4873 files, total size = 6.24GB, 38.0MB with NTFS compression
2019/09/24 : 3844 files, total size = 4.92GB, 30.0MB with NTFS compression
2019/09/25 : 4426 files, total size = 5.67GB, 34.5MB with NTFS compression
And now I put all those files in a subfolder which amounts to a total of 70874 files for a whopping 90.4GB (574MB with NTFS compression).
There should be a way to reuse them the next time around when the same commands are used (or if a single one is added for instance), if at all possible with no significant programming or performance issue, or at least Morpheus should apply the NTFS compression automatically on those files, and put them in a subfolder right away, so as to avoid clogging the main directory which contains the scripts and other related files which need to be easily accessible.
By contrast, the temporary files generated by FrameSurgeon / ClipClop are much smaller, 35KB at most, and only a few are created during each session (besides they only seem to contain a copy of the list of commands).
– When running the script in VirtualDub2 x64 through Avisynth+ 0.1 r2772 x64, I noticed that some frames processed by Morph (not all of them) {=> actually all of them : see below} were wonky – it looks like the n-1 frame (for instance frame 12345 with the command Morph(12345,12347) which is supposed to re-create frame 12346) but dimmer and with weird patterns in some of the lighter areas. Is there any obvious explanation ? According to member "StainleSS" in this other thread, "Jenyok functions sometimes have 'undocumented features', and tend to be a bit big and cumbersome" ; could it be related with this issue ?
Illustrations :
17099
17100
The first screenshot is taken from VirtualDub2 x86, the frame is correctly processed (in this case two frames are processed in a row with the command Morph(23302,23305)) ; the second screenshot is taken from VirtualDub2 x64 (using the PullInterpFrames analysis function which allows to review only the interpolated frames),
– A different issue, not directly related with interpolation functions, although it did cause me trouble when applying them : using LWLibavVideoSource to load M2TS video files, I found out that it would display a few frames wrong, typically as a duplicate of the next one. With FFVideoSource the same frame appears normal. And if one of those frames (they are few and far between, with no obvious pattern) is interpolated with FrameSurgeon, it can produce that kind of wonky output :
17101
17102
Here the first screenshot shows frame #239 after a FrameSurgeon interpolation of 3 consecutive frames with the command I3 237 ; the second case is even weirder as it shows frame #5410 after the interpolation of the next one, #5411, with the command I1 5411. If I remember correctly, disabling the FrameSurgeon command, reloading the script, then enabling it again and reloading again would correct the display.
So, is LWLibavVideoSource known to be unreliable ? If so, is it a general problem, or does it affect some file types in particular ? Is there a list of recommended source plugins for each video format, or a list of the known behaviour of each one with its advantages and drawbacks depending on the situation ? (If not, there should !)
Thanks for any insight (if hindsight is 20/20, how much is insight ?... and oversight ?...).
EDIT : Actually, I verified again, all the Morph commands produce an abnormal output through Avisynth+ 64 bits.
{*} As a side note, I don't understand why those PNG screenshots have to be recompressed to JPG, as their size is only about 800KB, which is not that much by nowadays standards.