Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 28th November 2020, 19:30   #4121  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
What do you mean by "more clever"? Like adding temporal consistency?
feisty2 is offline   Reply With Quote
Old 29th November 2020, 08:33   #4122  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Yes, exactly. Temporal temporal consistency while taking scene changes into account. I was looking for some 'auto' filters like AutoAdjust, AutoContrast, Autolevels, HDRAGC and similar for Vapoursynth and the frameeval example of autowhite was the only thing I found. Seeing that it was a 'simple per frame auto white balance', I was wondering if there are mor complex/clever solutions out there.
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 29th November 2020, 22:08   #4123  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by Selur View Post
Yes, exactly. Temporal temporal consistency while taking scene changes into account. I was looking for some 'auto' filters like AutoAdjust, AutoContrast, Autolevels, HDRAGC and similar for Vapoursynth and the frameeval example of autowhite was the only thing I found. Seeing that it was a 'simple per frame auto white balance', I was wondering if there are mor complex/clever solutions out there.
here: https://github.com/IFeelBloated/Auto...master/src.cxx, basically the same as that FrameEval stuff, but with temporal functionalities. writing vaporsynth filters is really easy, it's really no different from scripting in Python. you should consider writing a filter yourself next time you wanna add some basic extensions to an existing script/plugin.
feisty2 is offline   Reply With Quote
Old 2nd December 2020, 10:43   #4124  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Nice. Thanks!
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 9th December 2020, 16:59   #4125  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by feisty2 View Post
I think "id" should be a member of VSVideoFormat in API v4, otherwise it requires access to the core to determine if a clip has a constant format which is not very convenient
Only id = 0 mean variable format. This is strictly enforced by all the api functions. The reason the api function need the core pointer is to maintain V3 api compatibility where the id isn't a simple serialization so a V4 filter won't choke if it's fed a V3 id.
If it truly bothers you look at the macro used to create ids and parse them on your own, the only advantage the api gives you is that invalid combinations will be rejected instead of possibly passed on.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 9th December 2020, 18:23   #4126  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
I've unified the memory layout of VideoFormat and VideoInfo in my wrapper (which currently still relies on API v3) to match that of API v4:
https://github.com/IFeelBloated/vsFi.../Node.vxx#L109
https://github.com/IFeelBloated/vsFi...tadata.vxx#L18
it should be fully compatible with API v4 at the source code level.
feisty2 is offline   Reply With Quote
Old 14th December 2020, 15:39   #4127  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
Guys, please up vote my request for direct VapourSynth support for x265:

https://github.com/msg7086/x265-Yuuki-Asuna/issues/11
stax76 is offline   Reply With Quote
Old 18th December 2020, 09:42   #4128  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
I have started noticing this memory hungry behavior on Vapoursynth on preview editors like vseditor and virtualdub2. Whenever a script was loaded, Vapoursynth seems to allocate a big chunk of memory before any frame was even requested. For example, I have a script with 1080p video and QTGMC in vseditor, on script load the memory jumped straight to 4.5 GB of RAM, this is both unnecessary and unsustainable for multi tab editor, where it will be running out of memory for most people just by opening four or five similar scripts.
lansing is offline   Reply With Quote
Old 18th December 2020, 13:05   #4129  |  Link
edcrfv94
Registered User
 
Join Date: Apr 2015
Posts: 84
Can filter once, multiple output to simultaneous x265 encoding? e.g. different filter stage or resolution 720p 1080p 2160p.
edcrfv94 is offline   Reply With Quote
Old 18th December 2020, 13:27   #4130  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
yes, see: https://github.com/IFeelBloated/Oyst...a/Alpha.py#L49
feisty2 is offline   Reply With Quote
Old 18th December 2020, 14:45   #4131  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
I'm not understanding why line 1+2 are needed here, isn't that exactly what altsearchpath=True in line 3 is supposed to do? Without line 1+2 it's not working.

Code:
import os.path
os.environ["PATH"] = r"C:\Anime4KCPP VapourSynth" + os.pathsep + os.environ["PATH"]
core.std.LoadPlugin(r"C:\Anime4KCPP VapourSynth\Anime4KCPP_VapourSynth.dll", altsearchpath=True)
clip = core.anime4kcpp.Anime4KCPP(clip, GPUMode = 1, ACNet = 1, zoomFactor = 2, HDN = 1, HDNLevel = 2)
stax76 is offline   Reply With Quote
Old 18th December 2020, 16:09   #4132  |  Link
Are_
Registered User
 
Join Date: Jun 2012
Location: Ibiza, Spain
Posts: 321
Doesn't documentation say that's for dependencies?
Are_ is offline   Reply With Quote
Old 18th December 2020, 17:02   #4133  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
I think I understand it now, what altsearchpath=True does is telling the LoadLibrary function to search dependencies in PATH.

Last edited by stax76; 18th December 2020 at 17:04.
stax76 is offline   Reply With Quote
Old 18th December 2020, 22:39   #4134  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 321
Quote:
Originally Posted by lansing View Post
I have started noticing this memory hungry behavior on Vapoursynth on preview editors like vseditor and virtualdub2. Whenever a script was loaded, Vapoursynth seems to allocate a big chunk of memory before any frame was even requested. For example, I have a script with 1080p video and QTGMC in vseditor, on script load the memory jumped straight to 4.5 GB of RAM, this is both unnecessary and unsustainable for multi tab editor, where it will be running out of memory for most people just by opening four or five similar scripts.
Nowadays memory should not be an issue. Or you just limit it. For my Preview I use that below, first free memory is evaluated and if not enough, it is limited in vapoursynth. No more memory problems. Sure it might be not optimal, but on the other hand it never freezes, there is no other way if not enough memory, like using QTGMC for HD etc, and it is automatic, no more guessing or setting max cache size.

Code:
import vapoursynth as vs
from vapoursynth import core
try:
    import psutil
except ImportError:
    pass
import  os

def freeRAM():
    '''
    getting free RAM
    first it uses non standard library cross platform psutil
    if modul is not installed, it falls back using Linux or Windows ways to get free RAM
    so for Mac it needs psutil: to install psutil: pip3 install psutil
    '''
    available_RAM = None
    
    #cross platform try if psutil is installed
    try:
        mem = psutil.virtual_memory()
        available_RAM = int(mem.available/1024/1024)
        if available_RAM and isinstance(available_RAM, int):
            return available_RAM
    except:
        pass

    #windows fallback
    try:
        proc = os.popen('wmic.exe OS get FreePhysicalMemory')
        l = proc.readlines()   #l should be a list
        proc.close()
    except:
        pass
    else:
        for i, item in enumerate(l):
            try:
                available_RAM = int(l[i])
                available_RAM = int(available_RAM/1024)
                if available_RAM and isinstance(available_RAM, int):
                    return available_RAM
            except:
                pass
               
    #linux fallback
    try:
        meminfo = dict((i.split()[0].rstrip(':'),int(i.split()[1])) for i in open('/proc/meminfo').readlines())
        available_RAM = int(meminfo['MemAvailable']/1024)
        if available_RAM and isinstance(available_RAM, int):
            return available_RAM
    except:
        pass
                      
    #failed to get free RAM            
    return None

def limit_core_cache(core):
    '''
    Returns tuple (core, log).
    Sets core.max_cache_size for less than available RAM,
    log is string type ready for print.
    '''
                
    log = []
    available = None
    vs_cache = core.max_cache_size
    log.append(f'vapoursynth cache is set to: {vs_cache}MB')
    available_RAM = freeRAM()
    if available_RAM is not None:
        log.append(f'free RAM: {available_RAM}MB')
        deduct = 0
        if available_RAM < 200:
            deduct = 50                   
            log.append('almost no RAM, system likely to freeze')
        elif 200 <= available_RAM < 400:
            deduct = 120                   
            log.append('not much RAM at all, freezing likely, lagish performance')
        elif 400 <= available_RAM < 1024:
            deduct = 220
            log.append('more RAM would give better performance')
        elif 1024 <= available_RAM < 1536:
            deduct = 280                       
        else:
            deduct = 350
        new_cache =  max(50, available_RAM - deduct)          
        if new_cache < vs_cache:
            log.append(f'setting Vapoursynth cache to: {new_cache}MB')
            core.max_cache_size = new_cache
    else:
        log.append('\nWARNING, failed to get available free RAM,')
        log.append('         Vapoursynth cache was not limited if needed,')
        log.append('         RAM overrun or freeze possible\n')
    return core, '\n'.join(log)
So it might be put as some extra module and just calling it:
Code:
import my_module
core, info = my_module.limit_core_cache(core)
if info: print(info) #or log or something

Last edited by _Al_; 18th December 2020 at 23:00.
_Al_ is offline   Reply With Quote
Old 19th December 2020, 01:15   #4135  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
Quote:
Originally Posted by _Al_ View Post
Nowadays memory should not be an issue. Or you just limit it. For my Preview I use that below, first free memory is evaluated and if not enough, it is limited in vapoursynth. No more memory problems. Sure it might be not optimal, but on the other hand it never freezes, there is no other way if not enough memory, like using QTGMC for HD etc, and it is automatic, no more guessing or setting max cache size.
I think for editor, the limiting of cache size should be frame based rather than memory based. The same QTGMC filter running in avspmod only takes 915 MB on playback because avs+(I think) only caches 3 frames.
lansing is offline   Reply With Quote
Old 19th December 2020, 03:05   #4136  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 321
I'm no developer, grateful for I got available.
Some ideas, but sure they could be dead wrong, lacking proper developer background and experience:

-Saying 3 frames sounds like an arbitrary solution, because frame could be any size, SD, HD, 4k. Folks have 32GB nowadays. I have 20GB in "cheaper" laptop. 16GB extra for something like $30.

-Couple of lines above this, feisty2 is storing RAW video so script does not run for more things. We got 4k on the horizon that's the way I guess.

-I mentioned that before. Having more instances of script and switching between them. I think much better is to have clip instances and switch between them and so adjust workflows and habits to do so. Python literally wants you to. Tons of filters above some point, denoise, QTGMC is already rendered and some clips are forked at the bottom of the script, below, to have many filters absolutely the same. If nothing, then at least source plugin is the same. No need to calculate something 5x because having 5 scripts. To get rid of avisynth workflows might help. A red-neck-faulty-spaghetti-code preview I did, switching between clips, not scripts, is very fast comparing clips, instant during playback.

-Also Qt should remember frames, some decent number, depending on size, same as opencv so switching and comparing frames ones were rendered should be instant.

Last edited by _Al_; 19th December 2020 at 03:08.
_Al_ is offline   Reply With Quote
Old 30th December 2020, 08:47   #4137  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
if an installed logger is never manually removed by removeLogHandler(), will it be automatically removed when the core is freed?
feisty2 is offline   Reply With Quote
Old 30th December 2020, 20:20   #4138  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by feisty2 View Post
if an installed logger is never manually removed by removeLogHandler(), will it be automatically removed when the core is freed?
Yes yes yes!
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 31st December 2020, 17:31   #4139  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by lansing View Post
I think for editor, the limiting of cache size should be frame based rather than memory based. The same QTGMC filter running in avspmod only takes 915 MB on playback because avs+(I think) only caches 3 frames.
But _Al_ does have some good points...

It's often faster for me to preview in avspmod using VSImport() for me when using multiple tabs and navigating, even with the overhead. It's less sluggish and it seems more frames are held in memory
poisondeathray is offline   Reply With Quote
Old 11th January 2021, 20:25   #4140  |  Link
Soliloquy
Registered User
 
Join Date: Apr 2010
Posts: 21
I'm having troubles running portable python / vapoursynth on win10 x64, version 1909.
I got python 3.9.1 and placed it in a folder, then extracted vapoursynth r52 in the same folder. Lastly i exctracted vapoursynth editor and upon running, it simply prints "Failed to initialize VapourSynth environment!".
Not sure what to try next - googled the issue and didn't really find anything to try besides typing import sys; sys.executable and see what it outputs, which is 'E:\\Encoding\\vapoursynth\\python.exe'.
What do i try next?
Soliloquy is offline   Reply With Quote
Reply

Tags
speed, vaporware, vapoursynth

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:27.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.