Vapoursynth

[Selur]

It's probably some filter with a memory leak.
Memory consumption doesn't run amok here when using 'QTGMC(Input=clip, Preset="Fast", TFF=True, opencl=True)', stays at around 4.3GB for Blu-ray content.
What QTGMC setting do you use?

Cu Selur

[_Al_]

Just Preset = "Slow"and TFF=True, but I am not complaining because I test things on my laptops which have ridiculous 4GB RAM. On the contrary by limiting RAM I am surprised that it works. Mostly there is about 1GB available for Vapoursynth. Sure performance is perhaps severely limited. Not sure now about workstation, but anyway for all scripts I limit RAM usage first, checking how much is available and setting available cache as a rule. So I might not even know, if RAM was leaking somewhere. I do not know how much control can Vapoursynth have above all those DLL's out there. Scripts are not crashing so I gather it kind of works.

[poisondeathray]

Originally Posted by _Al_

Same is hapening with QTGMC (havsfunc) and there might be others. I just simply tell Vapoursynth to limit RAM,
some cross platform method, like in python:
mem = psutil.virtual_memory()
available = int(mem.available/1024/1024) #MB
cache = available - some_wiggle_room_value
core.max_cache_size = cache
Vapoursynth then releases RAM, if I watch usage it goes all the way to the limit and then it is drastically released. It cycles like that for a some short time - filled/released,filled/released but strangely, after some short time, it settles somewhere underneath that set limit. Almost like someone wrote that in some intelligent way, or it is a coincidence, not sure.

I can't reproduce on x64 after 10 min . It just hovers around 1.2-1.3 GB RAM . Does not have characteristics of a memory leak

Maybe a problem with one of your plugin/filter dependencies ?

[_Al_]

There might be, I'm also one version behind with Vapoursyth and Python 3.6. Simple script to deinterlace avchd.M2TS with about 1GB available, it takes about couple of seconds to freeze. If cache is limited, no problem.

file = r'C:\vid\avchd.M2TS'
from vapoursynth import core
import havsfunc as haf
#core.max_cache_size = 800 #this works 
clip = core.lsmas.LWLibavSource(file)
clip = haf.QTGMC(clip, Preset='Slow', TFF=True)
clip.set_output()

[asarian]

Originally Posted by Selur

It's probably some filter with a memory leak.
Memory consumption doesn't run amok here when using 'QTGMC(Input=clip, Preset="Fast", TFF=True, opencl=True)', stays at around 4.3GB for Blu-ray content.
What QTGMC setting do you use?

Cu Selur

No memory leaks here on my end whatsoever. And I use QTGMC on a daily basis.

P.S. I see you're using 'opencl=True'. Interesting. Shouldn't be needed when you have 'Denoiser="KNLMeansCL"' in the parameters, though, right? Or does it something else too?

[jackoneill]

Originally Posted by _Al_

There might be, I'm also one version behind with Vapoursyth and Python 3.6. Simple script to deinterlace avchd.M2TS with about 1GB available, it takes about couple of seconds to freeze. If cache is limited, no problem.

file = r'C:\vid\avchd.M2TS'
from vapoursynth import core
import havsfunc as haf
#core.max_cache_size = 800 #this works 
clip = core.lsmas.LWLibavSource(file)
clip = haf.QTGMC(clip, Preset='Slow', TFF=True)
clip.set_output()

with about 1GB available

VapourSynth's cache will try to use more than that with high resolutions and complex scripts (QTGMC), at least until a few hundred frames have been processed. Then the cache figures out it doesn't need so much memory.

(The default max_cache_size is 4 GB on 64 bit systems and 1 GB on 32 bit systems.)

[_Al_]

thanks, did not realize that simple print(core.max_cache_size) could give it away, which indeed gives 4096

[Selur]

Shouldn't be needed when you have 'Denoiser="KNLMeansCL"' in the parameters, though, right? Or does it something else too?

OpenCL = True causes QTGMC to use NNEDI3CL instead of znedi3 (not sure atm. whether I did this modification myself in https://github.com/Selur/Vapoursynth...er/havsfunc.py or not )

[asarian]

Originally Posted by Selur

OpenCL = True causes QTGMC to use NNEDI3CL instead of znedi3 (not sure atm. whether I did this modification myself in https://github.com/Selur/Vapoursynth...er/havsfunc.py or not )

Every cycle shaved off is one.

[Dogway]

Is there a way to indicate Script path location? I'm using VS in portable mode but VSEdit can't find the modules. I tried placing vapoursynth_modules.pth and sitecustomize.py in root folder but still no luck.

[ChaosKing]

One way is to append a path in python in your script:

scriptPath = os.getcwd() + "/../scripts"
sys.path.append(os.path.abspath(scriptPath))

Usually you can place scripts here: C:\Python37\Lib\site-packages\vapoursynth

If you're using a python embedded zip you can set a path in python37._pth

[Dogway]

I'm unable to load them, my scripts are in "C:\Program Files (x86)\VapourSynth_R45(x64)\vapoursynth64\scripts"
so I write in python37._pth the following.

vapoursynth64\scripts
Scripts
Lib\site-packages
python37.zip
.

# Uncomment to run site.main() automatically
#import site

For the script I write:

import vapoursynth as vs
import havsfunc as haf

core = vs.get_core()

clip = core.ffms2.Source(r'D:\source.mp4')
clip = core.haf.SMDegrain(clip)
clip.set_output()

Edit: I'm the admin, so I have write access to Program Files.

Edit2: Ok, ok, it's clip = haf.SMDegrain(clip) without the core.

[ChaosKing]

And where is your python installation located? Have you tried with a full path?

[Dogway]

It's inside VapourSynth_R45(x64), embedded. It worked I just had to omit the core. namespace.
Too bad I didn't find your fatpack earlier, should be sticky. I will probably change to that once you update it to 2019 version.

I have a question though. When you load avisynth plugins or scripts into VapourSynth, will the performance be limited to that of avisynth or you get the benefits of better VS multithreading?

[ChaosKing]

Depends on the plugin. You will often see a msg with something like using slow method blah blah. Solution: use avsw.Eval() instead.

[asarian]

Odd. I have a source file of 624x354. But getting the following returned:

... line 3159, in MCTemporalDenoise
return core.std.Crop(smP, **crop_args)
File "src\cython\vapoursynth.pyx", line 1833, in vapoursynth.Function.__call__
vapoursynth.Error: Crop: cropped area needs to have mod 2 height offset

Wait, 354 is mod2, right?! Or am I going crazy?

[Myrsloik]

Originally Posted by asarian

Odd. I have a source file of 624x354. But getting the following returned:

... line 3159, in MCTemporalDenoise
return core.std.Crop(smP, **crop_args)
File "src\cython\vapoursynth.pyx", line 1833, in vapoursynth.Function.__call__
vapoursynth.Error: Crop: cropped area needs to have mod 2 height offset

Wait, 354 is mod2, right?! Or am I going crazy?

You have to add a print at the point where it happens. The internal clip that's cropped in MCTemporalDenoise isn't necessarily the same as the input.

[asarian]

Originally Posted by Myrsloik

You have to add a print at the point where it happens. The internal clip that's cropped in MCTemporalDenoise isn't necessarily the same as the input.

Didn't realize that.

Since this pertains to a pre-cropped area of a larger, 1080p source (so I can 'Oyster' it in several parts), I tried a 356 swatch, which didn't generate the error. So, I'll just go with that (it's mainly an overscan area anyway, as I only need a 346 height).

[_Al_]

Encoding using clip.output() right from vapoursynth script itself, where input video is ANY anamorphic video like HDV camcorder M2T video or NTSC DV or VOB gives jagged lines. As soon as there is resize involved , say to square pixel it is OK. If some odd resize is given , say 1000,600, again it might fail and jagged lines appear. That is not that important though because square pixel , resize to 16:9 (M2T video) or 4:3 (DVavi, VOB) seem to work. It encodes alright. Point is, without resize, just using SAR flag in encoder, gives jagged lines as presented in attachment.
Interlacing seem have nothing to do with it, using QTGMC or even assuming it as progressive using setFrameProp would not help. This behavior is also same using different source plugins.

import vapoursynth as vs
from vapoursynth import core
d2v_file  = r"path to file ..."
clip = core.d2v.Source(d2v_file)  #sourse was mpeg2 HDV camcorder video
import subprocess
import shutil
output = r'F:/Destination/test.264'

x264 = shutil.which('x264')
x264_cmd = [x264, '--frames', f'{len(clip)}',
             '--input-csp', 'i420',
             '--demuxer', 'raw',
             '--sar',  '4:3',
             '--input-depth', '8',
             '--input-res', f'{clip.width}x{clip.height}',
             '--fps', f'{clip.fps_num}/{clip.fps_den}',
             '--crf',    '18',
             '--colorprim', 'bt709',
             '--transfer', 'bt709',
             '--colormatrix', 'bt709',
             '--output', output,
             '-'] 
process = subprocess.Popen(x264_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
clip.output(process.stdin)
process.communicate()

for x265 the same:

x265_cmd = [x265, '--frames', f'{len(clip)}',
                 #'--input-csp', 'i420',
                 '--y4m',
                 '--input-depth', f'{clip.format.bits_per_sample}',
                 '--output-depth', f'{clip.format.bits_per_sample}',
                 '--input-res', f'{clip.width}x{clip.height}',
                 '--fps', f'{clip.fps_num}/{clip.fps_den}',
                 '--crf',  f'{str(crf)}',
                 '--output', output,
                 '-']

process = subprocess.Popen(x265_cmd,  stdin=subprocess.PIPE, stdout=subprocess.PIPE)
clip.output(process.stdin, y4m = True, progress_update=progressupdate)
process.communicate()

Previewing script is always ok, encoding within VSEditor also, just not directly using that output() function.

[_Al_]

I might found a culprit, but not sure what to do with it or how to fix it.

for example NTSC video there is 1 byte per sample, if I get_stride for all three planes:
f = clip.get_frame(0)
print(f.get_stride(0),f.get_stride(1),f.get_stride(2))
I get:
736 384 384 (expected: 720 360 360)
for HDV video from HDV tape camcorder (shooting 1440x1080) I get:
1440 736 736 (expected: 1440 720 720)

after resize I get even values, like resizing NTSC video to 640x480 I'd get:
640 320 320
or resizing that HDV video to 1920x1080, I'd get:
1920 960 960

Those are expected numbers and encoding is also ok, so this looks like a cause to offset the output if resizing is not done. If I resize to the same resolution values stay the same, it would not help.

Is there any way to get those values even without resizing? What are those data that are trailing (or preceding) each line in plane array? If it is not stride itself (wrong value) that offsets things.