Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 18th April 2018, 16:49   #3001  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Quote:
I have 16gb of ram.
+
Quote:
with core.max_cache_size = 32768
this seems strange,...
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 19th April 2018, 17:24   #3002  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
I did some tests on a Blu-ray source with my usual script to denoise and downsize to 720p, 2000 frames from the middle:

v44-test1, cache 32768:
Large pages - 10,74 fps
No large pages - 11,06 fps

v43
cache 32768 - 11,02 fps
default cache - 10,02 fps

So no use for me, but the cache size needs to be increased - so thanks for that tip I have 16 GB of memory and an AMD Ryzen 1800X system.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 21st April 2018, 08:10   #3003  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Still a bit confused about max_cache_size.
Quote:
max_cache_size

Set the upper framebuffer cache size after which memory is aggressively freed. The value is in megabytes.

set_max_cache_size(mb)

Deprecated, use max_cache_size instead.
so since max cache size is in MB and you have 16GB why set it to 32768 = 32GB, shouldn't your 16GB (- normal system memory usage) of RAM be the largest value one should set and not twice the available RAM or am I missing something?
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 25th April 2018, 16:34   #3004  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Quote:
Originally Posted by Selur View Post
Still a bit confused about max_cache_size.

so since max cache size is in MB and you have 16GB why set it to 32768 = 32GB, shouldn't your 16GB (- normal system memory usage) of RAM be the largest value one should set and not twice the available RAM or am I missing something?
It's a workaround for large pages in the current version only. If you aren't interested in large pages right now, or you're worried about a filter sucking up all of the cache, don't touch it.
foxyshadis is offline   Reply With Quote
Old 25th April 2018, 16:52   #3005  |  Link
DJATOM
Registered User
 
DJATOM's Avatar
 
Join Date: Sep 2010
Location: Ukraine, Bohuslav
Posts: 377
I found a bug. Well, not sure if that's really a bug, but might be unwanted behaviour of resizer.
Nowadays almost all anime blu-rays are upscales. I'd like to fight such upscales downscaling to original resolution using the mixed Descale and Spline36 clip, upscale back with eedi3 and merge it with source clip using lines mask.
Today my friend noticed slight shift in the source/AA comparisons. I tried to debug my script and found that resizer is just copying clip instead of fixing eedi3 shift. In that case 1920x1080 clip was downscaled to 960x540 and upscaled back (so we got 1920x1080 after eedi3 stuff).
As I understand, vapoursynth (zimg) resizer is just copying clip if input width and height matches with output. That might be unwanted if cropping options is set.
On my opinion resizer should check for crop options and do proper resizing in that case.
__________________
Me on GitHub
PC Specs: Ryzen 5950X, 64 GB RAM, RTX 2070
DJATOM is offline   Reply With Quote
Old 25th April 2018, 17:43   #3006  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
@foxyshadis: I get what max_cache_size is meant to do, what I don't get is why setting it to two times your RAM size is the right thing.
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 25th April 2018, 18:25   #3007  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Quote:
Originally Posted by Selur View Post
@foxyshadis: I get what max_cache_size is meant to do, what I don't get is why setting it to two times your RAM size is the right thing.
It's really not. It can easily eat all the RAM you have when the script is initialized and it takes quite a lot of time for VS to release the excess memory so your computer is pretty much frozen. I've settled down for 10240 on my 16 GB system as I can just manage that with BD to 720p encodes.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 26th April 2018, 10:00   #3008  |  Link
DJATOM
Registered User
 
DJATOM's Avatar
 
Join Date: Sep 2010
Location: Ukraine, Bohuslav
Posts: 377
Quote:
Originally Posted by HolyWu View Post
You probably need to provide an script to demonstrate what you really do, if you want the developers to resolve your doubts.
Okay. Script below is just stripped version to reproduce my problem.
PHP Code:
import havsfunc as haf
import mvsfunc 
as mvf
import descale 
as dsc

clip 
= <some 1920x1080 clip>

descale_str 0.32
kernel 
'bicubic'
descale_h 540 # 1080/2 to reproduce the problem

clip.width
clip.height
descale_w 
haf.m4((descale_h) / h)
descale_natural dsc.Descale(clipdescale_wdescale_hkernel=kernel)
descale_aa core.resize.Spline36(clipdescale_wdescale_h)
descaled core.std.Merge(clipb=descale_naturalclipa=descale_aaweight=descale_str)

dx w
dy 
h
ux 
descale_w 2
uy 
descale_h 2
rescale 
core.eedi3m.EEDI3(descaledfield=1dh=Truealpha=0.2beta=0.25gamma=1000.0vcheck=3sclip=core.znedi3.nnedi3(descaledfield=1dh=Truensize=0nns=4pscrn=1))
rescale core.std.Transpose(clip=rescale)
rescale core.eedi3m.EEDI3(rescalefield=1dh=Truealpha=0.2beta=0.25gamma=1000.0vcheck=3sclip=core.znedi3.nnedi3(rescalefield=1dh=Truensize=0nns=4pscrn=1))
rescale core.std.Transpose(clip=rescale)
#rescale = core.fmtc.resample(rescale, dx, dy, -0.5, -0.5, ux, uy, kernel='spline36').fmtc.bitdepth(bits=8) # always fixes eedi3 shift, but I need to call bitdepth function, that's not convenient.
rescale core.resize.Spline36(rescaledxdysrc_left=-0.5src_top=-0.5src_width=uxsrc_height=uy# leaves eedi3 shift untouched if rescale clip's width and height matches the source. 
__________________
Me on GitHub
PC Specs: Ryzen 5950X, 64 GB RAM, RTX 2070
DJATOM is offline   Reply With Quote
Old 26th April 2018, 11:16   #3009  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
You could also use edi_rpow2.py https://gist.github.com/YamashitaRen...7524e794779d9c (or just copy the shifting code)
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline   Reply With Quote
Old 26th April 2018, 11:50   #3010  |  Link
DJATOM
Registered User
 
DJATOM's Avatar
 
Join Date: Sep 2010
Location: Ukraine, Bohuslav
Posts: 377
Quote:
Originally Posted by ChaosKing View Post
You could also use edi_rpow2.py https://gist.github.com/YamashitaRen...7524e794779d9c (or just copy the shifting code)
zimg way is most likely broken, I didn't checked.
PHP Code:
Y=core.std.ShufflePlanes(clips=clipplanes=0colorfamily=vs.GRAY)
Y=core.resize.Spline36(clip=Y,width=clip.width,height=clip.height,src_left=hshift,src_top=-0.5
clip.width = Y.width, clip.height = Y.height, so resizer is just copying whole plane without doing the job. Another planes should be fixed if colorspace is not YUV444.
__________________
Me on GitHub
PC Specs: Ryzen 5950X, 64 GB RAM, RTX 2070
DJATOM is offline   Reply With Quote
Old 28th April 2018, 02:49   #3011  |  Link
hydra3333
Registered User
 
Join Date: Oct 2009
Location: crow-land
Posts: 540
Yay, vapoursynth ffmpeg integration seems to be in its way.

http://ffmpeg.org/pipermail/ffmpeg-d...il/229137.html

I also hold some hope for "for dummies" vapoursynth library build instructions leading to building of a static ffmpeg.exe (including python libs and any other dependencies) ...
hydra3333 is offline   Reply With Quote
Old 28th April 2018, 06:57   #3012  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Quote:
... leading to building of a static ffmpeg.exe (including python libs and any other dependencies) ...
I highly doubt that will happen, since you would probably need to include a whole Python portable environment (or at least a huge portion of it).
To open .vpy-scripts you at least some sort of a Python (runtime) environment.


Cu Selur

Ps.: To open .avs-scripts you need the avisynth.dll (and dependencies).
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 28th April 2018, 07:05   #3013  |  Link
hydra3333
Registered User
 
Join Date: Oct 2009
Location: crow-land
Posts: 540
Ah. I'd hoped, since youtube-dl manages to do it resulting in a standalone .exe ...

For some, unless maybe it could use vapoursynth portable in the same folder, non-static would be as useful as teats on a bull

Last edited by hydra3333; 28th April 2018 at 07:07.
hydra3333 is offline   Reply With Quote
Old 28th April 2018, 07:26   #3014  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Quote:
Ah. I'd hoped, since youtube-dl manages to do it resulting in a standalone .exe ...
youtube-dl isn't statically compiled it seems to wrap all the dependencies it requires.
Yes once could also wrap a whole hdd this way and create one large binary, but that isn't statically compiling that is basically the same thing Mac does with it's .app files.
(Provide mostly standalone packages which are basically a folder structure which contains dependencies&co.)

Cu Selur
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 30th April 2018, 00:36   #3015  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
Is it possible to compare the value of f.props.PlaneStatsAverage in a FrameEval function with the previous and the next frame (n-1 and n+1)? Sould I use get_frame_async() for that?

Code:
def g(n, f, clip, d_clip):
	clip=clip.text.Text("current: "+ f.props.PlaneStatsAverage)
	clip=clip.text.Text("\nPrev: "+ ???)
	clip=clip.text.Text("\n\nNext: "+ ???)
		
	return clip
clip = clip.std.FrameEval(functools.partial(g, d_clip=d_clip, clip=clip), prop_src=x)
Edit
This works but feels more like a hack.
Code:
x= clip.std.PlaneStats()
x1= clip.std.PlaneStats().std.DuplicateFrames(0) #prev
x2= clip.std.PlaneStats().std.Trim(1) #next

clip = clip.std.FrameEval(functools.partial(g, d_clip=d_clip, clip=clip), prop_src=[x, x1, x2])
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database

Last edited by ChaosKing; 30th April 2018 at 01:11.
ChaosKing is offline   Reply With Quote
Old 1st May 2018, 22:39   #3016  |  Link
amayra
Quality Checker
 
amayra's Avatar
 
Join Date: Aug 2013
Posts: 284
Quote:
Python development mode
A new command-line switch for the Python interpreter, -X, lets the developer set a number of low-level options for the interpreter. With Python 3.7, the option -X dev enables “development mode,” a slew of runtime checks that normally have a big impact on performance, but are useful for a developer during the debugging process
as vapoursynth user is this benefits me ?
__________________
I love Doom9
amayra is offline   Reply With Quote
Old 1st May 2018, 22:44   #3017  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by ChaosKing View Post
Is it possible to compare the value of f.props.PlaneStatsAverage in a FrameEval function with the previous and the next frame (n-1 and n+1)? Sould I use get_frame_async() for that?

Code:
def g(n, f, clip, d_clip):
	clip=clip.text.Text("current: "+ f.props.PlaneStatsAverage)
	clip=clip.text.Text("\nPrev: "+ ???)
	clip=clip.text.Text("\n\nNext: "+ ???)
		
	return clip
clip = clip.std.FrameEval(functools.partial(g, d_clip=d_clip, clip=clip), prop_src=x)
Edit
This works but feels more like a hack.
Code:
x= clip.std.PlaneStats()
x1= clip.std.PlaneStats().std.DuplicateFrames(0) #prev
x2= clip.std.PlaneStats().std.Trim(1) #next

clip = clip.std.FrameEval(functools.partial(g, d_clip=d_clip, clip=clip), prop_src=[x, x1, x2])
That's the right way.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 1st May 2018, 23:03   #3018  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
@ amayra:

Users will probably not benefit directly from debugging switches, because users will usually prefer the fastest possible execution. You may have read that these additional runtime checks will "have a big impact on performance" (see your quote), which means that it will run a lot slower and report a lot of warnings, even of less probable conditions.

But if a developer can use this mode to discover risky code and fix bugs (possibly even before they cause noticeable issues), then users may benefit indirectly from an advanced release build.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 2nd May 2018, 14:16   #3019  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
@Myrsloik
Ok, thx.
I found a strange behaviour while playing with FrameEval func.

The very same function gives different output if it's called within FrameEval.
Is it a bug or did I miss something?
Here's an animated gif of my problem (correct output is in the middle):


And my script (You'll need also Single Precision MVTools https://github.com/IFeelBloated/vapo...h-mvtools-sf):
Code:
import sys
import os

import functools
import vapoursynth as vs

import mvsfunc as mvf #https://github.com/HomeOfVapourSynthEvolution/mvsfunc/blob/master/mvsfunc.py
import havsfunc as haf
import mvmulti as mv #https://github.com/IFeelBloated/vapoursynth-mvtools-sf/blob/master/src/mvmulti.py
import fvsfunc as fvf #https://github.com/Irrational-Encoding-Wizardry/fvsfunc/blob/master/fvsfunc.py

core = vs.get_core()

def myFilter(clip):
	pre = haf.SMDegrain(clip, tr=2, pel=2, contrasharp=False, thSAD = 1200, thSADC = 1200, prefilter=2)
	pre = pre.flux.SmoothT(temporal_threshold=6)
	pre = haf.DitherLumaRebuild(pre, s0=1)
	pre = fvf.Depth(pre, 32, range_in=1)
	
	clip = mvf.Depth(clip, 32)
	
	super = core.mvsf.Super(pre,16,16,1,0)
	vectors = mv.Analyze(super, blksize=16, overlap=8, search=4, tr=6)
	super = core.mvsf.Super(clip,16,16,1,1)
	blur_clip = mv.DegrainN(clip, super, vectors, thsad=600, tr=6)
	
	grain = blur_clip.grain.Add(var=1200.0, constant=True)#.std.BoxBlur(hradius=1, hpasses=1, vradius=1, vpasses=1)
	#return grain
	diff_clip = core.std.Expr([clip, blur_clip], 'x y - abs').std.Inflate(threshold=200/255).std.Inflate(threshold=200/255)
	mask_clip = diff_clip.std.Binarize(threshold=[3.3/219, 3.3/224], v0=0, v1=80/255)
		
	clip = core.std.MaskedMerge(clipa=blur_clip, clipb=grain, mask=mask_clip)
	
	return mvf.Depth(clip, 8)
	
	

clip = core.std.BlankClip(format=vs.YUV420P8, width=120*2, height=80*2, length=100, color=[206,235,135])
g_static  = clip.grain.Add(var=100.0, constant=True)
g_dynamic = clip.grain.Add(var=100.0, constant=False)


clip = core.std.StackHorizontal([
	core.std.StackVertical([g_static, g_static]), 
	core.std.StackVertical([g_static, g_dynamic])
	])
goal = core.std.StackHorizontal([
	core.std.StackVertical([g_static, g_static]), 
	core.std.StackVertical([g_static, g_static])
	])
orig=clip




def asd(n, c):
	return myFilter(c)
	
feval = clip.std.FrameEval(functools.partial(asd, c=clip))

def comp(a, b, crop=0):
	return core.std.StackHorizontal([
		core.std.CropRel(a, crop,crop,0,0), \
		core.std.CropRel(b, crop,crop,0,0), \
		])
		
nofeval = myFilter(clip)

clip = comp(feval.text.Text("Function inside FrameEval").std.AddBorders(right=2), nofeval.text.Text("No FrameEval").std.AddBorders(right=2), crop=0)
clip = comp(clip, orig.text.Text("unfiltered"), crop=0)

clip.set_output()
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database

Last edited by ChaosKing; 2nd May 2018 at 14:22.
ChaosKing is offline   Reply With Quote
Old 2nd May 2018, 14:22   #3020  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by ChaosKing View Post
@Myrsloik
Ok, thx.
I found a strange behaviour while playing with FrameEval func.

The same function gives different output if it's called within FrameEval.
Is it a bug or did I miss something?
Here's a animated gif of my problem (correct output is in the middle):


And my script (You'll need also Single Precision MVTools https://github.com/IFeelBloated/vapo...h-mvtools-sf):
...
Should be identical in theory, yes. Unless some filter stores temporal state incorrectly. Then you get this. Gradually remove each filter and see when the output is once again identical, then report which filter it was. I'm going to bet mvtools or fluxsmooth right away.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Reply

Tags
speed, vaporware, vapoursynth

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:14.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.