Vapoursynth

[Myrsloik]

I released R31 (http://www.vapoursynth.com/2016/02/r31-boring-maintenance/)!

It mostly just fixes the that some format conversions were erroneously rejected. Changelog in the first post and all that...

[Boulder]

Regarding the performance related changes made in R30 affecting source filters, is it just that the performance improvements require new versions of source filters or will they stop working altogether (or have some weird behaviour)?

[Myrsloik]

Regarding the performance related changes made in R30 affecting source filters, is it just that the performance improvements require new versions of source filters or will they stop working altogether (or have some weird behaviour)?

All old filters will continue working exactly the way they always have. The only change is that filters now can request that a different cache strategy is used that minimizes the number of "out of order" frame requests.

[speedyrazor]

To separate out vspipe and ffmpeg's output you need to create separate subprocess objects and pipe them together. Something like (Untested):


#!python3
import threading
from subprocess import Popen, PIPE, STDOUT

vspipe_cmd = ['VSPipe', '--progress', '--y4m', 'vpyScript.vpy', '-']
ffmpeg_cmd = ['ffmpeg', '-f', 'yuv4mpegpipe', '-i', '-', '-c:v', 'prores', '-an', '-y', 'outputMovie.mov']
vspipe_process = Popen(vspipe_cmd, stdout=PIPE, stderr=PIPE)
ffmpeg_process = Popen(ffmpeg_cmd, stdin=vspipe_cmd.stdout, stderr=PIPE)
vspipe_process.stdout.close() #Allow vspipe to receive a SIGPIPE if ffmpeg exists

def output_stderr(process, name):
for line in process.stderr:
print(name, ":", line)

vspipe_output = threading.Thread(target=output_stderr, args=(vspipe_process, "VSPIPE"))
ffmpeg_output = threading.Thread(target=output_stderr, args=(ffmpeg_process, "FFMPEG"))

vspipe_output.start()
ffmpeg_output.start()


Thanks for the code, it works well, with a small tweak. One question though is do you know how I would send a SIGINT signal to terminate this process?

[splinter98]

Thanks for the code, it works well, with a small tweak. One question though is do you know how I would send a SIGINT signal to terminate this process?

Try:


from signal import SIGINT

vspipe_process.send_signal(SIGINT)


or if you just want to kill the process you can also use one of the following:


#Both of these are the same on Windows
vspipe_process.terminate() #Sends SIGTERM on POSIX
vspipe_process.kill() #Sends SIGKILL on POSIX

[speedyrazor]

To separate out vspipe and ffmpeg's output you need to create separate subprocess objects and pipe them together. Something like (Untested):


#!python3
import threading
from subprocess import Popen, PIPE, STDOUT

vspipe_cmd = ['VSPipe', '--progress', '--y4m', 'vpyScript.vpy', '-']
ffmpeg_cmd = ['ffmpeg', '-f', 'yuv4mpegpipe', '-i', '-', '-c:v', 'prores', '-an', '-y', 'outputMovie.mov']
vspipe_process = Popen(vspipe_cmd, stdout=PIPE, stderr=PIPE)
ffmpeg_process = Popen(ffmpeg_cmd, stdin=vspipe.stdout, stderr=PIPE)
vspipe_process.stdout.close() #Allow vspipe to receive a SIGPIPE if ffmpeg exists

def output_stderr(process, name):
for line in process.stderr:
print(name, ":", line)

vspipe_output = threading.Thread(target=output_stderr, args=(vspipe_process, "VSPIPE"))
ffmpeg_output = threading.Thread(target=output_stderr, args=(ffmpeg_process, "FFMPEG"))

vspipe_output.start()
ffmpeg_output.start()


This works, but it doesn't print in realtime on my Windows 7 system, the sdterr is blocked whilst the transcode is running. Is there anyway to get the VSPipe stderr out in realtime whilst it happening?

[LigH]

No clue about python; but other programming languages may need an output flush, possibly even a cooperative cycle to process messages.

[LoRd_MuldeR]

No clue about python; but other programming languages may need an output flush, possibly even a cooperative cycle to process messages.

That's right. However, the explicit fflush() is required inside the child process that writes something to stdout/stderr, not in the parent process that tries to read from the child's stdout/stderr.

So, this might be a problem in VSPipe - which is written in C++, by the way:
https://github.com/vapoursynth/vapoursynth/blob/master/src/vspipe/vspipe.cpp

If the child process is missing the required fflush()'s, there is nothing you can do on the parent's side! You have to fix the child process. And, indeed, after a quick look, it seems like VSPipe does not explicitly flush its stderr after the fprintf()'s.

(AFAIK, this is a Windows-only quirk)

[speedyrazor]

That's right. However, the explicit fflush() is required inside the child process that writes something to stdout/stderr, not in the parent process that tries to read from the child's stdout/stderr.

So, this might be a problem in VSPipe - which is written in C++, by the way:
https://github.com/vapoursynth/vapoursynth/blob/master/src/vspipe/vspipe.cpp

If the child process is missing the required fflush()'s, there is nothing you can do on the parent's side! You have to fix the child process. And, indeed, after a quick look, it seems like VSPipe does not explicitly flush its stderr after the fprintf()'s.

(AFAIK, this is a Windows-only quirk)

Thanks for looking into this. In that case, is there a way to get VSPipe to write to a 'realtime' log, I can then read from that?

[speedyrazor]

Thanks for looking into this. In that case, is there a way to get VSPipe to write to a 'realtime' log, I can then read from that?

What I am trying to do is get the sdterr out from VSPipe in realtime so I can see when L-SMASH source gives me an error (when encountering a bad file), currently it just continues to process and ffmpeg encodes a stuck frame for the duration of the file.

[Myrsloik]

Here's a vspipe you can test. (https://dl.dropboxusercontent.com/u/73468194/VSPipe64_fflush.exe) It calls fflush on stderr after every frame.

[Selur]

Is there something similar to 'LoadDll(...)' in Vapoursynth.
Using the portable version and for example DFTTest I have to put libfftw3f-3 next to the DFTTest.dll, but I would prefer to keep it inside a separate folder and load it explicitly, is something like that possible?

[Myrsloik]

I really don't recommend doing this. I think it's very wrong.

But calling the windows api loadlibrary using ctypes directly from python should do what you want.

[Selur]

Can you be a bit more specific?
atm. my script looks like this:
# Imports
import vapoursynth as vs
import sys
core = vs.get_core()
# Loading Plugins
core.std.LoadPlugin(path="G:/Hybrid/Vapoursynth/vapoursynth64/plugins/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="G:/Hybrid/Vapoursynth/vapoursynth64/plugins/SourceFilter/FFMS2/ffms2.dll")
# Loading Source: F:/TestClips&Co/test.avi
clip = core.ffms2.Source(source="F:/TESTCL~1/test.avi",cachefile="H:/Temp/avi_0197468a3716844ade54f3f30f60eeda_491.ffindex",fpsnum=25)
# Denoising
clip = core.dfttest.DFTTest(clip=clip)
# Output
clip.output(sys.stdout, y4m=1)
and I would like to place libfftw3f-3 at
G:/Hybrid/Vapoursynth/vapoursynth64/plugins/libfftw/libfftw3f-3.dll
looking at https://docs.python.org/2/library/ctypes.html#module-ctypes I don't see how to load the dll in a way that Vapoursynth can use it for in example DFTTest or other filters.

[Myrsloik]

The key here is that if a library with a certain filename, regardless of path, has been loaded into a process then loading the dll again without specifying an absolute path will simply return a handle to the already loaded copy.

This is of course advanced windows and insane python but I suspect this is the same thing loaddll does.

This actually has nothing to do with vapoursynth. You're just messing with the dll loading of the process.

My guess. Don't forget to assign it to a global variable to ensure it stays loaded. I guess.

import ctypes
Dllref = ctypes.windll.LoadLibrary(path)

[Selur]

Will do. Thanks! :)

[Selur]

std.Levels(clip clip[, int min_in=0, int max_in, float gamma=1.0, int min_out=0, int max_out, int[] planes=[0, 1, 2]])
source: http://www.vapoursynth.com/doc/functions/levels.html

What are the defaults for max_in/max_out or are there really none? 255? (seems strange to not have no defaults for parameters that are followed by parameters which have defaults)

[jackoneill]

source: http://www.vapoursynth.com/doc/functions/levels.html

What are the defaults for max_in/max_out or are there really none? 255? (seems strange to not have no defaults for parameters that are followed by parameters which have defaults)


The default value of max_in and max_out is the format’s maximum allowed value.


Source: http://www.vapoursynth.com/doc/functions/levels.html

[Selur]

Missed that, but still unsure what the 'the format’s maximum allowed value' is,...

[feisty2]

Missed that, but still unsure what the 'the format’s maximum allowed value' is,...

255 uint8_t
65535 uint16_t
1.0 float

[Selur]

Thanks!

[mawen1250]

There's actually no limitation for float format.
As for the conversion between float and 8-bit int:
0.0~1.0 corresponds to 16~235/0~255 for Y/R/G/B.
-0.5~0.5 corresponds to 16~240/0.5~255.5 for U/V.

[jackoneill]

There's actually no limitation for float format.
As for the conversion between float and 8-bit int:
0.0~1.0 corresponds to 16~235/0~255 for Y/R/G/B.
-0.5~0.5 corresponds to 16~240/0.5~255.5 for U/V.

How are you going to have 0.5 and 255.5 in 8 bit int?

[mawen1250]

How are you going to have 0.5 and 255.5 in 8 bit int?
You won't have it as a single pixel value though.
It's the definition of full range UV where 128±(255/2) corresponds to 0±0.5.

[2-perf]

Hi,

I'm a total newbie to video encoding. A friend at work heard me saying that I was using Handbrake to encode some Quicktime files to Blu-ray and told me I should be using a frame server... WTF? I replied to him!

Anyway he came to my place, set me up with:

- python-3.5.1-amd64.exe
- ffms2-2.22-msvc (I'm using the x64 flavor)
- vapoursynth-r29.exe (x64 flavor I guess)
- x264_launcher.2016-02-06.exe

He wrote me a VapourSynth script whish looks like this:

import vapoursynth as vs

# get the core instance
core = vs.get_core()

# load a native vapoursynth plugin, you can also use the auto-loading
# you should use absolute paths as the working directory may not be what you think it is
core.std.LoadPlugin(r'S:\Dropbox\softwares\portableapps\codec_encoder\ffms2-2.22-msvc\x64\ffms2.dll')

# open a video file; ret is now a clip object

path = r'D:\restauration\projet\tropico\deliveries\master\tropico.mov'

ret = core.ffms2.Source( source = path )

# flip the video a bit
#ret = core.std.Transpose( ret )

# set the clip to be output
ret.set_output()

The things is it fails 9 times out of 10 when trying to start the second pass.

Here's the .log (Well, I hope). Sorry I'm a digital colorist not a computer geek. :p

Simple x264 Launcher (Build #1012), built 2016-02-06

Job started at 2016-02-18, 22:19:28.

Source file : D:\encodage\projet\tropico\script.vpy
Output file : D:\encodage\projet\tropico\output\script.264

--- SYSTEMINFO ---

Binary Path : C:\Program Files (x86)\MuldeR\Simple x264 Launcher v2
Avisynth : No
VapourSynth : Yes

--- SETTINGS ---

Encoder : x264 (H.264/AVC), x64, 8-Bit
Source : VapourSynth (vpy)
RC Mode : 2-Pass
Preset : slower
Tuning : Film
Profile : High
Custom : --level 4.1 --ref 4 --subme 10 --psy-rd 1.00:0.15 --merange 24 --deadzone-inter 21 --deadzone-intra 11 --no-fast-pskip --cqm flat --chroma-qp-offset 3 --threads 12 --lookahead-threads 1 --slices 4 --no-dct-decimate --bluray-compat --vbv-maxrate 38000 --vbv-bufsize 30000 --bframes 3 --b-pyramid 1 --weightb --open-gop --weightp 1 --keyint 24 --min-keyint 1 --rc-lookahead 24 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --qcomp 0.00 --qpmin 0

--- CHECK VERSION ---

Detect video encoder version:

Creating process:
"C:\Program Files (x86)\MuldeR\Simple x264 Launcher v2\toolset\x64\x264_8bit_x64.exe" --version

x264 0.148.2665 a01e339
(libswscale 4.0.100)
(libavformat 57.21.101)
(ffmpegsource 2.22.0.1)
built by Komisar on Jan 18 2016, gcc: 4.8.4 (multilib.generic.Komisar)
x264 configuration: --bit-depth=8 --chroma-format=all
libx264 configuration: --bit-depth=8 --chroma-format=all
x264 license: GPL version 2 or later
libswscale/libavformat/ffmpegsource license: GPL version 2 or later

Detect video source version:

Creating process:
"C:\Program Files (x86)\VapourSynth\core64\vspipe.exe" --version

VapourSynth Video Processing Library
Copyright (c) 2012-2015 Fredrik Mellbin
Core R29
API R3.2
Options: -

> x264 revision: 2665 (core #148)
>
VapourSynth version: r29 (API r3)

--- GET SOURCE INFO ---

Creating process:
"C:\Program Files (x86)\VapourSynth\core64\vspipe.exe" --info D:\encodage\projet\tropico\script.vpy -

Width: 1920
Height: 1080
Frames: 7491
FPS: 24000/1001 (23.976 fps)
Format Name: YUV422P10
Color Family: YUV
Bits: 10
SubSampling W: 1
SubSampling H: 0

Resolution: 1920x1080
Frame Rate: 24000/1001
No. Frames: 7491

--- ENCODING PASS #1 ---

Creating input process:
"C:\Program Files (x86)\VapourSynth\core64\vspipe.exe" --y4m D:\encodage\projet\tropico\script.vpy -

Creating encoder process:
"C:\Program Files (x86)\MuldeR\Simple x264 Launcher v2\toolset\x64\x264_8bit_x64.exe" --bitrate 35000 --pass 1 --stats D:\encodage\projet\tropico\output\script.stats --preset slower --tune film --profile high --level 4.1 --ref 4 --subme 10 --psy-rd 1.00:0.15 --merange 24 --deadzone-inter 21 --deadzone-intra 11 --no-fast-pskip --cqm flat --chroma-qp-offset 3 --threads 12 --lookahead-threads 1 --slices 4 --no-dct-decimate --bluray-compat --vbv-maxrate 38000 --vbv-bufsize 30000 --bframes 3 --b-pyramid 1 --weightb --open-gop --weightp 1 --keyint 24 --min-keyint 1 --rc-lookahead 24 --colorprim bt709 --transfer bt709 --colormatrix bt709 --qcomp 0.00 --qpmin 0 --output D:\encodage\projet\tropico\output\script.264 --frames 7491 --demuxer y4m --stdin y4m -

y4m [info]: 1920x1080p 0:0 @ 24000/1001 fps (cfr)
resize [warning]: converting from yuv422p16le to yuv420p16le
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x264 [info]: profile Main, level 4.1
x264 [info]: frame I:339 Avg QP:10.89 size:467818
x264 [info]: frame P:2145 Avg QP:13.96 size:277862
x264 [info]: frame B:5007 Avg QP:18.17 size:120544
x264 [info]: consecutive B-frames: 5.1% 5.4% 35.4% 54.0%
x264 [info]: mb I I16..4: 41.5% 0.0% 58.5%
x264 [info]: mb P I16..4: 43.0% 0.0% 0.0% P16..4: 35.5% 0.0% 0.0% 0.0% 0.0% skip:21.5%
x264 [info]: mb B I16..4: 12.5% 0.0% 0.0% B16..8: 26.5% 0.0% 0.0% direct:26.8% skip:34.2% L0:10.4% L1:14.9% BI:74.8%
x264 [info]: direct mvs spatial:99.7% temporal:0.3%
x264 [info]: coded y,uvDC,uvAC intra: 94.9% 39.3% 19.9% inter: 47.8% 11.9% 5.6%
x264 [info]: i16 v,h,dc,p: 14% 9% 65% 12%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 12% 9% 20% 10% 11% 11% 9% 10% 10%
x264 [info]: i8c dc,h,v,p: 65% 16% 16% 4%
x264 [info]: Weighted P-Frames: Y:3.0% UV:0.8%
x264 [info]: kb/s:34776.07
encoded 7491 frames, 25.61 fps, 34776.07 kb/s
vpyp [info]: Output 7491 frames in 292.32 seconds (25.63 fps)
Final file size is 1.26 GB bytes.

--- ENCODING PASS #2 ---

Creating input process:
"C:\Program Files (x86)\VapourSynth\core64\vspipe.exe" --y4m D:\encodage\projet\tropico\script.vpy -

Creating encoder process:
"C:\Program Files (x86)\MuldeR\Simple x264 Launcher v2\toolset\x64\x264_8bit_x64.exe" --bitrate 35000 --pass 2 --stats D:\encodage\projet\tropico\output\script.stats --preset slower --tune film --profile high --level 4.1 --ref 4 --subme 10 --psy-rd 1.00:0.15 --merange 24 --deadzone-inter 21 --deadzone-intra 11 --no-fast-pskip --cqm flat --chroma-qp-offset 3 --threads 12 --lookahead-threads 1 --slices 4 --no-dct-decimate --bluray-compat --vbv-maxrate 38000 --vbv-bufsize 30000 --bframes 3 --b-pyramid 1 --weightb --open-gop --weightp 1 --keyint 24 --min-keyint 1 --rc-lookahead 24 --colorprim bt709 --transfer bt709 --colormatrix bt709 --qcomp 0.00 --qpmin 0 --output D:\encodage\projet\tropico\output\script.264 --frames 7491 --demuxer y4m --stdin y4m -

y4m [error]: bad sequence header magic
x264 [error]: could not open input file `-'

WARNING: Input process exited with error (code: -1073741819), your encode might be *incomplete* !!!

IMPORTANT: The Vapoursynth process terminated abnormally. This means Vapoursynth or one of your Vapoursynth-Plugin's just crashed.

PROCESS EXITED WITH ERROR CODE: -1

Thanks for your patience.

[jackoneill]

Hi,

I'm a total newbie to video encoding. A friend at work heard me saying that I was using Handbrake to encode some Quicktime files to Blu-ray and told me I should be using a frame server... WTF? I replied to him!

Anyway he came to my place, set me up with:

- python-3.5.1-amd64.exe
- ffms2-2.22-msvc (I'm using the x64 flavor)
- vapoursynth-r29.exe (x64 flavor I guess)
- x264_launcher.2016-02-06.exe

He wrote me a VapourSynth script whish looks like this:

...

The things is it fails 9 times out of 10 when trying to start the second pass.

Here's the .log (Well, I hope). Sorry I'm a digital colorist not a computer geek. :p

...

Thanks for your patience.

If that's all you do in the VapourSynth script you don't need VapourSynth at all. Your copy of x264 can probably open tropico.mov directly (depends on how it was compiled).

Anyhow, the log you provided doesn't contain anything to indicate why vspipe fails. You should run vspipe directly a few times, until it prints something other than "Output N frames in M seconds": "vspipe.exe script.vpy NUL"

[stax76]

I have a problem updating StaxRip to latest VS and vslsmashsource, code below works using ffms2 but not using vslsmashsource, with old versions it has worked fine.


import vapoursynth as vs
core = vs.get_core()
core.std.LoadPlugin(r'D:\Projekte\GitHub\staxrip\bin\Apps\Plugins\vs\vslsmashsource\vslsmashsource.dll')
clip = core.lsmas.LibavSMASHSource(source = r'D:\Temp\Video\Clips\test.mp4')
clip = clip.resize.Bicubic(format=vs.COMPATBGR32)
clip.set_output()

[Myrsloik]

I have a problem updating StaxRip to latest VS and vslsmashsource, code below works using ffms2 but not using vslsmashsource, with old versions it has worked fine.


import vapoursynth as vs
core = vs.get_core()
core.std.LoadPlugin(r'D:\Projekte\GitHub\staxrip\bin\Apps\Plugins\vs\vslsmashsource\vslsmashsource.dll')
clip = core.lsmas.LibavSMASHSource(source = r'D:\Temp\Video\Clips\test.mp4')
clip = clip.resize.Bicubic(format=vs.COMPATBGR32)
clip.set_output()


Now describe how it doesn't work

[stax76]

that was quick :), returns blackness it seems, culprit is 'format=vs.COMPATBGR32', without that it's fine but StaxRip uses VFW...

[stax76]

vslsmashsource version I use is x64 r875

http://www.dropbox.com/sh/3i81ttxf028m1eh/AAABkQn4Y5w1k-toVhYLasmwa?dl=0

[Myrsloik]

vslsmashsource version I use is x64 r875

http://www.dropbox.com/sh/3i81ttxf028m1eh/AAABkQn4Y5w1k-toVhYLasmwa?dl=0

What if you update VS but not lsmashsource? That should work properly too.

[stax76]

What if you update VS but not lsmashsource? That should work properly too.

neither works with VS R31

[stax76]

Can it be fixed or is there a workaround?

[Myrsloik]

Can it be fixed or is there a workaround?

Your script works perfectly here when testing in 64bit versions of things. Unless your test.mp4 is magical or something I don't see any problem.

[stax76]

Your script works perfectly here when testing in 64bit versions of things. Unless your test.mp4 is magical or something I don't see any problem.

I've tried also LWLibavSource and another mp4 and one vob, always blackness, which OS are you using?

[Myrsloik]

I've tried also LWLibavSource and another mp4 and one vob, always blackness, which OS are you using?

Win10 with all updates. Can you cut a piece of one of the failing files?

[stax76]

really strange, can you try to open the preview window of the latest StaxRip build? Here it's totally black.

latest build: http://1drv.ms/1OqPDOe

[Myrsloik]

really strange, can you try to open the preview window of the latest StaxRip build? Here it's totally black.

latest build: http://1drv.ms/1OqPDOe

The preview works just as expected but I did notice some odd things:

Says python is version 3.5.0 when it's 3.5.1. Should check for msvcp120.dll and not msvcr120.dll for symmetry and to be more certain all of the runtime is actually installed. (some applications only install the C runtime and not the C++ part)

I have to install avisynth+ BEFORE I can get far enough to select to preview with lsmash in vapoursynth. Makes no sense to me.

[stax76]

Thanks for trying and reporting. I updated to Python 3.5.1 and msvcp120.dll. Regarding VS integration the hard part is done, a few small issues are left to do. I've no idea about the blackness issue at the moment but I keep looking.

[jackoneill]

Maybe put core.text.FrameNum(clip) before you convert to RGB, to see if the blackness comes from the source filter or somewhere else.


clip = clip.resize.Bicubic(format=vs.COMPATBGR32)

I hope you're passing a clip there, in your local copy of the script.

[stax76]

With adding core.text.FrameNum it shows blackness without framenumbers.


import vapoursynth as vs
core = vs.get_core()
core.std.LoadPlugin(r'D:\Projekte\GitHub\staxrip\bin\Apps\Plugins\vs\vslsmashsource\vslsmashsource.dll')
clip = core.lsmas.LibavSMASHSource(source = r'D:\Temp\Video\Clips\test.mp4')
clip = core.text.FrameNum(clip)
clip = clip.resize.Bicubic(format=vs.COMPATBGR32)
clip.set_output()


I'll ask StaxRip users to provide log files to narrow it down, if all fails I have to get rid of VFW which will cost 1-2 days.

[LigH]

If you export the resulting clip into a video file, instead of trying to display it directly in StaxRip, and then watch it in a different program (or use the script as input for VirtualDub, with disabled display speedups), does it contain useful content?

[stax76]

If you export the resulting clip into a video file, instead of trying to display it directly in StaxRip, and then watch it in a different program (or use the script as input for VirtualDub, with disabled display speedups), does it contain useful content?


D:\>"C:\Program Files (x86)\VapourSynth\core64\vspipe.exe" "D:\Temp\Video\Clips\sabo temp files\sabo_Preview.vpy" - --y4m | "D:\Projekte\GitHub\staxrip\bin\Apps\x265\x265_ml.exe" --crf 22 --preset ultrafast --frames 5062 --y4m --output "D:\Temp\Video\Clips\sabo temp files\sabo_out.hevc" -
Error: Can only apply y4m headers to YUV and Gray format clips
Output 0 frames in 0.04 seconds (0.00 fps)
x265 [error]: unable to open input file <->


If I open the vpy with VirtualDub sometimes my nvidia driver crashes so maybe the issue happens only on nvidia systems...

edit:

you mean disabling DirectX in VirtualDub's prefs dialog? Doesn't make a difference.

[LigH]

I did not think of using x265 to "save it as video", instead imagined a rather basic process, maybe some raw video in AVI. VSPipe being limited to YUV or Y8 is an additional unexpected limit here.

It looks like your script is supposed to output RGBA (or BGRA?); is that the only kind of video source you could display in StaxRip's preview window? Do you still use DIB functions?

At least it seems that the reason is related rather to displaying the result, less to handling it. Especially VirtualDub killing the display driver is highly suspicious.
__

P.S.: I can confirm "blackness" on W7 SP1 with Nvidia and some compression errors now and then. VirtualDub-AMD64 opens it, it is rather stable, the reported FourCC is more or less empty (I would assume 0x00000000). That may be the reason for the problems. Any more specific FourCC would be useful. I see no reason why your script shall not output e.g. YV12 ~ YUV422P8 (or at least YUY2 ~ COMPATYUY2), as long as your preview window uses a technology which can handle such video frames (okay - VapourSynth prefers planar YUV formats, which are not always supported by Windows easily and natively, that will certainly be a point where StaxRip64 will need an enhancement).

I'd like to point at sdk\include\vapoursynth\VapourSynth.h to emphasize:

typedef enum VSPresetFormat {
/* ... */

/* special for compatibility, if you implement these in any filter I'll personally kill you */
/* I'll also change their ids around to break your stuff regularly */
pfCompatBGR32 = cmCompat + 10,
pfCompatYUY2
} VSPresetFormat;

Wouldn't be surprised if this will be expanded to "filter or tool". :D
__

P.P.S.:

VirtualDub-AMD64 can display (format=vs.COMPATYUY2). More or less... it's vertically flipped (upside down). Reported decompressor is "Microsoft YUV (YUY2)".

[jackoneill]

And what if you pass matrix_in_s="709" to the resizer?

[LigH]

Ah, this does indeed display something useful in VD64 using COMPATBGR32 as output format.

So the reason was just some lack of initialization of attributes?

[stax76]

And what if you pass matrix_in_s="709" to the resizer?

my headache got better :)

but what if it's a SD source?

[LigH]

Then probably "601" instead of "709".

[stax76]

I've tried all documented options with a VOB source without success, 601 gives an error, 709 works with HD sources but gives blackness with a SD source.

http://www.vapoursynth.com/doc/functions/resize.html

[Myrsloik]

I've tried all documented options with a VOB source without success, 601 gives an error, 709 works with HD sources but gives blackness with a SD source.

http://www.vapoursynth.com/doc/functions/resize.html

470bg

You really should do this experimentation in a separate script with vspipe. Then it will tell you what the error is.