vsrawsource - Raw Video Reader for VapourSynth

[Chikuzen]

I wrote a VapourSynth plugin named vsrawsource.
vsrawsource-0.3.2.zip (http://www.mediafire.com/download/dxsb99ttb4qwp6e/vsrawsource-0.3.2.zip)

source code repository
https://github.com/chikuzen/vsrawsource

This plugin is the rewrited version of avisynth's rawsource.dll (http://forum.doom9.org/showthread.php?t=39798) to for VS from scratch.

I am inviting the patch for MacOS X build now.

EDIT:
2013/07/31
updated to 0.3.2

[active1]

cool! thank you Chikuzen :)
i compiled it successfully on linux, but why does the configure script will work only when i run it as root? (sudo sh configure)

[Chikuzen]

cool! thank you Chikuzen :)
i compiled it successfully on linux, but why does the configure script will work only when i run it as root? (sudo sh configure)

because I forgot to 'chmod +x' to configure :p

[kolak]

Which formats are supported?

[SassBot]

To query the list of formats:

supported color formats:
see core.raws.FormatList()

[Chikuzen]

or see source code.
https://github.com/chikuzen/vsrawsource/blob/master/format_list.c

[kolak]

Thanks-all clear now:)

[Myrsloik]

I looked at the source. It's a bad idea to write to stdout directly. If you want to return text you really should return it as a string value from the function because as it is it won't even interact with the python interpreter nicely.
My personal opinion is that this kind of stuff belongs in the documentation but that's up to you.

[Chikuzen]

Update to 0.2.0

* new option 'rowbytes_align'.
* support Windows Bitmap(.bmp)

BMP support is only for 24bit/32bit uncompressed RGB (palette,RLE and the others are not supported).

[Chirico]

OS X compiled version can be grabbed from here (http://www.sendspace.com/file/okjpi7).

[Robert Martens]

Just a quick note, it seems that your function registration is looking for parameters named "header_off" and "frame_off":

...

VS_EXTERNAL_API(void) VapourSynthPluginInit(
VSConfigPlugin f_config, VSRegisterFunction f_register, VSPlugin *plugin)
{
f_config("chikuzen.does.not.have.his.own.domain.raws", "raws",
"Raw-format file Reader for VapourSynth " VS_RAWS_VERSION,
VAPOURSYNTH_API_VERSION, 1, plugin);
f_register("Source", "source:data;width:int:opt;height:int:opt;"
"fpsnum:int:opt;fpsden:int:opt;src_fmt:data:opt;"
"header_off:int:opt;frame_off:int:opt;rowbytes_align:int:opt",
create_source, NULL, plugin);
}

Whereas the readme, and your argument setting code, use "off_header" and "off_frame":

...

vs_args_t va = { in, out, core, vsapi };

if (header > 0) {
set_args_int(&rh->vi.width, 720, "width", &va);
set_args_int(&rh->vi.height, 480, "height", &va);
set_args_int(&rh->off_header, 0, "off_header", &va);
set_args_int(&rh->off_frame, 0, "off_frame", &va);
set_args_int(&rh->row_adjust, 1, "rowbytes_align", &va);
set_args_data(rh->src_format, "I420", "src_fmt", FORMAT_MAX_LEN, &va);
}

...

In my quest to get my hands on planar RGB input to test my own plugin, I found the need to use those options, but they don't seem to work as expected, and it looks like it's just a mixup of the names.

[Chikuzen]

oops, sorry.
I didn't notice that typo.

It seems that VS-r15 is coming soon and API version is bumped up.
so i'll release new version with fixed it as soon as r15 is come.

[Robert Martens]

That's great, thank you! I'm in no rush, mind you, I just wanted to make sure I mentioned it. In the meantime, I've written a little command line application that strips out the headers of the files I'm working with, so I can work around it for as long as I need to. I appreciate your work on this plugin, it's proving to be very useful!

[Chikuzen]

updated to 0.3.0

* Alpha channel support
* fix typo

All alpha channel data will be stored in "_Alpha" prop.
You can extract it with 'alpha = core.std.PropToClip(clip)' as a Gray clip.

[Chikuzen]

updated to 0.3.1

* change alpha support method.

[Chikuzen]

update to 0.3.2

* fix memory leak
* cope with the filename which includes non-ASCII characters

[Myrsloik]

New compiles for 64 bit are available here (https://dl.dropboxusercontent.com/u/73468194/vsrawsource-64.7z).

[feisty2]

how do I compile this thing exactly?
I followed the instructions

$ cd ./vsrawsource
$ ./configure
$ make

and everything went okay actually, no error occurred, I got a 22kb .dll file
and I copied that file along with cygwin1.dll to the plugin directory and tried to run it
and vaporsynth just crashed without any warning

[jackoneill]

how do I compile this thing exactly?
I followed the instructions

and everything went okay actually, no error occurred, I got a 22kb .dll file
and I copied that file along with cygwin1.dll to the plugin directory and tried to run it
and vaporsynth just crashed without any warning

Maybe your changes are incorrect and that's why it crashed. (If you're not making any changes, just use the existing DLLs.) Since you have Cygwin, you can probably install gdb and find out more about the crash.

There are no secret compilation instructions that make it not crash. Probably.

[feisty2]

I don't know, after failed to run that modified version, I tried to compile Chikuzen's original code, and it still crashed
I'll get gdb and report back soon

[feisty2]

okay, I been messing with gdb for the last 1.5 hours and still got no clue about how to get it to debug, literally.
so I made a broken binary of Chikuzen's original code
here: https://github.com/IFeelBloated/vsrawsource/releases/tag/test
a little help here, tell me what I'm doing wrong plz?

[jackoneill]

Make sure to add -ggdb -g3 -O0 to the CXXFLAGS and don't let it strip the DLL.


gdb --args <command that crashes>
run
bt full

This prints the sequence of function calls that led to the crash, with the most recent at #0. It should give you the location of the crash as a file name and line number.

[feisty2]

sorry for the dumbness but, I'm still confused with gdb, like, what kind of command do I have to put in "<command that crashes>", sure it can't be that core.raws.Source.... vaporsynth command, right? it crashes vaporsynth if you try to load any file with core.raws.Source (...)
I updated a new broken x64 binary with debug symbols, someone kind plz, help me do the gdb test.

[Are_]

Something like "vspipe.exe something.py -"

[feisty2]

http://i.imgur.com/yixUMDT.png
okay, it got stuck like this for, 15 minutes.
what does it mean?

1.vpy
import vapoursynth as vs
core = vs.get_core()
clp = core.raws.Source("1.rgb",720,576,src_fmt="GRAY16")
clp.set_output ()

[jackoneill]

It means there was no crash.

[feisty2]

but 1.rgb is only about 30MB, vspipe should've done the work within seconds, not stuck for 15 minutes, and when I terminated vspipe.exe in taskmgr, t.rgb was still 0kb
http://i.imgur.com/cdqKiGB.png
and I loaded 1.vpy in vseditor and tried to preview it, it got stuck as well and taskmgr told me it stopped responding

[captainadamo]

Can you host your sample file somewhere and post exactly what your script is? I can try to help you diagnose your issues.

[feisty2]

1.rgb https://mega.nz/#!FxIRHZRS!l_Vu7N_0ASSccLv0KFHJLqmiY1CtaBSyK0pemTgDDtY
1.vpy

import vapoursynth as vs
core = vs.get_core()
clp = core.raws.Source("1.rgb",720,576,src_fmt="GRAY16")
clp.set_output ()


1.vpy vspipe.exe are in the same folder with 1.rgb

edit:
and broken x64 binary with debug symbols https://github.com/IFeelBloated/vsrawsource/releases/tag/test
in case you missed #21

[sl1pkn07]

works for me in linux (64 bits) with original code, with vsedit and pipe method

you clip is only 35frames?

[feisty2]

Yeah, 35 frames just for test

[sl1pkn07]

then works ok for me

EDIT: oh. wait. i use this fork https://github.com/myrsloik/vsrawsource

sorry :S

[feisty2]

Maybe I should uninstall windows and take Linux instead?
Okay, I'm joking...
But I can't get it to work on windows with gcc

[sl1pkn07]

and with clang?

[feisty2]

Didn't test with clang, but it's 10am now and I'm going out and I will test it and report back when I get home

[feisty2]

@HolyWu
still not working, I think my gcc might be corrupted, I'll reinstall it and report back

[feisty2]

reinstalled gcc and things worked out alright, finally
so, new version here
v0.3.3
update: added support to GrayS (tested) and GrayH (not tested, cuz I can't find any float16 clip and fmtc got no support for GrayH so no way to test)
https://github.com/IFeelBloated/vsrawsource/releases/tag/0.3.3

[ddubya]

I have fixed a bunch of bugs in vsrawsource and almost done adding tests. Also I have added capability to read from a pipe (not tested on windows yet).

I have not compiled or tested on windows yet, if you want to help go for it.

https://github.com/walisser/vsrawsource

[Efenstor]

I have fixed a bunch of bugs in vsrawsource and almost done adding tests. Also I have added capability to read from a pipe (not tested on windows yet).

I have not compiled or tested on windows yet, if you want to help go for it.

https://github.com/walisser/vsrawsource

Can you please give me a hand on the shell pipes (in Linux)? Surfed through a whole lot of sites and still unsuccessful. Here is what I try to do and it doesn't work.
video = core.raws.Source(sys.stdin, 1920, 1080, 30000, 1001)
# Here goes some processing
video.set_output()
Command line:
ffmpeg -i input.mts -pix_fmt yuv420p -f rawvideo - | \
env PYTHONPATH=/usr/local/lib/python3.4/site-packages vspipe --y4m proc_vs.py - | \
ffmpeg -i pipe: output.mkv
The error I get:
Script evaluation failed:
Python exception: 'utf-8' codec can't decode byte 0x80 in position 7291: invalid start byte

[jackoneill]

video = core.raws.Source(sys.stdin, 1920, 1080, 30000, 1001)
# Here goes some processing
video.set_output()

Script evaluation failed:
Python exception: 'utf-8' codec can't decode byte 0x80 in position 7291: invalid start byte

video = core.raws.Source("/dev/stdin", 1920, 1080, 30000, 1001)

[Myrsloik]

I have fixed a bunch of bugs in vsrawsource and almost done adding tests. Also I have added capability to read from a pipe (not tested on windows yet).

I have not compiled or tested on windows yet, if you want to help go for it.

https://github.com/walisser/vsrawsource

I've had a go at making it compile on windows. Btw, snprintf works just fine in vs2015 so don't just mindlessly copy and paste compatibility stuff.

I'm going to test my changes and then make a pull request I guess.

[Efenstor]

video = core.raws.Source("/dev/stdin", 1920, 1080, 30000, 1001)


Ouch. Never though it would be that simple and obvious.:thanks:

[ddubya]

Use "/dev/stdin" as the file path (it needs to be a path) instead of sys.stdin

[Myrsloik]

Here are compiled windows binaries (https://dl.dropboxusercontent.com/u/73468194/rawsource_binaries.7z).

[kolak]

:thanks:

Going to try.

[kolak]

import vapoursynth as vs
import sys
core = vs.get_core()

clip = core.raws.Source("/dev/stdin")
clip = core.fmtc.bitdepth (clip, bits=10)
clip.set_output()
enable_v210 = True

with:
ffmpeg -i C:\Users\Andy\Desktop\50i.avi -f yuv4mpegpipe - | "C:\Program Files (x86)\VapourSynth\core64\vspipe.exe" --y4m C:\Users\Andy\Desktop\rr.vpy - | C:\ffmpeg.exe -i - -c:v v210 C:\Users\Andy\Desktop\vv.mov

doesn't work, giving error:

Output #0, yuv4mpegpipe, to 'pipe:':
Metadata:
encoder : Lavf57.44.100
Stream #0:0: Video: wrapped_avframe, yuv422p, 1920x1080, q=2-31, 200 kb/s, 2
5 fps, 25 tbn, 25 tbc
Metadata:
encoder : Lavc57.51.100 wrapped_avframe
Stream mapping:
Stream #0:0 -> #0:0 (utvideo (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Script evaluation failed:
Python exception: raws: source does not exist.
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScrip
t (src\cython\vapoursynth.c:26897)
File "C:\Users\Andy\Desktop\rr.vpy", line 5, in <module>
clip = core.raws.Source("/dev/stdin")
File "src\cython\vapoursynth.pyx", line 1383, in vapoursynth.Function.__call__
(src\cython\vapoursynth.c:25204)
vapoursynth.Error: raws: source does not exist.

looks like /dev/stdin doesn't work.

[sl1pkn07]

/dev/stdin sounds a linux

[kolak]

Yes, but been told it should work here also :) I've read it won't on Windows.


More issues:

clip = core.raws.Source("C:\Users\Andy\Desktop\vv.raw", 1920, 1080, 30000, 1001, src_fmt='YUV422P10')

gives strange error:

Script evaluation failed:
Python exception: (unicode error) 'unicodeescape' codec can't decode bytes in po
sition 2-3: truncated \UXXXXXXXX escape (rr.vpy, line 4)
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 1490, in vapoursynth.vpy_evaluateScrip
t (src\cython\vapoursynth.c:26883)
File "C:\Users\Andy\Desktop\rr.vpy", line 4
clip = core.raws.Source("C:\Users\Andy\Desktop\vv.raw", 1920, 1080, 30000, 1
001, src_fmt='YUV422P10')
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in positio
n 2-3: truncated \UXXXXXXXX escape


Is this complaining about actual video data?

[ddubya]

Is this complaining about actual video data?

No, if you want to use a literal backslash in a string, you must escape it with another backslash.

You can use forward slash instead or format like this

"C:\\folder1\\folder2\\file.raw"

etc.

[ddubya]

looks like /dev/stdin doesn't work.

Yes, it needs the hack that treats the empty dash (-) as stdin, as is the usual convention.

In theory, you could run a pipe server and use names pipes as the ffmpeg ouput / vs input. I don't know of any command offhand that provides this.