Cross-Platform D2V Source for VapourSynth

[Mystery Keeper]

The former crash report was for x64 version. Here's one for x32 (http://paste.org.ru/?0hl67n).

[TurboPascal7]

The former crash report was for x64 version. Here's one for x32 (http://paste.org.ru/?0hl67n).

Please google what x32 means.

[Mystery Keeper]

Please google what x32 means.
It was produced by x32 VirtualDub, using x32 VapourSynth, using x32 d2vsource. No idea why it sees all my physical memory, but if you compare memory addresses in the report - they are all 32bit long. Why do you even bother saying something insulting, not helping and irrelevant to the topic?

[turbojet]

While I understand what you mean Mystery Keeper, x86 is technically correct. x32 is a linux only ABI (https://en.wikipedia.org/wiki/X32_ABI)

[Mystery Keeper]

Yup. Got it. Or course, I meant 32-bit version.

[Daemon404]

Sorry, I've been busier this week than expected. I've filed an issue (https://github.com/dwbuiten/d2vsource/issues/9) for it, and will poke it this weekend (I am currently on a transatlantic flight after a week of work).

It's been 'fixed' in the git repo. I want to add one more thing before I make a new release, so wait a day or so.

[Mystery Keeper]

Sure. As long as it takes. Thank you for your hard work!

[Daemon404]

New binaries.

Changelog here (https://github.com/dwbuiten/d2vsource/compare/beta6...beta7).

[Mystery Keeper]

No more crashes. Thank you!

[Mystery Keeper]

Running with rff=False does not change the framerate when it should.

[Myrsloik]

Running with rff=False does not change the framerate when it should.

Ignoring rff has no obvious framerate. It's only when you have a mode like force film where excess fields are discarded you get a nice ratio that makes sense.

[Mystery Keeper]

If video is not hard telecined and I tell the source filter to discard unneeded fields - isn't that an obvious case to change frame rate?

[jackoneill]

If video is not hard telecined and I tell the source filter to discard unneeded fields - isn't that an obvious case to change frame rate?

Ignoring rff is not the same thing as dgindex's "force film" option. Ignoring rff means you get exactly what is in the stream. That could be 100% 30 fps, or 100% 24 fps, or a mix. You don't know.

[lansing]

The multithread speedup really shocked me today. On my 720x480 dvd it ran at whopping 2800-3000fps! :eek:

When I change d2vsource's internal threads=1, speed drops to 1200-1300fps. Well, it's still a lot faster than mpeg2source in avisynth+ though, as it only ran at 530fps, don't know why.

[Sangan]

Hi, when I try to install (Mac OS 10.9), I get this error:$ make
g++ -Os -ffast-math -Wall -D__STDC_CONSTANT_MACROS -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -I. -I./vs -I./core -I/usr/local/Cellar/ffmpeg/2.8.5/include -I/usr/local/Cellar/vapoursynth/29/include/vapoursynth -I/usr/local/Cellar/zimg/2.0.2/include -c vs/applyrff.cpp -o vs/applyrff.o
vs/applyrff.cpp:239:87: error: use of undeclared identifier 'nfMakeLinear'
...out, "applyrff", rffInit, rffGetFrame, rffFree, fmSerial, nfMakeLinear, ...
^
1 error generated.
make: *** [vs/applyrff.o] Error 1

Is there a way to work this out?

[Are_]

Use non-git version. I think it is because you need vapoursynth git installed, but that line is said to be wrong in d2vsource code anyway. So you better wait for the fix (and install git vapoursynth), or install an older version.

[bilditup1]

So, I was bored and wrote a cross-platform D2V parser and decoder for VapourSynth.

Current Version: Beta 7

Windows Binary: https://github.com/dwbuiten/d2vsource/releases/download/beta7/d2vsource_beta7.zip - built with GCC 4.8.2 (MinGW-w64 v3.0.0) and FFmpeg. Contains 32-bit and 64-bit binaries.
Source: https://github.com/dwbuiten/d2vsource/


Latest VapourSynth apparently needs updated versions of a few source filters, including d2v. Are you still maintaining this, Daemon? Has someone else taken it up?

[jackoneill]

https://github.com/dwbuiten/d2vsource/releases/tag/v1.0

Daemon404 is indisposed, so I compiled the list of changes in his stead:

* Fix opening of d2v files and video files with non-English characters in Windows.
* Attach various properties to the frames.
* Make ApplyRFF work fully according to the H.262 standard.
* Fix seeking directly to the last frame.
* Add some sanity checks for the settings section of d2v files.

[STJAM]

I apologize for my lack of understanding of pulldown but I believe Mystery Keeper is correct RFF option appears not to be working.
Adding fpsnum/fpsden like ffms2 or lsmas might be appropriate.

[jackoneill]

I apologize for my lack of understanding of pulldown but I believe Mystery Keeper is correct RFF option appears not to be working.
Adding fpsnum/fpsden like ffms2 or lsmas might be appropriate.

What exactly isn't working? If it's the "force film" feature you want, that doesn't exist in d2vsource.

[STJAM]

The RFF mode seems to be stuck in forced film operation. RFF= True/False makes no difference. A video which is soft-telecined 100% Film it works great but for a video less than 100% lets say 99% a few frames may not be decimated. Consequently audio and video will be out of sync.
I have also found it necessary to add
c = core.std.AssumeFPS(clip = c, fpsnum=24000,fpsden=1001)
to all of my scripts something I have not normally found necessary with Beta 7. The durations and frame counts can be way off. I more then likely doing something wrong since no one else has chimed in. I will experiment some more and hopefully understand.

[jackoneill]

The RFF mode seems to be stuck in forced film operation. RFF= True/False makes no difference. A video which is soft-telecined 100% Film it works great but for a video less than 100% lets say 99% a few frames may not be decimated. Consequently audio and video will be out of sync.
I have also found it necessary to add
c = core.std.AssumeFPS(clip = c, fpsnum=24000,fpsden=1001)
to all of my scripts something I have not normally found necessary with Beta 7. The durations and frame counts can be way off. I more then likely doing something wrong since no one else has chimed in. I will experiment some more and hopefully understand.

There is no "forced film" feature here. No fields will ever be discarded. Can you share a small video where the rff parameter makes no difference? Also one where the video duration/frame count is off. Or is it the duration of frames that is off? Either way, video samples are needed to determine what is wrong.

[Keiyakusha]

Hey guys, DGIndex (The GUI) has 2 options for YUV-RGB conversion

Inside the DGindex application their meaning is:
PC scale: Levels should be expanded for display
TV scale: Levels should NOT be expanded for display

In other words they do not flag the source content, but rather the operation that must be performed.

a) PC option selected in GUI writes YUVRGB_Scale=1 in D2V file (= Yes, convert levels)
b) TV option writes YUVRGB_Scale=0 in D2V file (= No, do not convert)

In case A, D2VSource reads the value 1 and flags the video as Fullrange, which leads to levels not being expanded in VS Editor preview
In case B, D2VSource reads the value 0 and flags the video as LimitedRange, which leads to the correct output. But this will not happen because you want PC to be selected in DGindex gui since most of the time you do want to expand levels.
So it assumes that the flag is for source content, not for the desired operation on it.

Maybe D2VSource should ignore this flag or at least reverse its behaviour? Since it is too late to fix DGIndex gui button text/meaning
Edit: or maybe this is VS Editor is doing something it is not supposed to? Anyway, I find the current behavior to be confusing.

[jackoneill]

Apparently you can ask mpeg2source to output RGB, so try that and see what it does with YUVRGB_Scale.

[Keiyakusha]

Apparently you can ask mpeg2source to output RGB, so try that and see what it does with YUVRGB_Scale.

If you ask for RGB24 output from mpeg2source:
YUVRGB_Scale=1 --> RGB with correct levels (they will be expanded)
YUVRGB_Scale=0 --> RGB with TV levels (dull image)
In other words it performs the same as DGindex GUI.

Regardless of this option, input is always assumed to be TVscale, just the operation on it differs. So if VS property reflects the current state of the video, current D2VSource behavior should be incorrect.

[jackoneill]

The metadata tells you what is in the stream. From that you deduce what you should do with it when you convert to RGB. The metadata does not tell you directly what to do.

[Keiyakusha]

The metadata tells you what is in the stream. From that you deduce what you should do with it when you convert to RGB. The metadata does not tell you directly what to do.

Exactly. But the flag in .D2V file has an opposite meaning. And D2VSourse insn't aware of it.

To sum it up:

we have limited range dvd
we set PC levels in DGindex gui because we want to preview it right and in case of mpeg2source RGB24 output - we want it to be converted right
In this case YUVRGB_Scale=1 is written in D2V file
D2VSource reads it and sets metadata "fullrange" (which is not)
VS Editor (and possibly other tools) reads metadata, sees fullrange and not performing levels expansion.

So I think whatever D2V source should ignore this D2V flag, or reverse its behavior.
Edit: Personally I like ignoring, because in this case unflagged YUV would probably be assumed TVscale, which is what we have most of the time - so less confusion.

[Mystery Keeper]

VS Editor (and possibly other tools) reads metadata, sees fullrange and not performing levels expansion.Starting with r6, VS Editor is using VS internal format conversion for preview.

[Keiyakusha]

Starting with r6, VS Editor is using VS internal format conversion for preview.

Not sure what do you mean. I am of course using r8. I haven't followed VS editor closely, but I believe how the conversion performed is irrelevant. As long as you honor metadata - you get wrong output because if DGindex is set up correctly, D2Vsource outputs limited range flagged as fullrange

BTW, we can also see in DGindex docs that the meaning of PC/TV scale option is not what D2Vsource expects.

[Keiyakusha]

Here (http://i.imgur.com/J3vp1RF.jpg) is a fancy picture for you

[jackoneill]

Here (http://i.imgur.com/J3vp1RF.jpg) is a fancy picture for you

Sorry. I thought you misunderstood things, but in fact DGIndex is thinking about this backwards, as explained in a section of the manual I hadn't read before.

[hydra3333]

Hello.
The first post shows
Windows Binary: https://github.com/dwbuiten/d2vsource/releases/download/beta7/d2vsource_beta7.zip - built with GCC 4.8.2 (MinGW-w64 v3.0.0) and FFmpeg. Contains 32-bit and 64-bit binaries.

https://github.com/dwbuiten/d2vsource/releases shows
Downloads
1.33 MB d2vsource-1.0-windows.zip https://github.com/dwbuiten/d2vsource/releases/download/v1.0/d2vsource-1.0-windows.zip

I should be using v1.0 ?

[Reel.Deel]

I should be using v1.0 ?

Yes. 1st post has not been updated.

[cwk]

When attempting to 'make' the D2V Witch on Ubuntu 14.04, I'm getting this error:

src/D2VWitch.cpp:88:59: error: no matching function for call to ‘std::vector<int>::erase(__gnu_cxx::__normal_iterator<const int*, std::vector<int> >)’
audio_ids.erase(audio_ids.cbegin() + j);
^
src/D2VWitch.cpp:88:59: note: candidates are:
In file included from /usr/include/c++/4.8/vector:69:0,
from src/D2VWitch.cpp:24:
/usr/include/c++/4.8/bits/vector.tcc:134:5: note: std::vector<_Tp, _Alloc>::iterator std::vector<_Tp, _Alloc>::erase(std::vector<_Tp, _Alloc>::iterator) [with _Tp = int; _Alloc = std::allocator<int>; std::vector<_Tp, _Alloc>::iterator = __gnu_cxx::__normal_iterator<int*, std::vector<int> >; typename std::_Vector_base<_Tp, _Alloc>::pointer = int*]
vector<_Tp, _Alloc>::
^
/usr/include/c++/4.8/bits/vector.tcc:134:5: note: no known conversion for argument 1 from ‘__gnu_cxx::__normal_iterator<const int*, std::vector<int> >’ to ‘std::vector<int>::iterator {aka __gnu_cxx::__normal_iterator<int*, std::vector<int> >}’
/usr/include/c++/4.8/bits/vector.tcc:146:5: note: std::vector<_Tp, _Alloc>::iterator std::vector<_Tp, _Alloc>::erase(std::vector<_Tp, _Alloc>::iterator, std::vector<_Tp, _Alloc>::iterator) [with _Tp = int; _Alloc = std::allocator<int>; std::vector<_Tp, _Alloc>::iterator = __gnu_cxx::__normal_iterator<int*, std::vector<int> >; typename std::_Vector_base<_Tp, _Alloc>::pointer = int*]
vector<_Tp, _Alloc>::
^
/usr/include/c++/4.8/bits/vector.tcc:146:5: note: candidate expects 2 arguments, 1 provided



Here's my ffmpeg and gcc version info:

chris@jupiter:/storage/code/vsynth/D2VWitch$ ffmpeg -version
ffmpeg version N-81036-g2b14204 Copyright (c) 2000-2016 the FFmpeg developers
built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
configuration: --enable-gpl --enable-version3 --enable-shared --enable-nonfree --enable-postproc --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid
libavutil 55. 28.100 / 55. 28.100
libavcodec 57. 50.100 / 57. 50.100
libavformat 57. 42.100 / 57. 42.100
libavdevice 57. 0.102 / 57. 0.102
libavfilter 6. 47.100 / 6. 47.100
libswscale 4. 1.100 / 4. 1.100
libswresample 2. 1.100 / 2. 1.100
libpostproc 54. 0.100 / 54. 0.100


Any ideas?

[jackoneill]

Any ideas?

Yes. You're in the wrong thread. :P Anyway, I'll fix that shortly. In the meantime, replace "cbegin" with "begin".

[xekon]

I am new to vapoursynth, so for the last couple days I didnt know where the problem was originating, and Ive been reinstalling and tinkering trying to narrow it down. I believe its the d2v source plugin.

here is a screenshot of the output:
http://i1208.photobucket.com/albums/cc361/xekon/vsedit_zpsfhnn5j8x.jpg

same d2v file in Avisynth/AvsPmod using MPEG2Source:
http://i1208.photobucket.com/albums/cc361/xekon/avsp_zps5cuhuhkh.jpg

I read through the entire post here, hoping to figure it out on my own, and tried adding the ApplyRFF but it did not change the result.

import vapoursynth as vs
core = vs.get_core()
ret = core.d2v.Source(input=r'/media/sf_enc/005s.d2v')
ret = core.d2v.ApplyRFF(ret,d2v=r'/media/sf_enc/005s.d2v')
ret.set_output()

I have a attached a couple small samples, need anything else and I would be happy to test or provide it. Any help is greatly appreciated.

http://www.mediafire.com/file/1azgp6i418wbwj1/enc001.7z
http://www.mediafire.com/file/wl90qp7t4d1o518/enc005.7z

[xekon]

I just noticed that under my windows build of vapoursynth and vsedit that the d2v source plugin appears to be working:

http://i1208.photobucket.com/albums/cc361/xekon/vseditgood_zpsawdoobtw.jpg

So it is only a problem on my linux builds, I have tried on both Ubuntu server 16.04.1(x264 output) and Kubuntu 16.04.1

I really need this to work on my linux builds because I primarily use linux and I plan to use GNU Parallel to do parallel encoding across multiple machines.
Any help is greatly appreciated, please let me know if you need me to test or provide anything further, I am stumped at what could be the problem.

From a clean/fresh install of kubuntu-16.04.1-desktop-amd64 I used the following script to install Vapoursynth/Vsedit/Plugins:

before executing the script I installed prereqs:
sudo apt-get install openssh-server
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install git libxvidcore-dev mercurial cmake build-essential yasm libass-dev python3-pip python3-dev cython3 autoconf libtool pkg-config
sudo apt-get install libmagick++-dev qt5-default libfftw3-dev nasm libpng-dev libsndfile1-dev zlib1g-dev libopencv-dev ocl-icd-libopencl1 opencl-headers

chmod +x ~/VSInstall && ./VSInstall


https://sites.google.com/site/jakzite/VSInstall
http://pastebin.com/wDttQwHP

[Are_]

Did try your sample and encountered the same problem, I fixed it with rff=False:

import vapoursynth as vs
core = vs.get_core()

clip = core.d2v.Source("005s.d2v", rff=False)

clip.set_output()

[xekon]

YAY! I didn't realize RFF was applied by default, since the syntax looked the same between my windows and ubuntu builds.

I also didnt realize rff=False was a parameter, only that you could use the ApplyRFF.

This fixed it for me too, thank you so much, I really appreciate it!

[jackoneill]

I am new to vapoursynth, so for the last couple days I didnt know where the problem was originating, and Ive been reinstalling and tinkering trying to narrow it down. I believe its the d2v source plugin.

here is a screenshot of the output:

[...]

I have a attached a couple small samples, need anything else and I would be happy to test or provide it. Any help is greatly appreciated.

http://www.mediafire.com/file/1azgp6i418wbwj1/enc001.7z
http://www.mediafire.com/file/wl90qp7t4d1o518/enc005.7z

Using your two samples, I was unable to reproduce the garbled output. However, I did discover that ApplyRFF is broken in some way. Please test again with my fork (https://github.com/dubhater/d2vsource). It's possible the same bug manifested in a different way on your computer.

[xekon]

jackoneill, I tried your fork, upon loading the sample in VSedit that I provided here, and pressing F5 to generate a preview, it at first appeared to be a proper frame, no graphical anomilyies.

But if you use the seek bar to change a frames a few times, it appears I still get the garbled output.

Both your fork, and the d2vsource I was originally using, output good frames (no garbled picture) if I add 'rff=False' from this:
import vapoursynth as vs
core = vs.get_core()
ret = core.d2v.Source(input=r'/media/sf_enc/005t.d2v')
ret.set_output()

to this:

import vapoursynth as vs
core = vs.get_core()
ret = core.d2v.Source(input=r'/media/sf_enc/005t.d2v', rff=False)
ret.set_output()

could It be something to do with my install method?
I shared it here: http://pastebin.com/wDttQwHP
running kubuntu 16.04.1

[jackoneill]

jackoneill, I tried your fork, upon loading the sample in VSedit that I provided here, and pressing F5 to generate a preview, it at first appeared to be a proper frame, no graphical anomilyies.

But if you use the seek bar to change a frames a few times, it appears I still get the garbled output.

Both your fork, and the d2vsource I was originally using, output good frames (no garbled picture) if I add 'rff=False' from this:
import vapoursynth as vs
core = vs.get_core()
ret = core.d2v.Source(input=r'/media/sf_enc/005t.d2v')
ret.set_output()

to this:

import vapoursynth as vs
core = vs.get_core()
ret = core.d2v.Source(input=r'/media/sf_enc/005t.d2v', rff=False)
ret.set_output()

could It be something to do with my install method?
I shared it here: http://pastebin.com/wDttQwHP
running kubuntu 16.04.1
I don't see anything wrong in your install method.

Perhaps you should load the new d2vsource plugin directly, to be sure you're testing the right one.

Does it only happen with random access, or can you get the garbled images by moving one frame at a time with the right arrow? If yes, can you get the garbled images by using vspipe instead of vsedit?

vspipe 005t.py 005t.y4m --y4m

[xekon]

I just made sure to remove the original d2vsource:
sudo rm /usr/local/lib/libd2vsource.so
sudo rm /usr/local/lib/libd2vsource.la
sudo rm /usr/local/lib/vapoursynth/d2vscan.pl
sudo rm /usr/local/lib/vapoursynth/d2vsource.so

at this point I could no longer use core.d2v.Source()

I then reinstalled your fork, same result, I tried pressing the right arrow key, as i go frame by frame, roughly half of them display garbled.

I would be happy to explicitly load your plugin if you need, what is the code to do that?

wasnt sure how to view the output of 005t.y4m so i did this:
vspipe --y4m 005t.vpy - | x264 --demuxer y4m - --output 005t.mkv

here is a video without using rff=False, oddly the output is not garbled like it is in vapoursynth, there just appears to be some kind of interlacing visible(look at raditz black hair as the frame zooms out.)
http://www.mediafire.com/file/8g7285x2dlr99cc/005t.mkv

here is one using rff=False, this one does not show interlacing or garbled output in vsedit.
http://www.mediafire.com/file/58hi1wgvxil80ia/005t_rff_False.mkv

[jackoneill]

After updating everything, I figured it out. The problem is not in d2vsource but in VapourSynth's resizer, which vsedit uses to convert to RGB for display.

Just for future reference, this is how you load a plugin manually:

core.std.LoadPlugin("/some/place/libd2vsource.so")

[xekon]

Right on Jackoneill! way to go dude.

[Mystery Keeper]

After updating everything, I figured it out. The problem is not in d2vsource but in VapourSynth's resizer, which vsedit uses to convert to RGB for display.With resizer itself or with the way I use it?

[jackoneill]

With resizer itself or with the way I use it?

The resizer itself. vsedit is not required to reproduce the bug. See https://github.com/vapoursynth/vapoursynth/issues/256

[xekon]

so I'm trying to compile this as a DLL for use on windows, having never used mingw or msys before, was hoping somebody could help.
$ ./autogen.sh
./autogen.sh: line 3: autoreconf: command not found

[JoeyMonco]

You don't have autotools installed.

[JoeyMonco]

oh I just assumed that was part of the msys2 package, i will have to try and figure out how to get that installed then, thank you.

Or maybe your PATH isn't set up correctly? *shrug*