Hybrid: Input -> x264/x265/Xvid/VP8/VP9

[Selur]

Hybrid rev 2011.05.05.1:

*fixed*
error when detecting if Hybrid homePath is writeable

*updated*
windows packages: mp4box, frameCounter and x264

-> downloads: http://www.selur.de/downloads

Cu Selur

[flebber]

Love the update only 1 request, would it be possible to have spline36resize or blackman as available option?

[Selur]

If mencoder or ffmpeg have them: yes (not sure if they do nowadays)
otherwise: no, since Hybrid doesn't create avisynth files,...

[devaster]

are u plannig for use of GPU decode on single/multiple GPUs ?
i have 3 GPUs and it would be nice one used for decoding and one for encoding ....

[Selur]

Don't think that will happen in the near future since mencoder&ffmpeg which are mainly used don't have gpu decoding and I probably won't have the time to read-up and write a decoder (which would need to be able to output raw yv12 via std::out) myself.
And since DGDecNV is avisynth only and doesn't have the ability to output via std::out that's probably also not gonna happen anytime soon.
That said if someone writes a command line decoder I would add support for it inside Hybrid.

Cu Selur

[devaster]

okay, can u send me some specs what must decoder meet , ill try to write one ...

[Selur]

these would be needed features:
1. it should be able to open raw h.264 files
2. output to pipe (std::out) and file
3. everything that's not file output should go to std:err not std::out
4. output file format should be something x264/ffmpeg/mencoder supports (i.e. NV12/Yv12)
these would be nice features:
1. being able to open raw mpeg-2/vc-1 files or everything else the decoder can handle
2. accept input via pipe, so one could use i.e. mencoder/ffmpeg to open a container and pipe the content to the decoder and from the decoder to i.e. x264
3. progress indication to std::err
4. deinterlacing support
these would be too good to be true features:
1. ability to read stream that are still inside a container (so one would have a basic DGDecNV replacement)
2. combine it the cuda encoder Hybrid uses (I share the the code, which is mainly a Qt port of the cuda encoder example from the cuda sdk)
3. additional filtering through the decoder (cropping, denoise, whatever is possible)

about license: don't care as long as it's free to use for personal purposes and I get no trouble if I add it to the Hybrid packages/installer
open source: would be nice, but not a needs-to-be

Cu Selur

[devaster]

why is encoding on gpu so slow when i use filters ? - it uses the gpu only on ~25%. when i switch off the filtering it is used for ~100% (dvd source)...
btw mencoder is using only one core ???

[Selur]

why is encoding on gpu so slow when i use filters ?
currently the dvd source will be opened and decoded with mencoder (which is probably the bottleneck when decoding)
btw mencoder is using only one core ???
What does '-lavdopts threads=X' in the decoder part of the call say ?
(on linux threads=1 is forced since multi-threaded mpeg decoding seems to be broken there from time to time, but on windows X should be number of cpu core iirc.)
Depending on your filtering, it might also be that one of the filters used doesn't support multi-threading.

Cu Selur

[devaster]

it is setted to 8 threads and i am using sharpen and denoise - hqdn3d ....

[Selur]

it's probably hqdn3d,.. (but I'm not 100% sure since I frankly never tested which filters in mencoder interact with multi-threading in a "bad" way)

[meshaun]

Hello Selur,
is that problem with the encoding finished?

[devaster]

it's probably hqdn3d,.. (but I'm not 100% sure since I frankly never tested which filters in mencoder interact with multi-threading in a "bad" way)

no its sharpen. when i use only hqdn3d, it is better (50%)

[Selur]

@meshaun:should be fixed in latest public release,...

@devaster: good to know,.. damn, switching to chroma&luma sharpen instead of (un)sharpen doesn't help either,.. :/

Found a way :) use mencoder_mt (and rename it to mencoder) from i.e. http://ps3mediaserver.googlecode.com/svn/trunk/ps3mediaserver/win32/ (or http://sourceforge.net/projects/mplayer-win32/files/MPlayer%20MT/revision%2032848/) and the speed is a lot better ;)

=> does anyone have any experience with mencoder_mt ? (if it doesn't have any known problems I'll switch to mencoder and mplayer MT for hybrid)

[devaster]

mt mencoder seems doesnt help, will try better tomorrow

[Selur]

strange, more than doubled the encoding speed for me when sharpening and encoding with the cuda encoder,..

[devaster]

i had it tested on another system and yes there is double speedup (and double GPU usage - 50%)

[devaster]

i was looking on nvidia docs about decoder. It will be easier to incorporate the decoding and maybe filtering part into your encoder - maybe lower overhead with frame transferring ...

[Selur]

Did some extensive testing on why I used the normal mencoder and not the mencoder_mt build and I found out why. :)
-> sometimes mencoder_mt doesn't start up properly, i.e. when calling
mencoder.exe "G:\Hybrid\test - clips\test.avi" -ovc raw -noskip -vf scale,format=i420 -forcedsubsonly -nosub -nosound -mc 0 -lavdopts threads=8 -ofps 25 -aspect 1.81818:1 -of rawvideo -o - | x264.exe --crf 42 --f
ps 25 --input-res 640x352 --output NUL -
over and over again, around each third or forth time mencoder just doesn't send any frames,..
which will cause Hybrid to abort current since no output (and or stats file for 2pass 1st pass) will be created.
so not using mplayer_mt/mencoder_mt was a stability choice. :)
(will keep on testing new mt versions in the case that I find updated binaries ;))

Cu Selur

Ps.: @devaster: please also add an option to use the decoder without the encoding part, this would allow to use the decoder part and pipe the output to x264/xvid/mencoder/.. instead of using the nvidia encoder. ;)

[Selur]

Hybrid rev 2011.05.11.1

*fixed*
- mkvextract: removed '--raw' from extraction cl
- problem when importing chapters

*changed*
- frameCounter: limit buffer to 20MB + better frame detection

*added*
- win7/vista: show progress in taskbar
- hibernate as an additional option on queue finished

*updated*
- windows packages: frameCounter


-> downloads: http://www.selur.de/downloads

[flebber]

Will you be updating hybrid for the new x264 presets sharktooth is discussing here. http://forum.doom9.org/showthread.php?p=1499552#post1499552

[Selur]

Judging from:
1. a x264 download from x264.nl
and
2. the git summary (http://git.videolan.org/?p=x264.git;a=summary)
these device presets are not preset in x264 atm.

When they are included, and there are working binaries with them, I will adjust Hybrid accordingly to support them.

Cu Selur

[Selur]

Hybrid rev 2011.05.19.1:

*fixed*

aspectRatio in cropView
Blu-ray autoCrop
Blu-ray chapter list import
cropView WASD
small memory leak

*changed*

autoCrop

*added*

x264: 2pass crf support again
autocrop: option to set how many frames are analysed

-> downloads: http://www.selur.de/downloads

[Selur]

Hybrid 2011.05.19.3

*fixed*

x264: broke 1pass modes when I added 2pass crf again


Cu Selur

[Brazil2]

Hybrid 2011.05.19.3
Typo in the Win32 file name:
Hybrid_1105191_3.zip should be Hybrid_110519_3.zip

[Selur]

Thanks, fixed the link. :)

[hifi25nl]

I am using last version 2011.05.19.3 in archlinux with last git version of x264. The other apps are from arch official repository.

All is apparently working fine but If I try to encode for blu ray the resulting video has colors wrong: skins are almost blue...like in some sf movies...

The original file has been taken from a Sanyo Xacti video camera:
AVC, Progressive, Bitrate 12166, Resolution 1920x1080, PAR 1, 29,97, aac.

The options changed in Hybrid are:
1) Structure: Blu ray on Main tab
2) Audio handling auto add first on Main tab
3) Hardware blu ray on x264 tab
4) Constant rate factor (2 pass) on the x264 tab
5) Scale output frame rate to 23,976 or 24 in filtering tab (is this right?)
6) Encoding to ac3 in audio tab.

The same movie demuxed with tsmuxer, changed aac to ac3 and authored with easyBD is playing fine with right colors in my panasonic BD60 blu ray player (no re-encoding!)

[Selur]

the resulting video has colors wrong: skins are almost blue..
it's probably a problem with the decoding part.
How do the complete encoding calls look like ?
(disable 'Minimize job command lines' and post the complete lines of the encoding calls)

5) Scale output frame rate to 23,976 or 24 in filtering tab (is this right?)
Since the input is progressive 29.97fps I would say: Yes.

The same movie demuxed with tsmuxer, changed aac to ac3 and authored with easyBD is playing fine with right colors in my panasonic BD60 blu ray player (no re-encoding!)
which doesn't help at all, since you want to reencode the clip :)

4) Constant rate factor (2 pass) on the x264 tab
dump that, it's an old relict from when vbv didn't work with crf encoding (-> will fix description)

Cu Selur

[hifi25nl]

Under x264 tab:
x264 --pass 1 --bitrate 16781 --profile high --level 4.1 --bluray-compat --keyint 60 --b-pyramid strict --direct auto --slices 4 --mvrange 511 --sync-lookahead 15 --weightp 1 --aq-mode 0 --stats "/tmp/SANY0169.stats" --fps 24000/1001 --input-res 1920x1080 --output NUL -

x264 --pass 2 --bitrate 16781 --profile high --level 4.1 --bluray-compat --keyint 60 --b-pyramid strict --direct auto --slices 4 --partitions i4x4,p8x8,b8x8 --mvrange 511 --subme 5 --sync-lookahead 15 --trellis 0 --no-fast-pskip --weightp 1 --aq-mode 0 --vbv-maxrate 40000 --vbv-bufsize 30000 --colorprim bt709 --transfer bt709 --colormatrix bt709 --stats "/tmp/SANY0169.stats" --fps 24000/1001 --input-res 1920x1080 --output "/tmp/SANY0169.264" -

Under Jobs:

MP4Box -raw 2 "/media/linux/home/olmeda/Downloads/BluRay/test/SANY0169.MP4" -out "/tmp/SANY0169_eng_aid_0__11_53_09_131_01.aac"

m2ts;false;100_:_/tmp/SANY0169_eng_aid_0__11_53_09_131_01.aac

mencoder "/media/linux/home/olmeda/Downloads/BluRay/test/SANY0169.MP4" -ovc raw -vf filmdint=io=29.97:23.976,softskip -forcedsubsonly -nosub -nosound -mc 0 -lavdopts threads=2 -really-quiet -ofps 24000/1001 -aspect 1.77778:1 -of rawvideo -o - | x264 --pass 1 --bitrate 16781 --profile high --level 4.1 --bluray-compat --keyint 60 --b-pyramid strict --direct auto --slices 4 --mvrange 511 --sync-lookahead 15 --weightp 1 --aq-mode 0 --stats "/tmp/SANY0169_11_53_09_131_03.stats" --fps 24000/1001 --input-res 1920x1080 --output /dev/null -

mencoder "/media/linux/home/olmeda/Downloads/BluRay/test/SANY0169.MP4" -ovc raw -vf filmdint=io=29.97:23.976,softskip -forcedsubsonly -nosub -nosound -mc 0 -lavdopts threads=2 -really-quiet -ofps 24000/1001 -aspect 1.77778:1 -of rawvideo -o - | x264 --pass 2 --bitrate 16781 --profile high --level 4.1 --bluray-compat --keyint 60 --b-pyramid strict --direct auto --slices 4 --partitions i4x4,p8x8,b8x8 --mvrange 511 --subme 5 --sync-lookahead 15 --trellis 0 --no-fast-pskip --weightp 1 --aq-mode 0 --vbv-maxrate 40000 --vbv-bufsize 30000 --colorprim bt709 --transfer bt709 --colormatrix bt709 --stats "/tmp/SANY0169_11_53_09_131_03.stats" --fps 24000/1001 --input-res 1920x1080 --output "/tmp/SANY0169_11_53_09_131_04.264" -

"/tmp/SANY0169_11_53_09_131_03.stats"

"/tmp/SANY0169_11_53_09_131_03.stats.mbtree"

/tmp/SANY0169_11_53_09_131_07.meta ### MUXOPT --no-pcr-on-video-pid --new-audio-pes --blu-ray --vbr --vbv-len=500
V_MPEG4/ISO/AVC, "/tmp/SANY0169_11_53_09_131_04.264", fps=23.976, insertSEI, contSPS
A_AAC, "/tmp/SANY0169_eng_aid_0__11_53_09_131_01.aac"

tsMuxeR "/tmp/SANY0169_11_53_09_131_07.meta" "/media/linux/home/olmeda/Downloads/BluRay/test/SANY0169"

"/tmp/SANY0169_11_53_09_131_04.264"

"/tmp/SANY0169_eng_aid_0__11_53_09_131_01.aac"

"/tmp/SANY0169_11_53_09_131_07.meta"

[Selur]

"-vf filmdint=io=29.97:23.976,softskip" is probably wrong for your input -> try disabling Filtering->Deinterlace/Telecine Settings->Auto deinterlacing
Strange thing is, Hybrid shouldn't insert this unless the source is detected as interlaced,.. -> would be cool if you could upload a small sample to MultiUpload (or similar site) so I can do some tests. :)

DOOHHH,.. it's not added by the deinterlacer but by the frame rate changer :)
-> think I know where the problem is. :)
("-vf filmdint=io=29.97:23.976,softskip" needs to be a "-vf scale,format=i420,filmdint=io=29.97:23.976,softskip")
What version 32/64bit of linux do you use, so I can send you a test version of a hopefully fixed Hybrid. :)


Cu Selur

[hifi25nl]

I tried your previous suggestion, but the skins are always blu....

I am using archlinux 64 bit.

[Selur]

I'm fixing a small bug atm. but will send you a test version in around an hour.
-> send you a pm with a download link to an updated version

[hifi25nl]

I am testing now but...most of time the original file will disappear, it is deleted when I start a job!

[Selur]

That should not happen unless source and targetDir are the same and the input&ouput are named equally,...

[hifi25nl]

I have made a test and the resulting file was fine now!

However I confirm that most of times this experimental version has some problems. To be sure I have deleted the configuration file .hybrid in my home directory. When I load a file this will be deleted at the end of analyzing.

I have tried to find some errors. The only ones are those given by dmesg:


[ 3648.285767] Hybrid[13891]: segfault at 8 ip 00007f7d04e1dc1d sp 00007fff70c9ff58 error 4 in libQtCore.so.4.7.3[7f7d04d68000+27f000]
[ 5553.830557] Hybrid[26999]: segfault at 8 ip 00007f27f90dac1d sp 00007fffbf52b688 error 4 in libQtCore.so.4.7.3[7f27f9025000+27f000]
[ 5731.870442] Hybrid[28411]: segfault at 8 ip 00007fcc06d8cc1d sp 00007fff95151288 error 4 in libQtCore.so.4.7.3[7fcc06cd7000+27f000]
[ 5802.830814] Hybrid[29545]: segfault at 8 ip 00007f9d1fccdc1d sp 00007fffbf915e38 error 4 in libQtCore.so.4.7.3[7f9d1fc18000+27f000]
[ 5996.223695] x264[30699] general protection ip:42cbbf sp:7f435854eeb4 error:0 in x264[400000+15f000]
[ 5996.223713] x264[30700] general protection ip:42cbbf sp:7f4357d4deb4 error:0
[ 5996.223717] x264[30701] general protection ip:42cbbf sp:7f435754ceb4 error:0 in x264[400000+15f000]
[ 5996.225135] in x264[400000+15f000]

[Selur]

However I confirm that most of times this experimental version has some problems.
"most of the times" means what?

To be sure I have deleted the configuration file .hybrid in my home directory.
(the should be a configuration folder .hybrid, not a configuration file)
use Hybrid->Config->Defaults->Reset->All to be sure you deleted the right stuff.

I really need some debug output to trace a problem.

Important info needed to fix&reproduce bugs:

What Hybrid version did you use?(Windows/Mac/Linux 32/64bit + versions number)
What did happen? If an error message came up what did it exactly say?
What did you do to get to the bug? (best thing is a detailed step-by-step guide so that I can reproduce the problem)
Debug output would be helpful. (Config->Misc Settings->create debug file; enable it, directly after starting Hybrid)
If possible a small sample file would be great.

All these infos might be needed to reproduce the problem and reproduction is needed to find a fix. :)
If you find a bug, I'll try to fix it and send you a link to a hopefully fixed version for testing.
source: http://www.selur.de/support

Cu Selur

[hifi25nl]

Ok, now I have sent the debug file. Thank you very much for your assistance.

[hifi25nl]

New version now is fine!!

[Arti]

After turning CUDA is 650 fps :D

GeForce 9600 GT :D

[hifi25nl]

Is CUDA working on linux? I see a cuda command line but...

[Selur]

CUDA in general is working on Linux assuming you use the NVIDIA driver, the cuda encoder isn't available for Linux (mainly because none of my linux machines got an NVIDA card in it, afaik the nvcuvenc library that is used is also available on linux),... -> correction, the nvcuvenc library is only available for Windows

[Selur]

Hybrid rev 2011.06.18.1:

*changed*

added warning when output path equals program folder
allow vc-1 in m2ts container
extract videostream from blu-ray before reencoding (add additional overhead, but fixes synch problems when mpls spans over multiple files)

*added*

using mp4fpsmod to extract mp4 timecodes (http://sites.google.com/site/qaacpage/junk)

*fixed*

a bunch of problems related to separate 2pass 1st and 2pass 2nd pass encoding
autocorp: handling of raw h.264
error popup when adjusting preset an bluray restrictions enabled,..
problem with .mpls and autocrop
missing color space conversion when changing frame rate and not doing any other filtering
problem when extracting/importing chapters from non-dvd sources
filtermodel not completely ignored when using passthrough
fps of input ignored if video handling was set to 'passthrough' on input load and then changed to something else before adding a job
adding -tmp TempPath to MP4Box muxing call to make sure that the right temp folder is used
.dts-hd/.dtshd not detected as raw audio
blu-ray synch problem

*updated*

windows packages/installer
hobarrera started a archlinux package


-> downloads: http://www.selur.de/downloads

Cu Selur

[Selur]

Hybrid rev 2011.06.20.1:

*cosmetics*

changed progress indication in title/taskbar
Win7 32bit: progress in taskbar - accidentally build I didn't build the last Win32Bit build with Windows XP, not Windows 7

*fixed*

video passthrough bug
x264: 2pass 1st pass calls not generated


-> downloads: http://www.selur.de/downloads

Cu Selur

[Selur]

Hybrid rev 2011.06.22.1:

*cosmetics*

reworked the internal message system

*fixed*

when extraction Audio with tsMuxeR audio was moved to Hybrid home on rename :(
cuda: force AVC Level 'auto' for 'cq'-encoding mode


-> downloads: http://www.selur.de/downloads

Cu Selur

[Selur]

Hybrid rev 2011.06.23.1:

*cosmetics*

x264 encoding progress indication

*fixed*

video profiles import (bitrate adjustment)


-> downloads: http://www.selur.de/downloads

Cu Selur

[Selur]

Hybrid rev 2011.06.26.1:

*fixed*

don't create output file in the 1st pass of a two pass encoding
missing chapter DataNodes on some input files without chapters


-> downloads: http://www.selur.de/downloads

Cu Selur

[RanmaCanada]

How do I add multiple mkv files that have srt subs? Whenever I try, and use the add and extract, the program only extracts the subs from the first file, and then names it after the last file. So it will be ripped from 01.mkv and be named 19.srt. I am trying to do a huge batch conversion for bluray from files I created a few years ago, and if I do them one at a time it works great, but I have almost 150 files do to.

Conversely if I do not use the add and extract options, it does not even recognize that there are subs in the stream at all. All it sees it the video and 2 audio streams. I would post a log but for some reason it does not want to save one, and I always have to reset everything when the program opens because it does not remember any of my previous settings.

[Selur]

How do I add multiple mkv files that have srt subs?
Huha,.. to be frank I never thought about batch processing when I wrote the subtitle add/extract routines. :)
-> I'll try to look at it next week.

Conversely if I do not use the add and extract options, it does not even recognize that there are subs in the stream at all.
Are you sure? The Subtitle-Spinbox is grayed out for each file if you cycle through the sources through the Source-Spinbox ?
(just tried it here, and it seems to be fine here,..)


I would post a log but for some reason it does not want to save one,...
Config->Misc Settings->Create Debug File; enable it, directly after starting Hybrid
see: http://www.selur.de/support

... and I always have to reset everything when the program opens because it does not remember any of my previous settings.
than you probably didn't tell Hybrid to save your settings,...

Cu Selur

[RanmaCanada]

ok thank you I will see what I can do. :) That was one fast response haha.

[Selur]

I only need a debug output regarding the 'not even recognize that there are subs in the stream at all', since I can reproduce the extract&co problem. (like I said I didn't think about multiInput when designing the extraction methods ;))

Cu Selur