Hybrid: Input -> x264/x265/Xvid/VP8/VP9

[Selur]

definitely not a display error, Hybrid got that information from somewhere.

My script contains this : assumefps(24000,1001)
Which script? I need more details.

[Music Fan]

Just an avisynth script I made, no problem when opened in Virtual Dub (which shows well that the timecode of the 24th frame is 0:00:01.001).
assumefps(24000,1001) is the last line of this script so I don't understand why Hybrid detects another framerate than 23.976.

[Selur]

try adding a:
last
to the call.

To see the infos Hybrid will get for an avs script, you can call:
"PATH TO Hybrid FOLDER/avsInfo.exe" "PATH TO avs script"

[Music Fan]

try adding a:
last
to the call.

To see the infos Hybrid will get for an avs script, you can call:
"PATH TO Hybrid FOLDER/avsInfo.exe" "PATH TO avs script"
It indicates "framerate : 24000/1001", whatever I put last or not in the script.

[Selur]

I see,.. there's a typo somewhere which causes the rounded and unrounded fps to be concatenated. (for avs input only)
-> will fix

[Selur]

Hybrid rev 2014.08.06.1:

*added*

avisynth: added option to disable Avisynth during normal preview, this way audio and subtitles from the source can be viewed.

*changed*

x265: support for '--[no-]slow-firstpass'

*fixed*

input: ffmpeg addressing video id and temp file detection

-> downloads: http://www.selur.de/downloads

Cu Selur

[Music Fan]

I guess this version also includes the fix about the detected framerate when opening avs scripts.

[Selur]

yeah, forgot to mention that. :)

[Music Fan]

There is a bug in rev 2014.08.06.1 : nothing happens when I click on + (add current) in audio tab after having opened a wav and changed pitch.

[Selur]

will look into it tomorrow :)

[Music Fan]

Actually I found what was wrong : I had to open first the wav in the main window and not directly in audio tab, then I click on +, go back to main window, create output file, add it to queue, go in jobs tab.;)

[Selur]

Ah, okay.
Yes, the Audiotab is always ment as addition to the main input.
-> no Main-Input, no Audio processing. ;)

[Music Fan]

Ok, so the file opening in Audiotab is only done to choose another audio track than the one integrated in the video file opened in main window (or to add an audio track when there is only video) ?

[Selur]

Exactly. :) If you got an Audio only sources, open them in the main tab.

[Selur]

@digital.h: posted a temporary solution for the selection stuff, see: http://forum.doom9.org/showthread.php?p=1689823#post1689823
So in theory I could now write some sort of 'compression test'.
Remaining questions are:

What to do with the X% encode of the source? (delete it after some analysis?)
What infos would a user want?
Only thing that pops into my mind atm. is "estimated output size of the whole file" which would be:
estimatedSize = (outputSizeOfEncodedChunks/X)* 100

[DJ-1]

Hi, think I found a bug with VFR inputs,

running default Global settings
Video = x264
audio = Passthrough All

x264 preset = x264 Hybrid default

VFR is detected by Hybrid as per the info pane and when encode is finnished I get `Constant frame rate mode output`

(I did check the "config/Input tab" and CFR is unchecked.)

debug: https://www.dropbox.com/s/0povdnh2kwjcz72/Output%20constant.txt

[Selur]

Looking at the log:
1. Hybrid detect the content as vfr
2. Hybrid extracts the time codes and checks if they really indicate vfr (they do)
3. Hybrid reencodes the video
4. Hybrid muxes the video
5. seems like Hybrid doesn't add the time codes again (normally a mp4fpsmod call your be there)
-> will look into it

=> seems like I accidentally commented the line which adds the time codes when reencoding with x264 out. :)

[DJ-1]

tried test version and still the same, cfr output

debug: https://www.dropbox.com/s/molfn077cvnfmt0/1st%20test%20version.txt

[Selur]

According to the log file, Hybrid:

1. reencoded the video
2. muxed the video into an mp4 container using mp4box
3. used mp4fpsmod to add the time codes from the input to the output without any problem
4. added the audiostream to the mp4 using mp4box
-> I see no problem,...

Is the output async?

[DJ-1]

hmmm, just updated mediainfo (was promting me for an update..)

on the output file mediainfo shows

Frame rate mode : Constant
Frame rate : 23.976 fps
Original frame rate : 25.000 fps
...also if i use the output .mp4 as source in Hybrid its detected as CFR



the source file is

Frame rate mode : Variable
Frame rate : 23.976 fps
Minimum frame rate : 23.974 fps
Maximum frame rate : 23.981 fps

[Selur]

since you use avisynth, try if it changes if you use LibavSource instead of FFVideoSource

[DJ-1]

...sorry, cant find that yet, which tab is that under ?

[Selur]

Filtering->Avisynth->Misc->Misc->LibavVideoSource instead of FFmpegSource

[DJ-1]

same, with the same mediainfo details also....

incase you want it for comparison or something..

debug:https://www.dropbox.com/s/03hkel297zrovuv/Libavsouce%20checked.txt

[Selur]

during that encode avisynth isn't used,...

[Selur]

Also:
1. What happens if you enable "Config->Automation->On Load->Always extract timecoides form mp4/mkv/flv files" and then load the reencode?
Hybrid should then check the time codes.
2. Still waiting for the answer to: "Is the output async? "

[DJ-1]

Doh, yeah forgot to enable avisynth:scared:

missed the question about async, I have a look in a min, but its only a 4 min clip.....usually vfr source that ends up as cfr starts in sync but slowly drifts, not sure if It`ll be noticeable on this clip.


will try the timecode setting in a mo....

[DJ-1]

enabled the `extract timecodes` setting loaded the source and still reports cfr 23.976 (in the hybrid info panel.)

didnt see any async issues, but maybe down to the clip duration..

avisynth should be active now:

debug:https://www.dropbox.com/s/6p4d3ud93vkau3l/avisynth%20active%20and%20Libav.txt

[Selur]

Uploaded a new version which still produces the same mediainfo output, funny thing is, that if I simply passthrough the video, audio and timecode streams this still happens.
-> my guess is that this isn't really a Hybrid bug, but just one of many cases where MediaInfos vfr detection isn't that good.

[DJ-1]

hmmm ok cheers, I got the version, only enabled the `extract timecodes from source` option.

Ran a test, detected as cfr same....
https://www.dropbox.com/s/6sc6db0848vte6h/new%20test%20verson.txt

are there any alternative to Mediainfo ?


just a thought, is there a chance of having an option to `force VFR` ?( like the CFR output option thats pretty much "force mode").

I cant find the original longer source vfr clip i had so i had to find a short clip online, but couldnt find much `sample vfr clips`, do you have a longer vfr clip i could try so i can try an async test by eye after a test ?

cheers.

[Selur]

a. Got a sample on my GoogleDrive (https://drive.google.com/folderview?id=0B_WxUS1XGCPASUZibG5XZkRfeTg&usp=sharing) which I like to take for vfr testing.
b. No Hybrid got no 'force VFR' option and will probably not get one since vfr is rarely worth the effort since newer codecs don't really waste much space when encoding duplicates.

are there any alternative to Mediainfo ?
Not really and MediaInfo got some limits due to the way it works. (vfr detection is one of them since it only checks the first few time codes if those have the same distance content is assumed to be cfr otherwise vfr)
Only way to reliably know if a file is vfr or cfr is to look at the time codes.
If you enable "Config->Automation->On Load->Always extract timecoides form mp4/mkv/flv files" and load an mkv/mp4/flv file Hybrid will extract the time codes and check if they indicate vfr or not, if they don't Hybrid will output: "timecodes from input do not indicate vfr and will therefore be ignored."

You can also manually extract time codes yourself using flvextract for .flv, mkvextract for .mkv and mp4fpsmod for .mp4 files

Cu Selur

[DJ-1]

ok cheers.

[digitall.h]

@digital.h: posted a temporary solution for the selection stuff, see: http://forum.doom9.org/showthread.php?p=1689823#post1689823
So in theory I could now write some sort of 'compression test'.
Remaining questions are:

What to do with the X% encode of the source? (delete it after some analysis?)
What infos would a user want?
Only thing that pops into my mind atm. is "estimated output size of the whole file" which would be:
estimatedSize = (outputSizeOfEncodedChunks/X)* 100

Great!.
I already run some tests with your previous ffmpeg code. I encoded 2% of the film, and even it is not perfectly accurate (I would say that encoded test is a little oversized) it is very useful, when encoding in crf mode, to anticipate the approximate final x264 file size.

The 'easiest' thing to do would be the one you suggest: encode the requested X% of the movie, using Y seconds chunk size, and show the user (in a window?) what would be the final file size with given encoder settings. And then delete the test file or not (it could be an option to be selected in the GUI). I also imagine this could be a functionality in the x264 encoder tab, with two input items (for % and chunck size in seconds) and a 'Go test!' button...

The more complete solution would be to ask the user the % of the movie to be encoded (the bigger the more accurate), the chunck length in seconds, and the approximate final video (or mkv/mp4) file size. Then an internal routine would run tests changing the crf value until finding the crf value that generates the test file with the size that predicts the requested final size. And would keep the test file encoded with the crf value that will probably generate the file with the requested size.
:D

It is easy to have ideas. It is difficult to code them. But as you asked...
:p
;)

This is a functionality that could be added to Hybrid. But if you think that it is not a matter to be discussed in this thread, please let me know and point me in the right direction.

[Selur]

The more complete solution would be to ask the user the % of the movie to be encoded (the bigger the more accurate), the chunck length in seconds, and the approximate final video (or mkv/mp4) file size. Then an internal routine would run tests changing the crf value until finding the crf value that generates the test file with the size that predicts the requested final size. And would keep the test file encoded with the crf value that will probably generate the file with the requested size.
to me: that does not make any sense
If you aim for a specific size use 2pass encoding, which probably is faster than multiple times encoding 5% with full settings.

[digitall.h]

Of course, to target an specific size is not the objective.

But in crf mode sometimes you get a really oversized file, that you have to reencode. This functionality would help to avoid it.
In my tests I did not even use crf decimal values, just tested crf=20, 21 or 22.
And when I speak about a target file size I'm really referring to an approximate final file size. If I encode a file with 14 GB size, I wonder if I can generate a 10 GB file while keeping good quality. If I make this choice before the final (whole movie) encoding, looking at the test file, I will save time and CPU life time.
:D

[Selur]

-> Don't think anything like that will come anytime soon.
May be I'll add an option to create video only jobs which encode x% of the input, using y-sec chunks, but that's probably it.
(the normal log will show the final size of the output, but the whole estimating stuff would be up to the user)

[digitall.h]

It would be enough.
If nobody else shows interest in this kind of functionality, I don't think it does merit further effort.

Would they be just video test-encoding jobs?, would it keep the test for further analysis?.

[Selur]

Would they be just video test-encoding jobs?,
yes, it would be video only (cutting subtitles&co would just slow things down a lot and not really help)

would it keep the test for further analysis?.
What kind of analysis? Atm. I see no further use for the output file, so Hybrid would delete it.

[digitall.h]

It would be useful to play the test file and evaluate the quality at a given crf value.

[Selur]

Hybrid rev 2014.08.25.1:

*added*

audio: option to not cut external audio
audio: don't use DGIndex when for audio extraction when chapters are selected


*removed*

encoder: support for cuda encoder since latest drivers dropped support for nvcuenc (which is the old api), will add a new cuda encoder when I stumble over nvcenc using command line application or I write one myself; for those who still use old nvida drivers I added a misc.ini option to enable cuda support
avisynth: DGDecNV got removed due to request from the author, see: http://rationalqm.us/board/viewtopic.php?f=12&t=364
It's still supported, but the files that belong to it now have to be manually downloaded and added to the avisynthPlugins-folder for Hybrid to be able to use it.


*fixed*

audio: using raw external audio and mkv a/v input
input: wrong avs frame rate detection on decimal frame rates
cut: mkvmerge cut/split call
x265: 2pass fast 1st pass
x264: time codes accidentally got ignored when encoding with x264
decode: mencoder->ffmpeg->encoder call, removed map-call inside ffmpeg call
cuda: cq encoding


*changed*

subtitle: subtitle tab show grab language from file name in sidebar


-> downloads: http://www.selur.de/downloads

Cu Selur

[digitall.h]

Latest version Linux 64bit 2014.05.25.1 working great here.
:D
I don't know if it was there before, but I see now, in crf mode (I don't know in other modes) 'estimated size' when encoding x264, good!.

It was just first encode, but I realised that with a bluray source, video encoder x264, crf mode, preset very slow and tune film, it now uses 5 reference frames instead of the 4 ref frames it used. Did you change anything in the command Hybrid passes to x264, or was it a change in x264 presets and tunes?.

[Selur]

2014.05.25.1
you meant, 2014.08.25.1, right?

I haven't change anything and afaik. the presets&tunes haven't changed either. My guess is that you changed something in your settings. :)

[digitall.h]

Of course, you're right, it's 2014.08.25.1 (a typo...)

I would swear a did not change anything.
Anyway I did a second encode, applying the same settings, and the x264 encoding command line now shows 4 reference frames...
Mysterious happenings...
I'll keep trying and report back
;)

It happened again.
Initially it was showing correctly --ref 4. But after some tweaking and loading again profile, preset and tune, it shows --ref 5.
I checked in ~/.hybrid my profile and there everything is Ok.
Send you debug, in case it helps...

[Music Fan]

Hi Selur,
x264 encodings crash with Lagarith videos in avi container, whatever I open directly the avi in Hybrid or via an avs script.
Here is a debug file level 4 ;
http://paste2.org/dhHMp2Oh

[Selur]

before I look at it, try if it works if you use the x264 version from http://download.videolan.org/x264/binaries/

[Music Fan]

Which version ?
Could you just look the end of the debug file before I try to change x264 ?

[Selur]

Which version ?
latest matching your OS

Looked at the end:
"C:\PROGRA~1\Hybrid\mencoder.exe" -lavdopts threads=4 -really-quiet -of rawvideo -o - -ovc raw -noskip -vf scale,format=444p,scale,format=i420 -forcedsubsonly -nosub -nosound -mc 0 "F:\ZOOCUT~2.AVS" | "C:\PROGRA~1\Hybrid\x264.exe" --pass 1 --slow-firstpass --bitrate 2000 --profile high --level 4.0 --ref 4 --keyint 100 --qcomp 0.5 --no-mbtree --tff --nal-hrd vbr --partitions i4x4,p8x8,b8x8 --no-fast-pskip --subme 5 --trellis 0 --weightp 0 --aq-mode 0 --non-deterministic --stats "E:\t2_12_09_18_9710_01.stats" --input-csp i420 --fps 25000/1000 --input-res 720x576 --output "E:\t2.264" -
the only thing strange is that the input seems to be 444 and you convert it to 420.
Since I have no clue what is inside the .avs you feed Hybrid with I can't say more.

[Music Fan]

latest matching your OS
Ok but can I take a version from win 64 folder if I use Avisynth 32 bit ?

the only thing strange is that the input seems to be 444 and you convert it to 420.
Since I have no clue what is inside the .avs you feed Hybrid with I can't say more.
Strange because it's a 4:2:0 file and I didn't change it in the script, I just wrote avisource("F:\file.avi").assumetff().
Here is the debug file when I open the avi in Hybrid ;
http://paste2.org/IyUaVt4a

Is this normal (line 858) ;
"C:\PROGRA~1\Hybrid\mencoder.exe" -lavdopts threads=4 -really-quiet -of rawvideo -o - -ovc raw -vfm ffmpeg -noskip -vf scale,format=i420
I'm talking about the - without anything after, between -o and -ovc raw.

[Selur]

Ok but can I take a version from win 64 folder if I use Avisynth 32 bit ?
Since Hybrid uses mencoder for decoding: Yes

Is this normal (line 858)
Yes.

I just wrote avisource("F:\file.avi").assumetff().
a. add 'ConvertToYv12()' to the end
b. check the output of avsInfo when you feed it with the .avs file

Starting Main@12:04:06.053:
"C:\PROGRA~1\Hybrid\x264.exe" --pass 1 --slow-firstpass --bitrate 2000 --profile high --level 4.0 --ref 4 --keyint 100 --qcomp 0.5 --no-mbtree --tff --nal-hrd vbr --partitions i4x4,p8x8,b8x8 --no-fast-pskip --subme 5 --trellis 0 --weightp 0 --aq-mode 0 --non-deterministic --stats "E:\test_12_03_56_4810_01.stats" --input-csp i420 --fps 25000/1000 --input-res 720x576 --output "E:\test.264" -
2014.09.19 - 12:04:06_Windows 7-64bit_2014.08.25.1 - level 4: Main call started,..
2014.09.19 - 12:04:06_Windows 7-64bit_2014.08.25.1 - level 4: Starting Helper:
"C:\PROGRA~1\Hybrid\mencoder.exe" -lavdopts threads=4 -really-quiet -of rawvideo -o - -ovc raw -vfm ffmpeg -noskip -vf scale,format=i420 -forcedsubsonly -nosub -nosound -mc 0 "F:\ZOOCUT~2.AVI"
2014.09.19 - 12:04:07_Windows 7-64bit_2014.08.25.1 - level 4: Helper call 1 started,..
2014.09.19 - 12:04:07_Windows 7-64bit_2014.08.25.1 - level 4: startJobs,...
2014.09.19 - 12:04:07_Windows 7-64bit_2014.08.25.1 - level 4: postponeJobs:
2014.09.19 - 12:04:07_Windows 7-64bit_2014.08.25.1 - level 4: Queue is empty,...
2014.09.19 - 12:04:07_Windows 7-64bit_2014.08.25.1 - level 4: No parentjobs -> nothing to do
2014.09.19 - 12:04:07_Windows 7-64bit_2014.08.25.1 - level 4: 12_03_56_4810_01_video ### F:\test.mkv ### x264 [error]: could not open output file `E:\test.264'
2014.09.19 - 12:04:07_Windows 7-64bit_2014.08.25.1 - level 4: 12_03_56_4810_01_video ### F:\test.mkv ### finished after 00:00:01.526 with exitCode -1
2014.09.19 - 12:04:07_Windows 7-64bit_2014.08.25.1 - level 4: -> 12_03_56_4810_01_video crashed:Crash: 0
-> seems like x264 can't write to E:\test.264

Cu Selur

Ps.: Also do not forget to check the x264 version, there have been other users having issues with the x264 version which atm. comes with Hybrid,...

[Music Fan]

Thanks but I discovered that all files crashed, I searched why and I found a few minutes ago that the problem was actually the temp folder which was E: (the root, no folder actually), probably because I have 2 OS (XP and 7) in dual boot and E: contains XP (E: is actually C: when running XP and becomes E: when running 7) but I believed it didn't matter. I don't why Win 7 can't write on this partition.
I probably changed temp folder's path some time ago without doing encodings at this moment, that's why I didn't realize before that this path was problematic.
Everything is ok now.