h264enc for Linux

[Rasi]

The movie is 01:59:23...

The numbers above were guesses: Here are the real numbers i used:

262 mb for audio/subtitles - resulting in 1138 mb for video.
The lenth of 1:59:23 results in:
1138 * 8192 / 7163 = 1301.4 - which is ~100 MB less.

[microchip8]

Hi Rasi,

I think you are doing something wrong here, or maybe misinterpreting. h264enc calculates the video bitrate for a target filesize. You give it a target filesize in Megabytes and it gives you a bitrate value which will reach this target filesize.

so in your case, you want 1138 MB (this is your target filesize) * 8192 / 7163 (the movie length in seconds) and h264enc has calculated a video bitrate of 1301 kbps and presented to you so you can keep or change it. If you keep it, the 1301 kbps value will reach the requested 1138 MB filesize. Why? because if you calculate the target filesize from the the video bitrate (1301 kbps in this case) and movie length it comes at about 1137.5 MB, eg

1301 * 7163 / 8192 = 1137.5 MB

below is the formula

VIDEO_BITRATE * MOVIE_LENGTH_IN_SEC / 8192

so in this case, the calculated video bitrate (1301 kbps) by h264enc is correct.

[Rasi]

Originally Posted by froggy1

Hi Rasi,

I think you are doing something wrong here, or maybe misinterpreting. h264enc calculates the video bitrate for a target filesize. You give it a target filesize in Megabytes and it gives you a bitrate value which will reach this target filesize.

so in your case, you want 1138 MB (this is your target filesize) * 8192 / 7163 (the movie length in seconds) and h264enc has calculated a video bitrate of 1301 kbps and presented to you so you can keep or change it. If you keep it, the 1301 kbps value will reach the requested 1138 MB filesize. Why? because if you calculate the target filesize from the the video bitrate (1301 kbps in this case) and movie length it comes at about 1137.5 MB, eg

1301 * 7163 / 8192 = 1137.5 MB

below is the formula

VIDEO_BITRATE * MOVIE_LENGTH_IN_SEC / 8192

so in this case, the calculated video bitrate (1301 kbps) by h264enc is correct.

I cant check now, since the logs have been deleted, but i simply hit return when it asked me for bitrate (which was prefilled with the calculated one) - iirc

[microchip8]

Well, I cannot see anything wrong with the way h264enc calculates this (or maybe I'm just retarded at math and need to go to math class). Like I said before, I always get the correct target filesize when I let h264enc do the calculation, whether I encode with or without audio, it doesn't matter.

[Rasi]

Ok.. i tried it again and i made sure that i tell h264enc to use the 1301 bps.
After adding my audio and subtitles to the movie, the file is still 1.3 GB...


Same with another movie, here i accidently told h264enc to hit 1,4 GB with no sound and no subtitles checked. It told me to use around 1850 bps. I realised that i told h264enc the wrong filesize (since i had to substract my audio files) and calculated the bitrate manually. I told h264enc my new bitrate and the resulting file was AGAIN 1.3 GB

My versions:

x264 0.60.0+891 2ca4593
MEncoder 1.0rc2-4.3.2
h264enc 8.5.4


*edit* actually its not THAT bad. dolphin just says 1.3GB. The actual file size is:

1438024127 bytes which is
1371.4 megabytes

But still this is 28.6 MB off my desired filesize

[Rasi]

Another try, this time i will let you know what i am doing by providing a shot




In the lower right you see the files that will be added to the final mkv file. As you can see they are 223MB big, which means my final video file has to be 1177MB

This *should* give me a video that is ~1177MB, right? I will update this post tommorow.



*edit* and here is the result, AGAIN, 26MB smaller:

[microchip8]

Hi Rasi,

Yes, I see it. I don't know why this is happening though. I also did a few tests myself yesterday and compared h264enc's calculation method to the one used in Avidemux. For a test sample (without audio) of 299 seconds (00:04:59) and target filesize of 1177 MB, h264enc gave me 32247 kbps while Avidemux gave me 32883 kbps - that's 636 kbps higher than h264enc. I have no idea how Avidemux calculates the values, but if you do 32883 * 299 / 8192 = 1209 MB, which is higher than the requested. If you take h264enc's value and do 32247 * 299 / 8192 = 1176 MB which almost hits the requested filesize. Also keep in mind that if the bitrate is high enough, depending on the movie, the encoder may not need to use the full bitrate thus lowering it which results in smaller filesizes.

I do not know any other formula to calculate the target filesize, besides the one used in h264enc. If you know a better one, please tell me. If you keep getting files which are 20-30 MB smaller than the requested, try compensating for this by providing a target filesize which is 20-30 MB higher than the one you want. Also most people will be happy that their files end up being smaller than intended. In your case it is 26 MB smaller, is that such a huge problem?

[Rasi]

It's no problem at all.. Mainly i wanted to report a possible bug

Plus that i am used to hit exactly the filesize i want from megui

[microchip8]

Will have to look how MeGUI is doing it, if the author provides the source code of course...

[microchip8]

@ Rasi

been trying all day long with various formulas (my head hurts lol), none of them seems to hit the exact target size. I asked buzzqw (dev of AutoMKV) for his formula and the calculated bitrate comes extremely close to the one in avidemux (within 10-20 kbps). However, even this formula misses the target by 25 MB ( requested size: 1177 MB, got 1152 MB with is 25 MB less than requested). I think you'll have to compensate for that, sorry :/

[buzzqw]

on h264enc are you encoding to h264 or directly to mp4/mkv ?

also.. mkv/mp4 and have a less overhead then avi (the 24bytes*frames).. you can rise the bitrate * 1.005

good work froggy1!

BHH

[microchip8]

Hi buzzqw

h264enc outputs directly to AVI, then remuxes it to mkv/mp4. I'm still doing some tests here. Will see how it turns out

[microchip8]

ChangeLog for version 8.5.9

* Rewrote the DVD part of the video_subtitles_func() function. It now supports
dumping two subtitles to disk
* Updated the MP4/MKV muxing code to support importing of two subtitles
* Dramatically simplified the MP4/MKV/OGM batch exporting code. This reduces a
lot of code duplication
* Bugfix in the video_subtitles_func() function for SRT subtitles for video files. Wrong variable used
* When encoding samples, make sure to take care of any subtitles that might have been dumped to disk
* Don't exit if the user provides a floating point value for the Vorbis audio codec. Instead warn 
and use the default value.
* New formula to calculate the video bitrate for a target filesize. The previous formula, which was
directly taken from the ripdvd project, incorrectly calculates the bitrate which often results by
missing the target filesize by as much as 30 MB. The new formula also uses floating point precision
and has been tested agains the calculators of MeGUI and Avidemux
* Removed support for me-prepass
* Removed neroAacEnc quality-based encoding mode. It is incompatible with video bitrate calculation
for a target size and I cannot find any info on quality -> average bitrate mappings.
* Updated the HW compatible presets. Changed H.264 Level for the SD presets from 3.1 to 3.0 and
for the HD presets from 4.1 to 4.0. Also updated the vbv_maxrate values according to Level specs
* Restructured the Blu-ray presets. Now there are four different Blu-ray presets available: bd40, bdhq40,
bd41, bdhq41. bd40 is Main profile @ Level 4.0, bdhq40 is High profile @ Level 4.0, bd41 is Main
profile @ Level 4.1 and bdhq41 is High profile @ Level 4.1
* Allow CRF encoding to be used with some device presets (ps3, xbox360, appletv)
* Updated the man page
* Some cleanups

[mocham]

froggy,

I kind of talked to you in another thread about mencoder batch GUIs/scripts, and the lack thereof under Linux. Your script is the most useful frontend for mencoder on Linux, but really only friendly to single file jobs. I wanted to ask you if you considered adding any features to make batch encoding with your script more flexible? I was thinking if it could do something like encode all the files located in a specified directory, tell h264enc I want to encode a "directory" of files, go though the y/n options, accept those as the default for all the files to be encoded, resulting file names will just be the original file names with appropriate extensions, then to avoid any problems with duplicate file names or overwriting just put all the encoded files into a "h264enc" sub-directory of the directory that was chosen. Is this a doable feature request with your script? Thanks.

[microchip8]

Originally Posted by mocham

froggy,

I kind of talked to you in another thread about mencoder batch GUIs/scripts, and the lack thereof under Linux. Your script is the most useful frontend for mencoder on Linux, but really only friendly to single file jobs. I wanted to ask you if you considered adding any features to make batch encoding with your script more flexible? I was thinking if it could do something like encode all the files located in a specified directory, tell h264enc I want to encode a "directory" of files, go though the y/n options, accept those as the default for all the files to be encoded, resulting file names will just be the original file names with appropriate extensions, then to avoid any problems with duplicate file names or overwriting just put all the encoded files into a "h264enc" sub-directory of the directory that was chosen. Is this a doable feature request with your script? Thanks.

Hi,

I have no idea if this is doable in the code itself at the moment (I think it may be possible). I will have to look though. BUT you can already do such a thing by modifying the batch file and use a "for loop". For example, you configure one of the files you want to encode. Save the settings to the batch file, open it up and add a loop to so it will look something like this

################### START OF COMMANDS ###################

# Change to dir with video files
cd /path/to/dir/with/video/files

# Make an output dir inside the dir with video files
mkdir -p h264enc

for i in *.avi; do

test -d "/home/microchip" || mkdir -p "/home/microchip"
mkdir -p "/home/microchip/.h264enc/$i"
test -e "/home/microchip/H264-20485.avi" && mv -f "/home/microchip/H264-20485.avi" "/home/microchip/H264-20485.avi.old"

/usr/local/bin/mencoder "$i" -o /dev/null -vf softskip,harddup -oac faac -faacopts mpeg=4:br=110:object=2 -ovc x264 -x264encopts bitrate=1000:me=umh:me_range=24:nodct_decimate:nointerlaced:8x8dct:nofast_pskip:trellis=1:partitions=p8x8,b8x8,i8x8,i4x4:mixed_refs:keyint=250:keyint_min=25:psy_rd=0.5,0.3:frameref=4:bframes=16:b_adapt=1:b_pyramid:weight_b:direct_pred=auto:subq=7:chroma_me:cabac:deblock:level_idc=41:threads=auto:ssim:psnr

/usr/local/bin/mencoder "$i" -o "h264enc/$i" -vf softskip,harddup -oac faac -faacopts mpeg=4:br=110:object=2 -ovc x264 -x264encopts bitrate=1000:me=umh:me_range=24:nodct_decimate:nointerlaced:8x8dct:nofast_pskip:trellis=1:partitions=p8x8,b8x8,i8x8,i4x4:mixed_refs:keyint=250:keyint_min=25:psy_rd=0.5,0.3:frameref=4:bframes=16:b_adapt=1:b_pyramid:weight_b:direct_pred=auto:subq=7:chroma_me:cabac:deblock:level_idc=41:threads=auto:ssim:psnr -passlogfile "/home/microchip/.h264enc/$i/h264.log"

sleep 3

test -e "/home/microchip/H264-20485.mkv" && mv -f "/home/microchip/H264-20485.mkv" "/home/microchip/H264-20485.mkv.old"
/usr/local/bin/mkvmerge  "$i"    -o "h264enc/$i.mkv"
sleep 3

/usr/local/bin/MP4Box -aviraw video "$i" -out "/home/microchip/.h264enc/$i/h264.h264"
/usr/local/bin/MP4Box -aviraw audio "$i" -out "/home/microchip/.h264enc/$i/aac.raw"
mv -f "/home/microchip/.h264enc/$i/*.raw" "/home/microchip/.h264enc/$i/audio.aac"
test -e "/home/microchip/H264-20485.mp4" && mv -f "/home/microchip/H264-20485.mp4" "/home/microchip/H264-20485.mp4.old"
/usr/local/bin/MP4Box -fps $(/usr/local/bin/mplayer "$i" -identify -nosound -vo null -nocache -frames 1 | grep '^ID_VIDEO_FPS' | tail -1 | awk -F= '{print $2}') -tmp /home/microchip/.h264enc -add "/home/microchip/.h264enc/$i/*.h264" -add "/home/microchip/.h264enc/$i/*.aac"   "h264enc/$i.mp4"

sleep 3

rm -rf "/home/microchip/.h264enc/$i"

done

#################### END OF COMMANDS ####################

This is an example of a two-pass encoding with mkv/mp4 remuxing. Everything set in bold is modified/changed to work with a "for loop". The example also assumes that you stick to using a MEncoder audio codec instead of using an external audio encoder like neorAacEnc or oggenc or aacplusenc. It also assumes that all the files you want to encode in the directory are AVI files (though you can add very easily others to the loop too)

The output of each file will be stored in the h264enc directory inside the directory with video files you want to encode. If remuxing to mkv/mp4 the mkv's or mp4's will be named something like file.avi.mkv or file.avi.mp4 so you'll have to remove the .avi extensions yourself

If this is too much, you can always configure one file, save the options to the batch file and post it here. I will modify it for you

Also make sure you have enough disk space if you're also going to remux to mp4/mkv/whatever

The above is just and example and I have not tested it, but it should work

[mocham]

This is what I was thinking of, a for loop. I just thought it could be implemented within the script. I'll give this a try. Thanks.

[microchip8]

It may be possible to implement this, however I have to look since I deal with a lot of audio codecs and container formats, so it won't be that easy... and I need a little break too, been working on h264enc for the past one and a half week

you may also want to try Autox264 from buzzqw, which IIRC has a Queue where you can add jobs

[microchip8]

ChangeLog for version 8.6.0

* Bugfix: add the $ofps variable to MEncoder when dumping a second subtitle.
Also enable pcm as audio to maintain sync with the video. A small performance hit but it's worth it
* Added support for dumping one more subtitle. This brings it to a total of 3 subtitles that can be dumped
to disk and imported into the MKV and/or MP4 container
* Dramatically simplified the TS/Blu-ray muxing code
* Implemented initial real batch support for files. As of now, one can encode a bunch of video files in a
directory and automatically remux them to MKV/MP4/OGM. The TS container is not supported yet due
to me being lazy. The user only needs to configure one of the files in a given directory, and all the settings
will apply to the other files as well
* Moved the DVD device detection code into its own function to reduce some code duplication
* Made the encoding priority level global. It now applies to audio/video dumping, audio encoding, subtitles
dumping and container muxing

[mocham]

Originally Posted by froggy1

ChangeLog for version 8.6.0

* Implemented initial real batch support for files. As of now, one can encode a bunch of video files in a
directory and automatically remux them to MKV/MP4/OGM. The TS container is not supported yet due
to me being lazy. The user only needs to configure one of the files in a given directory, and all the settings
will apply to the other files as well

Cool! Seems to be working on some test files I did. Thanks!

[microchip8]

ChangeLog for version 8.6.1

* Empty variables cleanups 
* Bugfix: the vobsubout3 variable was not tested in the MKV/MP4 muxing code. This leads
to the third subtitle not being imported into these containers 
* Added support for TS/Blu-ray muxing when batch encoding files in a directory 
* Small improvements for batch encoding of video files in a directory 
* New formulas to calculate the video bitrate for a target filesize. The user can choose between
MeGUI's formula, h264enc's refactored MeGUI formula and the old h264enc/ripdvd formula