A few queries about x264 CLI setup

[jdlk]

My primary use for the tools we discuss here is to batch process MPEG2 files (TV recordings) that I want to archive. I'm sure many here do the same.

I'm now in the process of moving over to H.264 (x264) for archiving -- until now I have used ASP (XviD). The resulting files have to be usable on both my TV and my Archos 605. The Archos now deals with H.264 pretty well, but still can't handle High Profile or the MKV container, just MP4.

For the last couple of years I've been using the following options to obtain satisfactory AVI files:


-sws 9 -vf scale=624:352,harddup -ovc xvid
-xvidencopts fixed_quant=3 max_bframes=1 packed
-oac mp3lame -lameopts vbr=0 br=128
For the last few days I've been playing with HandbrakeCLI and I've ended up with this option list:

-e x264 -f mp4 --rate 25
-q 0.588 [means crf=21 in HandbrakeCLI-speak]
-x bframes=3:weightb:subme=7:ref=3:threads=2:mixed-refs:
no-dct-decimate:no-fast-pskip:keyint=250
--color-matrix 601 --width 624 --height 352
--aencoder faac --ab 128
My questions: Firstly, does this look a reasonable setup, or are there any schoolboy errors? I'm new to H.264.

Secondly, the files are processing at about 25fps, compared to about 50fps for my xvid settings. I'm perfectly happy with that so long as my setup is okay, but if there is anything in my options that is drastically slowing down my x264 processing for little benefit, then I'd be grateful for guidance. (I realise the answer here is to try removing everything myself one-by-one and seeing what happens, but many threads here mention the inter-dependency of x264 options (particularly when using crf), so it's a bit bewildering for x264 beginners).

Finally, as I said above, I've been using HandbrakeCLI, which seems to be one of the very few "all-in-one" H.264 CLI tools that I can use to batch process MPEG2 files in the same way I can use Mencoder for XviD (i.e. Mencoder/Handbrake automatically call the necessary tools for audio encoding/muxing etc).

When you try to create MP4 files in Mencoder, it flashes up lots of warnings about its broken Libavformat implementations, don't use bframes etc, so it seems I should avoid Mencoder. Are there alternative all-in-one H.264 CLI tools (I don't want to use a GUI, just batch files and a CLI tool).

[audyovydeo]

My primary use for the tools we discuss here is to batch process MPEG2 files that I want to archive. I'm sure many here do the same.

I'm now in the process of switching from ASP (XviD) to H.264 (x264). The resulting files have to be usable on both my TV and my Archos 605. The Archos now deals with H.264 pretty well, but still can't handle High Profile or the MKV container, just MP4.



I don't use Handbrake at all, so I'll let others answer, but, as one who has been tempted to do exactly what you are doing, allow me to question the usefulness of transcoding in general, and ASP to AVC in particular :

you know you are going to lose quality, no matter how little,
you no longer care about storage space (cost/GB is peanuts compared to when you were busy encoding with DivX).


So, while it makes some sense to transcode MPEG2, why dont you just demux/remux your ASP.avi to ASP.mp4 files ?

cheers
audyovydeo

[jdlk]

I'll let others answer on the technicalities, but, as one who has been tempted to do exactly what you are doing, allow me to question the usefulness of transcoding in general, and ASP to AVC in particular.

Sorry, I obviously didn't make myself clear. I have no intention of transcoding my ASP files to AVC. My XviD files will remain as they are. I just plan to transcode new MPEG2 files to H.264 rather than ASP. Thanks for your response.

[audyovydeo]

Sorry, I obviously didn't make myself clear. I have no intention of trancoding my ASP files to AVC. My XviD files will remain as they are. I just plan to transcode new MPEG2 files to H.264 rather than ASP.


While I now prefer DGxxxIndex and x264 cli, I have tried and liked Avidemux in the past. You might want to give it a try.

As far as the cmdline is concerned, there isn't really a one-set-of-options-fits-all, if anything because the single most important factor is the type of source : cartoons? b/w french intellectual movie ? high-action-packed ?

when I transcode, I tend to use my own "High profile" + "Best" settings, where :


set intra=-8 -m 8 -t 1 -I 1 --direct spatial
set base=-A i4x4,p4x4,p8x8 -t 0 --no-cabac
set main=-A i4x4,p4x4,p8x8,b8x8 -t 1 -b 16 --b-pyramid -w
set high=-A all -8 -t 1 -b 16 --b-pyramid -w

and

::set the mode
set none=--aq-mode 0
set fast=-r 1 --me hex -m 5 --direct auto --aq-mode 0
set good=-r 3 --me hex -m 5 --direct auto --aq-mode 0
set best=-r 4 --me umh -m 5 --direct auto --aq-mode 0
set blast=-r 4 --me esa -m 7 --direct auto --aq-mode 0
set psy=-r 4 --me esa -m 7 --direct auto --psy-rd 1.0:0.0 --aq-mode 1 --aq-strength 1.0


with a --crf of 18 to 22, according to how much time I'm willing to waste on any given movie (special case : anything with Pam Grier gets crf 16)


cheers
audyovydeo

[nm]

For the last few days I've been playing with HandbrakeCLI and I've ended up with this option list:

-e x264 -f mp4 --rate 25
-q 0.588 [means crf=21 in HandbrakeCLI-speak]
-x bframes=3:weightb:subme=7:ref=3:threads=2:mixed-refs:
no-dct-decimate:no-fast-pskip:keyint=250
--color-matrix 601 --width 624 --height 352
--aencoder faac --ab 128
My questions: Firstly, does this look a reasonable setup, or are there any schoolboy errors? I'm new to H.264.
Looks pretty reasonable to me.

Secondly, the files are processing at about 25fps, compared to about 50fps for my xvid settings. I'm perfectly happy with that so long as my setup is okay, but if there is anything in my options that is drastically slowing down my x264 processing for little benefit, then I'd be grateful for guidance.
no-fast-pskip is such an option, unless the x264 library used by your HandBrake build is so ancient that it doesn't have VAQ.

threads=auto or threads=0 (they mean the same thing) may also have a small speed gain over threads=2 if you have a dual core CPU. Threads=2 is probably optimal on a single-core hyperthreading CPU (P4 HT, single-core Atom models), but threads=3 may be slightly better on a true dual core. Threads=auto selects the optimal number of threads automatically.

You might want to use b-adapt=2. It is slower than the default b-adapt=1, especially with many b-frames, but it should increase quality notably.

When you try to create MP4 files in Mencoder, it flashes up lots of warnings about its broken Libavformat implementations, don't use bframes etc, so it seems I should avoid Mencoder. Are there alternative all-in-one H.264 CLI tools (I don't want to use a GUI, just batch files and a CLI tool).
MEncoder is fine if you only output a raw H.264 stream and use MP4Box to mux it with a separately encoded audio. ffmpeg doesn't have this limit, but using x264 through its command-line interface is torture (now there are some helpful presets though). HandBrakeCLI is a very nice tool, although it doesn't have all the filters and features that MPlayer/MEncoder contains.

[Sagekilla]

2 things: drop no-dct-decimate and no-fast-pskip. no-fast-pskip is just a waste of speed (for very little improvement) and no-dct-decimate is going to bloat your file size unnecessarily. Also, add in 8x8dct: It gives a nice compression boost for practically no cost.

[Blue_MiSfit]

Indeed.

Consider you're switching to AVC which has significantly higher efficiency than ASP.

So, I don't think you need to resize your sources anymore. Unless you're shooting for very low bitrate (why?), you should just leave them as-is, and encode at full frame 720x480 (NTSC) or 720x576(PAL).

I strongly suggest b-adapt=2 and 3 b-frames (provided your hardware supports 3 b-frames). I also recommend subme 9, trellis 1, and psy-trellis (i.e. psy-rd 1.0:0.2)

There's lots of nice GUIs out there. I think Ripbot264 is very good for intermediate users.

Also, isn't FAAC pretty terrible in terms of quality? I'd use LAME if you really want 128kbps audio. If your hardware device ONLY supports AAC in the MP4 container, and you're okay with the quality - fine then :)

Finally, regarding your speed - what are your system specs? Are you getting 100% CPU Utilization when encoding?

~MiSfit

[nm]

Also, add in 8x8dct: It gives a nice compression boost for practically no cost.
jdlk wanted the video to be compatible with Archos 605. which doesn't (currently) support High Profile. Also resolutions over 720x480 may not be supported. If that is true, PAL sources need to be resized at least vertically, but as Blue_MiSfit said, don't go lower than necessary. Your next player might already have better capabilities.

If the player requires AAC audio, you could use neroaacenc (http://www.nero.com/eng/downloads-nerodigital-nero-aac-codec.php) to encode it. It's much better than FAAC, especially at lower bitrates.

[jdlk]

Thank you for all the responses. Much to think about. I will try all the option changes suggested, although as nm says, 8x8dct: is not an option for me as the Archos doesn't support High Profile.

Is FAAC really that bad? I thought that at 128kbps, AAC was so far ahead of MP3 that I didn't even consider sticking with MP3 audio for MP4 files.

HandbrakeCLI only offer FAAC for AAC, so are you suggesting I should stick with Lame for 128kbps audio? Using Nero AAC would entail using multiple programs to generate my MP4 files, which I'm trying to avoid.

Regarding scaling -- I can't agree with audyovydeo's assertion that the cost of storage space is so small that it is now irrelevant. It's certainly relevant to me. I'm up to 3TB of archived video and still growing rapidly. Plus a backup copy. So yes, cost still matters.

For me, the process of compressing files is all about getting files as small as possible, while still giving me a perfectly satisfactory viewing experience.

I did a lot of experimentation when I first started using XviD, and I found that scaling down to 624x352 and using the other settings in my original post gave me a completely satisfactory file (for viewing on my 42" plasma TV at a normal viewing distance).

Hence the reason I automatically opted for the same scaling option in x264.

I can understand people going for the full 720x if they're archiving DVD/HD material, but for SD MPEG2 TV material, scaling down makes sense to me.

[audyovydeo]

Regarding scaling -- I can't agree with audyovydeo's assertion that the cost of storage space is so small that it is now irrelevant. It's certainly relevant to me. I'm up to 3TB of archived video and still growing rapidly. Plus a backup copy. So yes, cost still matters.


3TB ? I guess you're right. But, the rips *are* the backup copies of movie on DVD/BR media, aint it ? ;-)


cheers
a/v

[jdlk]

3TB ? I guess you're right. But, the rips *are* the backup copies of movie on DVD/BR media, aint it ? ;-)

Nope. They're all archived TV recordings. I never watch TV series as they're broadcast. I record a whole series and then watch it. And I have a serious backlog!

[multimediaman]

Is FAAC really that bad? I thought that at 128kbps, AAC was so far ahead of MP3 that I didn't even consider sticking with MP3 audio for MP4 files.

Yes, it is. According to my ears at 64 kbps LAME (and nero-aac) is clearly better than FAAC. It gets narrower at higher bitrates. And I consider myself " deaf" when it comes to audio encoding stuff ;)

Using mencder as video encoder, ffmpeg or mplayer source to pcm decoder, nero as audio encoder and mp4box as muxer can be automated with one simple script.

[jdlk]

Using mencoder as video encoder, ffmpeg or mplayer source to pcm decoder, nero as audio encoder and mp4box as muxer can be automated with one simple script.

Thanks for the advice. I'd like to try this. Are there any beginner's guides around to creating such scripts? I can handle basic CLI stuff, and writing simple batch files to automate the work of a single program, but I would have no idea how to write a script to link the work of the above programs. I'm sure it's all very logical, but a pointer in the right direction would be a great help.

[multimediaman]

Thanks for the advice. I'd like to try this. Are there any beginner's guides around to creating such scripts? I can handle basic CLI stuff, and writing simple batch files to automate the work of a single program, but I would have no idea how to write a script to link the work of the above programs. I'm sure it's all very logical, but a pointer in the right direction would be a great help.
What is your operating system? I can handle bash but I don't have any clue when it comes to windows default dos-like stuff

[audyovydeo]

Thanks for the advice. I'd like to try this. Are there any beginner's guides around to creating such scripts? I can handle basic CLI stuff, and writing simple batch files to automate the work of a single program, but I would have no idea how to write a script to link the work of the above programs. I'm sure it's all very logical, but a pointer in the right direction would be a great help.

I have a system of windows batch files (but Cygwin needed) that takes my camcorder's DV.avi files as input and produces H.264_aac.mp4 files, given the specs I posted above. I'm sure you can add the MPEG2 extraction using neuron2's tools in cmdline mode.
PM me as needed.

cheers
a/v

[jdlk]

What is your operating system? I can handle bash but I don't have any clue when it comes to windows default dos-like stuff
I'm on Windows (XP).

[jdlk]

audyovydeo was kind enough to share his scripts with me, and I spent an hour or so trying to work my way through them and adapt them for my use.

But it all became a bit bewildering, so I decided to go back to what I knew -- writing small batch files. And here's what I came up with. It converts a folder of MPEG2 files to MP4, via DGIndex, FFMpeg, NeroAacEnc, x264 and MP4Box. Amazingly, it seems to work, so I thought I'd share it here.


001 for %%i in ("D:\MPEGS\*.mpg") do DGIndex -HIDE -IA=5 -FO=0 -OM=0 -YR=1 -DRC=0 -DSA=0 -IF=[%%i]
-OFD=[%%~dpi%%~ni] -AT=[D:\Video\template.avs] -EXIT
002 for %%i in ("D:\MPEGS\*.mpg") do ffmpeg -i "%%i" -vn "%%~dpi%%~ni".wav
003 for %%i in ("D:\MPEGS\*.avs") do x264 --crf 21 --bframes 3 --weightb --subme 7 --ref 3 --threads 3
--mixed-refs --keyint 250 --b-adapt 2 --thread-input --progress -o "%%~dpi%%~ni".264 "%%i"
004 for %%i in ("D:\MPEGS\*.wav") do neroAacEnc -ignorelength -q 0.4 -if "%%i" -of "%%~dpi%%~ni"audio.mp4
005 for %%i in ("D:\MPEGS\*.wav") do MP4Box -add "%%~dpi%%~ni".264 -add "%%~dpi%%~ni"audio.mp4 "%%~dpi%%~ni".mp4
006 DEL "D:\MPEGS\*.wav", "D:\MPEGS\*.avs", "D:\MPEGS\*.m2v", "D:\MPEGS\*audio.mp4", "D:\MPEGS\*.264", "D:\MPEGS\*.d2v"
007 move "D:\MPEGS\*.mpg" "D:\MPEGS\processed"
008 move "D:\MPEGS\*.mp4" "D:\Video\MP4"

Lines 6 to 8 just tidy up after processing. And I've removed paths from the script to make it easier to read here -- I have full paths to the various apps and other locations in my original script.

And this is the template for the AVS file (referred to in line 1 above).

DGDecode_mpeg2source("__vid__")
LanczosResize(624,352)
assumefps(25)

As I said in my original post, I'm a beginner with AVC stuff, so any thoughts on the above script would be appreciated. One concern that occurs to me (only based on previous experience with XviD), is whether using VBR for AAC can lead to any sync issues, and also whther I need any additional controls to double-check sync. However the short files I have generated with this script so far seem fine.

[J_Darnley]

FOR %%I IN ("D:\MPEGS\*.mpg") DO (
DGIndex -HIDE -IA=5 -FO=0 -OM=0 -YR=1 -DRC=0 -DSA=0 -IF=[%%~I] -OFD=[%%~dpnI] -AT=[D:\Video\template.avs] -EXIT
ffmpeg -i "%%~I" -vn "%%~dpnI.wav"
x264 --crf 21 --bframes 3 --weightb --subme 7 --ref 3 --threads 3 --mixed-refs --keyint 250 --b-adapt 2 --thread-input --progress -o "%%~dpnI.264 "%%~dpnI.avs"
neroAacEnc -ignorelength -q 0.4 -if "%%~dpnI.wav" -of "%%~dpnI_audio.mp4"
MP4Box -add "%%~dpnI.264" -add "%%~dpnI_audio.mp4" "%%~dpnI.mp4"
DEL "%%~dpnI.wav" "%%~dpnI.avs" "%%~dpnI.m2v" "%%~dpnI_audio.mp4" "%%~dpnI.264" "%%~dpnI.d2v"
MOVE "%%~I" "D:\MPEGS\processed"
MOVE "%%~dpnI.mp4" "D:\Video\MP4"
)
Has cosmetic changes and changes to the dgindex cmd line, it doesn't work if you quote the filename.
Why demux the source video? That is an unnecessary and slower step. If you want more quality from x264 in exchange for speed, you could change/add:
--subme 9
--me umh

[jdlk]

Very nice. Much neater and more compact. Mine is crude in comparison. And yours has the considerable advantage of processing each file in full before moving on to the next one. So I'll steal this!

(BTW, on the x264 line it needs to be "%%~dpnI.264" (missing ")


Why demux the source video? That is an unnecessary and slower step.
I don't know. I thought I needed to. I presume Avisynth takes care of it? (I've never used an Avisynth script before). I've amended my script and removed the demuxing.

[J_Darnley]

Very nice. Much neater and more compact. Mine is crude in comparison. And yours has the considerable advantage of processing each file in full before moving on to the next one. So I'll steal this!

(BTW, on the x264 line it needs to be "%%~dpnI.264" (missing ")


I don't know. I thought I needed to. I presume Avisynth takes care of it? (I've never used an Avisynth script before). I've amended my script and removed the demuxing.
You are very welcome do anything with it. Yes, it is missing the closing quote there. DGDecode can decode the source file directly.

[audyovydeo]

audyovydeo was kind enough to share his scripts with me, and I spent an hour or so trying to work my way through them and adapt them for my use.

But it all became a bit bewildering, so I decided to go back to what I knew -- writing small batch files.


I agree my initially simple file has evolved into something less lean, but it does a radically different job from the one below.
I dont use the same setting for my encodes, I like/need to specify Profile, sar, etc for each file, so my batch is mostly spend checking input & flow control. on the other hand, I dont edit it every time I encode something. you have to edit yours every time you change x264 settings

I'll use yours for the DGMPPGDec part.

cheers
a/v

[jdlk]

you have to edit yours every time you change x264 settings

I know from working with XviD that there are only three or four variations in settings that I use, depending on the input file.

So I'll just set up a few versions of this script (along with different versions of the AVS template if necessary).

Once that's done, I'm good for another few years, until the next best thing comes along!