Help with command-line x264 encoding from DVD to mp4 under Linux

[t3g]

Since I need options only available on the command-line, I have to move from an Avidemux encoding to direct use of x264.

I've read the doc and I'm ok with x264 options, but there are dark areas that I couldn't figure myself.

1. I'm used to work with vob files. However, x264 only work with raw YUV 4:2:0, YUV4MPEG 4:2:0 or AVI.
Under Linux, how do I convert my vobs to something usable by x264? Out of the different options available, is there one that would speed-up a 2 or 3-pass encoding?

What program can I use under Linux (command line or GUI) to rip my DVDs directly in the good format for x264?

2. How do I crop the video before encoding, since x264 doesn't have any option for that?

3. Is there an option to specify a desired file size instead or an average bitrate when doing multi-pass encoding?

4. How do one apply external filters when using x264 (i'd rather not use ffmpeg)?

The first two questions are real blockers for me - I could do without 3, and 4 I'd only need later.
Thanks a lot.

[Dark Shikari]

1. Raw YV12 is going to be huge. Your best option is to either a) encode through mencoder, or (b) set up some sort of pipe between mencoder and x264.

2. Mencoder.

3. Yes, use a calculator to calculate the correct average bitrate to get the filesize you want.

4. Mencoder?

[t3g]

Thanks.

I understand Raw YV12 would be around 50go - but would speed multi-passes. Since I'd rather separate the processes (so I can try different options with x264), I'd really like to have data prepared as much as possible before using x264 (if cropping and such could be made in advance on the data lying on the hard drive, that would be even better, but I guess I'm dreaming too much).

I couldn't find YUV 4:2:0 or YUV4MPEG 4:2:0 output in mencoder options. I guess if I just use mencoder to do a copy of the video with an AVI output, it should be fine to process it with x264 afterwards. However, I don't know how to input multiple vobs in mencoder (the documentation only address the case of a dvd, or of an iso of the dvd).

Encoding through mencoder is exactly what I want to avoid - I really want to be able to encode with x264. How would I set up a pipe between mencoder and x264, and is it really the only solution? It really doesn't sound convenient. If it really is the only solution under linux, does mencoder give access to all x264 options, and does it uses x264 latest code?

Sorry to be such a noob :)

[Dark Shikari]

Thanks.

I understand Raw YV12 would be around 50go - but would speed multi-passes.
Probably not; you'd be disk-limited, and the conversion to raw YV12 requires almost no CPU compared to the encoding. Likely it would slow down multiple-pass encoding a lot.
Since I'd rather separate the processes (so I can try different options with x264), I'd really like to have data prepared as much as possible before using x264 (if cropping and such could be made in advance on the data lying on the hard drive, that would be even better, but I guess I'm dreaming too much).

I couldn't find YUV 4:2:0 or YUV4MPEG 4:2:0 output in mencoder options. I guess if I just use mencoder to do a copy of the video with an AVI output, it should be fine to process it with x264 afterwards. However, I don't know how to input multiple vobs in mencoder (the documentation only address the case of a dvd, or of an iso of the dvd).

Encoding through mencoder is exactly what I want to avoid - I really want to be able to encode with x264. How would I set up a pipe between mencoder and x264, and is it really the only solution? It really doesn't sound convenient. If it really is the only solution under linux, does mencoder give access to all x264 options, and does it uses x264 latest code?

Sorry to be such a noob :)
I'm not much of a Linux encoding person, but I think x264 takes input through standard input, and it will also accept a FIFO pipe, I believe.

I generally encode on Windows because Avisynth makes all of this unbelievably easy.

[t3g]

Sadly Avisynth is on Windows, and I'd rather not try it with wine :)

[nm]

I couldn't find YUV 4:2:0 or YUV4MPEG 4:2:0 output in mencoder options. I guess if I just use mencoder to do a copy of the video with an AVI output, it should be fine to process it with x264 afterwards.

You can output raw YUV with -ovc raw -of rawvideo -vf scale,format=i420. With a FIFO, this should do it:mkfifo video.fifo
mencoder input -nosound -vf scale=720:304,format=i420 -ovc raw -of rawvideo -o video.fifo & x264 -o output.264 video.fifo 720:304 --crf 20
rm video.fifo
IIRC, piping directly from MPlayer is also possible, but I'd recommend the fifo approach.

However, I don't know how to input multiple vobs in mencoder (the documentation only address the case of a dvd, or of an iso of the dvd).
Just give it all the files you want to process. You can even use specific options for each file, see: http://lists.mplayerhq.hu/pipermail/mencoder-users/2006-November/004509.html

Encoding through mencoder is exactly what I want to avoid - I really want to be able to encode with x264.
Why? Do you need to use a patched x264?

How would I set up a pipe between mencoder and x264, and is it really the only solution? It really doesn't sound convenient.
Piping with or without a FIFO is pretty simple if you write your own encoding scripts that do it for you.

If it really is the only solution under linux, does mencoder give access to all x264 options, and does it uses x264 latest code?
Yes, when both x264 and MEncoder are compiled from SVN.

And using AviSynth in Wine is probably not that hard either. See avs2yuv (http://akuvian.org/src/avisynth/avs2yuv/)

[t3g]

Thanks for the detailed explanation. It will help me get a hand on it.


Why? Do you need to use a patched x264?



No, I was under the false impression that you only had access to all x264 options while using it from the command line. I also didn't want to learn another tool than x264 :)
If all options are available through mencoder, I guess I'll have to learn to use it too.

You can output raw YUV with -ovc raw -of rawvideo -vf scale,format=i420. With a FIFO, this should do it:mkfifo video.fifo
mencoder input -nosound -vf scale=720:304,format=i420 -ovc raw -of rawvideo -o video.fifo & x264 -o output.264 video.fifo 720:304 --crf 20
rm video.fifo

I assume it's the full command line using x264 through a pipe - starting from "x264", I can just pass x264 options with x264 syntax, isn't it (that way my learning of mencoder can remain minimal for the moment, and I can keep with x264 syntax I know a bit)?
Also, can I omit scale=720:304,format=i420 since I'm never resizing, just cropping?

[nm]

If all options are available through mencoder, I guess I'll have to learn to use it too.
AFAIK all current options are available and the parameters of -x264encopts are similar to the long options of x264.

I assume it's the full command line using x264 through a pipe - starting from "x264", I can just pass x264 options with x264 syntax, isn't it (that way my learning of mencoder can remain minimal for the moment, and I can keep with x264 syntax I know a bit)?
Yes.

Also, can I omit scale=720:304,format=i420 since I'm never resizing, just cropping?
Colorspace conversion with -vf format=xxxx (which needs to be done) may require the scale filter to actually work, or at least so the MPlayer manual page tells. However, it works without scale for me, so just try it (or use scale without a target size: -vf scale,format=i420). Note that you still need to tell the final frame resolution to x264, which is not necessary if you use x264 through MEncoder.

[Manao]

Probably not; you'd be disk-limited, and the conversion to raw YV12 requires almost no CPU compared to the encoding. Likely it would slow down multiple-pass encoding a lot.Disk limits ? I don't think so.

Worst case : x264 encodes 1920x1080 at 10 fps ( that's already really fast ). That means 1920x1080x10x1.5 bytes/seconds, ie 33 MB/sec. It's not disk limited. So it's faster.

[Sharktooth]

33MB/sec is a very high value. recent SATA disks can keep that throughput but only if the file is not fragmented...

[Manao]

Because 1920x1080 @ 10 fps isn't a very high value ?

[Dark Shikari]

Disk limits ? I don't think so.

Worst case : x264 encodes 1920x1080 at 10 fps ( that's already really fast ). That means 1920x1080x10x1.5 bytes/seconds, ie 33 MB/sec. It's not disk limited. So it's faster.Since your first pass is going to go a lot faster than 10 FPS without the disk-limited aspect... it'll be slower, definitely, since the benefit of saving it all to disk is relatively dubious.

[t3g]

Disk limits ? I don't think so.

Worst case : x264 encodes 1920x1080 at 10 fps ( that's already really fast ). That means 1920x1080x10x1.5 bytes/seconds, ie 33 MB/sec. It's not disk limited. So it's faster.

Right. And actually, I'm on a 2000Mhz Sempron, and I encode my dvds with max options, so my first pass is around 5 fps, and second pass is 1 to 2 fps. That's why I've been happy with Avidemux, since I can pause it when I need it, and hibernate as I want (pausing the encode isn't necessary for hibernate though)
. It doesn't slow down my daily use of the computer, and since my Sempron doesn't support CoolSteep (I only realised it after I bought it) it doesn't increase the power usage :).

I know, I desperately need a Core2... By the way, anybody knows if Intel's Penryn sse4 4x4 SAD instruction will be supported in x264, and how much it boostes encoding?

[Dark Shikari]

I know, I desperately need a Core2... By the way, anybody knows if Intel's Penryn sse4 4x4 SAD instruction will be supported in x264, and how much it boostes encoding?IIRC the SSE4 instruction has been declared useless by Akupenguin because his sequential elimination method, used in x264 for ESA mode, is actually faster in software than the 4x4 ESA is in hardware due to better algorithmic efficiency.

[t3g]

Hi again,
I've been reading mencoder documentation the past few days, and I'm trying to encode movie credits from a single vob using this command line :
mencoder -vf crop=720:544:0:16 -vf scale,format=i420 -ofps 25 -nosound vts_01_7.vob -o credits.mp4 -ovc x264 -x264encopts bitrate=50:bframes=16:b-pyramid:ref=15:pass=1:partitions=all:8x8dct:direct=temporal:weightb:me=umh:merange=64:subme=7:b-rdo:mixed-refs:bime:trellis=1:no-fast-pskip:no-dct-decimate:nr=0:seek=5442:fps=25:sps-id=1

However, I get this error :

MEncoder 2:1.0~rc1-0ubuntu9.1 (C) 2000-2006 MPlayer Team
CPU: AMD Sempron(tm) Processor 2600+ (Family: 15, Model: 28, Stepping: 0)
CPUflags: Type: 15 MMX: 1 MMX2: 1 3DNow: 1 3DNow2: 1 SSE: 1 SSE2: 1
Compiled with runtime CPU detection.
Option x264encopts: Unknown suboption seek

WARNING: OUTPUT FILE FORMAT IS _AVI_. See -of help.
success: format: 0 data: 0x0 - 0x18bf7000
MPEG-PS file format detected.
VIDEO: MPEG2 720x576 (aspect 3) 25.000 fps 9800.0 kbps (1225.0 kbyte/s)
[V] filefmt:2 fourcc:0x10000002 size:720x576 fps:25.00 ftime:=0.0400
Opening video filter: [expand osd=1]
Expand: -1 x -1, -1 ; -1, osd: 1, aspect: 0.000000, round: 1
Opening video filter: [format fmt=i420]
Opening video filter: [scale]
==========================================================================
Opening video decoder: [mpegpes] MPEG 1/2 Video passthrough
VDec: vo config request - 720 x 576 (preferred colorspace: Mpeg PES)
The selected video_out device is incompatible with this codec.
Try adding the scale filter, e.g. -vf spp,scale instead of -vf spp.
VDecoder init failed :(
Opening video decoder: [libmpeg2] MPEG 1/2 Video decoder libmpeg2-v0.4.0b
Selected video codec: [mpeg12] vfm: libmpeg2 (MPEG-1 or 2 (libmpeg2))
==========================================================================
VDec: vo config request - 720 x 576 (preferred colorspace: Planar YV12)
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
SwScaler: using unscaled yuv420p -> yuv420p special converter
FATAL: Cannot initialize video driver.

Exiting...

seek option in x264 parameters seems to be the culprit, since it works without it. I'd like to encode only part of the vob, and I didn't find a seek option in mencoder documentation - the documentation doesn't even talk about the filters, and none appears with mencoder -vf help
I'd like to use x264 --zones options for some future encodings - are --seek and --zones options supported as -x264encopts parameters?

Also, I'm at 0.75fps - with 209min remaining. Is there a page detailing the options I can drop for the first pass, considering I'm going for 2 or 3 passes?

Thanks a lot

[nm]

mencoder -vf crop=720:544:0:16 -vf scale,format=i420 -ofps 25 -nosound vts_01_7.vob -o credits.mp4 -ovc x264 -x264encopts bitrate=50:bframes=16:b-pyramid:ref=15:pass=1:partitions=all:8x8dct:direct=temporal:weightb:me=umh:merange=64:subme=7:b-rdo:mixed-refs:bime:trellis=1:no-fast-pskip:no-dct-decimate:nr=0:seek=5442:fps=25:sps-id=1
No need to use format=i420 when you encode with -ovc x264. MEncoder does the format conversion automatically.

WARNING: OUTPUT FILE FORMAT IS _AVI_. See -of help.
Do not output to AVI. Use -of rawvideo to write a raw .264 stream and mux that to MP4 or Matroska with external tools (MP4Box and mkvmerge).

MEncoder describes the problem clearly, although not in the expected part of the output:
Option x264encopts: Unknown suboption seek

seek option in x264 parameters seems to be the culprit, since it works without it. I'd like to encode only part of the vob, and I didn't find a seek option in mencoder documentation - the documentation doesn't even talk about the filters, and none appears with mencoder -vf help
There is probably something fishy in your MEncoder build. Does mplayer -vf help display something? Also, all filters and options are thoroughly documented in the manual page (man mplayer)!

Seeking can be done with -ss and -endpos. Search the manual for better descriptions.

I'd like to use x264 --zones options for some future encodings - are --seek and --zones options supported as -x264encopts parameters?
Zones are supported in x264encopts (see manual), seeking is done with MEncoder.

Also, I'm at 0.75fps - with 209min remaining. Is there a page detailing the options I can drop for the first pass, considering I'm going for 2 or 3 passes?
Use turbo=1 or turbo=2 in x264encopts. See MPlayer/MEncoder manual page for more information. Three passes is complete overkill. Check out CRF if you don't need a specific filesize.

Edit: Noticed that your MEncoder build is quite old: MEncoder 2:1.0~rc1-0ubuntu9.1, and so is probably x264 if it is the official Ubuntu package. You may want to compile latest versions of both from their SVN repositories and uninstall the older packages.

[t3g]

No need to use format=i420 when you encode with -ovc x264. MEncoder does the format conversion automatically.


Do not output to AVI. Use -of rawvideo to write a raw .264 stream and mux that to MP4 or Matroska with external tools (MP4Box and mkvmerge).

Thank you. I'll change my command line.

There is probably something fishy in your MEncoder build. Does mplayer -vf help display something? Also, all filters and options are thoroughly documented in the manual page (man mplayer)!

Seeking can be done with -ss and -endpos. Search the manual for better descriptions.


I've been reading the man page again for the past 2 hours :) (and eventually noticed the part about filters). I didn't notice -ss, it was in mplayer's part. However, -ss only accept an input in seconds, and I'd rather input it in frames, like I used to do with x264.

--zones seems ok, I'll just need a precision about the way it affects overall bitrate, but I'll make a new thread.

Edit: Noticed that your MEncoder build is quite old: MEncoder 2:1.0~rc1-0ubuntu9.1, and so is probably x264 if it is the official Ubuntu package. You may want to compile latest versions of both from their SVN repositories and uninstall the older packages.

I didn't want to have to compile them. I purposely avoided installing Gutsy's beta because I knew the versions of the program I use would give me many itches, and I'd end up having to build the packages myself, then spending a few weeks trying to convince somebody to accept them or do an UVF. I just checked for Gutsy, and if x264 is a 09/2007 SVN (something exceptional), mencoder is still the 1.0~rc1. I'd only go and try to package mencoder if it's going to be useful to more than 1 person (me), and there's no chance a simple user can convince them to accept some code from a SVN (even for Gutsy+1) - I've tried before. Only hope in these situation is if upstream "artificially" increase the version number by releasing rc2 - then we'd get the new version for Gutsy+1.

Here is my new command line, thanks to your advices :
mencoder fin.mpg -vf crop=720:544:0:16 -ofps 25 -nosound -of rawvideo -o credits.264 -ovc x264 -x264encopts bitrate=65:bframes=16:b-pyramid:ref=15:pass=1:partitions=all:8x8dct:direct=temporal:weightb:me=umh:merange=64:subme=7:b-rdo:mixed-refs:bime:trellis=1:no-fast-pskip:no-dct-decimate:nr=0:fps=25:sps-id=1

Thanks to everybody's help, it now works and I'm getting more used to do things with mencoder.

I still have an error message (though the encoding continues), and I don't know if I should do something about it :
Opening video decoder: [mpegpes] MPEG 1/2 Video passthrough
VDec: vo config request - 720 x 576 (preferred colorspace: Mpeg PES)
Could not find matching colorspace - retrying with -vf scale...
Opening video filter: [scale]
The selected video_out device is incompatible with this codec.
Try adding the scale filter, e.g. -vf spp,scale instead of -vf spp.
VDecoder init failed :(
Opening video decoder: [libmpeg2] MPEG 1/2 Video decoder libmpeg2-v0.4.0b
Selected video codec: [mpeg12] vfm: libmpeg2 (MPEG-1 or 2 (libmpeg2))
I don't want to use scale, because when I used it it produced an uncropped video at 720:576 instead of the cropped 720:544 I was asking for.


BTW, is it advised to increase the bitrate on first pass (like 100kbps when aiming at 65kbps) when doing two-pass encoding (and about the utility of the 3rd pass, I've seen movies where it helps compared to 2 pass, esp at low bitrates)?
Thanks again

[nm]

I still have an error message (though the encoding continues), and I don't know if I should do something about it :
That could be due to the MEncoder version used or your settings. Looks like it tries to use the mpegpes passthrough first, which is obviously not going to work, and then switches to libmpeg2. Try specifying "-vc mpeg12," (the comma in the end allows MPlayer/MEncoder use other decoders if mpeg12 is wrong for the source) or even better, use the libavcodec MPEG-2 decoder, which handles errors much better: "-vc ffmpeg12,"

BTW, is it advised to increase the bitrate on first pass (like 100kbps when aiming at 65kbps) when doing two-pass encoding (and about the utility of the 3rd pass, I've seen movies where it helps compared to 2 pass, esp at low bitrates)?
It is best to use the same bitrate for all passes.

[foxyshadis]

65 kbps definitely sounds like 3-pass material; even if it's small that must be above avg q 30, which is my personal threshold for 3-pass. If you use a third pass, the first pass's importance is much less, it probably won't make any perceptible difference to use a somewhat larger or smaller bitrate.