Setting up mencoder x264 on HPC cluster (750 cores)

[Ecan]

Hi!

I am supposed to convert ~15 TB of data (or 70K hours of film if you like) from rawavi to somewhat more usable format. I have found a nice setting that fills my needs with mencoder and x264. This works perfectly on my desktop.
Now i´m trying to get the environment running on our HPC cluster. (250 nodes, 750 cores consistning of AMD Opteron Processor 250/252,280 and Intel Xeon CPU´s ,E5345 (most of them are the xeon version)

The cluster is running Linux for High Performance Computing 4.5 (Swift) (2.6.9-67.9hp.8sp.XCsmp) (HPXC).

I have had a lot of problems compiling this at all since i needed to change all installation, lib, codecs etc. directories to a place where all the nodes in the cluster can access them. After a lot of hours banging my head i have finally managed to compile mplayer (mencoder) with the x264 codec in the custom directory. It works if i give something like "mencoder in.avi -ovc x264 -o out.avi" but if i´m trying to add the "-x264encopts bitrate=300" it says that -x264encopts is not an option of mencoder.

Due to the inconsistency of processors in the cluster i ignored the warnings that there was no yasm installed during compilation which has started to get me thinking if the -x264encopts option is only availible if you have yasm?

When starting the encoding it gives me "using cpu capabilities: none".

The most important part is that i can alter the bitrate since that is essential for the system intended to use the converted videos.

Anyone have any ideas? Recompile with yasm? Or am i missing something else? I will use a clustered filesystem which can feed one core with one film (or at least thats what im intending to do) so i guess i dont need to bother about running the encoding on several threads..

When giving mencoder -ovc help it prints

Available codecs:
copy - frame copy, without re-encoding.
frameno - special audio-only file for 3-pass encoding, s
raw - uncompressed video. Use fourcc option to set
lavc - libavcodec codecs - best quality!
x264 - H.264 encoding

So it looks like I should have x264 support.

Hope there is someone who can give some suggestions! I would be happy to give some results in performance from the cluster coding video if anyone is interested..

Regards,
Eric

[Dark Shikari]

It would help to know what the "setting" is...

Also, why use mencoder? Why not use x264 directly?

[popper]

Hi!

I am supposed to convert ~15 TB of data (or 70K hours of film if you like) from rawavi to somewhat more usable format. I have found a nice setting that fills my needs with mencoder and x264. This works perfectly on my desktop.
Now i´m trying to get the environment running on our HPC cluster. (250 nodes, 750 cores consistning of AMD Opteron Processor 250/252,280 and Intel Xeon CPU´s ,E5345 (most of them are the xeon version)

The cluster is running Linux for High Performance Computing 4.5 (Swift) (2.6.9-67.9hp.8sp.XCsmp) (HPXC).

I have had a lot of problems compiling this at all since i needed to change all installation, lib, codecs etc. directories to a place where all the nodes in the cluster can access them. After a lot of hours banging my head i have finally managed to compile mplayer (mencoder) with the x264 codec in the custom directory. It works if i give something like "mencoder in.avi -ovc x264 -o out.avi" but if i´m trying to add the "-x264encopts bitrate=300" it says that -x264encopts is not an option of mencoder.

Due to the inconsistency of processors in the cluster i ignored the warnings that there was no yasm installed during compilation which has started to get me thinking if the -x264encopts option is only availible if you have yasm?

When starting the encoding it gives me "using cpu capabilities: none".

The most important part is that i can alter the bitrate since that is essential for the system intended to use the converted videos.

Anyone have any ideas? Recompile with yasm? Or am i missing something else? I will use a clustered filesystem which can feed one core with one film (or at least thats what im intending to do) so i guess i dont need to bother about running the encoding on several threads..

When giving mencoder -ovc help it prints

Available codecs:
copy - frame copy, without re-encoding.
frameno - special audio-only file for 3-pass encoding, s
raw - uncompressed video. Use fourcc option to set
lavc - libavcodec codecs - best quality!
x264 - H.264 encoding

So it looks like I should have x264 support.

Hope there is someone who can give some suggestions! I would be happy to give some results in performance from the cluster coding video if anyone is interested..

Regards,
Eric

Ecan its hard for people to help you If you dont bother reading the posts people make to your thread, assuming you had notify activated this will appear in your email soon, you might be interested in this thread too,
http://forum.doom9.org/showthread.php?p=1363932#post1363932
it may one day be useful to you and useable once again on your kit etc, and perhaps give you an incentive to help out, building the new code and easier work flow outlines etc using current tools and advances...

something to consider anyway if you ever return.

[Ecan]

The "setting" is pretty much just the use of x264 and a bitrate of 300. x264 since i have found that it works good to syncronize a video encoded with that codec with an extarnel frame syncronization vector which is available in my data.

About using x264 directly, is that possible? And as easy as using the mencoder?

[nm]

About using x264 directly, is that possible? And as easy as using the mencoder?

x264 CLI is probably easier in this case, since your MEncoder+x264 doesn't seem to get built properly. Are you using latest SVN/git revisions of both, and linking x264 statically?

I'd also sacrifice some time to get yasm working and build two x264 binaries: one for AMD and other for Intel CPUs. Or if that's too much work, make one generic build, but do include assembly optimizations. They will speed the encoding up a lot.

[Blue_MiSfit]

Yeah, whatever you do, DON'T do all this with all the ASM disabled :o

~MiSfit

[Hiritsuki]

Where to get new x264 in that the compile version?
I have two i7-980X PC can use to do.