Trivial parallel encoding tool for x264

[emmel]

Hello,

Inspired by the great work of Tobias Bergmann, the author of ELDER (see http://forum.doom9.org/showthread.php?t=102119 ), I wrote another small script for parallel usage of x264. I call it "x264_tptool.exe" or "trivial parallel encoding tool for x264".

I'd like to point out that there is nothing Elder can't do this script can. On the contrary - Elder is much more sophisticated and capable of supporting almost all x264 command line switches, whereas this particular script is limited to one-pass encoding. Yet, I wrote it cause I needed something as simple and robust as possible.

The idea is simple (as in the first stage of Elder): The script divides all frames of a clip uniformly into "chunks", and encodes them in parallel by multiple x264-threads. It scales almost linearly up to 32 cores (haven't tested more), if the number of chunks is at least 2..4 times the number of threds. Even with only 2 cores, it is usually about 15% faster than the native "--threads 2" option of x264. The technique behind the script is of course embarrasingly trivial with respect to that of x264.

Usage is simple. If you have an x264 command line, e.g.

x264 --threads 2 --thread-input -b 2 -r 4 --me dia -o c.mp4 c.avs

just replace "x264" by "x264_tptool" and add two switches:

x264_tptool --tp-chunks 8 --tp-threads 2 -b 2 -r 4 --me dia -o c.mp4 c.avs

The switch "--tp-chunks" defines the total number of frame-chunks, and "--tp-threads" the total number of parallel threads for encoding them.

Limitations:
- Only one pass encoding
- Only AviSynth input
- Only mp4 output
- x264 and MP4Box must be in path
- [it must be possible to add "trim()" in the end of the original AviSynt script] <-- [edit: removed]

I'd finally like to point out that the motivation behind the script is pure simplicity. If you need more advaced features, or more accurate load balancing, please check Elder instead.

Thank you, and sorry if this is something too trivial to be discussed on this forum.

The binary file "x264_tptool.exe" is available from http://www.mytempdir.com/932989 [edit: link updated] Source:

//--------------------------------------------------------------------------------
// x264_tptool: trivial parallel encoding tool for x264
//
// Divides the frames of a clip into chunks of (almost) equal size, and
// encodes them in multiple parallel threads (instances of x264). Scales
// quasi-linearly if the number of chunks (N) is greater than the number
// of threads (M), say N > 2..4 * M. Some rules of thumb:
//
// - Choose N small with respect to M, if you prefer better rate control
// - Choose N big with respect to N, if you prefer better scaling
//
// All verbose output directed to stderr
//
// Usage:
//
// - Command line arguments as in x264
// - Additional arguments:
// --tp-threads int: number of parallel threads
// --tp-chunks int: total number of frame chunks
//
// Sample:
//
// x264_tptool --tp-threads 4 --tp-chunks 40 --thread-input \
// --crf 22 --me dia -b 2 -r 2 -o clip.mp4 clip.avs
//
// Assumptions:
//
// - x264.exe must be in path (not checked)
// - MP4Box.exe must be in path (not checked)
//
// Limitations:
//
// - Only 1-pass encoding
// - Only AviSynth input
// - Only mp4 output
// - Script name must be the last argument in command line
//
// Compile under MinGW with gcc. Link to libvfw32:
// gcc x264_tptool.c -O -o x264_tptool.exe -lvfw32
//
// Written by emmel, 16. Sept. 2006
//--------------------------------------------------------------------------------
#include <stdio.h>
#include <math.h>
#include <windows.h>
#include <vfw.h>

#define MAXTHREADS 32
#define MAXSTRLEN 2048
#define RELAX 1 // sec
#undef TEST

DWORD WINAPI LaunchEncoder( LPVOID lpParam );

//--------------------------------------------------------------------------------
int main( int argc, char **argv ) {
HANDLE threads[MAXTHREADS];
DWORD ExitCode;
int start, end, thread, chunk, done = 0;
char avs_cmd[MAXSTRLEN], avs_name[MAXSTRLEN];
char x264_cmd[MAXSTRLEN], MP4Box_cmd[MAXSTRLEN];
char x264_cmd_line[MAXSTRLEN];
AVISTREAMINFO avs_info;
PAVISTREAM ppavi;

// Parse command line arguments:
//------------------------------
memset( x264_cmd_line, '\0', MAXSTRLEN );
char *avs, *mp4;
int chunks = 0;
int num_threads = 0;

while( *++argv ) {
if( !strncmp( *argv, "--tp-threads", 12 ) ) {
num_threads = atoi( *++argv );
} else if( !strncmp( *argv, "--tp-chunks", 11 ) ) {
chunks = atoi( *++argv );
} else if( !strncmp( *argv, "--output", 8 ) ) {
mp4 = *++argv;
} else if( !strncmp( *argv, "-o", 2 ) ) {
mp4 = *++argv;
} else {
avs = *argv; // script name must be the last argument
if( *( argv+1 ) != NULL )
sprintf( x264_cmd_line, "%s %s", x264_cmd_line, *argv );
}
}

if( num_threads < 1 || num_threads > MAXTHREADS ) {
fprintf( stderr, "Illegal number of threads (max=%d).\n", MAXTHREADS );
return 0;
}

// Let AviSynth report the total number of frames:
//------------------------------------------------
AVIFileInit( );
if( AVIStreamOpenFromFile( &ppavi, avs, streamtypeVIDEO, 0, OF_READ, NULL ) ) {
fprintf( stderr, "Unable to open avisynth script %s\n", avs );
AVIFileExit( );
return 0;
}

if( AVIStreamInfo( ppavi, &avs_info, sizeof(AVISTREAMINFO ) ) ) {
fprintf( stderr, "Unable to retrieve stream info\n" );
AVIStreamRelease( ppavi );
AVIFileExit( );
return 0;
}

AVIStreamRelease( ppavi );
AVIFileExit( );

int frames = avs_info.dwLength;

if( frames < 1 ) {
fprintf( stderr, "No frames to encode\n" );
return 0;
}

if( chunks < 1 || chunks > frames ) {
fprintf( stderr, "Too few or too many chunks\n" );
return 0;
}

int chunk_size = (int)ceil( (double)frames/(double)chunks );

fprintf( stderr, "Script : %s\n", avs );
fprintf( stderr, "Frames : %d\n", frames );
fprintf( stderr, "Chunks : %d\n", chunks );
fprintf( stderr, "Chunk size : %d\n", chunk_size );
fprintf( stderr, "Threads : %d\n", num_threads );
fprintf( stderr, "Output : %s\n", mp4 );
fprintf( stderr, "x264 switches :%s\n", x264_cmd_line );

// Loop over chunks:
//------------------
for( thread = 0; thread < num_threads; thread++ ) threads[thread] = NULL;
fprintf( stderr, "Start encoding:\n" );
int next_chunk = 1;
while( !done ) {
for( thread = 0; thread < num_threads; thread++ ) {
GetExitCodeThread( threads[thread], &ExitCode );

if( !( ExitCode == STILL_ACTIVE ) && !done ) {
fprintf( stderr, "\n[x264_tptool] Thread %d ready to go: ", thread+1 );
start = (next_chunk-1) * chunk_size + 1;
end = next_chunk * chunk_size;
if( end >= frames ) {
end = frames;
done = 1; // last
}
fprintf( stderr, "encoding range %d...%d:\n", start, end );

// Launch encoder:
//----------------
sprintf( x264_cmd, "x264%s --seek %d --frames %d -o chunk%d.264 %s &",
x264_cmd_line, start, chunk_size, next_chunk, avs );
fprintf( stderr, "%s\n\n", x264_cmd );

threads[thread] = CreateThread( NULL, 0, LaunchEncoder, &x264_cmd, 0, NULL );
Sleep( RELAX );

next_chunk++;
fflush( stderr );
}
}
Sleep( RELAX );
}

// Ok. Wait for completion:
//-------------------------
Sleep( RELAX );
WaitForMultipleObjects( num_threads, threads, TRUE, INFINITE );
fprintf( stderr, "Done!\n" );

// Finally, concatenate results (scalar):
//---------------------------------------
#if defined TEST
fprintf( stderr, "Concatenate results...\n" );
#else
fprintf( stderr, "\nConcatenating results (%d chunks):\n", next_chunk-1 );
for( chunk = 1; chunk < next_chunk; chunk++ ) {
fprintf( stderr, "\nConcatenating chunk %d to %s:\n", chunk, mp4 );
fflush( stderr );
sprintf( MP4Box_cmd, "MP4Box -cat chunk%d.264 %s", chunk, mp4 );
system( MP4Box_cmd );
}
#endif

fprintf( stderr, "All done!\n" );
return 0;
}

//--------------------------------------------------------------------------------
DWORD WINAPI LaunchEncoder( LPVOID lpParam ) {
char *DataString = (char*)lpParam;

#if defined TEST
fprintf( stderr, "Launch encoder...\n" );
fflush( stderr );
#else
system( DataString );
#endif
return 0;
}

[Thunderbolt8]

since Im new here and dont have a clue about all the technical stuff, I actually got only 1 question: does or can this way to speed up encoding result into any kind of quality loss in the final file ?

if no, then I get it right that the use of this script has only advantages and no disadvantages ?

[akupenguin]

@emmel
Wouldn't it be simpler to use x264's --seek/--frames instead of messing with the contents of the avs script?

@Thunderbolt8
This method of threading reduces the precision of ratecontrol. i.e. 1pass ABR will provide lower quality and/or less predictable filesize. (2pass, CRF, CBR are less affected.)
It also inserts some extra I-frames, which wastes bits if the chunk size is small. Then again, slice-based threading also wastes some bits. I haven't computed the minimum chunk size to break even, but it's probably on the order of a thousand frames.

[Thunderbolt8]

@Thunderbolt8
This method of threading reduces the precision of ratecontrol. i.e. 1pass ABR will provide lower quality and/or less predictable filesize. (2pass, CRF, CBR are less affected.)
It also inserts some extra I-frames, which wastes bits if the chunk size is small.[/QUOTE]
so the lower quality (and the less predictable file size) does only apply when using 1 pass ABR, but not with constant quality e.g. -crf 21 or such ?

[emmel]

Suppose that your original content is of moderate or high bitrate, say a good quality DVD or a HDTV capture, and you wish to use 100-200 chunks and --crf 20 to encode it, the price to pay for using the "trivial method of threading" is about 1% increase in final file size. The chunk boundaries are not a problem at all from a visual point of view.

The gain in encoding speed, however, depends on the number of threads, and on the number of physical cpus / cores you have available. With 2-4 cores, for example, you get something like 15%-200% compensation in encoding speed for the slight increase in file size.

[emmel]

@emmel
Wouldn't it be simpler to use x264's --seek/--frames instead of messing with the contents of the avs script?

Indeed, thank you.
Script modified and edited in first post.

It also inserts some extra I-frames, which wastes bits if the chunk size is small. Then again, slice-based threading also wastes some bits. I haven't computed the minimum chunk size to break even, but it's probably on the order of a thousand frames.

Yes.

Here are some statistics from encoding a small DVB-sample (5902 frames) in parallel by the trivial method (x264_tptool --tp-chunks N --tp-threads 2 --crf 26 -b 2 -r 2):

N=1 -> Chunk size=5902 -> File size=10.563MB
N=2 -> Chunk size=2951 -> File size=10.572MB -> Relative increase = 0.091%
N=4 -> Chunk size=1476 -> File size=10.583MB -> Relative increase=0.189%
N=8 -> Chunk size=738 -> File size=10.632MB -> Relative increase=0.657%
N=16 -> Chunk size=369 -> File size=10.709MB -> Relative increase=1.387%
N=32 -> Chunk size=185 -> File size=10.972MB -> Relative increase=3.872%

For the same sample slice based threading (x264 --threads N --crf 26 -r 2 -b 2) gives:

N=1 -> File size=10.563MB
N=2 -> File size=10.783MB -> Relative increase=2.087%
N=4 -> File size=11.169MB -> Relative increase=5.742%

In this case, --threads 2 is about as expensive as using 250 frames chunks.

Bye.