Multithreaded H.264 transcoding with ffmpeg

[Dust Signs]

Hi,

I'm currently experimenting with ffmpeg to transcode raw H.264 HD streams which come from a company-internal software which extracts them from a DVB-S2 transport stream.
As SD (576i50) trancoding works fine I thought that 720p50 transcoding should not be much of a problem, since the SD transcoding consumed 50% of one core of a 16 core machine. When transcoding the 720p50 stream I see (htop) two threads which consume 80 to 100% of one core each, and 14 threads which seem to do nothing (0% CPU utilization). The frame rate I achieve is around 30fps which means that no real time transcoding is possible. I tried to break it down in order to figure out what the main issue may be, and tried the "ultrafast" preset (removing the lossless option of course) and /dev/null as output file just to make sure that it's not due to the encoder settings. Besides the fact that no output file is generated the frame rate remains around 30fps. CPU utilization does not seem to change much.
I use the following command:
./internal_program_which_outputs_raw_h264_to_stdout | ./ffmpeg -y -i - -vcodec libx264 -vpre ./mypresets.ffpreset -level 41 -b 6500000 -maxrate 6500000 -bufsize 2000000 -threads 16 -f mp4 -r 50 -an /dev/null

with the following preset:
coder=0
flags=+loop
cmp=+chroma
partitions=-parti8x8-parti4x4-partp8x8-partp4x4-partb8x8
me_method=dia
subq=0
me_range=16
g=25
keyint_min=25
sc_threshold=40
i_qfactor=0.71
b_strategy=1
qcomp=0.6
qmin=10
qmax=51
qdiff=4
directpred=1
flags2=+fastpskip

Any ideas why encoding in real time is not possible although enough CPU power is available? If I only decode the stream (no transcoding) everything is fine and I get my 50fps

Dust Signs

[Dark Shikari]

For a number of reasons, x264's threading capability becomes limited when using extremely fast encoding settings and large numbers of threads. You can try setting sc_threshold=-1.

In practice though, you should have no problem. Avail does 1080i30 realtime transcoding on 4-core systems with rather slow settings (~subme=7, etc).

[Dust Signs]

Setting sc_threshold -1 does not really change the fact that I only get 30 fps. What seems so weird to me is the fact that decoding alone works fine at 50fps (real time) - but as soon as I start transcoding the frame rate drops to 30fps. Of course this is not acceptable for our company, as it would exclude x264 from our list of possible encoders in the first place which is not my intention as I also use x264 privately and I personally think that it's a really good and fast encoder. Are there any other options which I could set in order to profit from parallelization so that I am at least able to encode in real time?

In practice though, you should have no problem
The problem is that this is a "real" live stream - to be specific even one of the streams we want to transcode in the near future.

Dust Signs

//EDIT: If this is of interest for you: using subq 7 decreases the frame rate by 7 and increases the CPU load by 20% (total), so it's now 23fps and 2*100% CPU utilization

[Dark Shikari]

The problem is that this is a "real" live stream - to be specific even one of the streams we want to transcode in the near future.As I said, there shouldn't be a problem; Avail uses x264 for live 1080i encoding (IPTV/cable broadcast), so whatever the problem is, it sounds like it's on your end.

What are the hardware specs of your live transcoding box?

Also, have you tried using raw uncompressed input from a file as a test case to see if the decoding could be the problem? The 50fps -> 30fps issue would make sense if the decoding was run in the same thread as the main encoding thread, preventing the two from being run simultaneously.

[Dust Signs]

//EDIT: Using a raw H.264 file as input I get even more than 50fps, although the CPU utilization seems to be 20% lower (160% now in total)

The machine I currently run my tests on is an 8 core Xeon machine.

[root@**********]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
stepping : 11
cpu MHz : 2666.934
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 5337.31
clflush size : 64

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
stepping : 11
cpu MHz : 2666.934
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 5333.64
clflush size : 64

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
stepping : 11
cpu MHz : 2666.934
cache size : 4096 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 5333.68
clflush size : 64

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
stepping : 11
cpu MHz : 2666.934
cache size : 4096 KB
physical id : 3
siblings : 2
core id : 1
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 5333.67
clflush size : 64

[Dark Shikari]

Here's a test encode on a machine with similar specs (3.2Ghz Core 2, 8 cores):

Encoder112 x264 # ./x264 --subme 0 --scenecut -1 --me dia --vbv-maxrate 6500 --bitrate 6500 --level 41 --vbv-bufsize 20000 --threads 16 --partitions none ../ffmpeg/out.y4m -o /dev/null
yuv4mpeg: 1920x1080@24/1fps, 1:1
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 PHADD SSE4.1 Cache64
x264 [info]: profile Main, level 4.1
x264 [info]: slice I:2 Avg QP:19.98 size: 70971 PSNR Mean Y:66.67 U:68.76 V:69.54 Avg:67.22 Global:37.46
x264 [info]: slice P:480 Avg QP:19.91 size: 32029 PSNR Mean Y:43.50 U:48.40 V:48.71 Avg:44.53 Global:40.07
x264 [info]: mb I I16..4: 72.8% 0.0% 27.2%
x264 [info]: mb P I16..4: 13.8% 0.0% 0.0% P16..4: 41.6% 0.0% 0.0% 0.0% 0.0% skip:44.6%
x264 [info]: SSIM Mean Y:0.9760006
x264 [info]: PSNR Mean Y:43.592 U:48.480 V:48.800 Avg:44.626 Global:40.055 kb/s:6180.51

encoded 482 frames, 59.11 fps, 6181.01 kb/s


That's why I'm pretty sure you should be able to get realtime encoding of 1080p on your box; it isn't as if it's over 2x slower than this one.

//EDIT: Using a raw H.264 file as input I get even more than 50fps, although the CPU utilization seems to be 20% lower (160% now in total)I meant raw video, i.e. raw YV12, not "raw H.264."

By the way, the main reason for the bottleneck at extremely fast settings is the lookahead used for VBV, B-frame decision, scenecut, and so forth. You can see how big a bottleneck this is by looking at the speed with --qp 25 instead of bitrate mode:

Encoder112 x264 # ./x264 --subme 0 --scenecut -1 --me dia --qp 25 --threads 16 --partitions none ../ffmpeg/out.y4m -o /dev/null
yuv4mpeg: 1920x1080@24/1fps, 1:1
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 PHADD SSE4.1 Cache64
x264 [info]: profile Main, level 4.0
x264 [info]: slice I:2 Avg QP:22.00 size:200515 PSNR Mean Y:71.55 U:73.89 V:74.48 Avg:72.13 Global:47.27
x264 [info]: slice P:480 Avg QP:25.00 size: 36630 PSNR Mean Y:42.61 U:47.86 V:48.05 Avg:43.69 Global:42.17
x264 [info]: mb I I16..4: 66.6% 0.0% 33.4%
x264 [info]: mb P I16..4: 10.5% 0.0% 0.0% P16..4: 35.0% 0.0% 0.0% 0.0% 0.0% skip:54.4%
x264 [info]: SSIM Mean Y:0.9813083
x264 [info]: PSNR Mean Y:42.734 U:47.966 V:48.163 Avg:43.808 Global:42.185 kb/s:7163.55

encoded 482 frames, 122.67 fps, 7163.97 kb/s

Avail is working on a patch to thread the lookahead in order to improve performance, but you should be able to get the necessary performance without that patch.

Also, note that with the settings I used on the top, --no-cabac doesn't affect performance because of the bottleneck (but in the second one, it raises the speed from 123 to ~134 fps!). The reason for this is that when the lookahead is the bottleneck, all other settings become "free" CPU-wise until the lookahead is no longer the bottleneck.

[Dust Signs]

Thanks - that's good to know.
I now also tested with raw (YV12) input. I'm quite close to real time (47fps), but this now seems to be due to the speed of the hard disk. Does this fact change anything? Does it mean that there may be a threading problem if ffmpeg is reading from a pipe and decoding H.264 at the same time?

Dust Signs

[Dark Shikari]

Thanks - that's good to know.
I now also tested with raw (YV12) input. I'm quite close to real time (47fps), but this now seems to be due to the speed of the hard disk. Does this fact change anything? Does it mean that there may be a threading problem if ffmpeg is reading from a pipe and decoding H.264 at the same time?

Dust SignsI am not 100% sure how ffmpeg handles multiple jobs at a time (encoding and decoding), but it's possible that the same thread that calls the x264 process is the thread that does the H.264 decoding, preventing them from being done at the same time.

Avail Media doesn't have this issue because they wrote their own control program that calls libavcodec and libx264 separately in different threads.

[Dust Signs]

So this means that I have to write something similar in order to deal with this issue?

Dust Signs

[Dark Shikari]

So this means that I have to write something similar in order to deal with this issue?

Dust SignsHave you tried piping from ffmpeg or mplayer directly to the x264 process and seeing if that gives any better performance?

E.g. mkfifo pipe.y4m;mplayer inputfile -vo yuv4mpeg:file=pipe.y4m -ao null & x264 pipe.y4m <options>

[Dust Signs]

I now tried the following:

./some_program | \
./ffmpeg -y -r 50 -i - -vcodec rawvideo -f rawvideo - | \
./x264 --progress --level 4.1 --bitrate 6500 --vbv-maxrate 6500 --vbv-bufsize 2000 \
--no-cabac --me dia --merange 16 --keyint 25 --partitions none --subme 0 --min-keyint 25 \
--scenecut 40 --qcomp 0.6 --qpmin 10 --qpmax 51 --qpstep 4 --direct spatial --threads 8 --fps 50 -o orf1hd.264 - 1280x720

The speed is now around 23fps, the CPU utilization of x264 is around 85%, ffmpeg 65% respectively. The output file seems to be ok in the first 10 seconds, but then starts jumping around in past and present pictures, blocking and blurring, mostly green and purple which I assume is due to the fact that there is some bottleneck in the data flow. Mkfifo doesn't make any difference here. I can't try mplayer as we had troubles compiling it for our platform. Any suggestions what to do now? Am I right if I assume that there is some sort of problem in ffmpeg in combination with pipes?

Dust Signs

[Hellworm]

Normal pipe has too small a buffer size to be useful to really detach the input, you could try a program like bfr (http://www.glines.org/wiki/bfr) as buffer which allows to set the buffer size. 1M buffer size is enough for 1080p.

[Dust Signs]

Good morning,

@Hellworm: thanks for this information, I'll give it a try as it sounds very likely to be able to solve my problem
@Dark Shikari: I recompiled ffmpeg and removed all codecs and formats which I didn't need and used my original parameters and finally the speed is sufficient :eek: :confused:

./some_program | \
./ffmpeg -y -r 50 -i - -vcodec rawvideo -f rawvideo - | \
./x264 --progress --level 4.1 --bitrate 6500 --vbv-maxrate 6500 --vbv-bufsize 2000 \
--me dia --merange 32 --keyint 25 --partitions all --subme 0 --8x8dct --min-keyint 25 --trellis 1 \
--scenecut 40 --qcomp 0.6 --qpmin 10 --qpmax 51 --qpstep 4 --direct spatial --threads 8 --fps 50 -o orf1hd.264 - 1280x720

This works but I will have a look at the information Hellworm provided. If I can make it work any better I will post the settings here

Dust Signs

[Dark Shikari]

Those are incredibly bizarre x264 settings. You're mixing slow options with absurdly fast options, giving a very suboptimal result in terms of speed vs quality.

[Dust Signs]

Originally I intended to use --subme 5, but then I run into the real time problem again. Maybe bfr will fix this, but I'm still trying to compile and test.

Dust Signs

[Dark Shikari]

Originally I intended to use --subme 5, but then I run into the real time problem again. Maybe bfr will fix this, but I'm still trying to compile and test.

Dust Signs./x264 --progress --level 4.1 --bitrate 6500 --vbv-maxrate 6500 --vbv-bufsize 2000 \
--me dia --merange 32 --keyint 25 --partitions all --subme 0 --8x8dct --min-keyint 25 --trellis 1 \
--scenecut 40 --qcomp 0.6 --qpmin 10 --qpmax 51 --qpstep 4 --direct spatial --threads 8 --fps 50 -o orf1hd.264 - 1280x720You're using partitions all, a very slow setting, with subme=0, an absurdly fast setting.

Try this for starters:

./x264 --progress --level 4.1 --bitrate 6500 --vbv-maxrate 6500 --vbv-bufsize 2000 \
--me hex --keyint 25 --subme 3 --partitions i4x4,i8x8 --8x8dct --threads auto --bframes 1 --no-b-adapt --scenecut -1 --fps 50 -o orf1hd.264 - 1280x720

Also, using such an absurdly low buffer size can be very detrimental to quality.

[Dust Signs]

Ok, now I tried with the settings you proposed in combination with bfr:

./some_program | \
./bfr | \
./ffmpeg -y -r 50 -i - -vcodec rawvideo -f rawvideo - | \
./bfr | \
./x264 --progress --level 4.1 --bitrate 6500 --vbv-maxrate 6500 --vbv-bufsize 2000 \
--me hex --keyint 25 --subme 3 --partitions i4x4,i8x8 --8x8dct --threads auto --bframes 1 --no-b-adapt --scenecut -1 --fps 50 -o orf1hd.264 - 1280x720

Result: ~30fps
The reason for my weird settings was the fact that I empirically derived them (based on the bit rate restrictions which I have consider) in order to achieve my 50fps (which I still do using the settings I mentioned above). //EDIT: Using the parameters you proposed and --subme 0 I'm at 50fps again

Regarding the low buffer size I have to say that there are certain network restrictions which I have to consider - and as vbv-maxrate alone was not enough I had to restrict the buffer size too.

Dust Signs

[Dark Shikari]

Ok, now I tried with the settings you proposed in combination with bfr:

./some_program | \
./bfr | \
./ffmpeg -y -r 50 -i - -vcodec rawvideo -f rawvideo - | \
./bfr | \
./x264 --progress --level 4.1 --bitrate 6500 --vbv-maxrate 6500 --vbv-bufsize 2000 \
--me hex --keyint 25 --subme 3 --partitions i4x4,i8x8 --8x8dct --threads auto --bframes 1 --no-b-adapt --scenecut -1 --fps 50 -o orf1hd.264 - 1280x720

Result: ~30fps
The reason for my weird settings was the fact that I empirically derived them (based on the bit rate restrictions which I have consider) in order to achieve my 50fps (which I still do using the settings I mentioned above). //EDIT: Using the parameters you proposed and --subme 0 I'm at 50fps again--subme 0 is an extreme last resort. If you can't get fast enough without it, you want to drop partitions to none, me to dia, bframes to 0, subme to 1, and use no-cabac before considering subme 0 as an option.

I find it odd that you can only get 30fps with those settings; Avail uses something similar to the following for real-time 720p60 encoding with a 3.2Ghz Core 2 8-core system:

./x264 --progress --level 4.1 --bitrate 8000 --vbv-maxrate 8000 --vbv-bufsize 8000 \
--me hex --keyint 50 --subme 7 --partitions i4x4,i8x8 --8x8dct --threads auto --bframes 1 --no-b-adapt --scenecut -1 --fps 50 -o orf1hd.264 - 1280x720

Do note, however, that part of your problem may be decoding the input h264 file; Avail transcodes from MPEG-2, which is much easier on the CPU. Have you tried using ffmpeg-mt for the decoding?

[Dust Signs]

Ok. Based on that, here some new test results:

Your settings from Avail: not fast enough

Based on the settings you posted previously:
partitions none: not fast enough
partitions none and me dia: not fast enough
partitions none and me dia and bframes to 0: not fast enough
partitions none and me dia and bframes to 0 and subme to 1: not fast enough
partitions none and me dia and bframes to 0 and subme to 0: 50fps

I think I'll give ffmpeg-mt a try, but this will take a while to compile for the target system. In the meantime I post my configure parameters in case I missed something crucial (CPU load when decoding the H.264 stream is around 90% of one core):

********# ./ffmpeg
FFmpeg version SVN-r16816, Copyright (c) 2000-2009 Fabrice Bellard, et al.
configuration: --enable-pthreads --disable-network --disable-ipv6 --disable-mpegaudio-hp --enable-gpl --enable-hardcoded-tables --disable-encoders --enable-encoder=libx264 --enable-encoder=rawvideo --disable-decoders --enable-decoder=mpeg2video --enable-decoder=h264 --disable-muxers --enable-muxer=mp4 --enable-muxer=yuv4mpegpipe --enable-muxer=rawvideo --disable-demuxers --enable-demuxer=rawvideo --enable-demuxer=h264 --enable-demuxer=mpeg2video --enable-demuxer=yuv4mpegpipe --disable-parsers --enable-parser=h264 --enable-parser=mpegvideo --disable-bsfs --enable-bsf=h264_mp4toannexb --disable-protocols --enable-protocol=file --enable-protocol=pipe --disable-indevs --disable-outdevs --disable-devices --disable-filters --enable-filter=crop --enable-libx264 --extra-version=hugo --cpu=i686 --disable-debug
libavutil 49.14. 0 / 49.14. 0
libavcodec 52.11. 0 / 52.11. 0
libavformat 52.25. 0 / 52.25. 0
libavdevice 52. 1. 0 / 52. 1. 0
built on Feb 4 2009 15:04:17, gcc: 4.1.1


//EDIT: ffmpeg-mt makes the situation a little better which means that the frame rate slightly increases, but it is still too low (~45 fps at most of the settings propsed except the original weird --subme 0).

Dust Signs

[nm]

//EDIT: ffmpeg-mt makes the situation a little better which means that the frame rate slightly increases, but it is still too low (~45 fps at most of the settings propsed except the original weird --subme 0).
Try adding more decoding threads (4, 6 or 8).

[Dust Signs]

@nm: That was it :) Thank you very much. It now works with

./some_program | \
./ffmpeg -y -r 50 -threads 8 -i - -vcodec rawvideo -f rawvideo - | \
./x264 --progress --level 4.1 --bitrate 6500 --vbv-maxrate 6500 --vbv-bufsize 2000 \
--me hex --keyint 25 --subme 3 --partitions i4x4,i8x8 --8x8dct --threads auto --bframes 1 --no-b-adapt --scenecut -1 --fps 50 -o orf1hd.264 - 1280x720

Setting subme to 4 causes instability (48fps), subme 3 is fine for now - i'll start optimizing later on. Thank you all for your help.

Dust Signs

[Manao]

Regarding the low buffer size I have to say that there are certain network restrictions which I have to consider - and as vbv-maxrate alone was not enough I had to restrict the buffer size too.
Huh ? network restrictions influence only vbv-maxrate. Vbv-bufsize is restricted only by the streaming application, the player application, and the allowable end to end latency.

[Dust Signs]

Sorry, my fault - I used the wrong word. I meant "device", but wrote network :o

Dust Signs