Massively parallel adaptive bitrate encoding

[Blue_MiSfit]

Hey folks!

I'm working on designing a system for massively parallel adaptive-bitrate encoding using ffmpeg and libx264. If you're not familiar, ABR encoding requires keyframe alignment since the output streams are fragmented at specific intervals.

The basic idea is that instead of running an analysis pass and then running through each output in series using --pass 3 to update the stats file as you go, you do the analysis pass then do all of the outputs in parallel on different machines.

However, you go even further, and split the outputs into ~5 minute chunks, then processes these chunks in parallel.

You can make adaptive GOP work by parsing the first pass stats file and using it to make qpfiles for each output chunk, so that's fine.

However, you have to be pretty clever to maintain the benefits of 2 pass encoding. Has anyone done this before? I'm sure you could just edit the stats file to split it into chunks, but is that really safe?

I'm digging into it now, but was wondering if anyone's gone through this before and could save me some time :)

[benwaggoner]

Hey folks!

I'm working on designing a system for massively parallel adaptive-bitrate encoding using ffmpeg and libx264. If you're not familiar, ABR encoding requires keyframe alignment since the output streams are fragmented at specific intervals.

The basic idea is that instead of running an analysis pass and then running through each output in series using --pass 3 to update the stats file as you go, you do the analysis pass then do all of the outputs in parallel on different machines.

However, you go even further, and split the outputs into ~5 minute chunks, then processes these chunks in parallel.

You can make adaptive GOP work by parsing the first pass stats file and using it to make qpfiles for each output chunk, so that's fine.

However, you have to be pretty clever to maintain the benefits of 2 pass encoding. Has anyone done this before? I'm sure you could just edit the stats file to split it into chunks, but is that really safe?

I'm digging into it now, but was wondering if anyone's gone through this before and could save me some time :)
Didn't we talk about this a few years back :).

It really depends on how much parallelism you are targeting, and what kind of end-to-end encoding latency you want. And networking and storage configuration have a HUGE impact. Having to copy files/segments locally versus being able to stream from ultra-fast NAS, for example.

One challenge for any split-and-stitch mechanism is to ensure VBV is maintained around all stitch points.

If you're just setting a frames range to encode a given chunk, I don't think you'd need to edit the stats file at all. x264 would just ignore the entries for frames you aren't processing. Ideally it'd even estimate VBV fullness from the frames before the section you're encoding, but I have no indication it is actually that clever.

You could potentially pick stitch points that are really easy to encode per the first pass, so VBV is less likely to be an issue.

Anyway, I don't know of anyone who has actually done a production implementation along these lines based on ffmpeg and x264.

[sneaker_ger]

I think the RipBot author tried something like that before and had some issues with the 2 pass mode. I don't remember the details so you might want to search the RipBot thread.

[Blue_MiSfit]

Hey Ben! I'll shoot you an email :)

[Blue_MiSfit]

EmuAGR on #x264 suggested simply using CRF across the board, with vbv-maxrate. Anyone have experience doing this?

[kieranrk]

Anyway, I don't know of anyone who has actually done a production implementation along these lines based on ffmpeg and x264.

Youtube, Vimeo...

[Blue_MiSfit]

Hi Kieran :)

Do you think those tools were done with off-the-shelf ffmepg / x264 or with in-house tools developed using libx264 etc?

[Atak_Snajpera]

Sounds like revive of x264farm ;) Are you sure that ffmpeg is 100% frame accurate without index file? This will be required if you want to encode multiple chunks simultaneously. From my experience I can tell you that even FFmpegSource is not 100% frame accurate in some cases. For example VC1 (https://github.com/FFMS/ffms2/issues/99). Also AVC stream can start from corrupted frame after seeking.

If I understand correctly you want to run first pass on single machine (CPU) and then split first pass log file for each chunk ,right?

[nevcairiel]

Anyone developing any sort of complex workflow would be silly not to use the ffmpeg libraries (ie. libavcodec, libavformat, etc) instead of relying solely on the ffmpeg CLI tool.
Flawless chunked encoding does need an index, which is not something the FFmpeg CLI tool for example does for you, but the APIs can be used for that without doubt.

[kolak]

Ffmpeg seeking is in 99% frame accurate, even with -ss option in front of -i. I had some issues with it which was due to some rounding/time stamps ( for fractional fps) used by ffmpeg.
You can also use 100% precise frame access by decoding video from the beginning (eg. using select filter). Depending on the source file format this may be "fast enough", so transcoding will pay off time wasted for decoding from the beginning.

For someone with programming skills this should be even easier:)

[Blue_MiSfit]

We're only looking at ProRes sources, which seem to work fine - so far anyway :)

[asarian]

Sounds like revive of x264farm ;) Are you sure that ffmpeg is 100% frame accurate without index file? This will be required if you want to encode multiple chunks simultaneously. From my experience I can tell you that even FFmpegSource is not 100% frame accurate in some cases.


Yes, x264farm is what it was called! :) What ever happened to it? I wouldn't mind doing networked x264 rendering over my two i7 CPUs now!

But yes, FFmpegSource/FFVideoSource are really not frame safe (even when I occassionally read claims to the contrary).

[benwaggoner]

Yes, x264farm is what it was called! :) What ever happened to it? I wouldn't mind doing networked x264 rendering over my two i7 CPUs now!

But yes, FFmpegSource/FFVideoSource are really not frame safe (even when I occassionally read claims to the contrary).
It is probably frame safe for a CFR intraframe only codec like ProRes. Testing and validating gets a whole lot harder as you get more diverse and older source types. Trying to get WMV3 to decode frame-exact and with identical results across multiple systems could likely be challenging.