New EMA spec for H.264 mezzanines

[benwaggoner]

Hello all,

The Entertainment Merchants Association (EMA) has just published 1.0.0 of its Mezzanine File Creation Specification and Best Practices document.

The technical goal was to provide a H.264 equivalent to existing MPEG-2 mezzanine files that simultaneously provides improved source quality while reducing file size. There were a ton of compromises made to make the files easy to decode in lots of workflows, which was a higher priority than maximum compression efficiency. The document is also a compromise based on the goals of different companies. I know there are all parts that we would have liked to be slightly different, but something good that works for everyone was better than anything that would be perfect for just some of us.

The business goal was to have a single mezzanine format that all the online video retailers could agree to take, so that post houses don't have to make dozens of slight variants for different specs and can instead focus their effort on one great file that everyone can use.

Anyway, I know it's an industry that some of your are involved in. And others may have thoughts about the technical specifics, and particularly the example encoding settings (which I'm already working on some updates for).

The obvious next step would be to add support for High422 and try to get the end-to-end workflow ecosystem to support that, so we can have an equivalent quality but much more efficient alternative to hauling huge ProRes files around.

My "quality" goal was always simple: the mezzanine should be good enough that deliverable files derived from the mezzanine should be visually indistinguishable from files derived from the source file the mezzanine was generated from. Thus, no visual loss to the second generation.

Overview:
http://www.entmerch.org/digitalema/committeescouncils/mezzanine-file-work-group.html

Document:
http://www.entmerch.org/digitalema/committeescouncils/ema-mezzanine-file-specific.pdf

[kieranrk]

Looks good!

Why b-pyramid strict? Would you still frame-reduce 59.94 material for distribution or are you planning to go full-framerate any time soon?
Have you done any work about whether keeping psy on or turning it off is better for multi-generation encodes, especially at low bitrates?

also why a scanned pdf :p (some sort of weird irony when you're trying to maintain mezzanine quality)

[benwaggoner]

Looks good!
*whew*

Why b-pyramid strict? Would you still frame-reduce 59.94 material for distribution or are you planning to go full-framerate any time soon?
If sources are truly 59.94p, I would like to receive the mezzanines like that. No reason to throw out good data!

also why a scanned pdf :p
No idea; I didn't make that :). I was a print nerd before I was a video nerd, and now you've probably induced a compulsion for me to do a proper layout and new PDF. Darn your eyes!

[kieranrk]

I thought of another thing which will become an issue as you go 5.1:

Audio metadata (downmix coefficients etc): You don't appear to have a way of storing these anywhere for PCM audio. Ideally these flags can be passed straight through to the distribution encoder so that the mix engineer's metadata stays though till the consumer.

[benwaggoner]

I thought of another thing which will become an issue as you go 5.1:

Audio metadata (downmix coefficients etc): You don't appear to have a way of storing these anywhere for PCM audio. Ideally these flags can be passed straight through to the distribution encoder so that the mix engineer's metadata stays though till the consumer.
My thought had been to include that in the XML manifest that has yet to be fleshed out. I'm not aware of any reliable way to include that in metadata.

[Biggiesized]

Is 62.5 Mbps enough for 1080p59.94 content?

[benwaggoner]

Is 62.5 Mbps enough for 1080p59.94 content?
No, probably not! Beyond the extra frames to code, this content tends to be sports or other fast-shutter high-detail high-motion content that needs a fair amount of bits in general. 60 Mbps would be ample for a delivery bitrate, but I doubt it would be adequate for a mezzanine.

We'd probably need to go to Level 5.0 for 1080p60. That's out of spec for this version of the document. We want to tackle UHD as well.

As it is, mezzanines are pretty much never delivered as >720p60 or >1080p30.

I'm running some 10-bit 4:2:2 tests first, though :).

[Blue_MiSfit]

Interesting that the container has switched over to MOV. I guess this makes sense, as it's entirely common to store PCM audio in MOV. I recall the earlier draft proposing MP4 with AAC audio as the standard, and there being a great deal of harrumphing coming from certain parties regarding using AAC as a mezzanine format :devil:. I can see the metadata options being quite useful as well, potentially.

Questions:

1) What about the damn captions? ;)

2) There's mention of an XML spec, but I don't see a schema / XSD / documentation etc for that. I imagine that's a work in progress?

I'd say overall this is pretty solid, and a step in the right direction. Maybe I'll soon be able to stop making 50 Mbps SD and 80 Mbps HD intra-only MPEG-2 mezzanine files :rolleyes:

[kieranrk]

there being a great deal of harrumphing coming from certain parties regarding using AAC as a mezzanine format :devil:

Why? At very high bitrates there's no problem with using it. In Live TV most places still use 384kbps MP2 audio and that's considered acceptable.
Not that long ago the use of MPEG-4/AVC as a mezzanine format would be considered outrageous.

[smok3]

good work,

The obvious next step would be to add support for High422 and try to get the end-to-end workflow ecosystem to support that, so we can have an equivalent quality but much more efficient alternative to hauling huge ProRes files around.

Absosmurfly, also it would be usefull if all the ffmpeg examples are also copy-pasted to a new txt/pdf for easier stealing.

[nhakobian]

But what about VBV-Bufsize? H.264 High Profile @ Level 4.1 allows for a maximum of 78125 kbit. Is that allowed for this Mezzanine specification as well?

If you look at the PDF file, that value is explicitly stated.

[Blue_MiSfit]

Why? At very high bitrates there's no problem with using it. In Live TV most places still use 384kbps MP2 audio and that's considered acceptable.
Not that long ago the use of MPEG-4/AVC as a mezzanine format would be considered outrageous.

TBH, I agree with you. I wasn't one of the parties harrumphing :)

I think this whole idea is great. If we can move towards shoveling around 62 Mbps 1080p24 H.264 mezzanines instead of 220 Mbps 1080p24 ProRes 422 HQ mezzanines it drives down costs for everyone. Less storage, less internet bandwidth, less file acceleration licenses (Aspera, et al.), and overall quicker workflows.

Whether this will be offered by post houses and / or accepted by those performing the 1:many transcodes or not is another question :)

[raffriff42]

(from the PDF)
No head or tail content like slates, bars and tone, ...etcetera are to be included with the mezzanine; the content should be only what the viewer would see and hearThis is surprising to me...I can see that metadata can replace the slate, but wouldn't you want a 'belt and suspenders' reference bars+audio tone+channel ID?

No mention of luma range or chroma matrix standards - isn't this a problem for you? Maybe you exchange these "off line," in a separate file?

Also: 23.976, 24, 25, 29.97, 59.94 only? No 30 fps? Is NTSC's shadow still hanging over us?

it would be useful if all the ffmpeg examples are also copy-pasted to a new txt/pdf for easier stealing.

Here's a working batch file with all commands laboriously transcribed :D

EDIT: there's at least one problem: the second audio stream should be stereo, but it's mono
EDIT: fixed. BTW source video & audio is from the link in the PDF. I used the 1080p version, but there's a 4K version available.
@echo off
set ffmpeg="C:\Program Files\ffmpeg\bin\ffmpeg.exe"
set qtfaststart="C:\Program Files\ffmpeg\bin\qt-faststart.exe"
:: qtfaststart for windows: http://ffmpeg.zeranoe.com/blog/?p=59

set in_video="ToS-4k-1920.mov"
set temp_video="temp.mov"

set in_audio_2ch="ToS-stereo.wav"
set in_audio_6ch="TOS-DVDSURROUND-Dolby 5.1.ac3"

set title="Tears of Steel test encode"
set artist="Blender Foundation"
set yyyy="2013"
set copyright="(CC) Blender Foundation"
set license="creative commons"
set description="test encode"

set temp_mov="temp2.mov"
set final_mov="ToS-mezzanine.mov"

::**** for testing, limit to first 300 frames ****
set framelimit=-frames 300

:VIDEO

set tune=film

set filt=-vf "setsar=1/1,scale=1920:-1,pad=1920:1080:(1920-iw)/2:(1080-ih)/2"

set opts1=-an -pix_fmt yuv420p -c:v libx264 -crf 6 -preset medium ^
-x264opts nal-hrd=vbr:vbv-maxrate=62500:vbv-bufsize=78125:keyint=48:ref=3:b-pyramid=strict:no-cabac:force-cfr=1 ^
-r 24 -tune %tune% -sws_flags +accurate_rnd %filt% -deblock -1:-1 -profile:v high ^
-psy 1 -psy-rd 1.00:0.00 -wpredp 0 -8x8dct 1 -partitions all -rc-lookahead 40 ^
-me_method hex -me_range 16 -subq 6 -chromaoffset -2 -bf 3 -sc_threshold 40 ^
-qcomp 0.60 -qmin 4 -qmax 51 -qdiff 4 -i_qfactor 0.71 -c:a pcm_s24le -f mov

if exist %final_mov% del %final_mov%

@echo on
%ffmpeg% -i %in_video% %framelimit% %opts1% %temp_video%
@echo off

:AUDIOMUX

set opts2=-channel_layout "5.1(side)" -i %in_audio_6ch% ^
-channel_layout "stereo" -i %in_audio_2ch% ^
-c:v:0 copy -c:a:0 pcm_s16le -c:a:1 pcm_s16le ^
-map 0:0 -map 1:0 -map 2:0 ^
-metadata title=%title% -metadata artist=%artist% -metadata date=%yyyy% ^
-metadata copyright=%copyright% -metadata license=%license% -metadata description=%description% ^
-metadata:s:a:0 language=eng -metadata:s:a:0 description="DVD surround 5.1 mix" ^
-metadata:s:a:1 language=eng -metadata:s:a:1 description="DVD stereo mix"

@echo on
%ffmpeg% -i %temp_video% %opts2% %temp_mov%
@echo off

:FASTSTART

@echo on
%qtfaststart% %temp_mov% %final_mov%
@echo off

pause

del %temp_video%
del %temp_mov%
::nop
::nop


MediaInfo report:
General
Complete name : D:\VideoProjects\work\ToS-mezzanine.mov
Format : MPEG-4
Format profile : QuickTime
Codec ID : qt
File size : 427 MiB
Duration : 12mn 14s
Overall bit rate mode : Variable
Overall bit rate : 4 874 Kbps
Movie name : Tears of Steel test encode
Performer : Blender Foundation
Recorded date : 2013
Writing application : Lavf55.8.102
Copyright : (CC) Blender Foundation

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.1
Format settings, CABAC : No
Format settings, ReFrames : 3 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 12s 500ms
Bit rate mode : Variable
Bit rate : 14.2 Mbps
Maximum bit rate : 62.5 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 24.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.286
Stream size : 21.2 MiB (5%)
Writing library : x264 core 133 r2334 a3ac64b
Encoding settings : cabac=0 / ref=3 / deblock=1:-1:-1 / analyse=0x3:0x133
/ me=hex / subme=6 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1
/ trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2
/ threads=3 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0
/ bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=1 / b_adapt=1 / b_bias=0
/ direct=1 / weightb=1 / open_gop=0 / weightp=0 / keyint=48 / keyint_min=4 / scenecut=40
/ intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=6.0 / qcomp=0.60 / qpmin=4
/ qpmax=51 / qpstep=4 / vbv_maxrate=62500 / vbv_bufsize=78125 / crf_max=0.0
/ nal_hrd=vbr / ip_ratio=1.41 / aq=1:1.00
Language : English

Audio #1
ID : 2
Format : PCM
Format settings, Endianness : Little
Format settings, Sign : Signed
Codec ID : sowt
Duration : 12mn 14s
Bit rate mode : Constant
Bit rate : 4 608 Kbps
Channel(s) : 6 channels
Channel positions : Front: L C R, Side: L R, LFE
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Delay relative to video : -1s 24ms
Stream size : 403 MiB (95%)
Language : English

Audio #2
ID : 3
Format : PCM
Format settings, Endianness : Little
Format settings, Sign : Signed
Codec ID : sowt
Duration : 12s 500ms
Bit rate mode : Constant
Bit rate : 1 411.2 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 44.1 KHz
Bit depth : 16 bits
Delay relative to video : -1s 24ms
Stream size : 2.10 MiB (0%)
Language : English

[benwaggoner]

Interesting that the container has switched over to MOV. I guess this makes sense, as it's entirely common to store PCM audio in MOV.
Precisely. While there is a mapping for PCM in MP4, we found that very few tools actually supported that. MOV was the most compatible container format for H.264 + PCM.

I recall the earlier draft proposing MP4 with AAC audio as the standard, and there being a great deal of harrumphing coming from certain parties regarding using AAC as a mezzanine format :devil:.
High bitrate AAC is still allowed. At maxed-out VBR, AAC-LC is perhaps 15% the bitrate of 24-bit PCM, and passes the "perceptually lossless to the second generation" test.

I can see the metadata options being quite useful as well, potentially.
Definitely. It was a relatively late attention to the spec, and I wish we'd had time to complete it. I thought sticking an early draft in as an appendix would be a good way to encourage feedback.

1) What about the damn captions? ;)
Captions are being handled by a different EMA working group:

2) There's mention of an XML spec, but I don't see a schema / XSD / documentation etc for that. I imagine that's a work in progress?
Yep. And you are invited to offer feedback! I'm hardly an XML expert myself.

I'd say overall this is pretty solid, and a step in the right direction. Maybe I'll soon be able to stop making 50 Mbps SD and 80 Mbps HD intra-only MPEG-2 mezzanine files :rolleyes:
Thanks!

And yes, the core goal of the 1.0 spec is to give the industry a superior alternative to MPEG-2. With H.264 we can get better quality and smaller file size.

Relatedly, it would be great to hear thoughts on the optimum x264 parameters to make these mezzanines. There are a couple in the doc itself. Here's one I did for a 1920x1080p24 Tears of Steel DNxHD source:
C:\Users\benwagg\Desktop\MeGUI\tools\x264\x264_64.exe --level 4.1 --preset slow --tune grain --force-cfr --crf 8 --no-cabac --keyint 48 --min-keyint 1 --bframes 3 --b-pyramid strict --ref 3 --vbv-bufsize 78125 --vbv-maxrate 62500 --aud --nal-hrd vbr --non-deterministic --colorprim bt709 --transfer bt709 --colormatrix bt709 --crop-rect 0,140,0,140 --output "ToS_EMA-1920x1080p24_CRF8-grain_slow_CropRect.mp4" ToS_DNxHD-1920x1080p24_5.1-stereo.mov

I'm not sure about the compatibility of --crop-rect, or the pro/con between --tune film and --tune grain for mezzanine use. It's pretty easy to wind up with a file that's >50 Mbps, and thus pretty close to CBR at Level 4.1 limits.

[benwaggoner]

The document only mentions 62500 Kbps as a bitrate constraint.

Which is the maximum VBV-Maxrate value for H.264 High Profile @ Level 4.1.

But what about VBV-Bufsize? H.264 High Profile @ Level 4.1 allows for a maximum of 78125 kbit. Is that allowed for this Mezzanine specification as well?
Yes, you can use the maximum bitrate and VBV allowed by Profile @ Level.

I'm doing some initial experiments with 10-bit 4:2:2 now. Now there are some bits to work with :)! We'll have to be careful if we don't want grainy content to wind up near ProRes bitrates.

[benwaggoner]

I think this whole idea is great. If we can move towards shoveling around 62 Mbps 1080p24 H.264 mezzanines instead of 220 Mbps 1080p24 ProRes 422 HQ mezzanines it drives down costs for everyone. Less storage, less internet bandwidth, less file acceleration licenses (Aspera, et al.), and overall quicker workflows.
Exactly!

Whether this will be offered by post houses and / or accepted by those performing the 1:many transcodes or not is another question :)
We've been talking to a number of post houses about this, and have already received compliant test clips from one. Post houses are generally fine with the idea of having to do the work only once and then selling the results multiple times :).

The spec was written by participants representing the majority of the VOD industry, with Apple as the significant exception. The goal was a format that was broadly compatible with lots of different workflows, so anything that can handle a MPEG-2 High Profile should be able to handle these files.

[benwaggoner]

This is surprising to me...I can see that metadata can replace the slate, but wouldn't you want a 'belt and suspenders' reference bars+audio tone+channel ID?
It's a challenge to get everyone to include the slates in a consistent enough way to make removal of the slates reliable.

No mention of luma range or chroma matrix standards - isn't this a problem for you? Maybe you exchange these "off line," in a separate file?
Color Matrix metadata is required to be specified in the bitstream.

We probably should have specified tv luma range, yes.

Also: 23.976, 24, 25, 29.97, 59.94 only? No 30 fps? Is NTSC's shadow still hanging over us?
Oops :)? In practice, we do see real 24.000 fps sources, but real 30.000 is exceedingly rare. But probably should be allowed.

Thanks for the example batch!

[paradoxical]

H.264 High 4:2:2 Profile @ Level 4.1 is --vbv-bufsize 250000 --vbv-maxrate 200000 :) ;).

Yes, which is far higher than he wants. Hence the comment about not wanting "ProRes bitrates" (http://documentation.apple.com/en/finalcutpro/professionalformatsandworkflows/index.html#chapter=10%26section=4%26tasks=true).

What? Ever thought of video-game footage ;)?

Why would that have any relevance? This spec is for people who are probably overwhelmingly dealing with live-action footage not Machinima. Despite what you seem to think, encoding video game footage is a niche use case.

[paradoxical]

Setting --vbv-maxrate 200000 does not automatically mean the resulting encode has to come out at 200 Mbps.

Where exactly did I say it would? Oh right, I didn't...

[benwaggoner]

What? Ever thought of video-game footage ;)?
It isn't something that gets normally used in mezzanine files, which is why it didn't come up. But I'd like to explicitly add 30.00 in the next revision.

Why did you cap --ref and --bframes at 3?
To constrain worst-case random access time and decode time for software decoders. That's the same reason that CABAC isn't used (very slow at high bitrates and offers relatively small gain at high bitrates).

[nhakobian]

That's the same reason that CABAC isn't used (very slow at high bitrates and offers relatively small gain at high bitrates).

I would assume that CABAC encoding/decoding time would be somewhat proportional to the bitrate used. Is the issue a tradeoff between compression efficiency and decoding time, or is there actually a saturation point where CAVLC approaches the compression efficiency of CABAC at high bitrates?

If so, I'm curious as to what would cause that (from a technical / compression standpoint). Maybe less redundant data in the raw bitstream? Maybe due to more high frequency features being retained?

[J_Darnley]

I'd like to explicitly add 30.00 in the next revision.

If you're going to add 30, might I suggest 60 too?

[benwaggoner]

If you're going to add 30, might I suggest 60 too?
Definitely on the list.