x264 for BD encoding - does it work now?

[jsevakis]

I've been pouring over the forums and it seems like there's been quite a bit of progress lately in adding BD compliance to x264.

From what I can gather, a compliant stream needs both --slices:4 (in the svn since early September) and --hal-nrd (patch available for some time, being worked into the SVN now). Is this accurate?

If so, has anyone released a binary with both of these features activated? Have the results been tested for replication? (BD is a bit of a mess, since authoring successfully doesn't necessarily mean it'll replicate OK.)

Any assistance would be really helpful. If replication hasn't been tested yet, I can get some test encodes submitted to a plant to find out.

[Dark Shikari]

I'm still waiting for some BD-verifier results from a contact of mine...

[jsevakis]

Awesome. Can I anticipate some noise being made when it passes?

I can't wait to put this into use. The encoder built into the new version of Compressor is... acceptable, but really chokes on noisy sources. Everything else is way out of my price range. T_T

[cacepi]

I can't wait to put this into use. The encoder built into the new version of Compressor is... acceptable, but really chokes on noisy sources. Everything else is way out of my price range.
There's a Quicktime component (http://www003.upp.so-net.ne.jp/mycometg3/) for x264, but I don't believe that the newer options like slices and the such are supported since it uses ffmpeg as a backend. The cli binary does have all the options, but you wouldn't be able to use Compressor with it. I can compile a binary of the latest git sources if you want.

jsevakis... is this for ImaginAsian or personal use?

[jsevakis]

lol, ImaginAsian is not even a company anymore. I work for Anime News Network, but I also do freelance DVD/BD authoring on the side. But even if it doesn't replicate successfully yet, I have a few personal projects I can put it to work on. ;)

I don't need to use it with Compressor; honestly I'd be thrilled if I can use it as a CLI on either Intel Mac or Windows. I guess Windows would come in handier, since I could feed it AVIsynth sources, but the Mac version would really come in handy too. (I'm stuck with XP, so multithreading is not gonna go all that fast for me on the Windows side.) I'd be most grateful for either one.

[shon3i]

I'm still waiting for some BD-verifier results from a contact of mine...
Did Scenarist not good enough? since check everything in stream including possible underflows in stream tought mux, btw i see many streams extracted from original blu-ray that not pass scenarist check or mux.

If so, has anyone released a binary with both of these features activated? Have the results been tested for replication? (BD is a bit of a mess, since authoring successfully doesn't necessarily mean it'll replicate OK.)Here is builds for you http://forum.doom9.org/showpost.php?p=1336131&postcount=2526, use patched bulds.

[Dark Shikari]

Did Scenarist not good enough? since check everything in stream including possible underflows in stream tought mux, btw i see many streams extracted from original blu-ray that not pass scenarist check or mux.Does the latest patch from Alex Giladi work? Trahald says that Alex's patch is much better and that we should get that one into git, not his.

[shon3i]

Does the latest patch from Alex Giladi work? Trahald says that Alex's patch is much better and that we should get that one into git, not his.
No :(, but i test only two builds posted here i don't know is there maybe newer since then. I tested many times and both scenarist and elecard reject stream, second time is more strange. If you want to do something about that i will test if need. Even Trahald last rev19 crashes x264 if --keyint 24 is set, with --keyint 48 work fine.

[Underground78]

Does the latest patch from Alex Giladi work?

Thahald wrote this on the mailing-list about the last version of the patch by Alex Giladi :

I think your x264_vbv_fullness() routine is broken. initial cpb removal delay is holding negative values. it is what is causing elecard to reject the stream.

[kolak]

Did Scenarist not good enough? since check everything in stream including possible underflows in stream tought mux, btw i see many streams extracted from original blu-ray that not pass scenarist check or mux.



Scenarist check is minimal (mainly: level, slices, buffer). I've tested it with "bad streams" end they were improted with no problems. You need verifier to test encoder for BD compliancy (the best one is Sony).

This is preaty impossible- extracted streams have to import and mux (even if BDs were created with Blu-print). It means there is something wrong with extracting.

[shon3i]

Scenarist check is minimal (mainly: level, slices, buffer). I've tested it with "bad streams" end they were improted with no problems. You need verifier to test encoder for BD compliancy (the best one is Sony).
I agree, but scenarist wont mux stream out of specs that is key thing, i don't care if stream is broken itself, that should be done by encoder, like DTS verifier after encoding. The key thing is that stream contain 100% compilacy parameters. btw did this Sony BD verifyer is avaible in public? as demo/trial or something? or with some app like Blu print?


This is preaty impossible- extracted streams have to import and mux (even if BDs were created with Blu-print). It means there is something wrong with extracting.I am not sure, because i just demux streams with several applications, and results are same. The problem is in muxing stage, and always at same place. I read from other users, and somebody notice that one stream have much higher buffer than should > 30000.

[kolak]

I agree, but scenarist wont mux stream out of specs that is key thing, i don't care if stream is broken itself, that should be done by encoder, like DTS verifier after encoding. The key thing is that stream contain 100% compilacy parameters. btw did this Sony BD verifyer is avaible in public? as demo/trial or something? or with some app like Blu print?


Muxing is one thing- there are many more restricions and rules- seamless connetion in playlists, multiangel, etc. Scenarist can produce out of the spec BD discs (evn if seperate assets are BD compliant) and you won't know about it unless you verify the imnage. It either comes to your knowledge or you need to use verifier.

BD verifier is not available in any trial/demo version- as far as I know. You can have trial version of Blu-print, but you need a company which has to go through credit check process.

If you make discs which are replicated in thousands copies, you have to be sure about your software.
If stream from x264 passes one verification it still doen't mean it can be safetly used for commercial use. We would need at least few tests with different settings and options used for encoding.

Andrew

[kolak]

I am not sure, because i just demux streams with several applications, and results are same. The problem is in muxing stage, and always at same place. I read from other users, and somebody notice that one stream have much higher buffer than should > 30000.

Hmmm- I don't know, but can you explain me how come Scenarist muxed it in the first place than? If it's commercial disc than it was done either with Scenarist BD or with Blu-print-both share exactly the same muxing engine.

There can be possibility that new version had bigger restricions, so some streams may not mux- I don't know.
I don't use 30Mbit for buffer, but slightly less and never had any problems with muxing, even if my assets peaks close to the limit.

Andrew

[shon3i]

Muxing is one thing- there are many more restricions and rules- seamless connetion in playlists, multiangel, etc. Scenarist can produce out of the spec BD discs (evn if seperate assets are BD compliant) and you won't know about it unless you verify the imnage. It either comes to your knowledge or you need to use verifier.Well we here talking of video stream itself, not what scenarist do, main thing is that x264 encode fully pass all restrictions.

Hmmm- I don't know, but can you explain me how come Scenarist muxed it in the first place than? If it's commercial disc than it was done either with Scenarist BD or with Blu-print-both share exactly the same muxing engine.

There can be possibility that new version had bigger restricions, so some streams may not mux- I don't know.
I don't use 30Mbit for buffer, but slightly less and never had any problems with muxing, even if my assets peaks close to the limit.

Andrew I don't know here is prue example of situation http://forum.doom9.org/showthread.php?t=149305&highlight=scenarist+underflow, Van Helsing BD retail is source. I have same situation with some BD's with various Scenarist versions. I thinked if i upgrade to newer maybe pass, since Scenarist BD 5.0 more weaker about restrictions.

[kolak]

Well we here talking of video stream itself, not what scenarist do, main thing is that x264 encode fully pass all restrictions.

I don't know here is prue example of situation http://forum.doom9.org/showthread.php?t=149305&highlight=scenarist+underflow, Van Helsing BD retail is source. I have same situation with some BD's with various Scenarist versions. I thinked if i upgrade to newer maybe pass, since Scenarist BD 5.0 more weaker about restrictions.

There are some PIP streams, so for me it's 99% problem with rebuilding process.
There can be many reasons with such a complicated title.
BD is new- all rebuilding tools are far from being perfect, so there are many possible problems. Bad assets from reatail BD are the last one to suspect in my opinion.

Andrew

[kolak]

Well we here talking of video stream itself, not what scenarist do, main thing is that x264 encode fully pass all restrictions.


Verifiers work on muxed streams not elementary stream, which makes sens. That's why you need all header information to be correct, becuase they're used during muxing process.

Old days we had problems with muxing for HD DVD, becuse muxing engine didn't understand some information in AVC header. Two companies (responsible for encoder and muxing tool) had to talk to each other and find the problem. It's end up tha muxer expected some information in the header (excatly value=2), but encoder was using different one=3. It was all becasue HD DVD spec didn't specify this, and AVC spec allowed for any number from 1-5. Muxing engine was written to accept only 2. At the end muxing engine was changed to accept all possible numbers, becuase different encoders could use different ones.

Some information are not specified in BD spec or they are only recommended, but not mandatory. x264 can produce very good AVC stream (correct header, ect.), but it can still don't work well with eg. Scenarist. That's why you need to verify few streams to have at least some prove that it works fine.

Andrew

[J_Darnley]

P.S.: When will x264 have native .ts and .m2ts container support? :D
There has been discussion of other output formats, but from what I recall of the discussion, no one has been able to find an existing muxer framework/library/source that works correctly.

Wouldn't this only be of use when full libav input is done?

[neuron2]

I have deleted all the OT posts spawned by kolak's original OT troll.

@kolak

Knock it off! You can make one thread about CCEHD, but don't spam unrelated threads.

[benwaggoner]

I can't wait to put this into use. The encoder built into the new version of Compressor is... acceptable, but really chokes on noisy sources. Everything else is way out of my price range. T_T
I don't believe that's it's a new H.264 implementation, just Apple's standard (meh) implementation with the correct defaults for BD compliance.

Among other things, MPEG-2 is still required for interlaced coding.

[jsevakis]

Among other things, MPEG-2 is still required for interlaced coding.

That's... not true. I just made an interlaced .264 file with it, and TSMuxer confirms it as 1080i. Haven't plugged it into authoring yet, but it appears OK.

Agreed that it's a meh h.264 implementation at best, tho.

[WorBry]

From what I can gather, a compliant stream needs both --slices:4 (in the svn since early September) and --hal-nrd (patch available for some time, being worked into the SVN now). Is this accurate?

I'm still waiting for some BD-verifier results from a contact of mine...

Does the latest patch from Alex Giladi work? Trahald says that Alex's patch is much better and that we should get that one into git, not his.

Sorry to be a bit ignorant of x264 development and BD-compliance requirements, but can it be expected that an official x264 build incorporating hal-nrd will bring with it true MBAFF interlace encoding (i.e. MB adaptive, not just MBAFF signal syntax) or at least an improvement in interlace encoding efficiency?

[LoRd_MuldeR]

I think there is hope for NAL-HRD. But if you want MBAFF in x264, then it won't be cheap:
http://forum.doom9.org/showpost.php?p=1337505&postcount=5

[Revgen]

^You're talking to x264 programmers. The same programmers who IMO jumpstarted the whole anti-VFW crusade that still starts flame wars occasionally.

Needless to say, outdated legacy standards like interlacing aren't exactly going to garner much sympathy either. This shouldn't surprise anybody.

[Dark Shikari]

Sorry to be a bit ignorant of x264 development and BD-compliance requirements, but can it be expected that an official x264 build incorporating hal-nrd will bring with it true MBAFF interlace encoding (i.e. MB adaptive, not just MBAFF signal syntax) or at least an improvement in interlace encoding efficiency?Put simply: I am probably turning down over 20-30 thousand dollars by not working on MBAFF.

Maybe it will become something to consider once everything else on my list is done, but with most television channels eventually looking to move to 1080p60, interlacing will eventually be dead anyways.

[WorBry]

.....but with most television channels eventually looking to move to 1080p60, interlacing will eventually be dead anyways.

Hopefully by that time also there will be more (affordable) HD camcorders with 50p/60p recording capability. At present, there are just a few models in the professional lower-price bracket offering native 720/50p and/or 720/60p formats - Panasonic AG-HMC-150/151 and AG-HMC-40/41 (AVCHD) and JVC GY-HM100 (MPEG-2).

[jsevakis]

Okay, so it appears that what you have more or less committed right now has a decent chance of passing verification and can therefore be used to make Blu-rays. Freaking awesome. Can't wait till those results come back.

With that in mind, what features can/should be used? What part of the h.264 spec is illegal in Blu-ray, and what can safely be tweaked for optimum quality? Any recommended starting points?

[Trahald]

Alex's code is way more clean was really my point. It should be able to be committed with much fewer changes. bad thing is Alex has been MIA. I thought about looking at the code and fix the exact issue, but i dont want to necessarily inherit his since due to RL issues i cant guarantee I'll be around enough/at all to finish. At this point I have no reason to believe he wont resurface (maybe on holiday?) and finish.

[shon3i]

With that in mind, what features can/should be used? What part of the h.264 spec is illegal in Blu-ray, and what can safely be tweaked for optimum quality? Any recommended starting points?Use search i think is explained many times, what settings is allowed and what can be changed.

Alex's code is way more clean was really my point.What is different between your rev16 and rev19 because rev19 crash x264 with --keyint 24. Can you fix that. Your rev16 patch work stable for now. VFRManiac says that you chaged some calucaltions between r16 and r19

Bad assets from reatail BD are the last one to suspect in my opinion.Well it's true, as you can see, i found few more titles with different codec's, but i have little collection of BD's :)

Verifiers work on muxed streams not elementary stream, which makes sens. That's why you need all header information to be correct, becuase they're used during muxing process.Agree, and if x264 produce compatible stream that accomplish all parameters i don't see what muxer can do to make incompatible BD, then muxer is bad.

[popper]

dont any free so called stand alone 'BR Verifiers' exist at this time?, and if not, do we Need at least one OSS licenced BR compliance Verifier today?

as for so called 'Non-compliant BR Encoders' it seems really odd that the professional global post production community would pay around the $5K per copy for 'Non-compliant BR Encoders' or even a wopping $25k per unit for Unverified Encoders.

it seems that $25k per unit could get you a reasonable amount of one time OSS compliant BR x264 code patches if your Org is/were in need of bulk purchase, and thats before the potential x264 licence chances for commercial users come in.
see:http://www.netblender.com/main/resources/wikipapers/blu-ray-encoding/

hell you might even get some of the worlds Devs actually interested in making a professional grade intigrated GUI generic IDE suite and related tools for generic post production timecode based (Virtual/remote) video capture,Transport, Decoding, editing , Encoding etc in time...., if you put your $25k per unit corporate money,backing and effort into OSS production in this finantial down turn...., and in time, turn a better profit on your corporate books in the next quarter or two perhaps.

[kolak]

Agree, and if x264 produce compatible stream that accomplish all parameters i don't see what muxer can do to make incompatible BD, then muxer is bad.

As I said before- it's not really true.
The same as x264 makes proper stream, muxer can be also good, but not work with x264, becuase some things are not specified in the BD spec.
Sony/Sonic works with all companies which has made pro encoder (and opposite) to make sure they do work well together.

It's quite obvious, that "good" BD compliant stream should work with Sonic/Sony muxer, but it needs some testing at the early stages.


Andrew

[Stacey Spears]

I thought I would provide an update. Using x264_x86_r1301_with_nal-hrd_16, I was able to import and mux with Sonic 5 and pass verification with the Sony verifier. I tested both Profile 4.0 and 4.1.

Here are the settings I used:
x264.exe --profile high --level 4.1 --thread-input --threads 1 --keyint 24 --min-keyint 2 --direct auto --aq-mode 1 --ref 4 --slices 4 --qpmin 1 --qpmax 1 --qpstep 1 --ipratio 1.0 --pbratio 1.0 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-mbtree --merange 32 --me tesa --subme 10 --partitions all --trellis 2 --no-fast-pskip --no-dct-decimate --psy-rd 1.0:0 --psnr --mvrange 511 --nal-hrd --aud --sar 1:1 --output foo_41.264 MB_Chroma.avs

x264.exe --profile high --level 4.0 --thread-input --threads 1 --keyint 24 --min-keyint 2 --direct auto --aq-mode 1 --ref 4 --slices 1 --qpmin 1 --qpmax 1 --qpstep 1 --ipratio 1.0 --pbratio 1.0 --vbv-bufsize 30000 --vbv-maxrate 24000 --no-mbtree --merange 32 --me tesa --subme 10 --partitions all --trellis 2 --no-fast-pskip --no-dct-decimate --psy-rd 1.0:0 --psnr --mvrange 511 --nal-hrd --aud --sar 1:1 --output foo_40.264 MB_Chroma.avs

I encoded synthetic static test patterns. My goal is to get as close as possible to mathematically lossless. I assumed that --no-psy would have provided a better psnr, but it drops psnr by almost 3 db.

[LoRd_MuldeR]

If your priority is good PSNR (instead of good subjective quality), then use "--tune psnr" ;)

[Stacey Spears]

For test patterns, PSNR is something I consider critical. They are used to measure (objective, not subjective) other devices. (Blu-ray player, video processor, display, etc...) I don't want compression getting in the way. I will give the setting a try.

For real images, I consider PSNR a worthless measurement. :)

[Dark Shikari]

For test patterns, PSNR is something I consider critical. They are used to measure (objective, not subjective) other devices. (Blu-ray player, video processor, display, etc...) I don't want compression getting in the way. I will give the setting a try.If you care about maximum PSNR, why are you using such insane bizarre commandlines?

More than half your options make no sense whatsoever.

[Stacey Spears]

If you care about maximum PSNR, why are you using such insane bizarre commandlines?

What do you recommmend? So far, the settings provided produce the highest PSNR.

Ignore the B frame option in the command line. I left the option in the example with it set to 0 incase I wanted to put it back in. (being lazy) I also thought, possibly incorrectly, that using fixed QP ignores my VBV settings, so I forced fixed QP this way.

If your priority is good PSNR (instead of good subjective quality), then use "--tune psnr"

PSNR goes down when I use this option.

w/o --tune psnr
x264 [info]: PSNR Mean Y:100.000 U:70.920 V:70.993 Avg:75.727 Global:75.727 kb/s:19192.22

w/ --tune psnr
x264 [info]: PSNR Mean Y:100.000 U:68.125 V:68.152 Avg:72.909 Global:72.909 kb/s:18761.98

This particular pattern is a zone plate that exists in chroma only.

[Dark Shikari]

What do you recommmend?Dump the whole thing and use --preset placebo --tune psnr --level 4.1 --vbv-bufsize 30000 --vbv-maxrate 40000, maybe with --qpmin 0?

PSNR goes down when I use this option.

w/o --tune psnr
x264 [info]: PSNR Mean Y:100.000 U:70.920 V:70.993 Avg:75.727 Global:75.727 kb/s:19192.22

w/ --tune psnr
x264 [info]: PSNR Mean Y:100.000 U:68.125 V:68.152 Avg:72.909 Global:72.909 kb/s:18761.98

This particular pattern is a zone plate that exists in chroma only.Well of course it does. Your PSNR is already absurdly high, so the only thing that matters is the chroma quantizer. You've set QPmin to 1, not 0, so the quantizer can't go all the way down. Psy-RD lowers the chroma quantizer by 2, which allows it to reach 0.

The only reason you're getting a PSNR benefit is because your options are weird in the first place.

[ajp_anton]

Both his qpmin and qpmax are 1 =)

[Dark Shikari]

Both his qpmin and qpmax are 1 =)Which makes the VBV settings rather pointless as well ;)

[Stacey Spears]

Both his qpmin and qpmax are 1 =)

I want my encodes to use a constant QP. Can't use the --qp option AND set the VBV.

Which makes the VBV settings rather pointless as well

Not if you a.) want to use a constant QP and b.) want to know if you underflow using a constant QP. :)

[Blue_MiSfit]

Why do you want to use a constant QP???

It seems to me that all you're really trying to do here is maximize PSNR, to replicate your test patterns as close to lossless as possible, while still working within the BluRay spec - right? If so, why re-invent the wheel??

Just dump your options and ideas, use --tune psnr and --preset placebo, and 2 pass target bitrate mode with VBV set for BluRay specs, plus the extra nal-hrd crap that's necessary for bluray compliance.

x264's rate control is very good, and will do a good job with your bit budget :)! If you must, you can lower qpmin, but I would suggest just leaving things as-is.

~MiSfit

[Sagittaire]

I want my encodes to use a constant QP. Can't use the --qp option AND set the VBV.

Constant qp doesn't mean constant quality for eyes ...



Not if you a.) want to use a constant QP and b.) want to know if you underflow using a constant QP. :)

... particulary if you use qp=1.

[shon3i]

Constant qp doesn't mean constant quality for eyes ...
he already knows that

For real images, I consider PSNR a worthless measurement.

btw i run muxed stream (sonic 5) with lastest Alex HRD patch (Techouse build) tought Sony verifier and passes verification. I do only for one move. I will do more testing.

[Stacey Spears]

It seems to me that all you're really trying to do here is maximize PSNR, to replicate your test patterns as close to lossless as possible, while still working within the BluRay spec - right?

That is correct.

My original point for posting was to update the thread to let the community know that x264 passes the Sony BD verifier, at least when muxed with Sonic 5.

The BDA will approve the 3D standard by the end of the year. Right now only PHL can encode 3D for BD and they are already maxed out for 2010 with work from Disney, Dreamworks and WHV. Any chance this support can/will be added to x264?

[shon3i]

My original point for posting was to update the thread to let the community know that x264 passes the Sony BD verifier, at least when muxed with Sonic 5.Can you test aslo with this build? http://forum.doom9.org/showthread.php?p=1346701#post1346701 because this second with different HRD patch have bigger chances to become a part of x264. I tested once and passes, but i want to see other expirience.

[Lyris]

That's absolutely excellent that we've had a successful verification result. Hopefully we can get more feedback on whether or not the most recent versions are passing. Stacey, thanks for the info.