Bizarre x264encoder problem

[killmoms]

Hey all,

So I know QuickTime isn't the most beloved piece of software around these parts, but I'm trying to solve a very, very bizarre problem which involves QT and the x264encoder component for it.

Basically, I used to have a setup that worked. I could use x264encoder to create .mov files (and mux them to .mp4) that looked great and played great in QuickTime on Mac or PC, or any other software that supported H.264.

Now, any .mov or .mp4 files I make by encoding with x264encoder (a newer version) chug horribly in QuickTime—even at the same profile and level with the same (or less strenuous) settings. I cannot for the life of me figure out what is going wrong or what setting I've got checked that's causing such sub-par playback.

I compared the encodes I'm having trouble with now to ones I did using x264encoder in the past. The "old" files identify as High Profile @ Level 3.1, CABAC encoding with 5 reference frames at 6,406kbit/s in MediaInfo Mac, and they run great. Activity Monitor shows low CPU-utilization spread out amongst all 8 cores. These new ones (again according to MediaInfo Mac) are Main Profile @ Level 3.1, CABAC encoding with 2 reference frames at 3,961kbit/s. Considering every element is less intense, you'd think these would be even easier to decode—but they're not. They chug along at 17 - 21fps and Activity Monitor shows one core being pinged at 100% activity. I've tried turning off numerous options in x264encoder's settings (FLAG_LOOP_FILTER, FLAG2_WPRED, FLAG2_MIXED_REFS, FLAG2_8X8DCT, FLAG2_FASTPSKIP, and FLAG2_MBTREE among others) and nothing seems to make a difference.

Weirdest of all, I don't think it's a QuickTime playback bug because those old files still play perfectly fine now, right alongside the new files that play badly.

My source files are 10-bit Uncompressed 4:2:2 QuickTime files. I thought perhaps some internal weirdness with 10-bit data was responsible for the issues (even though I don't believe x264 can even create 10bpc H.264), so I tried creating files from 8-bit Uncompressed 4:2:2 as well.

Basically I'm at my wits' end and have no idea what I'm doing wrong. If anyone has any experience at all with a workflow like this, would you mind giving me some pointers?

[Dark Shikari]

Paste the entire output of "strings file | grep x264" for both an old file and new file.

[killmoms]

Seems I jumped the gun on posting—I believe I found something that's made a difference. Changing "weightp pred:" from "Smart analysis" to "Disabled" seems to have improved matters significantly. I'm still testing with other options to see what I can turn off and on and how those adjustments affect performance.

Dark Shikari: I tried running that command on a couple of my old MP4 files that I described above, but strings returned no output whatsoever. Could this be because x264encoder runs within QuickTime (and generates .mov files, which I then have to re-mux to .mp4)?

[Dark Shikari]

Originally Posted by killmoms

Seems I jumped the gun on posting—I believe I found something that's made a difference. Changing "weightp pred:" from "Smart analysis" to "Disabled" seems to have improved matters significantly. I'm still testing with other options to see what I can turn off and on and how those adjustments affect performance.

Dark Shikari: I tried running that command on a couple of my old MP4 files that I described above, but strings returned no output whatsoever. Could this be because x264encoder runs within QuickTime (and generates .mov files, which I then have to re-mux to .mp4)?

Sounds like the muxer is stripping the userdata SEI then.

My guess would be that if the old x264encoder was sufficiently ancient, it would have generated multislice files by default, which Quicktime could have multithreaded on.

[killmoms]

Define "sufficiently ancient." A backup of my previous installation (before wiping for upgrades) has an x264encoder that dates back to only October 2, 2009.

Though, to be fair, the "old" 6mbit file I was talking about dates back to late July of 2009, so that was likely generated by an even older version of x264encoder.

The muxer in question is QuickTime's MPEG-4 exporter, which has a "Pass-through" option for video and audio data that's alread AVC/AAC. God only knows what data it strips out.

[Dark Shikari]

Originally Posted by killmoms

define "sufficiently ancient."

2006.

[killmoms]

Originally Posted by Dark Shikari

2006.

Well, I doubt it was THAT old. I didn't even work here until 2008, and x264encoder wasn't installed when I got here.

Anyway, I'm going to chalk this one up to user error for not knowing which specific options (out of the millions possible) was responsible for my problem. And for not having any idea what "weighted p-frame prediction" is.

Thanks for the help!

[LoRd_MuldeR]

Anyway, latest x264 supports slices again (optionally). So you could try something like "--slices 4" and check if QT likes that better...

[Blue_MiSfit]

I wasn't aware that the QuickTime H.264 decoder supported slice based multithreading! That's a reason to use slices, I guess!

I'm also playing with the "x264encoder" quicktime component for automated Quicktime processing via Compressor droplets. Interesting stuff... I wish the config GUI for "x264encoder" made a bit of sense... For example, there is a flag to enable CRF encoding, but there is no field to put the crf number in! Also, to get rate control to work at all, you have to put the QuickTime quality slider all the way up, otherwise you get massively undersized files. Meh....

Write me a ProRes decoder for libavcodec pls Dark_Shikari, then I can just use x264cli for everything

~MiSfit

[LoRd_MuldeR]

I wasn't aware that the QuickTime H.264 decoder supported slice based multithreading! That's a reason to use slices, I guess!

I'd say: If QuickTime doesn't support multi-threading without slices, that's yet another reason to get rid of QuickTime

As if there weren't enough reasons...