x264 letterbox border detail loss

[kabelbrand]

When encoding videos with a top letterbox size that is not a multiple of 16 there often is some noticable loss of detail within the first few lines.
This happens especially with sources containing grain and weirdly only if the runtime is longer e.g. a trailer encode will be ok but a full feature encode isn't.

For example if you have a 1920x1080 source with a top letterbox of 132 pixels (2.35:1) the first 12 lines of active picture seem blurred or not as detailed as the rest of the picture.
The obvious solution is to shift the picture up or down a few pixels to the next macroblock border. This might be ok with a large letterbox but if you have a small letterbox like 20 pixels (1.85:1) the viewer will notice the picture is not centered.

Can you think of any other solutions to this problem?

Settings: --preset slower --keyint 24 --slices 4 --bitrate 30000 --vbv-maxrate 40000 --vbv-bufsize 30000 --level 4.1 --bluray-compat

[LoRd_MuldeR]

Why not crop away the black borders completely?

[kabelbrand]

Originally Posted by LoRd_MuldeR

Why not crop away the black borders completely?

Unfortunately that's not an option since I am aiming for Blu-ray compatibility.

[detmek]

Blu-Ray compatibility or Blu-Ray Disc authoring? For compatibility you can use crop.

[LoRd_MuldeR]

Just an idea: Might this be related to Psy-RDO/Psy-Trellis? Those try to retain the same "complexity" in each compressed block as in the original block. Now, if you look at those blocks directly on the border between the content and the "black" area, about 1/2 of the block is just 100% black, which means zero complexity in that part of the block. Overall, these blocks probably are strongly biased to look like "low complexity" blocks to the encoder. So Psy-RDO/Psy-Trellis will try to keep them "low complexity", which might explain the blurring - even in the part of the block that is not 100% black.

Other than that: Have you tried with and without MB-Tree enabled?

[kabelbrand]

Originally Posted by detmek

Blu-Ray compatibility or Blu-Ray Disc authoring? For compatibility you can use crop.

BD Authoring, so after cropping (for clean black borders) the video is padded back to full resolution.

Originally Posted by LoRd_MuldeR

Overall, these blocks probably are strongly biased to look like "low complexity" blocks to the encoder.

That makes sense. I was wondering if x264 looks at the whole macroblock to calculate the complexity or just a smaller area which might be completely black.

Originally Posted by LoRd_MuldeR

Other than that: Have you tried with and without MB-Tree enabled?

Yes. While it makes fades look a bit better there is no quality gain at the letterbox border.

[Lyris]

kabelbrand, I had this exact same problem - a very textured film with borders top and bottom, and the grain would get stripped away inside the block with the black border.

As you say, with large letterboxing you can just shift the image a little, but with thin strips top and bottom, that'd look odd.

Jason and Kieran pointed me to the solution: AQ. From my experience, the setting you need to adjust is --aq-strength. The default is 1.0. --aq-strength 0.5 or so should resolve this to a satisfactory extent, although I believe the trade-off is coarser compression in dark areas (although you'd have to crank up the brightness and/or gamma on a badly adjusted TV for this to really be a problem).

[LoRd_MuldeR]

Shouldn't AQ tend to improve those blocks located directly on the border?

About 1/2 of these blocks is pure black, so those blocks should be biased to appear relatively "flat". And, AFAIK, the VAQ algorithm moves more bits into "flat" blocks, whilst taking away bits from "complex" blocks...

[kabelbrand]

Originally Posted by Lyris

--aq-strength 0.5 or so should resolve this to a satisfactory extent

Thanks. I remember testing with --preset grain (includes --aq-strength 0.5) unsuccessfully before but I'll give it another try.

[Dark Shikari]

Re-generate the borders so that they're on mod-16 pixel boundaries (e.g. slightly crop the film, or move it up/down a few pixels) and the problem will go away.

[kabelbrand]

Originally Posted by Dark Shikari

Re-generate the borders so that they're on mod-16 pixel boundaries (e.g. slightly crop the film, or move it up/down a few pixels) and the problem will go away.

I had hoped for a more elegant solution... Cropping/padding an additional 4 pixels on each side for a 2.40:1 feature might be ok but 12 pixel each for a 2.35:1 or 1.85:1 feature just feels wrong.

Anyway... Thank you all!

[benwaggoner]

Originally Posted by kabelbrand

I had hoped for a more elegant solution... Cropping/padding an additional 4 pixels on each side for a 2.40:1 feature might be ok but 12 pixel each for a 2.35:1 or 1.85:1 feature just feels wrong.

Not on the sides, but top/bottom.

For example, a 1920x1080 source with 140 pixels of black on the top and bottom. 140 isn't mod16. So you could move it down slightly to have 144 on the top and 136 on the bottom, both of which are mod16* aligned.

*"1080p" is really encoded as 1920x1088 pixels with the bottom 8 of dummy data that isn't displayed. So that 136 is really 136+8=144 which is mod16. The key thing is that the bottom black bar starts at 944 pixels from the top, which is mod16.

[kabelbrand]

Cropping/Padding to the next multiple of 16 is what I was doing in the past and I just wondered if there was a better way.

I even tried to add an additional 4 black lines to the top and explicitly set the bitstream cropping rectangle to crop these instead of the bottom 8 lines for 1088->1080 to have that 140 letterbox aligned with macroblock boundaries but this did not work as I expected and I ended up with weird resolutions like 1920x1076.

Originally Posted by benwaggoner

The key thing is that the bottom black bar starts at 944 pixels from the top, which is mod16.

At least with decent bitrates the bottom border does not seem to be problematic with x264.

[benwaggoner]

Originally Posted by kabelbrand

Cropping/Padding to the next multiple of 16 is what I was doing in the past and I just wondered if there was a better way.

That should work. Just remember to count mod16 from the top.

I even tried to add an additional 4 black lines to the top and explicitly set the bitstream cropping rectangle to crop these instead of the bottom 8 lines for 1088->1080 to have that 140 letterbox aligned with macroblock boundaries but this did not work as I expected and I ended up with weird resolutions like 1920x1076.

Your source was never really 1088. It was 1080 with 8 null lines at the bottom that contain no visual data and should be ignored.

At least with decent bitrates the bottom border does not seem to be problematic with x264.

As long as you hit it at least mod8 you should be okay, so you are luma block aligned.

[kabelbrand]

Originally Posted by benwaggoner

Your source was never really 1088. It was 1080 with 8 null lines at the bottom that contain no visual data and should be ignored.

I am aware of this. If your source is 1080 and you have a 140 pixel top letterbox you could add an extra 4 black lines on top and 4 on the bottom to get 1088 lines. The top letterbox will now be aligned with a macroblock border at 144 pixels.
Then encode using "--crop-rect 0,4,0,4" which should signal the playback software to discard the top and bottom 4 lines to render a 1080 picture. Unfortunately this won't be usable for Blu-ray Disc because the specification allows cropping at the bottom only.

Other software seems to ignore "frame_crop_top_offset" in the SPS as well: PowerDVD, VLC, DGIndexNV

[benwaggoner]

Originally Posted by kabelbrand

I am aware of this. If your source is 1080 and you have a 140 pixel top letterbox you could add an extra 4 black lines on top and 4 on the bottom to get 1088 lines. The top letterbox will now be aligned with a macroblock border at 144 pixels.
Then encode using "--crop-rect 0,4,0,4" which should signal the playback software to discard the top and bottom 4 lines to render a 1080 picture. Unfortunately this won't be usable for Blu-ray Disc because the specification allows cropping at the bottom only.

You're not cropping, but shifting. If you shift up 8 pixels by cropping 8 more at the top, you need to add 8 black pixels to the bottom before you reencode. The input frame to the encoder for BD always needs to be 1080 for 1080p.

[sneaker_ger]

I don't see anything wrong with kabelbrand's last post. He's shifting down (not up) by 4 (not 8) pixels. You can certainly manually tell x264 where to put the cropping - otherwise it will automatically do it on the right and bottom side. The only problem would indeed be that the Blu-Ray specs don't allow it (assuming that is correct, I don't know if it's true), i.e. it's no use here.

[kabelbrand]

Originally Posted by benwaggoner

You're not cropping, but shifting. If you shift up 8 pixels by cropping 8 more at the top, you need to add 8 black pixels to the bottom before you reencode. The input frame to the encoder for BD always needs to be 1080 for 1080p.

Shifting or Cropping&Padding - I think we are kinda talking about the same thing
Encoding a 1080 source with x264 is actually as good as encoding a 1088 source with "--crop-rect 0,0,0,8" specified.
Both are ok for BD - x264 (and other encoders) will repeat the last line of the 1080 source 8 times to fill 68 macroblocks vertically (1088 pixels) and specify 8 pixels bottom cropping at bitstream level internally.

But as I said, specifying bitstream cropping other than at the bottom is illegal for BD. Sadly...