Best settings for encoding at very low bitrates

[arranhs]

Hey, I have been lurking on Doom9 for a while now and I was hoping I could get some help!

I am looking to encode some 1080p video, after downscaling it to 720p, using x265 at a very low bitrate - between 750 Kbps and 1 Mbps. I have been trying to find the best settings to get the best quality at this bitrate and was hoping for some advice as there doesn't seem to be a lot of low bitrate advice so far on this forum as far as I can see.

I would like to mention that I am already aware that there will be a very significant amount of detail lost in the videos at this bitrate, but as the videos are for reference purposes only and I will have a lot of them, I am more interested in saving space at this point, and the bitrates I have mentioned give approximately the size I am looking for. That being said, I would naturally like to squeeze out as much quality as I can at this bitrate. The videos also have a mixture of light and dark, so this is a consideration.

These are the settings that I have so far that seem to give the best results (that I have tested) on some test footage I have:
2 pass, bitrate 750k, 10-bit
--preset=slow (Slower was a little bit too slow for me - I'm looking for a slow/slower middle ground)
--bframes=8
--ref=6
--rect (the slow default)
--amp
--psy-rd=1 (the slow default)
--psy-rdoq=1 (a larger value seemed to produce a lower quality by my comparison)

Extra notes:
- I tried aq-mode=3/4 but their quality seemed worse than the default aq-mode=2
- Tried increasing rd, this gave a worse output.
- Tried no-sao and deblock=-1:-1 that I have seen mentioned a lot around here, but these settings were definitely being used to retain detail at far higher bitrates than I am targeting.

If there are any suggestions on how I can further improve these settings, the help would be massively appreciated

[excellentswordfight]

I think that you actually have a rather good baseline there, that is already rather fitting to your target.

Originally Posted by arranhs

- I tried aq-mode=3/4 but their quality seemed worse than the default aq-mode=2

In my experience its not "worse", it just a different tradeoff and it will be somewhat source dependent on what would give the best result.

Originally Posted by arranhs

- Tried no-sao and deblock=-1:-1 that I have seen mentioned a lot around here, but these settings were definitely being used to retain detail at far higher bitrates than I am targeting.

And that is expected, as SAO is beneficial for very high compression. But instead of trying no-sao, you could play with --selective-sao, although you are at such low bitrates that it might be better just keeping it fully on.

[TDS]

Originally Posted by arranhs

Hey, I have been lurking on Doom9 for a while now and I was hoping I could get some help!

I am looking to encode some 1080p video, after downscaling it to 720p, using x265 at a very low bitrate - between 750 Kbps and 1 Mbps. I have been trying to find the best settings to get the best quality at this bitrate and was hoping for some advice as there doesn't seem to be a lot of low bitrate advice so far on this forum as far as I can see.

I would like to mention that I am already aware that there will be a very significant amount of detail lost in the videos at this bitrate, but as the videos are for reference purposes only and I will have a lot of them, I am more interested in saving space at this point, and the bitrates I have mentioned give approximately the size I am looking for. That being said, I would naturally like to squeeze out as much quality as I can at this bitrate. The videos also have a mixture of light and dark, so this is a consideration.

These are the settings that I have so far that seem to give the best results (that I have tested) on some test footage I have:
2 pass, bitrate 750k, 10-bit
--preset=slow (Slower was a little bit too slow for me - I'm looking for a slow/slower middle ground)
--bframes=8
--ref=6
--rect (the slow default)
--amp
--psy-rd=1 (the slow default)
--psy-rdoq=1 (a larger value seemed to produce a lower quality by my comparison)

Extra notes:
- I tried aq-mode=3/4 but their quality seemed worse than the default aq-mode=2
- Tried increasing rd, this gave a worse output.
- Tried no-sao and deblock=-1:-1 that I have seen mentioned a lot around here, but these settings were definitely being used to retain detail at far higher bitrates than I am targeting.

If there are any suggestions on how I can further improve these settings, the help would be massively appreciated

So just out of interest's sake, what encoding app are you using ??

[arranhs]

Originally Posted by excellentswordfight

And that is expected, as SAO is beneficial for very high compression. But instead of trying no-sao, you could play with --selective-sao, although you are at such low bitrates that it might be better just keeping it fully on.

Thanks for the helpful suggestion. I'll give selective-sao a go. There's no harm in trying!

Found your point on aq-mode interesting, because I think you are right having relooked. 3 and 4 are not "worse" per say, but the trade-offs definitely makes certain frames look "worse" for my eye.

I wonder if I should give aq-mode=1 a try and add that to the comparison.

Originally Posted by TDS

So just out of interest's sake, what encoding app are you using ??

I am using ffmpeg, with libplacebo for scaling (scale and zscale are much lower quality). I am using ewa_lanczos as the downscaled as it is by far the best quality.

If you are interested here are the first and second pass commands I am currently using (I am using bash substitution for the file),

ffmpeg \
  -init_hw_device vulkan \
  -i "${1}" \
  -map 0:v:0 \
  -c:v libx265 \
  -profile:v main10 \
  -b:v 750k \
  -preset:v slow \
  -x265-params "
    bframes=8:
    ref=6:
    rect=1:
    amp=1:
    psy-rd=1:
    psy-rdoq=1:
    pass=1
  " \
  -vf "
    hwupload,
    libplacebo=
      w=1280:h=-1:
      downscaler=ewa_lanczos:
      deband=true:
      colorspace=bt709:
      color_primaries=bt709:
      color_trc=bt709:
      range=limited:
      format=yuv420p10le,
    hwdownload,
    format=yuv420p10le
  " \
  -an \
  -f null \
  -

ffmpeg \
  -init_hw_device vulkan \
  -i "${1}" \
  -map 0:v:0 \
  -c:v libx265 \
  -profile:v main10 \
  -b:v 750k \
  -preset:v slow \
  -x265-params "
    bframes=8:
    ref=6:
    rect=1:
    amp=1:
    psy-rd=1:
    psy-rdoq=1:
    pass=2
  " \
  -vf "
    hwupload,
    libplacebo=
      w=1280:h=-1:
      downscaler=ewa_lanczos:
      deband=true:
      colorspace=bt709:
      color_primaries=bt709:
      color_trc=bt709:
      range=limited:
      format=yuv420p10le,
    hwdownload,
    format=yuv420p10le
  " \
  -map 0:a:0 \
  -c:a libfdk_aac \
  -profile:a aac_he_v2 \
  -vbr 3 \
  -ac 2 \
  "$2"

[HD MOVIE SOURCE]

Originally Posted by arranhs

Hey, I have been lurking on Doom9 for a while now and I was hoping I could get some help!

I am looking to encode some 1080p video, after downscaling it to 720p, using x265 at a very low bitrate - between 750 Kbps and 1 Mbps. I have been trying to find the best settings to get the best quality at this bitrate and was hoping for some advice as there doesn't seem to be a lot of low bitrate advice so far on this forum as far as I can see.

I would like to mention that I am already aware that there will be a very significant amount of detail lost in the videos at this bitrate, but as the videos are for reference purposes only and I will have a lot of them, I am more interested in saving space at this point, and the bitrates I have mentioned give approximately the size I am looking for. That being said, I would naturally like to squeeze out as much quality as I can at this bitrate. The videos also have a mixture of light and dark, so this is a consideration.

These are the settings that I have so far that seem to give the best results (that I have tested) on some test footage I have:
2 pass, bitrate 750k, 10-bit
--preset=slow (Slower was a little bit too slow for me - I'm looking for a slow/slower middle ground)
--bframes=8
--ref=6
--rect (the slow default)
--amp
--psy-rd=1 (the slow default)
--psy-rdoq=1 (a larger value seemed to produce a lower quality by my comparison)

Extra notes:
- I tried aq-mode=3/4 but their quality seemed worse than the default aq-mode=2
- Tried increasing rd, this gave a worse output.
- Tried no-sao and deblock=-1:-1 that I have seen mentioned a lot around here, but these settings were definitely being used to retain detail at far higher bitrates than I am targeting.

If there are any suggestions on how I can further improve these settings, the help would be massively appreciated

Could you do me a favor and try psy-rd=1.60 and --psy-rdoq=0.00 with the rdoq level set to 2. This does very well for me in film content.

I personally only use aq mode 3 for animation with no film grain. For film grain I like aq mode 0, basically off, but I'm encoding at high bit-rates.

I don't like sao or deblock, however, I think with low but-rates like yours it may be needed. As some have suggested with sao, you could use selective. The only reason I don't like it is because of blurring or softness.

Default deblock may be useful for low bit-rates I believe. But I'm not sure about how strong it needs to be based on the bit-rates.

[Boulder]

Some suggestions:

--aq-mode 2 is probably better than 1 for low-bitrate encodes. Mode 3 is a definite no-no because it requires more bits due to the dark bias.
You could try lowering aq-strength one or two notches, like 0.8-0.9.

Set --rskip 2 and see if it makes a difference.

At low bitrates, --rd 6 might be useful to couple with --rd-refine.

Increase the amount of b-frames to 10. In some cases, it may help a lot.

Try --deblock 0:0 or 1:1. Your encode will be blurry anyway.

[arranhs]

Thanks for all the replies so far, I have found them incredibly helpful. I've gone away and run some tests, but have only managed to complete 3 so far:

- aq-mode=3, aq-strength=0.8 - This was indeed a big improvement over aq-strength=1. I noticed a big improvement in dark scenes but sadly light scenes were worse. I think these settings would be perfect for if I know the video will be dark, but for general video, probably not as good as the baseline I currently have.

- psy-rd=1.6, psy-rdoq=0, psy-rdoq=2 - These settings gave a very slight (almost unnoticeable tbh) improvement in light scenes, but a significant decrease in dark scene quality. I wonder why this is. Why does psy options affect dark scenes so much? Due to how much it degrades dark scenes I'd be concerned using this over the baseline for the very minor light scene improvement it provides.

- bframes=10 makes dark scene noticibly highter quality. It also slightly increases quality of light scenes but the improvements are mostly negligible. Interestingly however it gets a lower vmaf and psnr score (the same ssim) as my baseline, but I wonder if that is just imperfect nature of these metrics.

Currently working on some tests for rd=6 and rd-refine, rskip=2 and selective-sao (I am going to give 2 a go).

On a separate note, I have seen it mentioned that --nr (noise reduction) could help at low bitrates, but I am unsure at what a reasonable initial setting might be to try. Any suggestions?

[microchip8]

if you get banding and/or bad quality in dark frames, you should be increasing psy-rd and psy-rdoq, not lower/disable it. Try something like psy-rd=4 and psy-rdoq=15

[Boulder]

Originally Posted by arranhs

- aq-mode=3, aq-strength=0.8 - This was indeed a big improvement over aq-strength=1. I noticed a big improvement in dark scenes but sadly light scenes were worse. I think these settings would be perfect for if I know the video will be dark, but for general video, probably not as good as the baseline I currently have.

There's jpsdr's modded build available which contains the aq-auto feature. It will automatically switch between various modes depending on the brightness and the edge densities detected. See the big x265 thread here.

- bframes=10 makes dark scene noticibly highter quality. It also slightly increases quality of light scenes but the improvements are mostly negligible. Interestingly however it gets a lower vmaf and psnr score (the same ssim) as my baseline, but I wonder if that is just imperfect nature of these metrics.

Metrics really can't be trusted, your eyes are what tell you what you like. Many option decrease the metrics but perceptual quality during playback increases.

[arranhs]

Originally Posted by microchip8

if you get banding and/or bad quality in dark frames, you should be increasing psy-rd and psy-rdoq, not lower/disable it. Try something like psy-rd=4 and psy-rdoq=15

Hmm, that's interesting to hear cos my experimentation and understanding was that the opposite was true. But I have never tried such large values so I might give them a go and see

[arranhs]

I have also just run a test on rskip=2 and sadly it didn't provide any benefit. Dark scenes particularly were very bad

[microchip8]

Originally Posted by arranhs

Hmm, that's interesting to hear cos my experimentation and understanding was that the opposite was true. But I have never tried such large values so I might give them a go and see

Same. I came up with these values after a lot of experimentation trying to get rid of banding in low-light/dark scenes. These are the values that completely got rid of the banding. Make sure to use rdoq-level=1.

All my encodes are virtually banding-free now, as I always use these values along with aq-mode=1 (at default strength).

[benwaggoner]

First, what kind(s) of content are you compressing? How will you be watching it? The more focused the scenario, the finer-tuned the recommendations we can make.

Your average bitrate isn't that low for 720p. My Compression Challenge has had some decent 1080p entries down to 500 Kbps ABR.

Banding issues tend to not be so bad in 10-bit. Have you checked that your preprocessed input is 10-bit and doesn't have banding before it hits x265?

Throwing in some --nr-inter and --nr-intra can help quite a lot at low bitrates. As they are integrated into quantization, they do a smart job of reducing small details that have a bigger bitrate impact, and so can lower QP improving quality overall. I'd try with something like --nr-intra 100 --nr-inter 400 and see if that yields an improvement.

I'd suggest just using --preset slower and iterating from there. You're already manually setting a number of parameters to their slower values anyway. And the presets have pretty well balanced and well-tested combinations of settings. Still, they haven't been refactored for a while, so there are some newer options that are worth adding:
--selective-sao 2: this is a small speed boost; SAO only really helps forward prediction, so 99% of the benefit comes from applying it to I and P frames. I wouldn't ever use 4, and would only use 3 if I was already at placebo.
--rskip 2: faster and more robust with noise than --rskip 1. --rskip-edge-threshold 3 or 2 (default is 5) can improve visual results a bit at the cost of a little quality.
--aq-mode 4: this seems slightly better than 2 for SDR content. You might want to lower --aq-strength a bit when using it, as it is a bit stronger than other modes.
--nr-intra 100
--nr-inter 400

[benwaggoner]

And as you're worried about speed, and 720p is a relatively lower resolution, you might try --pmode if you've got a bunch of idle cores during encoding.

Setting --frame-threads 1 can help improve quality somewhat, but probably not worth it if you're struggling to use --preset slower.

Also, if your source isn't 10-bit, you can also try using --dither.

[PatchWorKs]

...it would be great to have - if technically possible - such kind of preset for rigaya's HW-accellerated encoders too.

[HD MOVIE SOURCE]

Originally Posted by arranhs

Thanks for all the replies so far, I have found them incredibly helpful. I've gone away and run some tests, but have only managed to complete 3 so far:

- aq-mode=3, aq-strength=0.8 - This was indeed a big improvement over aq-strength=1. I noticed a big improvement in dark scenes but sadly light scenes were worse. I think these settings would be perfect for if I know the video will be dark, but for general video, probably not as good as the baseline I currently have.

- psy-rd=1.6, psy-rdoq=0, psy-rdoq=2 - These settings gave a very slight (almost unnoticeable tbh) improvement in light scenes, but a significant decrease in dark scene quality. I wonder why this is. Why does psy options affect dark scenes so much? Due to how much it degrades dark scenes I'd be concerned using this over the baseline for the very minor light scene improvement it provides.

- bframes=10 makes dark scene noticibly highter quality. It also slightly increases quality of light scenes but the improvements are mostly negligible. Interestingly however it gets a lower vmaf and psnr score (the same ssim) as my baseline, but I wonder if that is just imperfect nature of these metrics.

Currently working on some tests for rd=6 and rd-refine, rskip=2 and selective-sao (I am going to give 2 a go).

On a separate note, I have seen it mentioned that --nr (noise reduction) could help at low bitrates, but I am unsure at what a reasonable initial setting might be to try. Any suggestions?

Just wondering, what do use to measure quality? And tell the difference in the quality of light and dark scenes?

[benwaggoner]

Originally Posted by HD MOVIE SOURCE

Just wondering, what do use to measure quality? And tell the difference in the quality of light and dark scenes?

These bitrates are all about the psychovisually preferable tradeoffs, so eyeballs are the gold standard. x265 will log per frame PNSR and SSIM with --psnr --ssim --csv-log-level 2.

If I have too much content to viably eyeball, I'll parse the .csv files and look for spikes of high QP (higher is more artifacts) or low YUV-PSNR (only built-in metric that looks at chroma) and SSIM over a --keyint rolling average.

VMAF is better than any one of those, but is also luma only and not great at comparing adaptive quantization or penalizing banding as much as its psychovisual impact.

A per-encode metrics score isn't very useful for content longer than a couple of minutes, as subjective quality is much more dependent on how bad the worst shots look than the mean. Hence looking for spikes in rolling average.

And yeah, I have some really big Excel files for full-length movies ! Using .xlsb instead of .xlsx cuts file size in about half.

[benwaggoner]

Originally Posted by arranhs

TI am using ffmpeg, with libplacebo for scaling (scale and zscale are much lower quality). I am using ewa_lanczos as the downscaled as it is by far the best quality.

I personally prefer to use spline36 instead of lanczos for downscaling for recompression. Lanczos is "sharper" but increases risk of ringing artifacts. And higher frequencies are intrinsically harder to encode, and so can yield QP increases. Spline36 is quite neutral and handles 25%-400% scaling without aliasing. While it might have a little less apparent detail before encoding, at low bitrates it will generally be equal or better quality as it doesn't have artificial high frequencies to encode.