My x265 settings so far for UHD DV and 1080p SDR for Visually Lossless

[Cmptr]

So I've been experimenting for weeks now, reading the forum, (Started to read the spec) gathering information and doing tests to find "General" settings I can use to transcode Blurays to visually lossless. The goal is to have a smaller file size, but retain as much detail as possible.

So far, this is what I've been using for UHD:
--preset slower --no-rect --ref 8 --bframes 8 --psy-rd 2.0 --psy-rdoq 2.0 --aq-mode 0 --aq-strength 0 --no-cutree --no-deblock --no-sao --no-strong-intra-smoothing --rd 4 --subme 5 --limit-refs 0 --rskip 0 --no-limit-modes --tu-inter-depth 2 --tu-intra-depth 2 --limut-tu 0 --max-merge 3 --open-gop --rc-lookahead 72 --min-keyint 24 --keyint 240 --vbv-buffsize 160000 --vbv-maxrate 160000 --level-idc 5.1

Some settings I didnt list because they are included with preset slower.

Obviously all the other color stuff and what dovi_scripts uses. I don't think I'm forgetting anything. I originally was using:

--crqpoffs -3 --cbqpoffs -3

But not sure if it was worth it or not, so for now I stopped.

For 1080p SDR, it is the same, but with the following:
--rd 6 --subme 7 --tu-inter-depth 4 --tu-intra-depth 4 --max-merge 5 --merange 96


Obviously the crf depends on the movie. When I did Obsession I used 15.5

What are everyone's thoughts and opinions? What would you change? Feel free to criticize me as well. Any feedback would be great.

If I forgot anything, I'll update it.

1080p SDR Update:
--preset slower --no-rect --ref 7 --bframes 8 --psy-rd 2.0 --psy-rdoq 2.0 --aq-mode 0 --aq-strength 0 --no-cutree --no-deblock --no-sao --no-strong-intra-smoothing --rd 4 --subme 7 --limit-refs 0 --rskip 0 --no-limit-modes --tu-inter-depth 4 --tu-intra-depth 4 --limut-tu 0 --max-merge 5 --open-gop --rc-lookahead 72 --min-keyint 24 --keyint 240 --vbv-buffsize 160000 --vbv-maxrate 160000 --level-idc 5.1 --merange 96

I messed around with aq-mode 1, cutree, rskip 2 with rskip-edge-threshold (I used rect and amp), ctu 32 and qg-size 16 and 8. I found that though they kinda helped in some areas, I always found macroblocking when deeply analyzing while pausing. I found them much more so compared to when I didn't use these settings. There was also some weird shifts of detail. A whole ctu would be shifted 2 or 3 ctu over. You could also notice that some werent recursed enough as well even though edge threshold was at 1, which I found out converts into a decimal range of 0.01 to 1.

So far I'm happy. Sometimes color is changed, so I could probably lower cr/cbqpoffs.

[microchip8]

Your ref is too high. Not sure why you disable rect and AQ. Optimal bframes is 6. Not using cbqpoffs/crqpoffs for HDR will make it look like SDR. Lower it to -4 as UHD has a much broader color gamut. There is a problem with rd above 4 in x265, end result is it lowers quality a bit instead of increasing it.

[Cmptr]

Your ref is too high. Not sure why you disable rect and AQ. Optimal bframes is 6. Not using cbqpoffs/crqpoffs for HDR will make it look like SDR. Lower it to -4 as UHD has a much broader color gamut. There is a problem with rd above 4 in x265, end result is it lowers quality a bit instead of increasing it.

x265 will lower refs to make it compliant. Surprisingly, there was one movie that lowered it to 7.

I always thought 8 bframes was optimal. Using that many isn't going to net me less quality though, ya?

I find rect wayyy to slow, and I thought it wasn't worth it.

I'll lower crqpoffs/cbqpoffs to -4. Never heard of anyone using it that low though. You said UHD has a much broader color gamut. Did you mean HDR does?

That's strange how rd above 4 lowers quality. Thank you for letting me know. I'll change my SDR settings. I'll have more time to play around with. What d you recommend I enable? Going to me sea? Speaking of me, is it true umh yields better quality than star? Using rect and possibly with amp? Raising merange?

I disabled aq because from what I read it is bad for movies with grain, which is pretty much them all. Same with cutree. Do you recommend I try it atleast? What would you recommend I start with?

[microchip8]

I don't use more than 4 refs and 6 bframes. That's a good tradeoff between speed and compressability

You can considerably speed up rect using limit-modes. You can even throw in amp without losing too much speed.

Yes, I meant HDR has a broader color gamut so by lowering cbqpoffs/crqpoffs tends to better preserve the chroma plane (it gets less compressed), resulting in more expressive colors. Try it and compare to standard value of 0.

UMH is better than Star as it goes deeper into analyzing while Star skips earlier. So UMH is a bit slower but should provide better motion estimation than Star. SEA is mostly placebo

As for AQ, use mode 1 as the other modes are "broken" in x265 and are actually worse. I have no issues with using AQ and cutree. All my encodes look pretty detailed, even in the presence of grain/noise.

[GeoffreyA]

That's strange how rd above 4 lowers quality.

It seems to be a different algorithm. I remember it creating some strange effects with grainy material.

[Z2697]

Just a myth

[Cmptr]

I don't use more than 4 refs and 6 bframes. That's a good tradeoff between speed and compressability

You can considerably speed up rect using limit-modes. You can even throw in amp without losing too much speed.

Yes, I meant HDR has a broader color gamut so by lowering cbqpoffs/crqpoffs tends to better preserve the chroma plane (it gets less compressed), resulting in more expressive colors. Try it and compare to standard value of 0.

UMH is better than Star as it goes deeper into analyzing while Star skips earlier. So UMH is a bit slower but should provide better motion estimation than Star. SEA is mostly placebo

As for AQ, use mode 1 as the other modes are "broken" in x265 and are actually worse. I have no issues with using AQ and cutree. All my encodes look pretty detailed, even in the presence of grain/noise.


Refs and bframes dont effect quality, right? It's mainly compression?

I'll need to read more about aq, aq-strength and cutree before I start using them. Any reads you recommend?

[microchip8]

Refs and bframes dont effect quality, right? It's mainly compression?

I'll need to read more about aq, aq-strength and cutree before I start using them. Any reads you recommend?

They do, but using (very) high values are not beneficial. This is true for x264 as well

cutree is basically mbtree in x264 ported to x265.

For AQ and others, consult the doc at https://x265.readthedocs.io/en/master/

You can also lower lookahead-slices to 2 instead of the one x265 picks by default based on some parameters. Lower lookahead-slices makes x265 more precisely decide where to place/use bframes and scene-cut decision, at the cost of minor speed penalty

[Cmptr]

They do, but using (very) high values are not beneficial. This is true for x264 as well

cutree is basically mbtree in x264 ported to x265.

For AQ and others, consult the doc at https://x265.readthedocs.io/en/master/

You can also lower lookahead-slices to 2 instead of the one x265 picks by default based on some parameters. Lower lookahead-slices makes x265 more precisely decide where to place/use bframes and scene-cut decision, at the cost of minor speed penalty

Wasn't mbtree kinda broken in x264?

I've never messed with lookahead-slices, ill try that. I may keep the ref and bframes though.

Is there any other settings that you would recommend trying out?

[microchip8]

Wasn't mbtree kinda broken in x264?

I've never messed with lookahead-slices, ill try that. I may keep the ref and bframes though.

Is there any other settings that you would recommend trying out?

Never heard that mbtree is broken in x264.

The resently released 4.3 version of x265 has added motion compensated spatial-temporal filtering MCSTF. It supposedly deals with noise/grain better than the old and broken rc-grain ratecontrol.

Try with --selective-mcstf 1

[Cmptr]

Never heard that mbtree is broken in x264.

The resently released 4.3 version of x265 has added motion compensated spatial-temporal filtering MCSTF. It supposedly deals with noise/grain better than the old and broken rc-grain ratecontrol.

Try with --selective-mcstf 1

I was just reading about the MCSTF now. Might give it a shot.

You said levels above 1 for aq-mode new "Broken". So would they not be worth even trying out?

[Cmptr]

Just a myth

Which? Rd above 4 resulting in worse quality?

[microchip8]

I was just reading about the MCSTF now. Might give it a shot.

You said levels above 1 for aq-mode new "Broken". So would they not be worth even trying out?

The x265 devs changed some core values compared to x264's AQ 2 and above. This results in x265's AQ 2 and above smearing small patches in the picture. Since all other modes above 2 are based on 2, they're equally "broken". You can try it, but many others have (me including) and we do not like what we see.

[Z2697]

MCSTF is still broken (maybe even more broken), selective-mcstf is just hiding the problem

[GeoffreyA]

Is it like vvenc's MCTF?

[microchip8]

MCSTF is still broken (maybe even more broken), selective-mcstf is just hiding the problem

In what sense? I've done a full season encode of Stranger Things in HDR10 and it looks good, I could even notice some small improvement here and there.

[Cmptr]

In what sense? I've done a full season encode of Stranger Things in HDR10 and it looks good, I could even notice some small improvement here and there.

I would love to encode the series, though I'm still waiting for a "Release"

I may try the new setting out a bit. Ill mess around with aq-strength as well using aq-mode 1.

I havent done really any tests using aq. So not sure what to expect. I've got bad OCD when it comes to quality. So this should be interesting.

[GeoffreyA]

Started testing MCSTF. Denoising seems a bit more consistent than AV1.

https://workupload.com/archive/kCZdQkakfM

[Cmptr]

You can also lower lookahead-slices to 2 instead of the one x265 picks by default based on some parameters. Lower lookahead-slices makes x265 more precisely decide where to place/use bframes and scene-cut decision, at the cost of minor speed penalty

Because I'm using preset slower, I'm pretty sure the lookahead-slice is 1.

So far, I've changed rd to 4 from 6, kept aq-mode and cutree off. My OCD doesnt like how the bits arent spread evenly. When I think of aq-mode, I makes me of a "Distorted" image, wish essentially it is I guess, but in the sense that it will starve bits from certain areas, to give to other areas. I'd much rather just lower crf if an area is starving... BUT in saying that, I would much rather give that area more bits if the rest of the image looks fine, which I guess is what aq-strength is for... BUT at the sametime, in doing that, the area that wasnt starved, might starve now...

I just talked myself out of it, then back in, then back out... What a roller-coaster.

I know psy-rd and psy-rdoq can do bluring too, and I have noticed it having pay-rd 1. So far my default is 2 for both psy-rd and psy-rdoq. I think I can tell when I need to change psy-rd. If I see an area blurred, I raise it...

Now with pay-rdoq... I can't necessarily tell.. It "prevents RDOQ from blurring all of the encoding options which psy-rd has to choose from"... I don't exactly understand that.

The other change I made was use rect and amp...

[Cmptr]

Started testing MCSTF. Denoising seems a bit more consistent than AV1.

https://workupload.com/archive/kCZdQkakfM

I've thought about denoising, but I feel like I'll go down another rabbit hole worring about detail being lost and everything being too smoothed out.

[Z2697]

If that so you can just keep the original file.

[Cmptr]

If that so you can just keep the original file.

I'm not sure I understand what you mean.

If you're saying that I should just keep my original file and not encode it... I have got some 1080p down to 1/3 of the original, and UHD to 3/5. I'm just trying to get some recommendations and expand my knowledge much more and see how I can improve my encodes.

[microchip8]

Because I'm using preset slower, I'm pretty sure the lookahead-slice is 1.

So far, I've changed rd to 4 from 6, kept aq-mode and cutree off. My OCD doesnt like how the bits arent spread evenly. When I think of aq-mode, I makes me of a "Distorted" image, wish essentially it is I guess, but in the sense that it will starve bits from certain areas, to give to other areas. I'd much rather just lower crf if an area is starving... BUT in saying that, I would much rather give that area more bits if the rest of the image looks fine, which I guess is what aq-strength is for... BUT at the sametime, in doing that, the area that wasnt starved, might starve now...

I just talked myself out of it, then back in, then back out... What a roller-coaster.

I know psy-rd and psy-rdoq can do bluring too, and I have noticed it having pay-rd 1. So far my default is 2 for both psy-rd and psy-rdoq. I think I can tell when I need to change psy-rd. If I see an area blurred, I raise it...

Now with pay-rdoq... I can't necessarily tell.. It "prevents RDOQ from blurring all of the encoding options which psy-rd has to choose from"... I don't exactly understand that.

The other change I made was use rect and amp...

If you play with psy-rd/rdoq, make sure you always use rdoq-level 1, which is the most effective one for psy-rd/rdoq. I use a psy-rd of 3 and rdoq of 3.5 and it looks good to me

For HDR encodes, lower qg-size to 16 to better preserve finer details.

And even for 4k resolution, I still use a ctu of 32.

You can set rskip to 2 for a nice boost in speed and crank up lookahead all the way up to 240

I also use the more exotic cu-lossless option which can be a potential rate distortion optimization. It never hurts

These are the options I mostly use for HDR


x265 [info]: frame threads / pool features : 4 / wpp(33 rows)
x265 [info]: Slices : 1
x265 [info]: Coding QT: max CU size, min CU size : 32 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 4 inter / 4 intra
x265 [info]: ME / range / subpel / merge : umh / 52 / 7 / 5
x265 [info]: Keyframe min / max / scenecut / bias : 24 / 240 / 40 / 5.00
x265 [info]: Cb/Cr QP Offset : -4 / -4
x265 [info]: Lookahead / bframes / badapt : 240 / 6 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 4 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 1.1 / 16 / 1
x265 [info]: Rate Control / qCompress : CRF-20.0 / 0.60
x265 [info]: tools: rect amp limit-modes rd=4 psy-rd=3.00 rdoq=1 psy-rdoq=3.50
x265 [info]: tools: rskip mode=2 rskip-edge-threshold=0.01 cu-lossless signhide
x265 [info]: tools: tmvp b-intra lslices=2 deblock(tC=-3:B=-3)

[Cmptr]

Does using a ctu of 32 really help? Ive read about it a few times now.

Ive never used rskip 2. I havent been able to understand it that much. Can you explain it a bit?

Why is rdoq-level 1 better than 2? What do you mean more efficient? Better quality or just compresses more?

[microchip8]

Does using a ctu of 32 really help? Ive read about it a few times now.

Ive never used rskip 2. I havent been able to understand it that much. Can you explain it a bit?

Why is rdoq-level 1 better than 2? What do you mean more efficient? Better quality or just compresses more?

Yes, ctu of 32 helps but many stick to 64

rskip is recursion skip which tells when the encoder stops recursion, eg at what point. rskip 2 uses edge density and is considerably faster than rskip 1 which goes deeper. the best you can do with rskip 2 is setting the threshold to 1 (the lowest possible) to maximize quality

At rdoq-level above 1, psy-rdoq is less effective since it has to work with "less" due to many cuts being made. Level 1 is biased to preserve the energy of the residual better and makes psy-rdoq more precise/effective.

[Cmptr]

I actually do remember reading that rdoq-level 2 is less accurate than 1 for psy. How much of a difference in compression have you noticed going from 2 to 1? Same for ctu 32 vs 64?

[Z2697]

The main reason of using ctu=32 is to gain more "multithreadability" through WPP.
Quality wise it technically may be harmful.
But the max size of TU and intra PU is 32x32, so in practice it's basically free.

I feel a bit tired...

[rwill]

I find these endless discussion threads without concrete results amusing.

x265 being an encoder that exposes lots of internal development parameters to the user like aq-mode 1,2,3,4, rskip 0,1,2, qg-size, cutree, subme or whatever. Of course people are going to ask what to use. And no one knows for sure but some have formed a strong opinion.

If the encoder developer does not know what to use when and does not take the decision then how are the users going to figure it out? Its shifting core responsibility from the developer to the user.

I really hope that MCWs intention was to simplify internal x265 development and not relaying algorithmic tuning to the end user...

[Cmptr]

I find these endless discussion threads without concrete results amusing.

x265 being an encoder that exposes lots of internal development parameters to the user like aq-mode 1,2,3,4, rskip 0,1,2, qg-size, cutree, subme or whatever. Of course people are going to ask what to use. And no one knows for sure but some have formed a strong opinion.

If the encoder developer does not know what to use when and does not take the decision then how are the users going to figure it out? Its shifting core responsibility from the developer to the user.

I really hope that MCWs intention was to simplify internal x265 development and not relaying algorithmic tuning to the end user...

I aim to understand so that way I know why something does what it does. For the majority of the options ive listed I understand, unless I have said different.

Ctu 32 makes sence that it can help with finer detail, but also can hurt compression for flat areas.

Same with qg-size, except in my case because I'm not using aq, it doesnt do anything.

[microchip8]

I actually do remember reading that rdoq-level 2 is less accurate than 1 for psy. How much of a difference in compression have you noticed going from 2 to 1? Same for ctu 32 vs 64?

For some content, the difference is well spotted but I've never used level 2 much. For ctu, I recall 32 does better than 64 for up to 4k after which 64 "should" be used. I haven't done much comparisons but some claim 32 delivers better compression for a given quality

That said, x265 has many shortcomings and if you expect to get as close to the original, it's hard to achieve and you should know how to get around these shortcomings, if possible at all. That doesn't mean you can't get a very good picture but don't expect it to look like the input. Lossy compression is all about tradeoffs between quality and speed. I've settled with the params I showed earlier and am happy with the results. There are no visible issues when watching, unless you go pixel peeping but most people watch video on TV from a distance which makes it difficult to spot, more so if the "issues" are barely visible up-close.

[Cmptr]

For some content, the difference is well spotted but I've never used level 2 much. For ctu, I recall 32 does better than 64 for up to 4k after which 64 "should" be used. I haven't done much comparisons but some claim 32 delivers better compression for a given quality

That said, x265 has many shortcomings and if you expect to get as close to the original, it's hard to achieve and you should know how to get around these shortcomings, if possible at all. That doesn't mean you can't get a very good picture but don't expect it to look like the input. Lossy compression is all about tradeoffs between quality and speed. I've settled with the params I showed earlier and am happy with the results.

Using ctu 32 makes sence for 1080p, especially if youre going to be watching the video on a 4k screen. It will definitely preserve more fine details, especially if there is say a grain of hair in a ctu, but most of the ctu is a smooth gradient. If that ctu is 64, encoder could ignore the detail because it isnt worth it, but if it is in a ctu 32, it might not ignore it.

For 4k, using ctu doesn't make that much sense. I mean it does, but I feel like the compression loss would be noticeable.

I'm aware I wont get a 1:1 picture. Thats what lossy is. I'm merely trying to learn more and share.

That said, rdoq-level 1 being better than 2 for quality does make sence as you said because it is less cuts, meaning more data to work with, but at the same time, going deeper help the encoder know more of what is there.

[Cmptr]

Reading up more about rskip. Rskip 2 does seem actually really good. It evaluates the edges of the next depth, if they match the boundary of the current depth, recursion is skipped. That is actually really cool. Very curious to try it out, though would it not zero out the rest? I feel like this could cause some blocking issues. Only one way to find out.

Please feel free to correct me if I am wrong about something.

[Cmptr]

Very interesting results. I tried rskip 2 with threshold of 1, with rect and amp. With ctu 32 compared to no-rskip and ctu 64, the difference was hard to see. Maybe a bit more detail in some areas, and some color was kept as well. File was slightly smaller.

However.. using ctu 64 with rskip 2 and threshold 1, rect and amp, a 30 second clip got smaller by 5mb, BUT there was one frame that I noticed where detail was lost. Thats all it took for me to not want to use ctu 64 with rskip. Just gotta see which is better now thoguh. Though I didnt know you could specify a floating point value, so maybe if I try 0.01 and ctu 64 I wont get the detail loss and still get some more compression.

I tried cutree, and even with aq-strength 0.1... I noticed some flattening out. File was much smaller, much much smaller, but the flattened out part is what got me. There isnt even a way I could lower the strength even more for it to not be as flat. I'd have to lower crf which I should have tried.

[Z2697]

LOL what you compared files with drastically different size and have some "conclusion"
Trusting years old pass on myths instead of a x265 contributor?

[GeoffreyA]

I've thought about denoising, but I feel like I'll go down another rabbit hole worring about detail being lost and everything being too smoothed out.

I am hesitant to denoise myself. Often, just using higher quantisers creates comparable results, and the quantisation artefacts tend to be masked by the grain.

But for the sake of information in this thread, here is another MCSTF test with heavier grain:
https://workupload.com/archive/tJUHJveV4c

For my part, I don't see myself using this feature.

[Z2697]

I am hesitant to denoise myself. Often, just using higher quantisers creates comparable results, and the quantisation artefacts tend to be masked by the grain.

But for the sake of information in this thread, here is another MCSTF test with heavier grain:
https://workupload.com/archive/tJUHJveV4c

For my part, I don't see myself using this feature.

Now compare frame 13 closely and you'll see what's broken.

[GeoffreyA]

Now compare frame 13 closely and you'll see what's broken.

I see. The folder? It appears to "thin."

https://slow.pics/s/gbVifzta

[rwill]

I see. The folder? It appears to "thin."

https://slow.pics/s/gbVifzta

In Frame 346 (the last of your sample) the hair color from the person moving in from the left in the background is also changed from dark blonde to some blonde/gray by mctf.

[Cmptr]

LOL what you compared files with drastically different size and have some "conclusion"
Trusting years old pass on myths instead of a x265 contributor?

What are you talking about? I'm not just going by file size. I'm looking at quality too.

[Cmptr]

I messed around with aq-mode 1, cutree, rskip 2 with rskip-edge-threshold (I used rect and amp), ctu 32 and qg-size 16 and 8. I found that though they kinda helped in some areas, I always found macroblocking when deeply analyzing while pausing. I found them much more so compared to when I didn't use these settings. There was also some weird shifts of detail. A whole ctu would be shifted 2 or 3 ctu over. You could also notice that some werent recursed enough as well even though edge threshold was at 1, which I found out converts into a decimal range of 0.01 to 1.

I very well could be doing something wrong, but I was pretty thorough. I've doing it all through ffmpeg, not using frameserver, so maybe that could be the issue?

Would using ffmpeg compared to a frameserver into x265 directly make a difference?

[microchip8]

I messed around with aq-mode 1, cutree, rskip 2 with rskip-edge-threshold (I used rect and amp), ctu 32 and qg-size 16 and 8. I found that though they kinda helped in some areas, I always found macroblocking when deeply analyzing while pausing. I found them much more so compared to when I didn't use these settings. There was also some weird shifts of detail. A whole ctu would be shifted 2 or 3 ctu over. You could also notice that some werent recursed enough as well even though edge threshold was at 1, which I found out converts into a decimal range of 0.01 to 1.

I very well could be doing something wrong, but I was pretty thorough. I've doing it all through ffmpeg, not using frameserver, so maybe that could be the issue?

Would using ffmpeg compared to a frameserver into x265 directly make a difference?

No, ffmpeg just passes the stream itself to libx265 for encoding. It wraps it then in a chosen container like mp4 or mkv along with the audio

I suspect that the macroblocking occurs because you fully disabled deblocking. I wouldn't do that and instead lower it to -3:-3 or -4:-4 (less common)

qg-size below 16 is not recommended. I found it introduces a bit of ringing.

[Z2697]

Shift by ctu sounds like total nonsense, unless under extremely low bitrate which it's not, if you think it's a bug please share sample(s).

[Cmptr]

Shift by ctu sounds like total nonsense, unless under extremely low bitrate which it's not, if you think it's a bug please share sample(s).

You really are negative arent you. You're right, I'm lying and totally making this up.

Sarcasm by the way. Why would I lie and make it up? What would I gain?

I don't think it is a bug, I think that because most of the area was flat, and that there was a tiny bit of detail, I'm assuming because of cutree, the block was reused, and being that I was using rskip 2 and how most of the block was flat, it didnt recurse deeper to see the detail to make an accurate decision that the block shouldn't be reused.

If I thought it was a bug, I would open up an issue, mention how to replicate it, and maybe even fork and do a pull request if I were to locate, examine, test and come up with a solution.

[Cmptr]

No, ffmpeg just passes the stream itself to libx265 for encoding. It wraps it then in a chosen container like mp4 or mkv along with the audio

I suspect that the macroblocking occurs because you fully disabled deblocking. I wouldn't do that and instead lower it to -3:-3 or -4:-4 (less common)

qg-size below 16 is not recommended. I found it introduces a bit of ringing.

I know it's a wrapper, I should have specified more. I meant more of the decoding. I thought that maybe the bitstream would be different from decoder to decoder, where using a frameserver with an indexer would be better.

[microchip8]

I know it's a wrapper, I should have specified more. I meant more of the decoding. I thought that maybe the bitstream would be different from decoder to decoder, where using a frameserver with an indexer would be better.

I think ffmpeg's decoder is just fine. You can use a GPU accelerated decoder if you want.

[Z2697]

It's always FFmpeg.

[GeoffreyA]

In Frame 346 (the last of your sample) the hair color from the person moving in from the left in the background is also changed from dark blonde to some blonde/gray by mctf.

I see it. Almost as if the door, which should be occluded by the head, is being blended into the current frame.

[Cmptr]

I think ffmpeg's decoder is just fine. You can use a GPU accelerated decoder if you want.

Is GPU decoding like GPU encoding? Worse quality? Would I gain anything using a frameserver and use x265 over the built in decoder in ffmpeg, in terms of quality, efficiency or compression?

[Z2697]

any conforming decoder should produce same result
and it's always ffmpeg, the source filter you gonna find

[Cmptr]

any conforming decoder should produce same result
and it's always ffmpeg, the source filter you gonna find

What you mean "Source filter"?

[GeoffreyA]

Is GPU decoding like GPU encoding? Worse quality? Would I gain anything using a frameserver and use x265 over the built in decoder in ffmpeg, in terms of quality, efficiency or compression?

Decoding should always be reference quality. However, when it comes to encoding, one might choose to pipe to the binaries to avoid any issues with FFmpeg's wrappers.

[Cmptr]

Decoding should always be reference quality. However, when it comes to encoding, one might choose to pipe to the binaries to avoid any issues with FFmpeg's wrappers.

How often is there issues uses ffmpeg compared to x265? Would you say most people opt into just using x265 directly?

Thank you for your responses by the way. It's sad to see that not many people are into compression and just really on streaming. I'd like to see doom9 be lively, cause it is full of lots of information.

[Cmptr]

Okay. So I'm looking at some of my encodes and they look phenomenal. It's very hard to tell the difference between the original and the encode. The ONLY time I can tell is when there is a very smooth gradient in the background, and not the whole background, or where there is a smooth gradient in a very dark, almost black area.. Most of the time I can only notice it when I switch back and forth between the original frame, and encoded frame (Yes, most of the video even doing that, it is very hard to tell the difference, except the situations I listed)... Now.. This is the first time I thought that maybe aq-mode could help.

I know aq-mode takes away from complex scenes, and focuses on flat gradient areas... I always thought (And this maybe how it works) that it takes away from the complex and fast moving areas because as the video is playing, the change is so quick, you wont see the difference, compared to a less moving and less dynamic flat gradient area... Maybe I am kind of wrong... Maybe it does that, but not as bad. I always thought that it would be noticeable when comparing frame by frame. But maybe it isn't as noticeable as I may think.. Now I am thinking out loud here, this whole message I have been.. Without aq all bits are allocated mathematically throughout the frame, which even doing so, x265 has been able to do it where it is hard to tell... Maybe aq-mode is a step further... allocates the bits a bit differently, taking away from the areas you wont notice "As much" compared to what I'm noticing in the flat areas now...

Maybe I'll notice worse quality in the complex areas, for example lets say 5%... But the flat areas will look better by 20%.. Originally I could have just lowered the crf, but there would be lots of bits wasted just to make the the flat areas look better, because those bits would also be applied to the complex areas mathematically the same, that already look indistinguishable... Aq maybe counters that.. take away bits from complex area, put them to flat area. If complex area looks bad, lower crf. Whereas before, lowering crf would sure make flat areas look better, but bits would also be applied to an already "perfect" area... (Yes I know aq-strength can change the weight of where bits are allocated)

Sorry for rant.. Like I said.. Thinking out loud. Yes I need to test and am going to. But that's my thought process. If anyone wants to weigh in on it and inform and and correct me where I'm wrong and right, that helps a lot. I do know that different levels of aq do different things. I was just thinking out loud in general in terms of aq.

I apologize again for the rant. I'll probably delete if my thought process is very inaccurate.

[GeoffreyA]

There was a long discussion on AQ:

https://forum.doom9.org/showthread.php?t=186641

How often is there issues uses ffmpeg compared to x265? Would you say most people opt into just using x265 directly?

Thank you for your responses by the way. It's sad to see that not many people are into compression and just really on streaming. I'd like to see doom9 be lively, cause it is full of lots of information.

The issues are minor to nonexistent, more in terms of metadata and how the bitstream is written. Visually, identical. 99% can stick to FFmpeg; perfectionists might pipe. Some would also swear by piping the 16-bit output from the frameserver and letting x265 dither to 10 bit. But, the bigger issue at hand will be x265 itself, which is not as well tuned as x264 out of the box. The approach I've adopted is using different encoders depending on the film. For some, x264. Other times, x265 or SVT-AV1-HDR.

Doom9 was, as the site says, "THE place" for encoding; and encoders and indispensable tools have been developed here. The DVD days, the x264 era, and so on. Some legends, such as DarkShikari of x264 fame, left but some are still here. So, thank you for your questions; it helps to keep the discussions going.

[Z2697]

How often is there issues uses ffmpeg compared to x265? Would you say most people opt into just using x265 directly?

Thank you for your responses by the way. It's sad to see that not many people are into compression and just really on streaming. I'd like to see doom9 be lively, cause it is full of lots of information.

There's generally no problem with FFmpeg's libx265 wrapper.
People usually concerned about the automated filter insertion, frame dropping or duplication that may or may not happen.
But, if it doesn't happen (and it's fairly easy to confirm), then it's fine.

Source filters generally guarantee or aim to guarantee the frame accuracy, and it's usually the only difference.

Source filter is what you use to read the source clip, into the script, if by frameserver you mean the AVS or VS.

Of course there's much more things one can do with "synths".
I only talk about simplest transcoding here.

[Z2697]

Some would also swear by piping the 16-bit output from the frameserver and letting x265 dither to 10 bit.

LOL it's error diffusion (same as x264) and we should know it's bad for encoding (all error diffusion species).
Maybe with 10 bits it's not really an issue, but if they are relying on that they'd be better off just do a direct shift / LSB discard.

[GeoffreyA]

LOL it's error diffusion (same as x264) and we should know it's bad for encoding (all error diffusion species).
Maybe with 10 bits it's not really an issue, but if they are relying on that they'd be better off just do a direct shift / LSB discard.

Would be there any difference between, say, 8- and 10-bit input on one hand, and 16-bit input on the other? Not with respect to the dithering method but the encoding pipeline.

[Z2697]

Would be there any difference between, say, 8- and 10-bit input on one hand, and 16-bit input on the other? Not with respect to the dithering method but the encoding pipeline.

The encoding will be slower (for nothing) since the dithering in x264/x265 CLI is not optimized

[GeoffreyA]

The encoding will be slower (for nothing) since the dithering in x264/x265 CLI is not optimized

I have been using FFmpeg+libplacebo for HDR-SDR conversion, piping yuv420p16le output to the relevant encoder. Before, I used to do everything in FFmpeg. Libplacebo is still not working on VS for me.

[Z2697]

Before, I used to do everything in FFmpeg.

You should totally keep doing that.
Let libplacebo use ordered dithering and output yuv420p10le.
The default is static blue noise, not as bad as error diffusion, but ordered (8x8 bayer or something?) is still best for encoding, when it comes to keep all that important dither pattern.

[GeoffreyA]

For live action, error diffusion is not better?

[Cmptr]

After lots of test... I dont like cutree and I find that using any aq mode, though it may fix some dark area's, it lowers the quality overall compared to if I dont use it. Even if the bitrate is much higher. I know I can dial it in a bit more using aq-strength, but even having it at 0.5, again overall quality if I remember correctly was still lower, file much larger and barely fixed the flat areas I was trying to fix.

Next I'm just debating on ctu 32 and 64. It seems to go back and forth. .

Sometimes at crf 16.5 you can't even see the difference, but when its dark, you can a bit. I feel like the qp could be higher in some places, and lower in others.

Kinda wanna experiment with cqp...

[microchip8]

Then use what looks good to you?

[Z2697]

Talking about optimizing away from quality

[Cmptr]

Then use what looks good to you?

Yes. I'm just sharing my experience as well.

Talking about optimizing away from quality

I'm not really optimizing away from quality. Doing CQP definitely isn't efficient, I'll admit that.

I would like to share my results that I have so far. Where is the best place you guys recommend I upload the files to?

[Cmptr]

Decided to upload a few things:

Sample 1 Source (https://www.mediafire.com/file/hg66wsdrgulpbny/e3-077.mkv/file)

Sample 1 crf 16.5 (https://www.mediafire.com/file/8rrz59nuj5htdx9/e377-crf16.5.mkv/file)

Sample 2 Source (https://www.mediafire.com/file/mxsjn331q7zgp63/e3-006.mkv/file)

Sample 2 crf 16.5 (https://www.mediafire.com/file/tls8xy0pvmzvsaq/e36-crf16.5.mkv/file)

These are from the same episode. Snowpiercer Season 1 Episode 3. The source is 1080p bluray that I obtained through purchase.

You'll notice a big difference in quality between the source and crf 16.5 of sample 1, but for sample 2, you can barley tell the difference.

Both the encodes use the same settings verbatim.

Sample 1 needs less qp, whereas sample 2 can handle more, and still look indistinguishable. I tried sample 2 at crf 18, and again there was barely a difference. A tiny Tiny bit, but still nothing like sample 1 at crf 16.5. To be honest, sample 2 I could probably goto crf 18.5 and maybe even 19.

It's just a few scenes like sample 1, quality is noticeably bad. Most of the rest of the episode is as good as sample 2. So I could raise crf a bit, but that scene would suffer more. Is this because maybe the source for sample 1 has an average qp higher than that of what I'm trying to target? Is maybe that section already heavily compressed?

[microchip8]

Use sample 2, then. There is no magical parameter combination that will give you stunning quality on everything you throw at it. You're only testing on a single (or a few?) samples and tuning for them, not for everything else. People on here use different params and are pleased with them overall. There will always be artifacts here and there. The question is will you notice them when you watch it on a TV from a distance? Pixel peeping is useless...

[Cmptr]

Use sample 2, then. There is no magical parameter combination that will give you stunning quality on everything you throw at it. You're only testing on a single (or a few?) samples and tuning for them, not for everything else. People on here use different params and are pleased with them overall. There will always be artifacts here and there. The question is will you notice them when you watch it on a TV from a distance? Pixel peeping is useless...

Sample 1 and sample 2 are use the same settings, and are from the same video, just different areas in the video.

Ive tested a few different episodes.

Most params can stay the same though, no? With the exception of crf, psy-rd, psy-rdoq, maybe qcomp, ipratio and pbratio, aq-mode and aq-strength (If you use it), deblock. Is there anything else that I'm missing?

I would assume though that unless a bluray was terrible encoded for release, a 1080p movie would never need lower than crf 16.5. Have you experienced any 1080p movie where you needed lower?

[microchip8]

Sample 1 and sample 2 are use the same settings, and are from the same video, just different areas in the video.

Ive tested a few different episodes.

Most params can stay the same though, no? With the exception of crf, psy-rd, psy-rdoq, maybe qcomp, ipratio and pbratio, aq-mode and aq-strength (If you use it), deblock. Is there anything else that I'm missing?

I would assume though that unless a bluray was terrible encoded for release, a 1080p movie would never need lower than crf 16.5. Have you experienced any 1080p movie where you needed lower?

For 1080p/SDR I use a CRF of 23 and for 1080p/HDR a CRF of 20. I'm not trying to achieve near-lossless, just really good looking compression. I'm not encoding yet UHD as my PC is too slow for that and will have to wait a few days for a single encode. I scale down UHD to Full HD resolution.

[GeoffreyA]

Decided to upload a few things:

Sample 1 Source (https://www.mediafire.com/file/hg66wsdrgulpbny/e3-077.mkv/file)

Sample 1 crf 16.5 (https://www.mediafire.com/file/8rrz59nuj5htdx9/e377-crf16.5.mkv/file)

Sample 2 Source (https://www.mediafire.com/file/mxsjn331q7zgp63/e3-006.mkv/file)

Sample 2 crf 16.5 (https://www.mediafire.com/file/tls8xy0pvmzvsaq/e36-crf16.5.mkv/file)

These are from the same episode. Snowpiercer Season 1 Episode 3. The source is 1080p bluray that I obtained through purchase.

You'll notice a big difference in quality between the source and crf 16.5 of sample 1, but for sample 2, you can barley tell the difference.

Both the encodes use the same settings verbatim.

Sample 1 needs less qp, whereas sample 2 can handle more, and still look indistinguishable. I tried sample 2 at crf 18, and again there was barely a difference. A tiny Tiny bit, but still nothing like sample 1 at crf 16.5. To be honest, sample 2 I could probably goto crf 18.5 and maybe even 19.

It's just a few scenes like sample 1, quality is noticeably bad. Most of the rest of the episode is as good as sample 2. So I could raise crf a bit, but that scene would suffer more. Is this because maybe the source for sample 1 has an average qp higher than that of what I'm trying to target? Is maybe that section already heavily compressed?
Sample 1 will be harder because of the red chroma. I had a similar issue with "Last Night in Soho," where many scenes were red. Z2697 found that x265's chroma offset did not work as in x264 and submitted a fix. So, for an episode like that, try increasing the chroma offset (more negative).

This scene was a nightmare in terms of chroma:
https://workupload.com/file/gfR4MQe7tu2

[x264N00b]

Yeah those soft red light scenes are a nightmare for any encoder, dosen't matter if x264, svt-av1 or x265. Pulp Fiction bar scene, Wild at Heart Elvis "Love Me" scene, Fire Walk With Me Pink Room Scene et cetera... Only thing you can do is using zones and increase the bitrate heavily IMO.

Didn't made a real comparison but while VMAF score (95+ but I only trust my own eyes ;)) for sample1 is decent, those soft, dark and flat scenes usually don't get enough bitrate. To solve this you can lover CRF and limit the max. bitrate with bufsize and maxrate, to not blow up bitrates at grainy scenes or using zones. Lowering pb-ratio to ~1.20 can also help.

[rwill]

I don't know what you want really. The 'Sample 1' is 20Mbit and your re-encode is around 7Mbit. The sample already has problems, the encoder cannot magically fix its shortcomings. For that, the re-encode looks pretty good.

[Cmptr]

I guess the best thing to do would be to split the video in 30 second segments, find the smallest one, use that as a sample to test what params are best, compared to just testing on the largest?

[Z2697]

You can use YUView or VQanalyzer to see what's going on "inside".

YUView:
https://github.com/IENT/YUView

VQAnalyzer:
https://www.xlsoft.com/en/products/vicuesoft/eval_down.html
(This is a 3-day free trial, the full price is quite expensive.)
(They require email request to get this link, but trial needs network connection and the machine "fingerprint(s)" are recorded on the server, plus anyone can request the link, so I just post it.)

There are other tools but these 2 are ones I've used.

[Cmptr]

You can use YUView or VQanalyzer to see what's going on "inside".

YUView:
https://github.com/IENT/YUView

VQAnalyzer:
https://www.xlsoft.com/en/products/vicuesoft/eval_down.html
(This is a 3-day free trial, the full price is quite expensive.)
(They require email request to get this link, but trial needs network connection and the machine "fingerprint(s)" are recorded on the server, plus anyone can request the link, so I just post it.)

There are other tools but these 2 are ones I've used.


Ill try them out when I get home from work. What would I be looking for when using them?

[Z2697]

Ill try them out when I get home from work. What would I be looking for when using them?

Prediction, residual, QP, etc.

[Cmptr]

Prediction, residual, QP, etc.


Ahh, so find where the lowest qp average is, and use that as a sample to test on?

[Emulgator]

Residual ?
Plus you may simply diff sample minus encode and amplify the difference, this is free.
Still x265 with these parameters did quite well for a bitrate reduction to 1/3.

Beyond this, rwill already told you (and I found the same):
The offending AVC source already contains crossgrading kinks leading to blocks (haze in downward scene, hair from back) not yet visible on a 4K AUO309B, but clearly on a less forgiving LG 55C6V.
x265 might have been able to encode those areas better if such faults hadn't been on the plate in the first place.
Outside of that: some CCD striping (vertically along the dark suit) which may indeed help to judge encoding quality.
If this would be my job I would not leave source as-is and encode, rather call the big feisty2's deblockers to the rescue.

For parameter search maybe make your preferred test sequence including fault-free stock footage as well as synthetic stuff.
You can generate that yourself in AviSynth (while working in at least 16bit depth), BTW.
Dark Gradients, Moving noise stripes varying coarse/fine, thin colourful line art, surrounded by color halos, real water recording from a high-bitrate cam sensor, fading into dark, anything.

And there was the big pareto machine, just don't remember the name right now.

[Cmptr]

Hmmm... So I split the video into 1 second segements. Part of that red scene was was only like 2mb. There also other parts that were around 2mb. I encoded the 1 second red scene using crf 16, but also qpmin 16 and qpmax 16, and got a little over 1mb. I encoded a different scene that was right around the same size, but got around 800kb... It looked indistinguishable. I then tried crf 18, set qpmin and max to that as well.. Got around 400kb and still indistinguishable.

Just going by that, do you think the red scene I transcoded had a high qp to start with, and the other 1 second clip that I encoded that started out the same size, but compressed down to 400kb... Do you think that part had a really low qp?

[Cmptr]

Im thinking if I do this:

ffmpeg -debug:v qp -i input -an -v 48 -f null -

And I create a script to parse it and get the sum of each macroblock, and get the average... Would that be an accurate representation of the qp for that frame?