x265 or AV1 and gameplay encoding

[OvejaNegra]

HI! If this thread belong to another section, please the mods feel free to move it:

I'm going to compress the library of gameplay captures of my pc and the one from my wife pc. Most of the clips are encoded with h264 at 60mbps using nvenc.


Im testing x265 vs x264 to find the most convenient. I'll post my settings later.

In theory, x265 should beat x264 with this kind of content (sharp edges, no grain) but in fact there is not a big difference. That brings the question:

Should i keep testing x265 (and x264) or AV1 has some advantage here?

I'm aware that av1 removes grain and simulate it back during playback, so maybe is better tuned for sharp clean content, like animation, or my gameplay videos.


Gameplay videos are, by far, not the usual content that most of the codecs process: 1080p, 60fps of sharp edges pictures, where all the pixels move, all the time....


So far the lowest bitrate i could achieve was 11-12mbps (im trying to preserve the sharpnes of the image as long as possible ). And yes, 3 hours for enconding 5 minutes of video, sound like too much for me, so i try to keep the encoding speed reasonable.


Any advice for x265, or should i move to AV1

Thanks..

[benwaggoner]

HI! If this thread belong to another section, please the mods feel free to move it:

I'm going to compress the library of gameplay captures of my pc and the one from my wife pc. Most of the clips are encoded with h264 at 60mbps using nvenc.


Im testing x265 vs x264 to find the most convenient. I'll post my settings later.

In theory, x265 should beat x264 with this kind of content (sharp edges, no grain) but in fact there is not a big difference. That brings the question:

Should i keep testing x265 (and x264) or AV1 has some advantage here?

I'm aware that av1 removes grain and simulate it back during playback, so maybe is better tuned for sharp clean content, like animation, or my gameplay videos.


Gameplay videos are, by far, not the usual content that most of the codecs process: 1080p, 60fps of sharp edges pictures, where all the pixels move, all the time....


So far the lowest bitrate i could achieve was 11-12mbps (im trying to preserve the sharpnes of the image as long as possible ). And yes, 3 hours for enconding 5 minutes of video, sound like too much for me, so i try to keep the encoding speed reasonable.
Screen recording is very different than other kinds of content for video encoding. I think --preset animation is a better starting point. There are a number of optional features in HEVC that can be very powerful for efficient screen recording. Try doing a test encode in --preset placebo which turns most of those on, and see how the compression efficiency improves. I've seen it be able to drop bitrate by 80% for some kinds of synthetic content using proper tuning versus just the default --preset medium parameters.

That said, if the H.264 already has significant encoding, especially artifacts, those tools will be of much less utility, because softening and putting 8x8 block structure on the source gives much less for the advanced HEVC tools to do.

There are tons of cases where encoding in a different codec from lossless or near lossless source would have been a lot better, but reencoding from just that lossy output can't save a material number of bits without degrading quality.

[Z2697]

"Screen content" generally refers to the type where you record your desktop, like for presentation in a online conference or something. With mostly static/panning/scrolling content.
But of course anything can come out of that screen, not all of them are "screen content" when we discuss palette, IBC, etc. Anyways it's not like with HEVC they are widely available or even have easily accessible decoder.

3D rendered game graphics are more close to live action videos but harder.

Same bitrate can be of different quality levels for different types of content/genre/game.
I usually think x265's CRF 24 is where you draw the line. Or where lines of x265 and SVT-AV1 intersect.
If that 11-12M bitrate is a result of higher CRF, consider SVT-AV1. The lower the more x265 is preferable.

[rwill]

Sharp edges and no grain -> try --tune ssim.

[OvejaNegra]

Sorry for the delay: My settings so far:

x264:
--level 4.2 --preset placebo --crf 18 --deblock -2:-2 --keyint 300 --min-keyint 25 --bframes 4 --vbv-bufsize 78125 --vbv-maxrate 62500 --rc-lookahead 250 --aq-strength 1.2 --merange 64 --me umh --psy-rd 0.60:0 --colorprim bt709 --transfer bt709 --colormatrix bt709

Yes, that insane merange helps. Those settings give me 11.3 mbps in my lastest encode (fps game, 1080p, 60fps, everything moves).


Now, x265:

--crf 18 --output-depth 10 --no-strong-intra-smoothing --no-sao --rd 4 --psy-rd 0.6 --psy-rdoq 0.0 --rdoq-level 2 --deblock=-2:-2 --aq-mode 1 --aq-strength 1.2 --max-merge 5 --rskip 2 --rskip-edge-threshold 2 --limit-refs 0 --ctu 32 --qcomp 0.7 --bframes 3 --ref 5 --tu-intra-depth 2 --tu-inter-depth 2 --limit-tu 0 --subme 4 --merange 92 --me 1 --rc-lookahead 250 --keyint 300 --weightp --no-weightb --b-intra --lookahead-slices 4 --colorprim bt709 --colormatrix bt709 --transfer bt709

Those settings give me 11.1 mbps on the same sample.



Both settings give aprox 5fps on encoding speed.

On x265, qcomp 0.6 (default) can produce some noticeable degradation when an object suddenly moves on the screen (remember, 60fps, not the usual film). Rising to 0.7 solved the problem BUT, of course it also raises the bitrate.

If i use the default value (0.6) the overall bitrate lowers to 9.6 ~ 10 mbps, that includes more blurring on fast objects.

rect / amp lowers too much the encoding speed and does not help a lot, --tu-intra-depth 2 --tu-inter-depth 2 does actually make a difference.

Any advice is welcome. Being honest, i expected more from x265 with this kind of content, but maybe i'm missing something. Or i should be more tolerant with the blurring, since that's how x265 works.

On a side note: Im serving the original video using lsmashworks, im decoding with MPCH, the x265 encodes have some green tint and i dont know why. (The original looks ok on avspmod preview and on megui preview).

>>>i tested a quick encode with AV1 on megui using --progress 3 -i - --frames 1 --preset 5 --rc 0 --crf 18, the bitrate was higher (13 mb) but it took 80% usage on my cpu during playback , is that normal?????

i think i'll stay with x26x for now....

thanks as always..

[Z2697]

Looks like your video card doesn't have HW decoder for AV1.

[GeoffreyA]

>>>i tested a quick encode with AV1 on megui using --progress 3 -i - --frames 1 --preset 5 --rc 0 --crf 18, the bitrate was higher (13 mb) but it took 80% usage on my cpu during playback , is that normal?????.

Are they 4K videos?

[OvejaNegra]

Looks like your video card doesn't have HW decoder for AV1.


maybe, nvidia 1660, i will check later and report.

all the videos are at 1080p, 60 fps

[GeoffreyA]

With libdav1d and a contemporary CPU, 1080p AV1 decoding shouldn't take up all that much CPU usage.

[Z2697]

https://developer.nvidia.com/video-encode-decode-support-matrix
No it doesn't.

[Z2697]

With libdav1d and a contemporary CPU, 1080p AV1 decoding shouldn't take up all that much CPU usage.

dav1d really is very efficient, more efficient than FFmpeg's HEVC decoder (SW) actually.
(But we've heard that the HW decoder for AV1 is harder to design.)
Perhaps he's referring a single core usage?

[GeoffreyA]

It is very efficient, making decoding almost as easy as AVC.

[benwaggoner]

Sorry for the delay: My settings so far:

x264:
--level 4.2 --preset placebo --crf 18 --deblock -2:-2 --keyint 300 --min-keyint 25 --bframes 4 --vbv-bufsize 78125 --vbv-maxrate 62500 --rc-lookahead 250 --aq-strength 1.2 --merange 64 --me umh --psy-rd 0.60:0 --colorprim bt709 --transfer bt709 --colormatrix bt709

Yes, that insane merange helps. Those settings give me 11.3 mbps in my lastest encode (fps game, 1080p, 60fps, everything moves).


Now, x265:

--crf 18 --output-depth 10 --no-strong-intra-smoothing --no-sao --rd 4 --psy-rd 0.6 --psy-rdoq 0.0 --rdoq-level 2 --deblock=-2:-2 --aq-mode 1 --aq-strength 1.2 --max-merge 5 --rskip 2 --rskip-edge-threshold 2 --limit-refs 0 --ctu 32 --qcomp 0.7 --bframes 3 --ref 5 --tu-intra-depth 2 --tu-inter-depth 2 --limit-tu 0 --subme 4 --merange 92 --me 1 --rc-lookahead 250 --keyint 300 --weightp --no-weightb --b-intra --lookahead-slices 4 --colorprim bt709 --colormatrix bt709 --transfer bt709
You probably don't need to turn deblock so low for HEVC. Its 0,0 is less aggressive than for H.264. I'd try both -1,-1 and 0,0 before going that low. --amp and --rect can help a lot with screen recordings if they're clean. --ctu 64 might actually be helpful if you're doing at least 1080p, although I'd combine that with --tu-inter/intra-depth 4 and then --limit-tu 3 to get some speed back. Only 3 b-frames is really low for this kind of content. I'd do 4 at a minimum, and try to get to 8; I can't think of any content where ref>>bframe is optimal. Strong-intra-smoothing could be net beneficial; did you test with this content with it on/off? I'd want to try with --rskip off as well; it can sometimes act weird with non-standard content. --me 1 --max-merge 5 is a combo I've never seen before either. Did you come across that experimentally?

--tskip (with --tskip-fast is you need more speed) can also be extremely useful for screen recording content if it's a mathematically lossless, and could help if it's close. Easy to try on/off.

For any parameters you've not tried, I'd just start with --preset slower and iterate from there. That'll get you in the ballpark speed of x264 placebo.

Those settings give me 11.1 mbps on the same sample.
For tuning, sometimes a CBR encode is illustrative, as it'll show how the settings are responding to bitrate stress. try --bitrate 5000 with the correct profile @ level for --vbv-bufsize and --vbv-maxrate and you can see where things are really struggling and thus where you might gain the most benefit from further tuning.

On x265, qcomp 0.6 (default) can produce some noticeable degradation when an object suddenly moves on the screen (remember, 60fps, not the usual film). Rising to 0.7 solved the problem BUT, of course it also raises the bitrate.. You can always raise --crf to 19 or 20 to compensate.

rect / amp lowers too much the encoding speed and does not help a lot, --tu-intra-depth 2 --tu-inter-depth 2 does actually make a difference.
That is surprising. A lossy color space conversion that is introducing some rounding errors could account for some of that.

Any advice is welcome. Being honest, i expected more from x265 with this kind of content, but maybe i'm missing something. Or i should be more tolerant with the blurring, since that's how x265 works.
x265 isn't intrinsically blurry. You can try raising --psy-rd and --psy-rdoq which I see you have turned off.

On a side note: Im serving the original video using lsmashworks, im decoding with MPCH, the x265 encodes have some green tint and i dont know why. (The original looks ok on avspmod preview and on megui preview).
Likely a color space mismatch somewhere. I'd specify Rec. 709 on source and both x264 and x265 encodes. There should be no color shift, and when it happens, it is almost always a preprocessing or color space metadata issue. Are you coming from RGB 444 full range to YUV 420 limited range? If so, you might try doing Main10 encoding in both x264 and x265 and see what it gets you.