x264 Settings for Video Games footage

[andiandi]

Hi,

I've got several (realistic) video games footages to encode (PS2/GC/Xbox Era), I will not keep the lossless source files and I would like to have some settings recommandations. I think video game footages should be in a very good quality (to better appreciate the technical showcase), but I don't want a too high bitrate.
I assume using more ReFrames would help, but for the rest I don't really know. Maybe aq and psy since image is generally clean and fidelity is important for these sources ?


Also, a side question : Would it be possible to eventually implement some x265 settings to x264 ? The kind of settings which only affects the encoding but not the decoding (speed, cpu load) nor compatibility - like subme and me.

Thanks

[benwaggoner]

Does it need to be AVC? HEVC has some really good tools (SAO, transform skip, lossless CU, much more flexibility in block/TU size) that can get a HEVC <<50% the bitrate of a AVC of a similar quality with this kind of game footage. That era of games had TONS of aliasing, and those sharp edges are very hard on frequency transforms, which HEVC can bypass.

--preset animation may be a good start with x264, and you could tweak from there. You shouldn't have any noise (unless the game has a film grain effect; turn off if possible!), and frame-to-frame matches can be really good, so animation is the closest analog in the presets.

How are you capturing? If via a HDMI capture system, you should set the console to output the native resolution in progressive scan and capture that, without any upsampling. GC was always 480, Xbox original had some 720p (Halo: CE, IIRC), and I never played on a PS2. Getting the native pixels will make encoding easier and faster.

[andiandi]

Originally Posted by benwaggoner

Does it need to be AVC? HEVC has some really good tools (SAO, transform skip, lossless CU, much more flexibility in block/TU size) that can get a HEVC <<50% the bitrate of a AVC of a similar quality with this kind of game footage. That era of games had TONS of aliasing, and those sharp edges are very hard on frequency transforms, which HEVC can bypass.

--preset animation may be a good start with x264, and you could tweak from there. You shouldn't have any noise (unless the game has a film grain effect; turn off if possible!), and frame-to-frame matches can be really good, so animation is the closest analog in the presets.

How are you capturing? If via a HDMI capture system, you should set the console to output the native resolution in progressive scan and capture that, without any upsampling. GC was always 480, Xbox original had some 720p (Halo: CE, IIRC), and I never played on a PS2. Getting the native pixels will make encoding easier and faster.

Yes AVC preferably, I think it does the job pretty well in general (if well tuned/tweaked) and I tend to prefer my videos to remain compatible with older hardwares (even if it implies more bitrate and/or limit resolution). Also HEVC is high CPU consuming.

OK, I precisely planned something quite close to animation tune, at least :

- More Refs
- aq-strength 0.6 (Edges tending to be sharp + UI and I think lowering aq-strengh can be fine for SD since edges have less pixels anyway)
- psy-rd 0.5 (Not quite sure there, but I believe psy-rd distortions and artifacts are more noticeable for SD in general and I assume it could be more bothering for video game footages)
- deblock -1,-1 instead of 1:1 or 0:0

I chose to capture with emulators since they're very very accurate for these consoles nowadays and also because progressive scan require component cables whose output is not necessarily very good quality while forcing 4:2:2 colorspace as well. Emulators, on an other hand allows RGB capture (so 4:2:0 conversion from RGB will be better than from 4:2:2).
There're other things that consoles don't always do very well like scaling to correct aspect-ratio (sometimes too blurry).

And yeah I will encode in SD (especially as hardware upscaling is improving further and further) and since internal resolutions of these consoles are often unusual (640x448, 512x448 etc) I wondered if it was better to change the display aspect-ratio or the pixel aspect-ratio (but not sure that Lanczos is the most appropriate in this case).

[benwaggoner]

Originally Posted by andiandi

Yes AVC preferably, I think it does the job pretty well in general (if well tuned/tweaked) and I tend to prefer my videos to remain compatible with older hardwares (even if it implies more bitrate and/or limit resolution). Also HEVC is high CPU consuming.

Makes sense for your goals. Although for this sort of sub-SD encoding, HEVC should still be really fast.

OK, I precisely planned something quite close to animation tune, at least :

- More Refs
- aq-strength 0.6 (Edges tending to be sharp + UI and I think lowering aq-strengh can be fine for SD since edges have less pixels anyway)
- psy-rd 0.5 (Not quite sure there, but I believe psy-rd distortions and artifacts are more noticeable for SD in general and I assume it could be more bothering for video game footages)
- deblock -1,-1 instead of 1:1 or 0:0

That generally makes sense. More B-frames could also really help. I'm not sure about deblock. Generally stronger values are better for synthetic content and lower for natural image content.

I chose to capture with emulators since they're very very accurate for these consoles nowadays and also because progressive scan require component cables whose output is not necessarily very good quality while forcing 4:2:2 colorspace as well. Emulators, on an other hand allows RGB capture (so 4:2:0 conversion from RGB will be better than from 4:2:2).

Getting lossless frame captures of the rendered pixels is ideal. For SD console stuff, native RGB encoding could make sense, although that would be a big HW decoder compatibility hit.

There're other things that consoles don't always do very well like scaling to correct aspect-ratio (sometimes too blurry).

And yeah I will encode in SD (especially as hardware upscaling is improving further and further) and since internal resolutions of these consoles are often unusual (640x448, 512x448 etc) I wondered if it was better to change the display aspect-ratio or the pixel aspect-ratio (but not sure that Lanczos is the most appropriate in this case).

I think you're best off encoding the exact rendered pixels, and setting SAR as appropriate to get the proper aspect ratio on playback. That'll optimize clarity, compression efficiency, and encoding speed.

[andiandi]

Originally Posted by benwaggoner

Makes sense for your goals. Although for this sort of sub-SD encoding, HEVC should still be really fast.

I also think that for HEVC, it might still be possible to limit to settings that doesn't use more CPU than AVC while having a better result than with AVC (unless it's inherently more cpu-intensive than AVC - I didn't put much interest to HEVC until now).
Regarding compatibility, does it necessarily requires a modern hardware or can we use it on older hardwares as long as they're not too weak, I ask because I remember that sometimes on older phones for example, AVC was decoded smoothly but with some visual flaws (Btw I always keep an old smartphone for benchmark purposes just in case).

Originally Posted by benwaggoner

That generally makes sense. More B-frames could also really help. I'm not sure about deblock. Generally stronger values are better for synthetic content and lower for natural image content.

OK, for deblock I'll compare then (I guess it can change depending on the game)

Originally Posted by benwaggoner

I think you're best off encoding the exact rendered pixels, and setting SAR as appropriate to get the proper aspect ratio on playback. That'll optimize clarity, compression efficiency, and encoding speed.

Alright, that's good.

[benwaggoner]

Originally Posted by andiandi

I also think that for HEVC, it might still be possible to limit to settings that doesn't use more CPU than AVC while having a better result than with AVC (unless it's inherently more cpu-intensive than AVC - I didn't put much interest to HEVC until now).

x265 --preset medium is faster than x264 --preset veryslow, so quality @ perf @ bitrate should certainly be better with HEVC for this kind of pixel-accurate synthetic content.

Either x264 and x265 should be faster than real time with these low resolutions on a CPU that supports AVX2.

Regarding compatibility, does it necessarily requires a modern hardware or can we use it on older hardwares as long as they're not too weak, I ask because I remember that sometimes on older phones for example, AVC was decoded smoothly but with some visual flaws (Btw I always keep an old smartphone for benchmark purposes just in case).

Pretty much every discreet or integrated GPU has supported HEVC HW decode for 8-bit standard def for many years, and pretty much all mobile devices do (thanks to rapid replacement cycles). Any 4K HDR TV supports HEVC, as do pretty much all Smart TVs 2017+. You can embed it a web page for Edge, Safari, and Chrome now (but not Firefox). And VLC and other media player apps include software decoders even if hardware isn't there.

OK, for deblock I'll compare then (I guess it can change depending on the game)

Yeah. And while I generally recommend using SAO, you should definitely test with it off, which would help maintain the crispness of individual pixels. SAO mainly reduces ringing at higher QP/CRF, but I expect you'd be able to do perceptually lossless at <200 Kbps with a well-tuned x265 encode. I've been able to do 1080p24 at <150 Kbps with HEVC with simple content and appropriate tuning.

If you can share a couple of minutes of your uncompressed captures, we can offer some more specific tuning advice. Encoding to lossless H.264 or HEVC would be a compact way to transport them, but with these low resolutions, even a .y4m in 7z LZMA2 Ultra should be quite compact.

[andiandi]

Originally Posted by benwaggoner

x265 --preset medium is faster than x264 --preset veryslow, so quality @ perf @ bitrate should certainly be better with HEVC for this kind of pixel-accurate synthetic content.

Either x264 and x265 should be faster than real time with these low resolutions on a CPU that supports AVX2.

Faster at encoding you mean ? Because to me, encoding speed isn't an issue at all - even if it takes hours.

Originally Posted by benwaggoner

Pretty much every discreet or integrated GPU has supported HEVC HW decode for 8-bit standard def for many years, and pretty much all mobile devices do (thanks to rapid replacement cycles). Any 4K HDR TV supports HEVC, as do pretty much all Smart TVs 2017+. You can embed it a web page for Edge, Safari, and Chrome now (but not Firefox). And VLC and other media player apps include software decoders even if hardware isn't there.

It's compatible with less recent hardwares than I thought, even though I prefer to stick to AVC as much as possible. But it's a good point for HEVC (even though I consider using it mostly for videos at 1080p and higher). Also hardwares are replaced quite quickly.

Originally Posted by benwaggoner

Yeah. And while I generally recommend using SAO, you should definitely test with it off, which would help maintain the crispness of individual pixels. SAO mainly reduces ringing at higher QP/CRF, but I expect you'd be able to do perceptually lossless at <200 Kbps with a well-tuned x265 encode. I've been able to do 1080p24 at <150 Kbps with HEVC with simple content and appropriate tuning.

If you can share a couple of minutes of your uncompressed captures, we can offer some more specific tuning advice. Encoding to lossless H.264 or HEVC would be a compact way to transport them, but with these low resolutions, even a .y4m in 7z LZMA2 Ultra should be quite compact.

OK, here are some captures :


https://mega.nz/file/9rADlRZC#0HhRBJ...rA_KRUbRwX2ReI

https://mega.nz/file/BzRwGBJD#VIulP9...z7anxyium_weus

https://mega.nz/file/IzRzABqA#p3zDBx...E1LEmrz9xDaqgM

[benwaggoner]

Originally Posted by andiandi

Faster at encoding you mean ? Because to me, encoding speed isn't an issue at all - even if it takes hours.

At these low resolutions, x265 --preset placebo --tskip --cu-lossless should still be only a few multiples of realtime.

It's compatible with less recent hardwares than I thought, even though I prefer to stick to AVC as much as possible. But it's a good point for HEVC (even though I consider using it mostly for videos at 1080p and higher). Also hardwares are replaced quite quickly.

I think in 2023 it is safe to use HEVC by default unless you know you need compatibility with H.264-only clients, like Firefox or 7+ year old PCs where users can't install VLC.

OK, here are some captures :

https://mega.nz/file/9rADlRZC#0HhRBJ...rA_KRUbRwX2ReI

https://mega.nz/file/BzRwGBJD#VIulP9...z7anxyium_weus

https://mega.nz/file/IzRzABqA#p3zDBx...E1LEmrz9xDaqgM

I will check them out!

[andiandi]

Originally Posted by benwaggoner

I think in 2023 it is safe to use HEVC by default unless you know you need compatibility with H.264-only clients, like Firefox or 7+ year old PCs where users can't install VLC.

That's a good thing, it'll probably depend of the best I can get with AVC without having an excessive bitrate, but now I'm much more interested by HEVC (quality is more important to me for this particular case).

Originally Posted by benwaggoner

I will check them out!

Thanks !

[benwaggoner]

Originally Posted by andiandi

That's a good thing, it'll probably depend of the best I can get with AVC without having an excessive bitrate, but now I'm much more interested by HEVC (quality is more important to me for this particular case).

Yeah, good to try both.

I ran a bunch of x265 tests over the weekend, and whew, those are challenging clips. A huge amount of big differences in adjoining pixels. I'd underestimated the difficulty badly.

The solution is presumably in combining recursion down to 4x4 TUs plus using assymetric motion partitions to get as small a transform as possible. Plus using rskip to skip frequency transform entirely as needed, and potentially enabling --cu-lossless if there are blocks where lossless encoding is actually more efficient.

Even with all that thrown at it, I'm still getting >4 Mbps at crf 23 for these sub-SD sources. This is Sol Levante level difficulty! A very chewy encoding challenge. Thanks for sharing!

I've found setting --aspect 4:3 in ffmpeg before piping into the encoder will calculate the correct SAR irrespective of the actual rendered pixels.

[andiandi]

Originally Posted by benwaggoner

Even with all that thrown at it, I'm still getting >4 Mbps at crf 23 for these sub-SD sources. This is Sol Levante level difficulty! A very chewy encoding challenge. Thanks for sharing!

Yeah, from what I tested, it was barely watchable even at really high bitrates. I also tried 720p upscaling thinking it could be a better tradeoff after all but in CRF 22 with AVC it gives about 8000 kbps (even though it's 60 fps, it's still a lot)

[Emulgator]

SC3 PS2: I would guess that emulator add-generated a scanline-look which might not have been there in the original game output, making it harder to encode.
The other clips have their fair amount of resize flicker too. Not that much to save bitrate-wise I guess too, concurring.

[andiandi]

Originally Posted by Emulgator

SC3 PS2: I would guess that emulator add-generated a scanline-look which might not have been there in the original game output, making it harder to encode.
The other clips have their fair amount of resize flicker too. Not that much to save bitrate-wise I guess too, concurring.

For SC3 it's supposed to be dithering, IIRC it could be fixed with the emulator but I don't know if it's an issue or if it's in the game. Regardless, these are really problematic sources, maybe it would've been better to use emulator enhancements (better antialiasing/texture filtering/mip-mapping etc.). But it'd have been less representative of the real experience.

[andiandi]

Also, there's something I forgot for upscaling regarding YUV 4:2:0 colors converted from RGB : they're better preserved in HD than in SD. That's a good argument in favor of upscaling but given the complexity of the sources, it's not easy.

[benwaggoner]

Originally Posted by andiandi

Also, there's something I forgot for upscaling regarding YUV 4:2:0 colors converted from RGB : they're better preserved in HD than in SD. That's a good argument in favor of upscaling but given the complexity of the sources, it's not easy.

It'd love to have chroma subsampling become a problem, but the luma is so challenging to even notice any issues. A native full range 444 RGB encode could potentially do a lot better, although pretty much only SW decoders could display it.

H.264 wasn't any better.

I suspect a well-tuned AV1 encode could be a lot better. AV1 Main has a lot of screen encoding tools built-in that could really help. The HEVC screen encoding extensions would probably help a lot too, but I don't know of any practical encoders for it. All the old screen recording codecs were optimized before GUIs were full of transparencies and anti-aliasing, so wouldn't be much help.

Maybe something meant for video games like Bink could do well?

[andiandi]

Originally Posted by benwaggoner

It'd love to have chroma subsampling become a problem, but the luma is so challenging to even notice any issues. A native full range 444 RGB encode could potentially do a lot better, although pretty much only SW decoders could display it.

H.264 wasn't any better.

I suspect a well-tuned AV1 encode could be a lot better. AV1 Main has a lot of screen encoding tools built-in that could really help. The HEVC screen encoding extensions would probably help a lot too, but I don't know of any practical encoders for it. All the old screen recording codecs were optimized before GUIs were full of transparencies and anti-aliasing, so wouldn't be much help.

Maybe something meant for video games like Bink could do well?

And would 4:2:2 be a good compromise ? (what about compatibility ?), consoles used progressive scan in 4:2:2 with component cables so it'd remain close to the real experience.

Yes, I thought that other codecs could have more suitable tools for these kind of sources. For AV1 I assume we'll need to rely more on software decoders. Btw, I also wonder if software decoders are necessarily slower than hardware decoders, even if they are more recent? (especially on mobile devices) ?

In any case, I think the best thing to do is to encode with multiple codecs since I don't keep the lossless sources (fortunately, most of my captures are short), I would say:
In x265 in SD with the settings you recommended, then in x264 upscaled (to mitigate perceived quality loss) with a CRF of about 21-22 (8-10 Mbps, which is still less than Blu-ray, and I believe this kind of source have to be in very high quality by definition), and potentially in another codec to achieve the best possible compression before deleting the sources. Since x264 is the less efficient, I should try maxing it out and tweaking/tricking it as much as possible (maybe adjusting qpmin/qpmax? More rc-lookahead for mbtree ? Lowering psy-rd to prevent it allocating too much bits in aliased areas?).

[benwaggoner]

Originally Posted by andiandi

And would 4:2:2 be a good compromise ? (what about compatibility ?), consoles used progressive scan in 4:2:2 with component cables so it'd remain close to the real experience.

Maybe? Since they could fall back to composite output as well, generally they shouldn't have had too much chroma density to avoid issues with that. That's something to test and see. Note that 422 decode is less common, so there would be a compatibility hit.

Note VMAF entirely ignores chroma, so isn't of any use for that kind of testing,.

Yes, I thought that other codecs could have more suitable tools for these kind of sources. For AV1 I assume we'll need to rely more on software decoders. Btw, I also wonder if software decoders are necessarily slower than hardware decoders, even if they are more recent? (especially on mobile devices) ?

Yeah, a good share of new devices have AV1 HW decode, but the installed base is <<50%. Software decoding certainly can work if available, and at these low resolutions wouldn't require much CPU to decode, even on quite low-end devices.

In any case, I think the best thing to do is to encode with multiple codecs since I don't keep the lossless sources (fortunately, most of my captures are short), I would say:
In x265 in SD with the settings you recommended, then in x264 upscaled (to mitigate perceived quality loss) with a CRF of about 21-22 (8-10 Mbps, which is still less than Blu-ray, and I believe this kind of source have to be in very high quality by definition), and potentially in another codec to achieve the best possible compression before deleting the sources. Since x264 is the less efficient, I should try maxing it out and tweaking/tricking it as much as possible (maybe adjusting qpmin/qpmax? More rc-lookahead for mbtree ? Lowering psy-rd to prevent it allocating too much bits in aliased areas?).

Honestly, your lossless captures are't THAT much bigger than a transparent lossy encode might be. It might be cheaper to buy some 16 TB backup drives than paying for all the electricity to transcode.

A lossless HEVC RGB 444 might be somewhat smaller thanks to intraframe and interframe prediction and better entropy coding. I'd be hesitant to delete any sources without having nailed a transparent quality archival format.

[andiandi]

Originally Posted by benwaggoner

Maybe? Since they could fall back to composite output as well, generally they shouldn't have had too much chroma density to avoid issues with that. That's something to test and see. Note that 422 decode is less common, so there would be a compatibility hit

Note VMAF entirely ignores chroma, so isn't of any use for that kind of testing,.

OK

Originally Posted by benwaggoner

Honestly, your lossless captures are't THAT much bigger than a transparent lossy encode might be. It might be cheaper to buy some 16 TB backup drives than paying for all the electricity to transcode.

A lossless HEVC RGB 444 might be somewhat smaller thanks to intraframe and interframe prediction and better entropy coding. I'd be hesitant to delete any sources without having nailed a transparent quality archival format.

Yeah, I think I'm going to keep the Lossless sources after all. I did several other tests, and it seems almost impossible to do better with these sources (even with x265) and I guess there're no filters that can help a bit either. I think the real solution will probably come from newer codecs and tools (unless current and older codecs keep getting improved significantly, even through forks).

[benwaggoner]

Originally Posted by andiandi

Yeah, I think I'm going to keep the Lossless sources after all. I did several other tests, and it seems almost impossible to do better with these sources (even with x265) and I guess there're no filters that can help a bit either. I think the real solution will probably come from newer codecs and tools (unless current and older codecs keep getting improved significantly, even through forks).

I suspect that x265 doing RGB 8-bit lossless could be quite a bit smaller. Lossless mode skips frequency transform entirely, so should be better suited, and there is plenty of intraframe and interframe redundancy to compress with

[andiandi]

Originally Posted by benwaggoner

I suspect that x265 doing RGB 8-bit lossless could be quite a bit smaller. Lossless mode skips frequency transform entirely, so should be better suited, and there is plenty of intraframe and interframe redundancy to compress with

For x265 lossless (for archiving), I tested with VirtualDub2 and I didn't know that presets worked in lossless (I thought they were ignored and that only "lossless" settings applied). Thus, filesize could be significantly reduced in veryslow. However, Vdub detect the source as RGB32, but is limited to RGB24 for x265 lossless (there isn't any color loss in what I compared but I prefer to be sure).

As for lossy encodings (for watching) I think I'll just stick to x264 with a CRF around 16-18, as my sources are short overall (the longest ones are more or less 5 min) and with the most appropriate settings.