Parallel encoding to speed up AV1 encoders or x265

[Boulder]

I've now pushed the latest version to the repo. Quite a few small tweaks there, namely the ability to tweak the saturation of the FGS analysis clip and doing tone mapping of HDR sources (using DGHDRtoSDR) and adjustable downscaling in the scene change detection phase for faster processing and increased accuracy. The progress bar also works better now and gives a more accurate estimate of remaining time.

I've put several grain table files in the repo as well if anyone finds them useful.

[Blue_MiSfit]

Hmm... comparing those encodes against the reference in the free version of MSU VQMT I see a slight hue shift towards magenta in the AV1 version but not in the HEVC version... Not sure what could be the cause of that.

[Boulder]

Originally Posted by Blue_MiSfit

Hmm... comparing those encodes against the reference in the free version of MSU VQMT I see a slight hue shift towards magenta in the AV1 version but not in the HEVC version... Not sure what could be the cause of that.

Yes, I have noticed the same thing myself. I've been trying to find out the reason and I think I've caught it.. if you feed 8-bit input to aomenc (and encode at 10 bits as is recommended), the result gets this shift. If you convert the source to 10 bits before inputting it, the colors look pretty much the same in the encode as in the source.

I need to test this with different cases and try to replicate it with the vanilla aomenc so I can see if the problem's there or in the lavish fork.

[Boulder]

The hue shift issue occurs when using FGS and 8-bit input, works fine if the source is converted to 10 bits or an FGS table is not used. I didn't check if it applies to 8-bit encodes, as there's really no use doing those.

I've opened a case in the aomedia issue tracker, let's see if anything happens. For the time being, I need to add the conversion step to my tool. I still think that a B/W film grain table is better than one with chroma data included

[benwaggoner]

Originally Posted by Boulder

The hue shift issue occurs when using FGS and 8-bit input, works fine if the source is converted to 10 bits or an FGS table is not used. I didn't check if it applies to 8-bit encodes, as there's really no use doing those.

I've opened a case in the aomedia issue tracker, let's see if anything happens. For the time being, I need to add the conversion step to my tool. I still think that a B/W film grain table is better than one with chroma data included

Hmm, perhaps they're doing some sort of 4x scale factor for chroma instead of using bicubic interpolation or something. A straight multiple would round down chroma value a bit. Inverse dithering with interpolation is a largely untapped approach. Basically it should be like a bicubic, but in depth instead of area.

[Boulder]

I've developed the tool further along the way and the latest change was to add the support for encoding end credits using different Q and --cpu-used.

I think the next bigger step will be adding support for SVT-AV1, there is a huge performance boost incoming and also very interesting development regarding aq-mode 2 and low contrast sources which are often a problem for encoders.

[Boulder]

A lot more development has been done lately, now the tool supports svt-av1, rav1e and x265. In addition to that, you can create Dolby Vision compatible encodes using svt-av1-psy (https://github.com/gianni-rosato/svt-av1-psy).
More things coming soon, I've been moving the encoder parameters to a preset file and should be able to push that change to the repo during this week.

[Boulder]

I was finally able to finish the changes regarding moving settings to a separate preset file.

DoVi for x265 is still pending though, I think I'll get that working next.

[Boulder]

DoVi for x265 works now, so I think it's time to let the script rest a while unless something interesting pops up. The forthcoming SVT-AV1 v2.0 will include a huge quality boost when the variance boost feature will be included in mainline from the psy mod, so definitely check it out even if you don't use this script

[benwaggoner]

Originally Posted by Boulder

DoVi for x265 works now, so I think it's time to let the script rest a while unless something interesting pops up. The forthcoming SVT-AV1 v2.0 will include a huge quality boost when the variance boost feature will be included in mainline from the psy mod, so definitely check it out even if you don't use this script

In what way is DoVi not supported in x265? Optimization for Y'CtCp? I know x265 certainly has been used for commercial DoVi Profile 5 encoding. And it has full-fledged support for Profile 8.1 (which is just HDR-10 plus metadata from a sidecar file).

[Boulder]

Originally Posted by benwaggoner

In what way is DoVi not supported in x265? Optimization for Y'CtCp? I know x265 certainly has been used for commercial DoVi Profile 5 encoding. And it has full-fledged support for Profile 8.1 (which is just HDR-10 plus metadata from a sidecar file).

You can input an RPU file and create DoVi Profile 8.x encodes so yes, dynamic metadata like HDR10+. SVT-AV1(-psy) supports Profile 10 the same way, thanks to quietvoid's patch. As far as I know, there are no internal optimizations other than what PQ sources may have in all cases.

What the script simply does is that it splits the RPU based on the chunks so each chunk to encode gets proper metadata. mkvmerge must be used to concatenate the chunks as it does not remove the data while muxing.

[BlueSwordM]

Originally Posted by Boulder

You can input an RPU file and create DoVi Profile 8.x encodes so yes, dynamic metadata like HDR10+. SVT-AV1(-psy) supports Profile 10 the same way, thanks to quietvoid's patch. As far as I know, there are no internal optimizations other than what PQ sources may have in all cases.

What the script simply does is that it splits the RPU based on the chunks so each chunk to encode gets proper metadata. mkvmerge must be used to concatenate the chunks as it does not remove the data while muxing.

HDR10+ should also work with svt-av1-psy now, but I haven't tested it on Windows, only Linux.

[Boulder]

Some more development done based on the SSIMULACRA2-based auto-boost feature by trixoniisama (https://github.com/trixoniisama/auto-boost-algorithm). Works on SVT-AV1 and x265.

[LigH]

The media-autobuild suite now supports building Av1an. But there seems to be an issue, it produces no console output for me.

[Z2697]

I don't really understand why av1an chose rust though. A scripting language is enough and maybe more suitable. Considering the VapourSynth requirement of it, Python would be an almost perfect choice.

[Z2697]

Here's my weird approach, the main goal is to not wait for the scene change detection to complete but do encoding "on the fly", because for quite a long time and forseeable future before x266 is ready, I almost exclusively use x265 for any "serious business" encoding (i.e. probably using some heavy vapoursynth script), I use AV1 for some simple transcoding, thus speed is very important.

https://github.com/Mr-Z-2697/ideal-c...ain/segsaio.py

Maybe you can find it helpful. Not necessarily the "on the fly" thing, the scene change detection should be much faster, and perhaps better.
The mvtools method used by default is running 600+ FPS on my Ryzen 7950X.
Alternatively the deep learning method can be used, the speed of current default model and execution provider (DirectML) is 400+ FPS on RTX 4090 (not full utilized, only draws ~150W).
If you Gfx card is reasonablely new and have free resources during encoding, offloading scene change detection to it can free up some CPU resources for encoder.

[Boulder]

Just to let you know that I've been developing the script slowly but surely, and some new things like target quality for Butteraugli and CVVDP are now there. The latter one in particular is very interesting as it's a video quality metric with viewing setup parameters available, so the encodes can be targeted to your own setup instead of trying to achieve some general quality level. It is not perfect, but it's much more fun to use than trying to encode a TV series and figuring out things episode by episode. Set the target high enough and it's quite likely that the result will look good to you.

Using the svt-av1-hdr fork (https://github.com/juliobbv-p/svt-av1-hdr) encoder is highly recommended, although a lot of the things are already in mainline SVT-AV1

[charliebaby]

https://slow.pics/c/KN96LvKW

comparaison

AV1 : encode 7000kbps video
Source : BD REMUX 23000kbps video

99% quality Bluray source me setting AV1 :-)

[TEB]

Is it me or has AV1 become alot better in last years?
I recently did a test on AV1 (SVT-AV1 latest version) vs x264 from a prores source 1080p24

Where ate AVG-VMAF@95, the AV1 result was like 1/3 of the bitrate vs AVC and encoding AV1 was just a tad slower than x264. Used P4 in svt-av1
Dell R630 Dual Intel 6140 = 72 cores @ Redhat 9.
Is it AVX2/AVX512 support that is boosting the svt-encoder ?

[Boulder]

AV1 has definitely been improved a lot during the last year or so, thanks to the open source community

At preset 2, which is the sweet spot for SVT-AV1 (until v4.0.0, which probably changes this to P1), 1440p encodes run at 7-8 fps on my 5950X using chunked encoding and minor filtering. I'm sure AVX2 plays a big part here.