Feeding 10 or 16-bit AVS output to x265?

[`Orum]

I'm trying to move from x264 to x265, and run into a problem with feeding high depth clips as input. The tail end of my script is this right now:f3kdb(input_mode=1, input_depth=16, output_mode=1, output_depth=16, dither_algo=2, grainY=16, grainC=16)
ConvertFromStacked(16)
Prefetch(2)
And while avs2yuv detects this as YUV 4:2:0, 16-bit, x265 (run with -D 10 --input - --fps 24000/1001 --input-res 1920x1080) output never looks correct. So I've messed around with the following trying to get it to work.

Not using ConvertFromStacked(), and using -depth 16 to avs2yuv (which it claims is a hack for older AviSynths that don't support true 16 bit)
Using output_mode=2 from f3kdb, to stack horizontally instead of vertically (which at least gets the resolution correct), combined with the above
Trying 10-bit variants of the above (output_depth=10 / ConvertFromStacked(10)), as, rather curiously, there is no way to tell x265 what bit depth the input is.

In all cases my x265 output looks garbled, and clearly I'm doing something that's very basic incorrectly. Anyone see the obvious that I'm missing?

[sneaker_ger]

You haven't given us enough information.
Post MediaInfo of source, your complete AviSynth script, AviSynth and source filter and avs2yuv version, and avs2yuv + x265 command-line.

If you put ConvertBits(8) below ConvertFromStacked() and open it in e.g. VirtualDub does the image look correct?

[`Orum]

Post MediaInfo of source, your complete AviSynth script, AviSynth and source filter and avs2yuv version, and avs2yuv + x265 command-line.

MediaInfo of the source:
Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.1
Format settings : CABAC / 4 Ref Frames
Format settings, CABAC : Yes
Format settings, RefFrames : 4 frames
Muxing mode : Container profile=@0.0
Codec ID : V_MPEG4/ISO/AVC
Duration : 25 min 41 s
Bit rate mode : Variable
Bit rate : 21.6 Mb/s
Maximum bit rate : 39.0 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.434
Stream size : 3.87 GiB (98%)
Default : No
Forced : No

The entire script:
LWLibavVideoSource("01_video.mkv")
ColorYUV(levels="TV->PC")
AssumeFPS("ntsc_film")
Scripted_MDegrain(thSAD=150, tr=2, thSCD1=300, debugSCD=false, lsb=true)
f3kdb(input_mode=1, input_depth=16, output_mode=1, output_depth=16, dither_algo=2, grainY=16, grainC=16)
ConvertFromStacked(16)
Prefetch(2)

AviSynth version: AviSynth+ 0.1 r2580, MT, x86_64
Not sure on the exact version of LSMASHSource, but does it really matter?
avs2yuv version: Avs2YUV 0.24bm5

Edit: Whoops, forgot the command line. It is: "C:\Program Files\avs2yuv\avs2yuv64.exe" -depth 16 -seek 5000 -frames 1000 "01_vid.avs" -o - | "C:\Program Files\x265\x265-10b8b-PGO.exe" -p veryslow
--crf 32 -F 4 -D 10 -I 240 --open-gop -b 10 --aq-mode 3 --aq-strength 0.9 --psy-rdoq 5.0 -o test.hevc --input - --fps 24000/1001 --input-res 1920x1080

As I said I've been testing with and without -depth 16, though.

If you put ConvertBits(8) below ConvertFromStacked() and open it in e.g. VirtualDub does the image look correct?

Yes, that looks fine in AvsPmod. I can also open the actual 16-bit (without downconverting to 8-bit) output in MPC-HC. The problem appears to be between AviSynth and avs2yuv, or between avs2yuv and x265, or both. Apologies if this isn't the correct forum for the question, but I'm really not sure what the "right" one would be.

[qyot27]

You're trying to pipe raw instead of using y4m?

The alternative would be to simply use an FFmpeg that was compiled against a multibit-capable x265.

[`Orum]

You're trying to pipe raw instead of using y4m?
No, I am not using -raw on avs2yuv. However, x265 still requires --fps and --input-res for some reason, though it appears to override them by what it reads over the pipe. Probably just a minor bug?

[sneaker_ger]

Try:
"C:\Program Files\avs2yuv\avs2yuv64.exe" -seek 5000 -frames 1000 "01_vid.avs" -o - | "C:\Program Files\x265\x265-10b8b-PGO.exe" -p veryslow
--crf 32 -F 4 -D 10 -I 240 --open-gop -b 10 --aq-mode 3 --aq-strength 0.9 --psy-rdoq 5.0 -o test.hevc --input - --y4m

[`Orum]

Wow, that did it, thanks! I just realized that --log-level full is needed to get the full help, which actually documents the option.
:thanks:

[sneaker_ger]

Don't forget to dither to 10 bit (e.g. in f3kdb). x265 doesn't do that by default.

[`Orum]

Don't forget to dither to 10 bit (e.g. in f3kdb). x265 doesn't do that by default.

I was debating on whether or not it's better to dither and feed it 10-bit versus directly feeding it 16-bit. I assume the former is best, but I'll do some tests to verify.

[sneaker_ger]

I would probably set f3kdb output to 10 bit. In the end it comes down to which dither algo is better suited for the job. (Which I'm not actually sure with 10 bit encoding. With low bitrate 8 bit ordered dither brought best results.)

[Heaud]

Have you tried piping it raw?

"F:\avs2pipemod\avs2pipemod.exe" -rawvideo %1 | "F:\codecs\video\X265\x265.exe" --input-res 1920x1080 --fps 24000/1001 --input-depth 16 --output-depth 10 --dither - -o "%~dpn1_10bit.mkv"

Aside from the Windows CMD syntax, this should work, just replace what is necessary. Also please excuse my use of avs2pipemod as I do not use avs2yuv64.

[qyot27]

Have you tried piping it raw?

"F:\avs2pipemod\avs2pipemod.exe" -rawvideo %1 | "F:\codecs\video\X265\x265.exe" --input-res 1920x1080 --fps 24000/1001 --input-depth 16 --output-depth 10 --dither - -o "%~dpn1_10bit.mkv"

Aside from the Windows CMD syntax, this should work, just replace what is necessary. Also please excuse my use of avs2pipemod as I do not use avs2yuv64.
If y4m is capable of being used, it should be used. Then you don't have to bother manually specifying resolution, framerate, (and ideally, but maybe not always because of the client program's eccentricities) the pixel format. If you can't use y4m for some reason, only then should you try piping it raw.

And that was the problem here: avs2yuv defaults to piping out y4m, and the OP simply didn't give x265 the --y4m parameter to make it recognize that it was getting y4m.

Although I'll reiterate that using an FFmpeg with a multibit-capable libx265 would make this entire issue moot*. No piping, all the video properties preserved, give it a script outputting 10-bit and it'll be encoded as 10-bit without having to tell libx265 otherwise. The same is true if said FFmpeg was also built against a multibit-capable libx264 (which is new on x264's side).

*as would x265 builds with the LAVF input patch (see: x265-Yuuki-Asuna (https://github.com/msg7086/x265-Yuuki-Asuna), which based on the fact that you're outputting MKV, you're probably already using), if the LAVF it links against had AviSynth support enabled.

[Heaud]

If y4m is capable of being used, it should be used. Then you don't have to bother manually specifying resolution, framerate, (and ideally, but maybe not always because of the client program's eccentricities) the pixel format. If you can't use y4m for some reason, only then should you try piping it raw.

I never got y4m to work outside of 8-bit with my local setup consisting of:

Barough's X265 build (https://forum.doom9.org/showthread.php?p=1832772#post1832772)
Avisynth 2.6
avs2pipemod 1.1.1

If the overhead performance cost is negligible for using raw over y4m, then I am fine with piping raw.

Never knew about the Yuuki Asuna builds until reading your post so I might give that a try. It would be interesting to see the performance diferences of pipe encoding and this build on my setup.

As for the OP, it's good to see that the problem was resolved. I mistakenly skimmed over the post stating all that was required was --y4m before replying to the thread.

[`Orum]

And that was the problem here: avs2yuv defaults to piping out y4m, and the OP simply didn't give x265 the --y4m parameter to make it recognize that it was getting y4m.
Yes, x264 auto-detects y4m whereas x265 does not, so that caught me off guard, at least when combined with its unusual syntax for getting the full help.
*as would x265 builds with the LAVF input patch (see: x265-Yuuki-Asuna (https://github.com/msg7086/x265-Yuuki-Asuna), which based on the fact that you're outputting MKV, you're probably already using), if the LAVF it links against had AviSynth support enabled.
Ah, thanks, I wasn't aware there were builds linked against LAVF. I know the x264 builds I use, while supporting 10-bit input, still don't work directly with 10-bit output from an avs script though for some reason (can't recall right now what it was and I'm not on my usual encoding machine). I'm still piping via avs2yuv64 there, but if either x264/x265 supported the 10-bit input directly from the script, that'd be much more convenient.

I've thought about ffmpeg, but that has many codecs rolled into one, so I prefer the individual command-line utilities.

[qyot27]

I never got y4m to work outside of 8-bit with my local setup consisting of:

Barough's X265 build (https://forum.doom9.org/showthread.php?p=1832772#post1832772)
Avisynth 2.6
avs2pipemod 1.1.1

If the overhead performance cost is negligible for using raw over y4m, then I am fine with piping raw.
Of course it wouldn't work with >8bit, you're not using AviSynth+. x265 doesn't support inputting the Stacked/DoubleWidth hack formats and reconstituting them into high bit depth. There was never a patch for that (the one for x264 to allow that was implemented in the avs input module; x265 doesn't have a dedicated avs input), and AviSynth+ made such a patch irrelevant anyway.

And that build doesn't output MKV, since that's just a normal build from the main project's development tip, and there is no containered output support in the main project's bitbucket repo (Yuuki-Asuna supports both MKV and MP4 output). You're outputting raw .hevc with an incorrect extension.

Ah, thanks, I wasn't aware there were builds linked against LAVF. I know the x264 builds I use, while supporting 10-bit input, still don't work directly with 10-bit output from an avs script though for some reason (can't recall right now what it was and I'm not on my usual encoding machine). I'm still piping via avs2yuv64 there, but if either x264/x265 supported the 10-bit input directly from the script, that'd be much more convenient.
x264 is simply set up to always output 4:2:0 unless the user overrides it (even if the input wasn't 4:2:0). Normally, bit depth wasn't a part of it, because x264 only gained multi-bit support on December 24th, 2017 (https://git.videolan.org/?p=x264.git;a=commit;h=71ed44c7312438fac7c5c5301e45522e57127db4). Despite now having 8-bit and 10-bit in a single build, multi-bit builds still default to 8bit rather than outputting in the same depth as the input, likely for the same reason you have to override it so it doesn't automatically convert your input to 4:2:0 - hardware compatibility. H.264-capable hardware players are almost always restricted to 8-bit 4:2:0.

[Heaud]

Of course it wouldn't work with >8bit, you're not using AviSynth+. x265 doesn't support inputting the Stacked/DoubleWidth hack formats and reconstituting them into high bit depth. There was never a patch for that (the one for x264 to allow that was implemented in the avs input module; x265 doesn't have a dedicated avs input), and AviSynth+ made such a patch irrelevant anyway.

And that build doesn't output MKV, since that's just a normal build from the main project's development tip, and there is no containered output support in the main project's bitbucket repo (Yuuki-Asuna supports both MKV and MP4 output). You're outputting raw .hevc with an incorrect extension.


Never even knew AVS+ had such a feature, been using AVS 2.6 without trying out the alternatives so far due to not wanting to go hunting for plugins all over again. I will give it a look sometime in the future.

For the incorrect extension on the encode, I guess that explains why previewing the output on a media player had broken seeking, ha ha! It never gave me an issue outside of that since the video had to be muxed together with audio post encode. Going to fix the mislabel anyhow for the sake of correctness.

[sneaker_ger]

x265 doesn't support inputting the Stacked/DoubleWidth hack formats and reconstituting them into high bit depth.
You can use the "DoubleWidth hack format" because it is only really a hack within AviSynth. Neither x265 nor x264 ever needed a patch for that. That's the format they always expected 16 bit input in.

But as you say with AviSynth+ and y4m it is easier anyways.

Although I'll reiterate that using an FFmpeg with a multibit-capable libx265 would make this entire issue moot*.
Except for those of us still using 32 bit AviSynth. x265 only officially supports 8 bit on 32 bit binaries. You need some kind of hacked/patched x265 version if you want 32 bit + 10 bit.

[wonkey_monkey]

x265 only officially supports 8 bit on 32 bit binaries.

I'd move "only" to after "supports," otherwise it sounds like it doesn't support 8 bit on 64 bit binaries. But maybe that's just me.

[qyot27]

You can use the "DoubleWidth hack format" because it is only really a hack within AviSynth. Neither x265 nor x264 ever needed a patch for that. That's the format they always expected 16 bit input in.
Then why did this patch (https://github.com/astrataro/x264_tMod/commit/73ff8d0b8f5a546361c26a3abdef661b81828716) exist at all?

Except for those of us still using 32 bit AviSynth. x265 only officially supports 8 bit on 32 bit binaries. You need some kind of hacked/patched x265 version if you want 32 bit + 10 bit.
It doesn't require any patching, it just requires disabling assembly (https://github.com/qyot27/mpv/blob/extra-new/DOCS/crosscompile-mingw-tedious.txt#L2407) (which makes encoding >8bit prohibitively slow, but it's still do-able).

[sneaker_ger]

Then why did this patch (https://github.com/astrataro/x264_tMod/commit/73ff8d0b8f5a546361c26a3abdef661b81828716) exist at all?
I think we might have been talking about different things. I was talking about the raw video input, not AviSynth input. (And I thought you were, too, as x265cli doesn't have AviSynth input in the first place without patches.)

[`Orum]

x264 is simply set up to always output 4:2:0 unless the user overrides it (even if the input wasn't 4:2:0). Normally, bit depth wasn't a part of it, because x264 only gained multi-bit support on December 24th, 2017 (https://git.videolan.org/?p=x264.git;a=commit;h=71ed44c7312438fac7c5c5301e45522e57127db4). Despite now having 8-bit and 10-bit in a single build, multi-bit builds still default to 8bit rather than outputting in the same depth as the input, likely for the same reason you have to override it so it doesn't automatically convert your input to 4:2:0 - hardware compatibility. H.264-capable hardware players are almost always restricted to 8-bit 4:2:0.
What does chroma subsampling have to do with bit depth?

Anyway, my understanding of the "multi-bit" builds was that was only for selectable depth output, and hasn't had any effect on what x264 will take in. As long as I've used it, x264 with lavf or y4m support supports all depths that lavf / y4m support for output, and you can even feed it a full 16 bit video if you like. Whether or not this is a good idea depends on other factors, but I usually try to feed it the same depth I'm outputting to (usually 10 bit). Theoretically 16 bit would be better if it had internal support for it, but I'm unsure without looking at the code base.

The error that appears when I try to directly encode 10-bit output from avs scripts is vexing though: "avs [error]: not supported pixel type: YUV420P10"? I can only assume this is due to its avs demuxer not handling anything higher than 8 bit, while lavf/y4m handle it just fine.

[DJATOM]

10-bit y4m input is actually up-converted to 16-bit internally: http://git.videolan.org/?p=x264.git;a=blob;f=input/y4m.c;h=c688c2f50de7ef4a60e4a2f5c3f03e7b58f9b56d;hb=refs/heads/master#l282
On my observations 16->10->16 will produce lesser bitrate (against direct 16 bits) with near the same quality.

[sneaker_ger]

10-bit y4m input is actually up-converted to 16-bit internally:
Yes, but the 10->16->10 conversion is supposed to be lossless.

[qyot27]

What does chroma subsampling have to do with bit depth?

Anyway, my understanding of the "multi-bit" builds was that was only for selectable depth output, and hasn't had any effect on what x264 will take in. As long as I've used it, x264 with lavf or y4m support supports all depths that lavf / y4m support for output, and you can even feed it a full 16 bit video if you like. Whether or not this is a good idea depends on other factors, but I usually try to feed it the same depth I'm outputting to (usually 10 bit). Theoretically 16 bit would be better if it had internal support for it, but I'm unsure without looking at the code base.

The error that appears when I try to directly encode 10-bit output from avs scripts is vexing though: "avs [error]: not supported pixel type: YUV420P10"? I can only assume this is due to its avs demuxer not handling anything higher than 8 bit, while lavf/y4m handle it just fine.
The point was that since if you give x264 4:4:4, it still by default will convert to 4:2:0 unless you tell it otherwise, the same thing holds true for bit depth. Even if you have a multi-bit build (which yes, means the bit depth of the output), it will output 8-bit if you don't tell it otherwise.

The avs demuxer was only updated to accept the 16-bit high bit depth pix_fmts from AviSynth+. It'll downsample to 10-bit if you tell it to output 10-bit.

[`Orum]

10-bit y4m input is actually up-converted to 16-bit internally: http://git.videolan.org/?p=x264.git;a=blob;f=input/y4m.c;h=c688c2f50de7ef4a60e4a2f5c3f03e7b58f9b56d;hb=refs/heads/master#l282
On my observations 16->10->16 will produce lesser bitrate (against direct 16 bits) with near the same quality.
Interesting. However, it's a good example of where theory and practice can diverge, as upconverting alone doesn't handle some of the problems that can occur later in the process, such as lacking dithering.

The point was that since if you give x264 4:4:4, it still by default will convert to 4:2:0 unless you tell it otherwise, the same thing holds true for bit depth. Even if you have a multi-bit build (which yes, means the bit depth of the output), it will output 8-bit if you don't tell it otherwise.
Good to know, though I'm still using the strictly 10-bit builds so I haven't run into this issue yet. Also the problems I have now with doing 4:4:4 are all on the capture-side of things, but I'll keep that in mind if the problems there ever get fixed.

The avs demuxer was only updated to accept the 16-bit high bit depth pix_fmts from AviSynth+. It'll downsample to 10-bit if you tell it to output 10-bit.
Ah, I didn't realize it handled 16-bit but not 10. Assuming DJATOM is correct and handing it 10-bit sources is better than 16, I don't see any harm in doing this (assuming src is a 16 bit clip):
src.ConvertBits(10, dither=0).ConvertBits(16)
...which would let me avoid the awkwardness of using avs2yuv64. Of course, you could use other methods of hacking off the last 6 bits too, like f3kdb(). Now if only I could do the same for x265, though I'll have to try the patched builds first to see if any support avs directly.

[qyot27]

...which would let me avoid the awkwardness of using avs2yuv64. Of course, you could use other methods of hacking off the last 6 bits too, like f3kdb(). Now if only I could do the same for x265, though I'll have to try the patched builds first to see if any support avs directly.
x265 (with the LAVF patch) acts more or less exactly as x264 did in this case too.

10-bit avs script -> not using the -D option -> 8-bit output
10-bit avs script -> -D 10 -> 10-bit output
and it's detected correctly when you look at the [lavf] info line.

When you give it a 12-bit script, though, things go sideways. The script is incorrectly detected as 8-bit, and --input-depth does nothing to force it to behave (the output is bugged as well). That's going to have to be fixed in the LAVF patch, since it's completely isolated to that patch; libavformat itself is totally fine with 12-bit input from AviSynth+, and will pass it to libx265 correctly as 12-bit when it happens inside FFmpeg.