VDub2 Canopus HQX 16-bit negotiation issue

[Hotte]

Hi,
I've recently started to pass HBD-Footage (i.e. 10 bit) through AVS+ with VDub2 de- and recompressing with the Canopus HQX codec. Everything worked fine with 8-bit. The clip is export from Edius-X NLE using the Canopus (Grassvalley) HQX Codec which is capable of up to 8K, 10bit, 4:2:2 all-intra in AVI-Containers.

They arrive as YUV422P16 in AVS+ (due to info() and ffmpeg -i) loaded with FFmpegSource2(FILNAM, atrack=-1), further processing works fine. Results appear correctly in VDub2. The trouble starts with "Save as AVI" with constant "Video format negoation error". Output codec again is HQX, mode ist Fast Recompress.

Menu > Video > Decode Format... shows P216 as current format. Whatever I select as at least 10bit-output format: Error persists. Compression menu shows: "Valid pixel formats RGB RGBA YUY2" although the codec is listed under "Similar to Source: P216".

Any ideas ?

[StainlessS]

You might want to say which version VD2 you got.
[Latest is build 44282, 19 Mar 2020:- https://sourceforge.net/projects/vdf.../version%2020/ ]

[Hotte]

Hi StainlessS,
installed latest build 44282, no improvements

Originally Posted by Hotte

They arrive as YUV422P16 in AVS+ (due to info() and ffmpeg -i) loaded with FFmpegSource2(FILNAM, atrack=-1), further processing works fine. Results appear correctly in VDub2. The trouble starts with "Save as AVI" with constant "Video format negoation error". Output codec again is HQX, mode ist Fast Recompress.

Are you doing this further processing in VirtualDub2? Depending on the filter, you might not be able to use "fast recompress". Try another mode.

[Hotte]

@Richard
all processing / Filtering is done in the avs file, I only use VDub2 as AVS+ host.

Even the following avs:

FFmpegSource2("F:\myhqx.avi", atrack=-1)
Return info(last)

Will not be saved with fast recompress due to the negotiation error - although info() shows that YUV422P16 is being returned.

I`d be happy to change the mode to "Normal recompress" and adjust "Video > Decode format...", but I cannot find a 10-16bit mode that does not return "Unable to initialize video compression. Check that the video codec is compatible with the video frame size and that the settings are correct.."

This message seems inappropriate.

EDIT:
I was retrying to load the hqx-Exportfile directly (without avs+) into VDub2 to do a Fast Recompress with the same codec, no filters. This time HQX it is NOT in the P216 list.
But I am pretty sure the codec must be able to handle this as it produces YUV422P16 itselves. Maybe the handshake information is false or misinterpreted ?

[kolak]

Originally Posted by Hotte

Hi,
I've recently started to pass HBD-Footage (i.e. 10 bit) through AVS+ with VDub2 de- and recompressing with the Canopus HQX codec. Everything worked fine with 8-bit. The clip is export from Edius-X NLE using the Canopus (Grassvalley) HQX Codec which is capable of up to 8K, 10bit, 4:2:2 all-intra in AVI-Containers.

They arrive as YUV422P16 in AVS+ (due to info() and ffmpeg -i) loaded with FFmpegSource2(FILNAM, atrack=-1), further processing works fine. Results appear correctly in VDub2. The trouble starts with "Save as AVI" with constant "Video format negoation error". Output codec again is HQX, mode ist Fast Recompress.

Menu > Video > Decode Format... shows P216 as current format. Whatever I select as at least 10bit-output format: Error persists. Compression menu shows: "Valid pixel formats RGB RGBA YUY2" although the codec is listed under "Similar to Source: P216".

Strange enough: VDub2 has absolutely no problem to take any 10/16bit "hqxin.avi" and save it as "hqxout.avi" without AVS+ as Fast recompress with the same codec.

Any ideas ?

There must be some false detection/reporting.
VFW/DirectShow etc. version of Canopus HQX doesn't support anything above 8bit. It never did. Only Edius and Resolve can export/read 8bit+ data for HQX.

[Hotte]

Kolak, meanwhile I have the same impression: In vfw the HQX-codec is somehow false-detected:

VDub2 reports the codec with: Valid pixel formats: rgb rgba yuy2. This was true for its predecessors (Canopus-HQ/HQ lossless) but not for HQX.

Of course the producer is asked but they won´t probably give a damn about vfw and VDub...

So is there a way to override the check in VDub2 and just let compression go ?

[kolak]

Force YUY2 as pixel format. This will work.
There is no way to do any 8bit+ pixel formats with HQX- nothing you can do about it. It's the way how VFW/directshow codec is written.
Resolve uses native support over low level integration- it doesn't touch VFW/directshow.

rgb rgba yuy2 are still only valid pixels formats for HQX. Well- last time I touched it was was years ago, but I doubt anything changed. VFW/directshow are not very good technologies and pro companies don't touch them anymore.
As I said- HQX outside Edius (Resolve) is 8bit only.

[Hotte]

Thanks kolak.

I have had a look at other high-quality 10-bit all-intra codecs and there we have:

huffyuv, FV1
not compatible with Edius

cineform
compatible but very bad timeline performance

ProRes
compatible, relatively good timeline performance, very fast encoding, 422HQ has IQ pretty close to HQX at highest quality setting (to split hair: HQX superfine has a close to invisible advantage in preserving detail)

DNxHD
compatible but not available

Any other suggestions ?
HEVC / H.264 are not all intra so probably not the best intermediate choice...

My questions:
ProRes could be an candidate. I wonder why this proprietary codec is available in vfw from a technical pov (I thought vfw could not manage >8 bit ?) and from a license pov ? I mean 422HQ is pretty good and XQ freely available - how come ?
Are there any other hq-codecs available and installable - how does that work ?

Thanks for some elucidation

[poisondeathray]

Originally Posted by Hotte

cineform
compatible but very bad timeline performance

Interesting, it's faster in other programs than HQX. Probably the fastest CPU based high quality intermediate . (There are faster GPU ones, like Daniel2, NotchLC)

DNxHD
compatible but not available

You can use ffmpeg as alternative (unless you require something specific in vdub2 ?)

Any other suggestions ?
HEVC / H.264 are not all intra so probably not the best intermediate choice...

HEVC will be too slow to decode.

h264 can be set intra, but even with the fastest decode settings (tune fastdecode, intra) , is typically more sluggish on the timeline (high latency) compared to prores, cineform. (However, some editors can use GPU accelerated decoding of h264, but usually not 10bit422, only the 8bit variants)

But x264 can be significantly higher quality at qp 1 than prores xq4444 or HQX - PSNR is typically ~8-15 db higher)

My questions:
ProRes could be an candidate. I wonder why this proprietary codec is available in vfw from a technical pov (I thought vfw could not manage >8 bit ?) and from a license pov ? I mean 422HQ is pretty good and XQ freely available - how come ?
Are there any other hq-codecs available and installable - how does that work ?

The prores version in vdub2 is the libavcodec/ffmpeg version . It's not VFW.

I'd go with ProRes over DNxHD, which can cause problems with levels/gamma. It's a shame that Cineform gives you slow performance with Edius. Normally, I'd have picked that.

[Hotte]

Thanks posiondeathray and Richard.

Tried cineform again with other settings - doesn`t help: Too bad: Horribly slow on the timeline. Btw. Cineform is either interleaved YUV oder RGB - right ?

X264 wouldn't play at all if settings are very HQ / all-Intra but that might need more testing in detail.

DNxHD (generated with ffmpeg) is by far the fastest on the timeline - of course, it is the native codec of Grassvalley. Cannot see levels / gamma issues at first sight, but I will watch out for that, also I will be doing more detail comparison with ProRes - thanks Richard.

My question to @poisondeathray: With ffmpeg you mean I should call ffmpeg from outside taking the avs-script as input and output to DNxHD - right ?
For some setup reason I`d prefer to continue using VDub2 as host. So the question is if there is a way to integrate ffmpeg-DNxHD into the compression list of VDub2 ?

[poisondeathray]

Originally Posted by Hotte

My question to @poisondeathray: With ffmpeg you mean I should call ffmpeg from outside taking the avs-script as input and output to DNxHD - right ?
For some setup reason I`d prefer to continue using VDub2 as host. So the question is if there is a way to integrate ffmpeg-DNxHD into the compression list of VDub2 ?

Yes, ffmpeg directly, with avs input

It should be possible to implement DNxHD in avlib-1.vdplugin (I don't know how to write vdub plugin - it would have to be shekh or someone else)

[kolak]

Originally Posted by Hotte

cineform
compatible but very bad timeline performance

Where?

Cineform is top choice for editing. In app which properly supports it (Premiere, Resolve, Scratch, Vegas if I'm correct) you have amazing performance and at any time you can increase it by miles just by swapping decoding to 1/2 (or further) resolution.

[Hotte]

Originally Posted by kolak

Where?

Cineform is top choice for editing. In app which properly supports it (Premiere, Resolve, Scratch, Vegas if I'm correct) you have amazing performance and at any time you can increase it by miles just by swapping decoding to 1/2 (or further) resolution.

Horribly slow (1-2 Frames/s at 4K 10 bit) timline performance in Edius-X. But I am happy to learn.

So this is what I do:

- I do not have any raw video file but just a 4K 10bit 4.2.2 25bit mov out of a Panasonic G9
- I export this from Edius-X with Canopus HQX 10 bit. Normally this is perfect input for AVS+, but today I just load it into VDub2
- In VDub2 this becomes YUV422P16
- Select Cineform as compressor, 10 bit, YUV422, Fast Recompress: Negotiation Error
- Select Cineeform 16-bit, YUV422, Fast Recompress: Negotiation Error
- So what do you recommend to do next ? Normal / Decode Format - Which ?

[kolak]

Originally Posted by Hotte

So this is what I do:

- In VDub2 this becomes YUV422P16
- Select Cineform as compressor, 10 bit, YUV422, Fast Recompress: Negotiation Error
- Select Cineeform 16-bit, YUV422, Fast Recompress: Negotiation Error
- So what do you recommend to do next ? Normal / Decode Format - Which ?

You get YUV422P16 because you import over ffmpeg source plugin. Then when you try exporting you use VFW system HQX codec, which only supports 8bit.

Try before you hit import using options button- this may let you force VFW HQX decoder not ffmpeg.
If not when you go to Compression and choose GV HQX then go to Pixel Formats and switch to YUY2. This should let you export to GV HQX. If not try also forcing Decode format to YUY2. You will loose 10bit precision.

Instead all of this use virtual AVI file system in AVS. Mount your AVS script as fake AVI and then import that AVI back to Edius. Not sure if AVS+ supports it but in vapoursynth you can mount final script result as v210 format, so this way you can preserve 10bit. Edius should be fine with 10bit (v210 ) AVI. If not use YUY2. No need for any intermediate files. You can use "AVS filters" directly in Edius.

[poisondeathray]

Originally Posted by Hotte

- I do not have any raw video file but just a 4K 10bit 4.2.2 25bit mov out of a Panasonic G9
- I export this from Edius-X with Canopus HQX 10 bit. Normally this is perfect input for AVS+, but today I just load it into VDub2
- In VDub2 this becomes YUV422P16
- Select Cineform as compressor, 10 bit, YUV422, Fast Recompress: Negotiation Error
- Select Cineeform 16-bit, YUV422, Fast Recompress: Negotiation Error
- So what do you recommend to do next ? Normal / Decode Format - Which ?

For a native 10bit file (not stacked), Cineform in vdub2 requires normal or full recompress. It might be the same for stacked , not sure. I would avoid stacked all together if possible.

You can verify output with other tools, such as 10bit waveforms. ffmpeg has one, and avisynth does too now. If you test a 0-1023 gradient, you will see if there were problems somewhere , such as an 8bit intermediate step somewhere, which will manifest as gaps


One bizarre issue with some DCT based codecs like DNxHR/DNxHD and prores to a lesser extent is severe noise on some types of patterns like small checkers. It's not subsampling related (4:4:4 is affected too). Official Avid and ffmpeg implementations are affected. I've reproduced it on other patterns, other colors, other grid sizes. You can see some of the disussion here starting around post 672 . x264 (also dct based) , cineform (wavelet based) are unaffected
https://forum.doom9.org/showthread.php?p=1855173

[Hotte]

Originally Posted by poisondeathray

One bizarre issue with some DCT based codecs like DNxHR/DNxHD and prores to a lesser extent is severe noise on some types of patterns like small checkers. It's not subsampling related (4:4:4 is affected too). Official Avid and ffmpeg implementations are affected. I've reproduced it on other patterns, other colors, other grid sizes. You can see some of the disussion here starting around post 672 . x264 (also dct based) , cineform (wavelet based) are unaffected
https://forum.doom9.org/showthread.php?p=1855173

I fear if I dive into this, no codec will be left for me. So I am probably doing better to keep my eyes a bit shut

[shekh]

Hi,
maybe the issue with "Similar to Source: P216" was because the codec was selected before choosing "Similar to source"? This filtering of codecs does not remove the selected one even if it appears inappropriate.

[Hotte]

Originally Posted by shekh

Hi,
maybe the issue with "Similar to Source: P216" was because the codec was selected before choosing "Similar to source"? This filtering of codecs does not remove the selected one even if it appears inappropriate.

Yes that could pretty much have been the case. It does not appear anymore if I do things right.