Patch to force RGB instead of YUV420 when using an external video encoder

[theultramage]

For some reason, VirtualDub's external encoder engine hardcodes the output colorspace to nsVDPixmap::kPixFormat_YUV420_Planar and there is no way to change that. This means that lossless processing is not possible.

I have fiddled with the source code a bit and located the two offending places:
@@src\VirtualDub\source\DubOutput.cpp:
int VDAVIOutputCLISystem::GetVideoOutputFormatOverride() {
return nsVDPixmap::kPixFormat_YUV420_Planar;
}
@@src\VirtualDub\source\AVIOutputCLI.cpp:
IVDMediaOutputStream *AVIOutputCLI::createVideoStream() {
if (mpVideoOutput)
throw MyError("CLI: Only one video output is supported.");

VDAVIOutputRawVideoFormat rawFormat = {};
rawFormat.mOutputFormat = nsVDPixmap::kPixFormat_YUV420_Planar;
rawFormat.mScanlineAlignment = 1;
It is possible to change these to use "nsVDPixmap::kPixFormat_RGB888", or even make it configurable using "g_dubOpts.video.mOutputFormat". This will allow using full rgb lossless x264
--qp 0 --preset veryslow --output-csp rgb --demuxer raw --input-csp bgr --input-res %(width)x%(height) --fps %(fpsnum)/%(fpsden) -o "%(tempvideofile)" -

Since getting virtualdub to compile takes a bit of work, I also looked up the file offsets that need to be changed, so that you can just hexedit the official exe. 15 (0F) is yuv420, 7 (07) is rgb888.

VirtualDub-1.10.4.35491-Win32:
0001EE11 0F -> 07
000D0F98 0F -> 07

VirtualDub-1.10.4.35491-AMD64:
0002A4E1 0F -> 07
0011CE48 0F -> 07

[Emulgator]

And I thought that Virtual Dub -> Video -> Color Depth -> Output to Compressor/Display would set what it promises...
Am I wrong ? Ah, this is about external standalone encoders... Good find, I still have to dig into that workflow.

[Tup3x]

For some reason, VirtualDub's external encoder engine hardcodes the output colorspace to nsVDPixmap::kPixFormat_YUV420_Planar and there is no way to change that. This means that lossless processing is not possible.

I have fiddled with the source code a bit and located the two offending places:
@@src\VirtualDub\source\DubOutput.cpp:
int VDAVIOutputCLISystem::GetVideoOutputFormatOverride() {
return nsVDPixmap::kPixFormat_YUV420_Planar;
}
@@src\VirtualDub\source\AVIOutputCLI.cpp:
IVDMediaOutputStream *AVIOutputCLI::createVideoStream() {
if (mpVideoOutput)
throw MyError("CLI: Only one video output is supported.");

VDAVIOutputRawVideoFormat rawFormat = {};
rawFormat.mOutputFormat = nsVDPixmap::kPixFormat_YUV420_Planar;
rawFormat.mScanlineAlignment = 1;
It is possible to change these to use "nsVDPixmap::kPixFormat_RGB888", or even make it configurable using "g_dubOpts.video.mOutputFormat". This will allow using full rgb lossless x264
--qp 0 --preset veryslow --output-csp rgb --demuxer raw --input-csp bgr --input-res %(width)x%(height) --fps %(fpsnum)/%(fpsden) -o "%(tempvideofile)" -

Since getting virtualdub to compile takes a bit of work, I also looked up the file offsets that need to be changed, so that you can just hexedit the official exe. 15 (0F) is yuv420, 7 (07) is rgb888.

VirtualDub-1.10.4.35491-Win32:
0001EE11 0F -> 07
000D0F98 0F -> 07

VirtualDub-1.10.4.35491-AMD64:
0002A4E1 0F -> 07
0011CE48 0F -> 07

It would indeed be great if someone could put setting so that we can choose what format it will feed to external encoder (or that it would honour the currently used format and not convert it).

I tried to do that hex edit but unfortunately it just gave green mess. :scared:

[theultramage]

Yes, it's probably not worth it. The first location is easy to patch, but the other does not have enough room. If you really need it, might as well just compile the source code with that modification I mentioned above; that's what I'm currently using. Or, make several virtualdub exe copies, each patched to a different colorspace value. The 'enum VDPixmapFormat' is at the beginning of src/h/vd2/pixmap.h.

[shekh]

"Output format" is not quite appropriate for this task because then there are two ways
1) pass pixel format specification as a variable to command line - impossible (too difficult to generalize)
2) user must blindly match output format vs encoder profile - ugly (obliged to fail)

What I propose is pixel format selectable per profile, like yuv420, rgb24 and some more (not all).
I am going to make required changes in my version.