ffmpeg and VA-API: How to handle YUV 4:2:2 MJPG?

[LigH]

The question was originally posted in the German doom9/Gleitz board, but we seem to lack of experts, so I will forward it here (because the original poster is not fluent in English and also has to wait for 5 days after registration):

Originally Posted by ggrub

Hi,

my system:

Kernel: 4.16.11-1-ARCH 
Resolution: 2560x1440 
Terminal: xfce4-terminal 
CPU: Intel i7-6700K (8) @ 4.200GHz 
GPU: Intel HD Graphics 530 
GPU: NVIDIA GeForce GTX 750 Ti 
Memory: 31094MiB

 vainfo
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.1 (libva 2.1.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Skylake - 2.1.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :    VAEntrypointVLD
      VAProfileMPEG2Simple            :    VAEntrypointEncSlice
      VAProfileMPEG2Main              :    VAEntrypointVLD
      VAProfileMPEG2Main              :    VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline:    VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:    VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline:    VAEntrypointEncSliceLP
      VAProfileH264ConstrainedBaseline:    VAEntrypointFEI
      VAProfileH264ConstrainedBaseline:    VAEntrypointStats
      VAProfileH264Main               :    VAEntrypointVLD
      VAProfileH264Main               :    VAEntrypointEncSlice
      VAProfileH264Main               :    VAEntrypointEncSliceLP
      VAProfileH264Main               :    VAEntrypointFEI
      VAProfileH264Main               :    VAEntrypointStats
      VAProfileH264High               :    VAEntrypointVLD
      VAProfileH264High               :    VAEntrypointEncSlice
      VAProfileH264High               :    VAEntrypointEncSliceLP
      VAProfileH264High               :    VAEntrypointFEI
      VAProfileH264High               :    VAEntrypointStats
      VAProfileH264MultiviewHigh      :    VAEntrypointVLD
      VAProfileH264MultiviewHigh      :    VAEntrypointEncSlice
      VAProfileH264StereoHigh         :    VAEntrypointVLD
      VAProfileH264StereoHigh         :    VAEntrypointEncSlice
      VAProfileVC1Simple              :    VAEntrypointVLD
      VAProfileVC1Main                :    VAEntrypointVLD
      VAProfileVC1Advanced            :    VAEntrypointVLD
      VAProfileNone                   :    VAEntrypointVideoProc
      VAProfileJPEGBaseline           :    VAEntrypointVLD
      VAProfileJPEGBaseline           :    VAEntrypointEncPicture
      VAProfileVP8Version0_3          :    VAEntrypointVLD
      VAProfileVP8Version0_3          :    VAEntrypointEncSlice
      VAProfileHEVCMain               :    VAEntrypointVLD
      VAProfileHEVCMain               :    VAEntrypointEncSlice

Why is it not possible to decode this AVI file?

General
Complete name                            : /media/a5_Lwork/va/test_01_.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
File size                                : 30.6 MiB
Duration                                 : 4 s 120 ms
Overall bit rate                         : 62.3 Mb/s

Video
ID                                       : 0
Format                                   : JPEG
Codec ID                                 : MJPG
Duration                                 : 4 s 120 ms
Bit rate                                 : 59.2 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:2
Bit depth                                : 8 bits
Scan type                                : Progressive
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 1.142
Stream size                              : 29.1 MiB (95%)

Audio
ID                                       : 1
Format                                   : PCM
Format profile                           : Float
Codec ID                                 : 00000003-0000-0010-8000-00AA00389B71
Codec ID/Hint                            : IEEE 
Duration                                 : 4 s 120 ms
Bit rate mode                            : Constant
Bit rate                                 : 3 072 kb/s
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 kHz
Bit depth                                : 32 bits
Stream size                              : 1.51 MiB (5%)
Alignment                                : Aligned on interleaves
Interleave, duration                     : 45  ms (1.12 video frame)
Interleave, preload duration             : 500  ms

$ ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -hwaccel_output_format vaapi -i test_01_.avi -f null -
ffmpeg version 4.0 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 8.1.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa
 --enable-libass --enable-libbluray --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame
 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora
 --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid
 --enable-nvenc --enable-omx --enable-shared --enable-version3
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100
Input #0, avi, from 'test_01_.avi':
  Duration: 00:00:04.12, start: 0.000000, bitrate: 62304 kb/s
    Stream #0:0: Video: mjpeg (MJPG / 0x47504A4D), yuvj422p(pc, bt470bg/unknown/unknown), 1920x1080, 59783 kb/s, 25 fps, 25 tbr, 25 tbn, 25 tbc
    Stream #0:1: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, stereo, flt, 3072 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (pcm_f32le (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
ffmpeg: gen8_mfd.c:2602: gen8_mfd_jpeg_bsd_object: Assertion `0' failed.
Abort (memory dump created)

bye ggrub

Avoiding further chroma subsampling to 4:2:0 would be preferred; but first of all, decoding this material is already an issue in certain combinations of ffmpeg parameters, resulting in green/purple garbage.

[SeeMoreDigital]

Shame the original poster is not using VirtualDub2, as it can open the mjpg.avi container file fine

[LigH]

May work under Linux. But that's not his workflow... and VDub2 won't use VA-API. In the meantime ... he even wonders if it may be a bug in intel's VA-API part of the graphic driver.

[SeeMoreDigital]

If it helps... I'm currently using a very cheap Asus laptop with Intel(R) HD graphics, which can play and display the 'test 01' sample just fine....

[Rumbah]

I would guess that the GPU simply cannot decode 4:2:2 JPEGs.

I'm not even sure that it supports decoding MJPEG at all as I don't see a MJPEG entry point. Some MJPEGs have different headers for the JPEGs so that it's not even a valid JPEG on its own.

In addition to that I guess that the GPU just gets fed the direct video bitstream via the VA API. Even if the MJPEG only contains baseline JPEGs (the only ones supported) I doubt that a standard JPEG decoder could decode a MJPEG bitstream.


I'd just use software decode as I don't think that a GPU decode would be noticable faster/efficient for JPEGs on this CPU.

[LigH]

Indeed, before he knows whether the GPU is able or the driver lacks support ... software decoding may be the simpler solution, and possibly faster without the GPU transfer overhead.

[nevcairiel]

From a performance standpoint, (M)JPEG hardware decoding is usually not worth it anyway, unless you plan to do full zero-copy on GPU processing from decode to encode, and even then it will depend on the decoding speed of your CPU if its any faster.