[FIXED] mencoder: "The selected video_out device is incompatible with this codec."

[doragasu]

A lot of time ago (about 2 years ago, I think) I made a tiny script to transcode video files from BluRays to H264. It worked fine, and I didn't use it again until now, and guess what, it's not working. When it tries encoding the first pass, it executes:

mencoder video.mkv -vf crop=1920:816:0:132 -o /dev/null -of rawvideo -passlogfile x264_1st_pass.log -nosound -ovc x264 -x264encopts pass=1:subq=7:frameref=4:b_pyramid=strict:me=umh:me_range=16:partitions=p8x8,b8x8,i8x8,i4x4:bframes=4:b_adapt:keyint=240:keyint_min=24:scenecut=40:deblock:cabac:qcomp=0.6:qp_min=10:qp_max=51:qp_step=4:vbv_maxrate=40000:vbv_bufsize=40000:direct_pred=auto:weight_b:8x8dct:mixed_refs:trellis=1:no_fast_pskip:nr=500:level_idc=41:threads=auto:no_psnr:no_ssim

And mencoder spits this:

MEncoder SVN-r36498-snapshot-4.8.2 (C) 2000-2013 MPlayer Team
206 audio & 433 video codecs
success: format: 0 data: 0x0 - 0xb6e4716e
libavformat version 55.19.104 (internal)
libavformat file format detected.
[lavf] stream 0: video (h264), -vid 0
VIDEO: [H264] 1920x1080 0bpp 23.976 fps 0.0 kbps ( 0.0 kbyte/s)
[V] filefmt:44 fourcc:0x34363248 size:1920x1080 fps:23.976 ftime:=0.0417
Opening video filter: [expand osd=1]
Expand: -1 x -1, -1 ; -1, osd: 1, aspect: 0.000000, round: 1
Opening video filter: [crop w=1920 h=816 x=0 y=132]
Crop: 1920 x 816, 0 ; 132
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
libavcodec version 55.39.100 (internal)
Selected video codec: [ffh264] vfm: ffmpeg (FFmpeg H.264)
==========================================================================
[VD_FFMPEG] Trying pixfmt=0.
[VD_FFMPEG] XVMC-accelerated MPEG-2.
The selected video_out device is incompatible with this codec.
Try appending the scale filter to your filter list,
e.g. -vf spp,scale instead of -vf spp.
[VD_FFMPEG] Trying pixfmt=0.
[VD_FFMPEG] XVMC-accelerated MPEG-2.
The selected video_out device is incompatible with this codec.
Try appending the scale filter to your filter list,
e.g. -vf spp,scale instead of -vf spp.

Exiting...

I have been googling for a while, and found no clue. Any suggestions?

[microchip8]

do you happen to have VDPAU enabled at compile time? I find this often the cause of interfering with mencoder

[doragasu]

I don't know what switches mencoder has been built with. I'm using Arch Linux and have installed mencoder from its repository.

[Selur]

also have you tried using:
-vf crop=1920:816:0:132,scale,format=i420
instead of
-vf crop=1920:816:0:132
also read: http://forum.videohelp.com/threads/359120-%5BFixed%5D-Mencoder-makes-me-crazy!

[doragasu]

Unfortunately, using "-vf crop=1920:816:0:132,scale,format=i420" makes no difference. It looks like I'll have to build mencoder from sources, with --disable-vdpau switch, or try encoding with ffmpeg instead.

[doragasu]

OK, I have built mplayer/mencoder from sources using "--disable-vdpau" and it's working now, Thanks!

I'm giving ffmpeg a try anyway, it is a lot more simple, and looks a lot less prone to cause problems when updating.

[microchip8]

OK, I have built mplayer/mencoder from sources using "--disable-vdpau" and it's working now, Thanks!

I'm giving ffmpeg a try anyway, it is a lot more simple, and looks a lot less prone to cause problems when updating.

ffmpeg more simple? it's more complex than mencoder, especially its filters.

Also, ffmpeg is more prone to problems as it constantly changes. Mencoder doesn't change (at all). The only thing mencoder could do is stop working if something big changes and mencoder is not updated

[doragasu]

I know almost nothing about video encoding, but it looks like ffmpeg presets are dead easy to use. The only filter I have tried is "crop", and I had no problem with it. Maybe mencoder also has presets as easy to use as ffmpeg ones, I haven't dug mencoder man file yet.

I have run an encode using ffmpeg (on a 2 hour BluRay rip) and now I'm going to run the same one using mencoder to decide if I continue with mencoder or switch to ffmpeg.

[microchip8]

of course mencoder supports x264 presets... just use
-ovc x264 -x264encopts preset=slow:tune=film:profile=high for example

[doragasu]

Hummm, that looks pretty similar to what I fed to ffmpeg. Thanks for the clear example on preset usage!

[Selur]

Just ran into the same error message when trying to decoder H.264 content to 420p10le raw video.
Using:
mencoder -lavdopts threads=8 -of rawvideo -o - -ovc raw -demuxer lavf -noskip -vf scale,format=i420,scale,format=420p10le -forcedsubsonly -nosub -nosound "F:\TESTCL~1\MPEG-4~1.264\INTROC~1.MKV"
I get:
MEncoder SVN-r36933-4.8.2 (C) 2000-2014 MPlayer Team
success: format: 0 data: 0x0 - 0x2b7b61
libavformat version 55.33.100 (internal)
libavformat file format detected.
[lavf] stream 0: video (h264), -vid 0
[lavf] stream 1: audio (dca), -aid 0
VIDEO: [H264] 1280x672 0bpp 25.000 fps 0.0 kbps ( 0.0 kbyte/s)
[V] filefmt:35 fourcc:0x34363248 size:1280x672 fps:25.000 ftime:=0.0400
Writing to stdout
Opening video filter: [expand osd=1]
Expand: -1 x -1, -1 ; -1, osd: 1, aspect: 0.000000, round: 1
Opening video filter: [format fmt=420p10le]
Opening video filter: [scale]
Opening video filter: [format fmt=i420]
Opening video filter: [scale]
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
libavcodec version 55.52.102 (internal)
Selected video codec: [ffh264] vfm: ffmpeg (FFmpeg H.264)
==========================================================================
Pos: 0.0s 9f ( 2%) 0.00fps Trem: 0min 0mb A-V:0.000 [0:0]
The selected video_out device is incompatible with this codec.
Try appending the scale filter to your filter list,
e.g. -vf spp,scale instead of -vf spp.

Exiting...

What am I missing?

[microchip8]

compiled with VDPAU?

[Selur]

nope, it's a Windows binary, so no VDPAU :/