Encoding High@L3.1 for Playstation Vita

[verifonix]

I was trying to get x264 to encode H264 video for my Playstation Vita. I was surprised there wasn't as much information on the matter as I'd expect.
After some hours of getting "This file is not compatible" errors on the handheld, I figured I might as well break down what was the problem.

Edit: after some confusion, it seems things actually changed along with firmware.

Prior to 2.00 files were checked to be of High@L3.1 specifications, 720p files can play (downscaled)

After 2.00 files can be up to High@L4.1 specifications, 1080p files can play (downscaled)
B-pyramid works, and it doesn't look like you need to set any maxrate/bufsize.

[pandy]

I was trying to get x264 to encode H264 video for my Playstation Vita. I was surprised there wasn't as much information on the matter as I'd expect.
After some hours of getting "This file is not compatible" errors on the handheld, I figured I might as well break down what was the problem.

I first looked at this post (http://forum.doom9.org/showpost.php?p=1588604&postcount=38) that said the Vita system requires High@L3.1 video. Video resolution should be 960x540 since that's the resolution of the device.
On the wiki (H.264/MPEG-4 AVC Baseline/Main/High Profile Level 4.1, Maximum 1080p, AAC (From 2.0 system software)) though, it says "H.264/MPEG-4 AVC Baseline/Main/High Profile Level 4.1, Maximum 1080p, AAC (From 2.0 system software)".
Kind of confusing. I think it can only play the 1080p videos streaming sites like YouTube output. Better not use 4.0/4.1 for encoding since it'll likely not play at all.

In any case, I ran an encode including various options (don't remember every single option exactly) and not a single video would play. I eliminated every option one by one, until I had just --level 3.1 and it still didn't work. Apparently it needed more specific encoding. Eventually I found that the program I was using, Ripbot264 (http://forum.doom9.org/showthread.php?t=127611), has its own set of presets when you change the level. Uping selecting 3.1, it lists the following:
--level 3.1 --aud --nal-hrd vbr --vbv-bufsize 17500 --vbv-maxrate 17500 --b-pyramid none

Following that, encodes started to play. Not sure it needs every single of those options but I implemented them just to be safe. I eventually ended up with this, which compresses (anime) well enough. using tMod x264

--level 3.1 --aud --nal-hrd vbr --vbv-bufsize 17500 --vbv-maxrate 17500 --preset veryslow --filter -2,-2 \
--b-pyramid none --aq-mode 3 --aq-strength 0.8 --fade-compensate 0.4 --psy-rd 0.4:0.0


Hmmm - kind of strange... i remember similar exercise for PSP...
From my perspective it works always (resoslution 480x272):

level=30:bitrate=432:vbv_maxrate=6000:vbv_bufsize=2000:qpmin=4:cabac=1:b-pyramid=0:weightp=1:weightb=1:trellis=2:nal_hrd=none:bframes=7:ref=3:partitions=all:8x8dct=0:bluray-compat=1:threads=auto

(yes i know that some parameters are contradictory but... it works)

[verifonix]

Hmmm - kind of strange... i remember similar exercise for PSP...
From my perspective it works always (resoslution 480x272):

level=30:bitrate=432:vbv_maxrate=6000:vbv_bufsize=2000:qpmin=4:cabac=1:b-pyramid=0:weightp=1:weightb=1:trellis=2:nal_hrd=none:bframes=7:ref=3:partitions=all:8x8dct=0:bluray-compat=1:threads=auto

(yes i know that some parameters are contradictory but... it works)

The PSP took high compression encoding like a champ. Never had a problem with the PSP. Here's my profile which works like a charm
--level 3.0 --vbv-bufsize 10000 --vbv-maxrate 10000 --preset veryslow --filter -2,-2 \
--ref 3 --bframes 12 --b-pyramid none --partitions p8x8,b8x8,i4x4 --no-8x8dct
B-pyramid none is crucial, and "partitions all" actually gives video artifacts from time to time. This is on an old FAT psp-1000.
You can actually go as high as --bframes 16, but again the video decoder can give an occasional artifact bug if you really max everything out.
Also, 2-pass encoding near the 10000 kbps range ended up in files the PSP could no longer read.

[LoRd_MuldeR]

Note that VBV is orthogonal to the Level/Profile. So you need to ensure that your encoded video sticks within the supported Level/Profile of the target device and complies to the required VBV parameters.

Also note that "Level 4.1, Maximum 1080p" is kind of misleading. Level 4.1 does support (at most) 1080p, but only at 30 fps (or lower). At a higher frame rate, a lower resolution is required to stick within the Level 4.1 limits!

That's because the Level not only limits the maximum "Macroblocks per frame" but also the maximum "Macroblocks per second".

[vivan]

That's odd. Before 2.0 vita rejected all streams that were higher than level 3.1. Since 2.0 this check was removed. However it's limited by the number of reframes. Usually I encode 720p for my vita with
--preset veryslow --ref 4 (--crf 17)

I'll recheck everything later today, but for now it's kinda hard to believe that simple --level 3.1 doesn't work.

Also note that "Level 4.1, Maximum 1080p" is kind of misleading. Level 4.1 does support (at most) 1080p, but only at 30 fps (or lower). At a higher frame rate, a lower resolution is required to stick within the Level 4.1 limits!Vita started supporting 1080p only since FW 2.0 (before maximum resolution was 720p), I think this is the reason why it's worded that way.

UPD: just checked (--preset veryslow --ref 1-16) for 720p and 540p. It didn't reject them even with 16 reframes (540p (https://www.mediafire.com/?9u5vwgiqcatnqfu)/720p (https://www.mediafire.com/?wzrx787j79vt10a)).
Now I have to find something heavy to check stability. Because even if files passed initial check they still could be rejected by the decoder after some time.
Oh, and don't forget to always include audio that works (aac with ~ 192 kbit/s bitrate or less). Vita rejects video without audio or with high bitrate audio (I still don't know real limit).

[verifonix]

That's odd. Before 2.0 vita rejected all streams that were higher than level 3.1. Since 2.0 this check was removed. However it's limited by the number of reframes. Usually I encode 720p for my vita with
--preset veryslow --ref 4 (--crf 17)

I'll recheck everything later today, but for now it's kinda hard to believe that simple --level 3.1 doesn't work.

Hmm. This was tested on a just recently unboxed VITA still on firmware 1.62. I never went over level 3.1 and files just didn't play.
In that previous thread someone mentions a limit of up to 11 reference frames... huhh

Edit: Looks like you're right though, ref seems to be the deciding factor. Updated first post.

[vivan]

Videos with ref way above 4 played fine so I'm quite sure ref isn't the deciding factor. In that previous thread someone mentions a limit of up to 11 reference frames.It was me :D
When I just got vita I did similar test, that's how I got that number.

Btw, they've added an option to stream video from PC (since 2.60 IIRC), so you can simply put files for testing into CMA video directory and then check them on vita (without copying them). It works fine for watching too, but on video with high peak bitrate (like 20 mbit/s) it could pause during playback for buffering.

Finished testing with slowed down to 24 fps crowdrun (and crf 20, which resulted in 8/13/28 mbit/s bitrate).
720 (https://www.mediafire.com/?wny7iq4yd8rn722)/540p (https://www.mediafire.com?coaolxmnay3b5g8) with 16 refs works fine. Quite impressive. That means that it can decode any 720p video, except with really high bitrate and fps (it can do 50 mbit/s or 60 fps, but not at the same time).
For 1080p the best it could do was 7 (https://www.mediafire.com/?7flvffclv66vn9e), on 8 (https://www.mediafire.com/?cnow18p30kcpq3t) I'm getting error.

[verifonix]

Awesome, thanks for testing. I was just about to give up making test files and copying them over through CMA every time.
Didn't even know about the streaming. Nice.