Yeah, that's just bad YV12 to RGB conversion.
The only way to make that look better is to encode into 720x480 or similarly 'bigger than native' resolution (so that the PSP has more information from which to create your 480x272 RGB picture).
TL;DR Blame Sony
__________________
[I'm human, no debug]
|