H.264 encoding problems by newbie

[papilio]

Hi all,

This is my first post, so please forgive me if I'm in the wrong forum.

I've spent a ton of time searching the forums, getting lost in the vastness and/or not finding relevant answers, although no doubt they're there.

I know this is a pretty long post, sorry if I'm over-explaining myself.

I'm trying to encode a number of 1920x1200 screen capture segments, done with the amazing, apparently lossless Fraps. I generally work with segments of (Fraps compresed) 4096MB files. The content is Google Earth touring (of the Inca Trail this time around -- cruising apparently inches over the ridges of the Andes is quite a rush!).

Using VirtualDub (and Avisynth on rare occasions), my video processing stages as as follows (in case it's relevant).

Screen capturing is done at 1/4 speed to help produce a smoother final result (rarely a need to multidecimate with Avisynth), and Fraps set at 30fps.

1 - Increase frame rate to 120 to restore proper speed.

2 - Decimate by 4 to return to 30fps.

3 - Crop off the Google watermark and change the color depth to YUY2 4:2:2 YCbCr.

4 - Smart-sharpen borders, resize to 1920x1080 w/smart-resize bicubic, also including in this step adjustments to brt/contr and color saturation.

5 - Unsharp mask in 3 separate VD stages, 3 then 5 then 7, all at medium strength and minimum necessary clipping threshold.

6 - Gentle spatial and temporal denoising.

Quite gorgeous final images I must say, at full screen on a 24-inch monitor.

Now the trouble -- compression!

I'm using the registered version of MainConcept H.264, v2.00.00.06. Certainly it's merely a case of my not having the faintest idea what I'm doing, nor how to choose the proper settings. For reference, I'm using a Dell XPS410 with 3GB RAM.

The only setting I've found (haven't tried them all) which will play back smoothly is HD1 720p (and why is there an option for HD2 1080i, but none for 1080p?). I find that letting the MC software do the resizing from 1080 gives much cleaner results than resizing first to 720 with VD. The output video is certainly the best I've seen with any codec I've tried, still it's a pathetic rendering of the pristine original 1080 image. I suppose I could live with a 720p video version if I absolutely had to, if that video had anything approaching a 1-pixel resolution anywhere in the frame, but the whole image is a bit mushy. (And for some reason, even using different players, the tint is always shifted considerably toward the red. The video/overlay settings for the nVidia 7600GT are set at neutral hue.) At a 3Meg bitrate my computer generally uses about 15% of its CPU.

Encoding at H.264 high (1080p) and program video gives a truly wonderful image, and the resulting file size is, amazingly, smaller than the HD1 720p's, but I've never gotten anything approaching a smooth playback (with WMP11). Still just a 3Meg bitrate. Changing down from H.264 high seems not to help, only hurts the image. The CPU in this case runs at about 50%. I've tried a few of the advanced settings without improvement. I initially had a number of the more common codecs installed, but I've removed most of them so that I can start with a clean slate once I learn what to use.

Now first of all, I know next to nothing about keyframes, which may well be a large or even the entire cause of my problems. VirtualDub makes every frame a keyframe, or at least I don't know how to change that. I know of the option when opening a file with VD to show advanced options, including re-derive keyframe flags, still I end up with every frame being a keyframe (I load and save with direct stream source).

BTW, my impression is that having a keyframe once or twice a second gives the cleanest video, at the expense of file size, of course (which I suspect will be reasonable in any case), and probably bitrate too -- what frequency recommendations would any of you have?

I guess that's everything I can say for now. Sorry again for the long post! Can anyone tell me what to try?

Thanks,
Michael

[foxyshadis]

Vdub only tells you what the source's keyframes are. In fraps, every frame is an independent keyframe to make editing simple - fraps is basically a high quality MJPEG as a codec. That won't affect your encoding gop length, though, which is what the Mainconcept config controls. Note the chosen length will be a maximum (x264 has max and min options), the codec chooses the right place to insert keyframes if needed.

The less keyframes, the higher the quality at a given bitrate. The only real purpose keyframes serve is for quick seeking; you just choose the distance by how much lag you can tolerate. (Or by how bad you expect network transmission to be.)

For higher quality 1080p, I'm sad to say but I suspect bumping the bitrate is the only fix. 3Mbps is, as you've seen, enough for watchable 720p but not over 2x as many pixels. 6Mbps should have similar quality in 1080p, for pristine quality you might need anything up to 20Mbps. Google Earth isn't very amenable to compression. However, since cpu usage goes up with bitrate, it's very doubtful you could use that effectively. If CABAC isn't being enabled in main profile, that could boost the quality, but that's the very setting that costs so much decoding time. Different motion methods don't cost anything at decode though.

Smart filtering, using specialized less-destructive avisynth filters, can greatly reduce visible encoding artifacts and softening, and once in a while work real magic, at the expense of more complexity and encoding time.

If you really want good playback, consider investing in a mid-range DX10 gpu with built-in AVC decoding. This way you can throw as much bitrate at the problem as it needs, and have perfectly smooth playback. (Getting it to work on XP is a bit of a bodge job for the moment though.) Otherwise, you might look into CoreAVC, which is the fastest decoder available otherwise, or find out if you can get Mainconcept hardware decoding working with the graphics card you have now.

[Dark Shikari]

If you are having playback speed issues try using FFDshow if you're not using it already, and if that's not fast enough use CoreAVC which is so fast that my laptop can play 1920x1200 High Profile video at full speed while on battery mode (1Ghz Core 2 Duo).