h.264 encoding on Mac OS X, and my failure at it

[pikoer]

First, I want to apologize for what may be an elementary-level question. I've tried looking through the guides here (and other places), as well as searching older posts, but a lot of what I've found goes into terms and instructions that are way over my head at this point. So, for anybody who is kind enough to try to help me out with this, please understand that a lot of what may be "common" encoding terminology might be a bit over my head. I know what bitrate is, I know (and think I understand) multi-pass encoding, but beyond that things get a bit dicey.

My situation
I'm big into Asian music, and have purchased a number of DVDs (or CDs that include a DVD) with music videos on them. Problem is, one disc may have only a couple videos on it, or as little as one. What I'd like to do is get the videos off of the DVDs, and into a file format that I can then watch in a variety of different ways. I'm looking to go MPEG4 h.264, because (a) it seems like a good, high quality new codec, and (b) I want encoded files that will be playable on my Mac, my Xbox 360, and my PS3, and that's a format that is supported on all three.

(Ending up with files that will play on all three is very important to me, just as a note for possible encoding tips/suggestions.)

The problem is, I don't have a lot of high quality video to practice encoding with, so up until now, I've not really noticed problems with the little bit of h.264 encoding that I've done. Either using QuickTime Pro or VisualHub, I've done stuff like video file -> PSP or video file -> Xbox 360/PS3 on an SDTV, so either I haven't noticed the quality issues or I just didn't have them with the files I was working with before.

My progress so far
I ripped on of the DVDs I have to use as a test before I started trying to encode everything. I got the DVD ripped to my drive, then opened it up in MPEG Streamclip, fixed the timecode breaks, and from there my original plan was to see if I could pass up all other steps and just encode straight to h.264 from there. When I tried it, though, I was very disappointed with the resulting video quality, so I saved my cropped clip back out as an MPEG2 file. (Which, I'm assuming, didn't require a re-encode, since the resulting file is the same quality wise as the original .vob file.) I then de-muxed the MPEG2 file, so that I now have three pieces to work with: an .mpeg file for apps that can handle muxed files, an a video .m2v file plus an audio .aiff file for those that can't (such as QT Pro).

Here are two images, showing examples of the results I've gotten from the test clip that I'm working with. Following that is a basic rundown of the encoding I did for each clip in order.

http://storage.gomorning.com/Compare01.jpg (596k)
http://storage.gomorning.com/Compare02.jpg (560k)


QuickTime Pro - 3000kbps.mp4
Profile: Main
Bitrate: 3000 kbps
Encoding: Multi-pass
Keyframe: Auto


MPEG Streamclip 1.9b3
Profile: ?
Bitrate: 3000 kbps
Quality: 100%
Encoding: Multipass, B-frames


VisualHub
Profile: ?
Bitrate: 3722 kbps
Quality: Go Nuts
Encoding: Two pass
Extra ffmpeg Flags: Best possible AVC/H.264


x264Encoder
(Encoded through MPEG Streamclip, since the x264Encoder export crashed for me in QT Pro. Same settings as Streamclip above.)


...so, from those results, what I'm seeing is a total death of the skin detail (looks like poor Sonim has been super airbrushed), as well as terrible quality of the shadows. I know a DVD-ripped source isn't the highest quality video to work from, but it doesn't seem like the results should be this bad.

Not having done a lot of video encoding up to this point, I'm not really sure where to go from here. I've tried upping the bitrate, but even getting into insane numbers isn't getting me all that great of results. I've tried a number of encoding methods, and while some do produce a bit better results (I was the happiest with Visual Hub), since I don't have the money to try out an expensive software encoding package, I'm basically working with what I've got available. I'm not sure if my problem here is the source video, the settings I'm using for encoding, all of my current encoder choices being terrible, or what.

Obviously, it might be impossible for somebody else to tell me the exact problem without being in my position and having access to my machine, but for anybody else out there who does h.264 encoding on Mac OS X, and advice you might have for me that could point me in the direction of better encodes? Am I making a really silly mistake? Is there a better (free or cheap) encoder that I've overlooked?

Again, excuse the probable high "noob" factor of this, but I appreciate any help that anybody could give me.

[Sagekilla]

Another oddity I noticed was somehow her hair was tinted to an odd coloration. I don't know how the OS X encoder works, but I would try to fiddle with the deblock filter and lower it a bit, to something like -2:-1. That would prevent her face from being smeared so much and losing skin detail.

[[P]ako]

I may suggest to disable the deblocker (replace --filter -2:-1 for --nf) and try again.

[pikoer]

With what I've got, I'm not totally sure how to do that. Are either of you familiar with any of those Mac apps?

In VisualHub, there's a line for "Extra ffmpeg Flags." If I set it to "Best AVC/h264," the following show up:

-refs 5 -me umh -subq 6 -me_range 21 -me_threshold 6 -i_qfactor 0.71428572

I tried to put "-nf" in there (not using two -'s since none of the other commands do), but that didn't work.

When I use the x264Exporter encoder through MPEG Streamclip, I get the following screen, which might as well be a foreign language to me. *heh*

http://storage.gomorning.com/x264.jpg

Also, since my original post, I've also tried ffmpegX, but with the fiddling I've done with it so far, I'm getting about the same results.

[J_Darnley]

Uncheck "FLAG_LOOP_FILTER" which should be the same as not using -flags loop (--nf in x264). The deblock filter is also sometimes called the in-loop or just the loop filter. You should try running ffmpeg -h as that will give a nice long list of options and a short description of each.