2 questions, PSNR and Options for x264

[morph166955]

Ok so thus far i have read as much as i can find on here about the options to use for both passes and psnr and all that but i have a few questions.

Im trying to encode an episode of the hit show House (before anyone starts pointing fingers, i own the dvd and im just using it to test this stuff), and for those wondering, the episode im testing is 1x14 - Control (Disc 2, Side B, Chapter 2).

The system that does the encoding is a FreeBSD 6.0 System with a Intel P4 3.2 Hypterthreaded CPU, 4 gig ram, 2TB Storage Raid5 Array on a Highpoint RocketRaid 2220 with 6 Western Digital 400Gig Raid Edition SATA Drives, a 200Gig Raid1 system drive (2x WD 200Gig SATA Raid Edition Drives Mirroed), 2x Lite-On DVD-RW Drives, Mitsumi Media Reader 7-in-1 Floppy, an Ecrix VXA-1e on a Adaptec SCSI Card, plus the usual fans, power supplys etc. So as far as im concerned based on this system I should have rather good speed and processing time (which as far as i can see im getting). I barely use any swap space ever due to the amount of ram in the system and the fact that BSD doesnt really like to swap stuff out (it is working though i have seen things go into the swap).

Im doing the encoding with a version of mencoder I compiled off off the cvs with the following version info:

MEncoder dev-CVS-060326-19:49-3.4.4 (C) 2000-2006 MPlayer Team
CPU: Intel Pentium 4/Celeron D Prescott; Pentium D/XE Smithfield; Xeon Nocona,Irwindale (Family: 15, Stepping: 1)
CPUflags: Type: 15 MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
Compiled for x86 CPU with extensions: MMX MMX2 SSE SSE2

For my passes im using:

Pass 1: mencoder dvd:// -dvd-device /drives/iso5 -vf crop=720:352:0:66,scale=720:396,expand=0:-4:0:2 -oac mp3lame -ovc x264 -af volnorm=2:.40 -lameopts preset=standard -x264encopts pass=1:turbo=1:threads=2:me=1:subq=1:bframes=3:psnr:b_pyramid:weight_b -o /dev/null

Pass 2: mencoder dvd:// -dvd-device /drives/iso5 -vf crop=720:352:0:66,scale=720:396,expand=0:-4:0:2 -oac mp3lame -ovc x264 -af volnorm=2:.40 -lameopts preset=standard -x264encopts pass=2:threads=2:frameref=5:me=3:8x8dct:subq=6:bframes=3:b_pyramid:mixed_refs:brdo:bime:nofast_pskip:psnr:weight_b:4x4mv:bitrate=BITRATE -o House-SB-CH2-FP.avi

where /drives/iso5 is a mount built from mdconfig pointing to an iso image of that one chapter that I created and BITRATE is what i choose for that specific run.

Now that i have given you all the info (wow i didnt think it would be that much)...Here are my questions.

1) I ran two runs on it, I did one first pass run and used that for each of my two second pass runs since there were not options that changed on either first pass runs. The first run was done at 500kbit/s and the second run was done at 1000kbit/s. The following is the PSNR info that mencoder spit out from each run...what im asking is how exactly do i read this to understand it and make the best decision on bitrate in the future.

Pass 1: x264 [info]: PSNR Mean Y:42.153 U:45.035 V:46.554 Avg:42.901 Global:42.593 kb/s:486.51

Pass 2 (500 kbit/s): x264 [info]: PSNR Mean Y:44.049 U:45.625 V:47.369 Avg:44.530 Global:43.948 kb/s:500.04

Pass 2 (1000 kbit/s): x264 [info]: PSNR Mean Y:46.024 U:46.916 V:48.931 Avg:46.366 Global:45.905 kb/s:1000.05

Both movies look very good based on me putting them up on my projector (Infocus SP5000) and watching them. My big question is what in these numbers really is the one im looking for and what do i want it to be? Also what is the bitrate from Pass 1 showing? Is that giving me an "optimal" bitrate or is that just a garbage number that i should pay no attention too? I know when i used to do Xvid that being up around 1000 kbit/s was a good place to be but due to x264 being much better at encoding and pass 1 showing a 486kbit/s number im hoping that 500 is a good # (or 600 or 700 or w/e is good, im really looking to learn here so any info is great)

Question 2: Are the options that im using above for mencoder the best? The Crop/Scale/Expand values were done using -vf cropdetect to get the crop values, then i scaled it to get it from the 720x480 to the 720x540 that the aspect ratio change normally does (i prefer not to have the aspect ratio resize movie because it causes hell with my media center so i pre-scale it). The expand value adds 2 pixel lines worth of black on top and bottom to get the height divisible by 16 so that the area of the image is divisible by 16x16 cubes. As for the volnorm i find that those options work rather well for ripping and fixing movies/tv shows that have low volume. The rest of the stuff being the x264 options is really what im looking for advice on. Ive compiled the best list i could based on info from this and other sites but a professional look can't hurt right? Also im noticing that the threads=2 thing doesnt actually do anything based on the info from "top". The total cpu usage barely gets over 50% and thats primarily because i have a few other things like samba running that are generating the load.

Thanks for the read on this lengthy post and any info you may give me in advance.

[akupenguin]

The expand value adds 2 pixel lines worth of black on top and bottom to get the height divisible by 16 so that the area of the image is divisible by 16x16 cubes.
No! x264 can add borders by itself in order to get mod16 resolution. Manually padding with black is the worst thing you can do. When people say that mod16 is preferred, we mean actually having content filling a mod16 resolution, e.g. scale=720:400. And if you don't want to mess with aspect ratio, then just encode whatever resolution it comes out without padding.

[morph166955]

really....interesting i didnt know that! so being that my input video is 720x396 i should just push that to x264? Just out of sheer curiosity whats so bad about doing the expand filter and making it 720:400 to meet mod16?

[foxyshadis]

Borders introduce hard transitions (video->black) that DCT (mpeg's main compression algorithm) has a hard time with, add extra residue to be saved to motion blocks, and as well as potentially throwing off motion prediction. All of that means extra bitrate and potentially a lot of extra ringing. The encoder has the option of mirroring the edges or doing whatever it wants, so it can pad to mod16 in the most efficient way that eliminates the sharp transitions.

Not all do, so the bottom row of many codecs looks a little worse than the rest if you don't use their preferred mod. Still beats having the top and bottom show that by adding borders to both sides.

[morph166955]

also just to throw this in, my ultimate goal (which is mostly done once i get all this stuff figured out) is to create a bash script that will be able to take any dvd-iso image or avi file (encoded in another codec or something of that nature) and make a 264 rip out of it automatically. All i need to do to finish this up is finalize my x264 options and figure out a good way to decide upon what bitrate to use for each individual file. My hope was that if the pass 1 bitrate from the psnr info (shown above) is a good estimation of the bitrate that i should use i could just bump that up to the nearest 50/100 kbit/s (eg 530 -> 550 or 600) and just go with that. Either that or i was hoping that i could some how figure out a good rate using psnr for say a 5 minute section of the movie if i encoded it at different bitrates and compared their psnr info. Or something else completely if someone has a really good way to predict a good bitrate. Im not worried about the bitrate going high for highquality movies (i consider 1200 kbit/s a good upper limit for my needs but seeing how x264 works i doubt i would get much over 1000) but i also dont wanna waste space going with like 1000 kbit/s for something that 500-600 will do just as good with. Im also not worried about the size of the file exceding the 700/800 meg limit or anything like that (eg. i dont mind if a long movie hits 1-1.5gig as long as the space isnt wasted by having too high of a bitrate).

Oh and should this script work out really well i have no problems posting it up here for everyone who wants to make their life easier with linux/unix/bsd x264 encoding.

Im a person who likes to learn and im very open to criticism about my methods or approach or anything else so if im doing this 100% wrong or im not seeing something that i should or anything else , please, let me know.

[morph166955]

thank you foxy, that makes alot of sense. im removing the expand filter from my stuff now. that effectively means the video that im sending to be encoded is 720x396 rather then the 400 originally.

Now knowing this i throw out another question. on some movies mplayer would like to expland the video to 854x480 to get the 16:9 ratio. 854 is obviously not divisible by 16. My solution was to simply remove (actually drop, not scale down) 3 pixels from each side to make it 848 pixels wide using the crop filter AFTER the scale filter (so it goes crop, scale, crop). i consider removing a few pixels thus maintaining the original image better then augmenting the entire image seeing as its less then a 1% loss of image...my math says its a ~.9% loss.

is there anything i should be adding to my x264 options to help with this?

[smok3]

if you dont care about the final size, then there is no need for 2pass, x264 inlcudes quality switch as well as constant quant switch, somebody else will have to tell you when to use each thought....

[morph166955]

well thats the thing, i do care, but i dont care at the same time. I care about file size becuase i dont want my movies/shows to be too large to the point where i have issues streaming them to my media center or something else, but i also dont want them to be to small and look horrible, i want the happy medium if thats possible.

I always understood that 2 pass was generally better because the second pass had info on the entire movie to go by thus it was able to better predict and compress the movie. I dont want a movie or a tv show to get larger then it really needs to be and i figured calculating bitrate from a firstpass run would be easier then trying to calculate a quality switch before anything was done to it (unless theres a way to predict this by viewing the original and extracting data from it with mplayer which im able to do with my script already...i just need to know what data to pull and what to do with it)

[nm]

You should try the crf parameter like smok3 suggested. The resulting quality is similar to two-pass encoding. Try values 20 and 25, for example. The higher, the worse quality. With this method, you don't even need to adjust the value much for each new encoding project since x264 will keep about the same quality level. You also won't get huge files unless the source is very difficult.

About the first pass bitrate you were wondering about; as you didn't specify it (you should, by the way), x264 uses constant quantizer with the default value of 26. This results in the bitrate that you see in the logs. If you want to specify a bitrate and use two-pass encoding, you'll get better quality if you use the same bitrate for both passes.

[morph166955]

interesting, i was always told that mencoder ignores any bitrate on the first pass (i think it even says, "First pass, bitrate ignored" when you run it). Ill take a poke at the crf parameter (is that qp_constant on mencoder?). Thanks.

Also, as to my original post...how should i read the PSNR info (eg what does each mean, whats an optimal # for the important ones) and does anyone got an idea why threads=2 isnt doing anything?

[nm]

interesting, i was always told that mencoder ignores any bitrate on the first pass (i think it even says, "First pass, bitrate ignored" when you run it).
Only if you encode with XviD. Lavc and x264 both use the specified bitrate in the first pass.
Ill take a poke at the crf parameter (is that qp_constant on mencoder?). Thanks.
No, it's crf in Mencoder too.

Also, as to my original post...how should i read the PSNR info (eg what does each mean, whats an optimal # for the important ones)There is no "optimal" PSNR value. If you don't mind crappy quality, 40 dB or lower could be just fine. If you want perfect quality no matter the file size, aim for 50 dB. It is better to just look at the video and decide what bitrate (or quality level for crf) is best for your requirements. PSNR can be used for comparing different encoding settings or encoders, but you can't feed the desired PSNR level to x264 directly.

and does anyone got an idea why threads=2 isnt doing anything?How do you know that there aren't two threads running? Are you sure that top should report separate threads within a process in FreeBSD? There could be a problem if x264 was not compiled with pthreads, but I think it should warn about that when you specify threads=2.

EDIT: Ok, I read your description about the threading problem again. I agree that generally the load should get higher than 50% with HT and several threads, so there seems to be something wrong. Maybe the threading code doesn't get properly configured on FreeBSD or you are missing some libraries?

[morph166955]

wow, thanks nm that post was very informative. so other then the bitrate are all of the other options that im using good? also, since the manual for mencoder doesnt mention it, whats the default crf? it shows the default qp_constant to be 26 but doesnt list a default crf value. im also assuming it works like qp_constant where 1 is the best and 50 is the worst for it?

is there any command i can run with x264's cli to show which options its been compiled with or a library i can look for or something? i know other programs tell ya but im not sure if theres one with x264. im mostly sure that i compiled it with threads but im not 100% sure and before i go recompiling everything i would like to check.

As for the psnr stuff, 500 put it at ~44db, 1000 put it at ~46. if you had to guess, where would 50 be and how much better is it? Also, whats a normal average psnr for a high quality rip? last question, if i run mencoder with qp=0 to get a lossless rip of the movie, can i assume that the psnr value of that run is the best i can possibly get (do i need to specify the subq stuff with qp=0 or can i simply do "-x264encopts qp=0")?

[morph166955]

ok so i just learned that when running qp=0 psnr info is not recorded and outputed at the end, so im now running at crf=1 to see what the "best" i can get with that is in comparison to the default.

[foxyshadis]

qp=0 is lossless, so there's no reason to output PSNR. ^^ 100 dB is the max. However, numerous studies have shown that PSNR has a weak relation to actual perceptual video quality. SSIM is stronger (but isn't built into x264). Stills only consistently look "better" at a glance with at least 1 dB diff, and in motion it can be 2 dB or more. Below that it can look better, worse, or the same, but there's only weak correlation. So basically, maximizing PSNR isnt the same as maximizing quality.

[morph166955]

ok then, sounds good. right now based on some tests i ran overnight last night im going to use a crf of 21-24 probably. I also had a realization of how i can make this a hell of alot easier on myself with this script. the way the script works now is by doing something to the effect of "./264encoder.script myiso.iso". Well now its going to be "./264encoder.script myiso.iso CRF/Bitrate/QP" where i assume anything 50 and below is a crf number, any negative number is a QP (absolute value of course) and anything above 50 is a bitrate (although i really cant see any use for a bitrate below 3-400). At that point the script can do the rest based on the requested bitrate/crf/qp to give as good of a rip as possible.

On that note (since im going to be working on this script over the next week or so when i have some free time) does anyone know the commands linux uses to mount iso's as drives? freebsd uses mdconfig to create a read-only vnode that points to the iso file (eg it makes /dev/md5 where 5 is a number that you specify). once you have that you simply just mount it to some point that you want. Im going to make the script with variables at the top that you can change to meet the specs you want (eg if you want to change your x264 options, program locations, stuff like that. My ultimate goal is to give *nix people a fast, foolproof way to do 264 encoding (as well as myself since frankly i really hate sitting there and tinkering when its mostly a pre-determined alg. that the computer can do for me).

Thanks for all the info people. It was very informative and very helpful.

[nm]

On that note (since im going to be working on this script over the next week or so when i have some free time) does anyone know the commands linux uses to mount iso's as drives? freebsd uses mdconfig to create a read-only vnode that points to the iso file (eg it makes /dev/md5 where 5 is a number that you specify). once you have that you simply just mount it to some point that you want.
Why do you need to mount the ISO image? Do you need to use other tools than MPlayer/Mencoder to read the contents, since they do support reading the image directly. And you know, if you need disk info, MPlayer can output a lot with -v.

[morph166955]

well, i honestly didnt know you could read an iso image with mplayer...how exactly do you do that? everything i always read said you had to either mount a real dvd thats in a drive to a mountpoint or create a virtual mount and point to that.

[morph166955]

oh and one last thing, just so im clear...with crf...do i or do i not need to do 2-pass (if i dont need to, should i anyway?)

[nm]

well, i honestly didnt know you could read an iso image with mplayer...how exactly do you do that? everything i always read said you had to either mount a real dvd thats in a drive to a mountpoint or create a virtual mount and point to that.
mplayer -dvd-device image.iso dvd://

[morph166955]

cool...i was unaware that dvd-device had that capability, i always understood it to be the destination of a real mountpoint that the dvd was mounted to if it wasnt the default one. AWESOME! thank you all very much, i wouldnt doubt that in the next two weeks you see my script. Im going to have a week off next week that i plan on doing alot of work on this durring. THANKS ALL!