"Stealth" encode project, questions regarding advanced features

[S¡nTë£]

Hi,

I'm encoding a PAL DVD with very high motion scenes and large blue & dark surfaces. I was aiming to encode at 960x432 (spline36resize of 716x424 cropped image) to 1/3rd DVD size and Vorbis encoded audio, which grants me a bitrate of about 1700kbps. 1/4th DVD size gives me 1300kbps which I think won't be sufficient to preserve all the nice details of the source. Am not doing any other preprocessing besides cropping and resizing, since I think any softening being done by FFT3DGPU will also be done by x264 if it determines the bitrate is too low to preserve everything. I don't really care about grain retention.

Now as a test I've been encoding the first 4 minutes at this bitrate which has all the possible troublemakers. The test will come out worse as the final encode of course, since there aren't any low motion scenes in it to steal bitrate from for these high motion scenes. For the tests I've been using the HQ-Insane profile with 3 changes: corrected the bitrate, reduced the amount of reference frames from 16 to 5, increased the amount of b-frames from 3 to 16 (tho maybe I'm trusting the adaptive algorithm a bit too much then?). I left deblocking on -2,-1 but seems this causes some noticeable blocks in non-dark areas.

*Any comments on my original choices? Specifically the "no denoisers", resizing and x264 options. I chose to do the aspect ratio correcting and modifying to MOD16 resolution with an advanced "ringing-less" resizer since they're supposed to be better than any overlay resizing (just using resizers is the correct way to expand the 'non square pixels' i hope...). The downside to this is of course that there are more pixels to encode, but I was hoping my bitrate would be sufficient. I could also encode anamorphic if that will help things along noticeably. Downsizing is not an option for me.

*Is there a specific quantization matrix I could try out for my targeted bitrate? I've been using the default up till now but like to now what the fuss is all about.

*Is Adaptive Quantization the only thing different between the stickied builds and the newest x264 SVN builds? Because looking at the changelog, there have been some nice speed improvements since r635, and documentation on what has changed on the patched builds is sparse. Also, how effective is AQ? I read that it helps some but never really solves the blocking in dark areas.

dark area blocks (http://users.skynet.be/TailwinD/encodedsize.png) - problem area expanded 200% with lanczos resize (http://users.skynet.be/TailwinD/200.png) // solving these would be nice since there's also an extensive night scene.

*Trellis or DCT decimation? Trellis has a large performance hit, and looking at sheep's guide, the result is questionable. Most information I find by searching comes out of the XviD section of the forum, where they recommend to leave it always on. Do those comments also apply to x264? Also take into account the bitrate, what I read is that Trellis mostly helps at lower bitrates, true?

Most if not all of these can be found out by doing tests. However, I do not have the processing power to test every possible parameter combination..

EDIT: pressed submit new thread to soon, didn't discuss my test results yet, i'll add a new comment or edit this one once i have that finished

[S¡nTë£]

http://users.skynet.be/TailwinD/encode2.mkv is the largest part of the sample I encoded. It's pretty good already I think, I'm just mostly wondering if the dark parts and the blue dolls at the end can be better.

Also, I'm averaging 5fps for each of both passes, which will put the end encoding at about 20 hours... A bit much for one movie (have 4 others in the pipeline), even though I have an AMD 2.4ghz dual core. So increasing the pace to something along 8fps would be extremely nice, which is why I'm asking if Trellis etc is all that. Does encoding rate differ significantly between high and low motion scenes?

I'm still kinda new to x264 encoding..

[jeffy]

It would help for the testers if you cut out a small sample of the problematic video, unprocessed, eg. VOB.

DgIndex, click [ and ], File -> Save Project and Demux Video, you will have a video only m2v file.

BTW, do you mean this movie?
http://www.imdb.com/title/tt0382992/

[HisDivineShadow]

Yes that is the movie. Futuristic Top Gun, without Tom Cruise ;)

I also have the PAL DVD, I have upped a m2v of the section he is having issues with. Must admit I will be doing a encode of this at some stage, so I'm interested in suggestions too. Though I dont care about speed, and I use filters/matricies.

m2v Sample (~30mb) - http://www.mediafire.com/?8zczmekmnzm

[S¡nTë£]

The reason I'm not denoising anymore is that from what I read, the noise/grain is actually a help to prevent the blocking in dark areas. And besides grain, the source is really impeccable on this one. Could be that you're asking becoz you don't think it's a good movie, but I'm using it as a stepping stone to learn for other movies, since it has much of the trouble areas in x264 encoding :p

MPEG2Source("VTS_01_1.d2v",info=3)
ColorMatrix(hints=true,threads=2)
Trim(0,5500)
Crop(0,76,-4,-76)
Spline36Resize(960,432)

Is my testscript up till now.

Back 2 business: did a few tests, but couldn't actually see the results because megui ignored my different filenames for the output files when I queued up the jobs, so it overwrote everything. That's a really annoying bug. Did enable the PSNR and SSIM calculation tho, and what I got is this:

Default quant mtx - deblocking 0,0 - trellis
x264 [info]: SSIM Mean Y:0.9826298
x264 [info]: PSNR Mean Y:46.257 U:48.347 V:49.179 Avg:46.890 Global:45.833 kb/s:1682.53

Default quant mtx - deblocking 0,0 - dct decimation
x264 [info]: SSIM Mean Y:0.9822149
x264 [info]: PSNR Mean Y:46.064 U:48.216 V:49.071 Avg:46.711 Global:45.646 kb/s:1682.11

Sharktooths quant mtx - deblocking -2,-1 - trellis
x264 [info]: SSIM Mean Y:0.9795572
x264 [info]: PSNR Mean Y:45.391 U:48.748 V:49.624 Avg:46.273 Global:45.162 kb/s:1681.97

Sharktooths quant mtx - deblocking -2,-1 - dct decimation
x264 [info]: SSIM Mean Y:0.9794971
x264 [info]: PSNR Mean Y:45.332 U:48.126 V:48.994 Avg:46.101 Global:44.963 kb/s:1681.98

Which means, if I'm interpreting everything correctly, that according to SSIM and PSNR I get the best results with the original settings, and sharktooths mtx provides slightly worse results in this particular case.

Haven't tried *.mp4guy's mtces yet.

[bkman]

dark area blocks (http://users.skynet.be/TailwinD/encodedsize.png) - problem area expanded 200% with lanczos resize (http://users.skynet.be/TailwinD/200.png) // solving these would be nice since there's also an extensive night scene.

Mate, the only way you'd see any blocking in that shot is with an LCD monitor. I can very barely make any out on my CRT.

[S¡nTë£]

Which is incidentally the only kind of screens we have at home :p
I'm watching on a 20" 1680 lcd, while the htpc is connected to a 40" 720p lcd.

I'm not really disappointed with the quality, I'm just asking some general questions if it is possible to improve it even more. Remember, wanna apply everything I learn on this project on my other DVDs

[HisDivineShadow]

Try M4G's High Detail V3.1.

I'd suggest anamorphic over upscaling too. Just asking for trouble there.

[S¡nTë£]

Thx, am now in the middle of the HRM mtx and will try the HQ one as well. BTW, I'm not really upscaling, only doing mod16 and aspect ratio correction. You think 1700kbps is not enough for this?

Also, is my reasoning correct regarding the results of my previous tests? That the first one should have the least difference with the original and the last one the most, based on the psnr/ssim results?

[HisDivineShadow]

I go off what I can see. I dont really know what those numbers mean to tell the truth. My eyes tell me if its good enough or not.

Not upscaling? 716x424 -> 960x432 is as far as I know, since the source width would be 720.

Going by your settings, try this in your script to test an anamorphic encode (in case you arent sure).

Bold = New/changed

# Set DAR in encoder to 47 : 20. The following line is for automatic signalling
global MeGUI_darx = 47
global MeGUI_dary = 20
MPEG2Source("VTS_01_1.d2v",info=3)
ColorMatrix(hints=true,threads=2)
Trim(0,5500)
crop( 0, 76, -4, -76)
Spline36Resize(720,432) # Spline36 (Neutral)

Also, you can try a simple removegrain(mode=1) to give it a little more room to compress without affecting the visual quality. Tip I was told a while ago.

Just to give you an idea, I'm using a bitrate of 1318, to fit a 1/3DVDR size with AC3 audio. My script is below.

# Set DAR in encoder to 47 : 20. The following line is for automatic signalling
global MeGUI_darx = 47
global MeGUI_dary = 20
mpeg2source("C:\DVD RIPS\Stealth\VTS_03_1.d2v")
colormatrix()
crop( 2, 76, -2, -76)
Lanczos4Resize(720,432)
removegrain(mode=1)
fft3dfilter(bt=5, sigma2=0.5, sigma3=1.5, sigma4=3, bw=32, bh=32, ow=16, oh=16, ncpu=2)
dull = last
sharp = dull.LimitedSharpenFaster( strength=100 )
Soothe( sharp, dull, 30 )

I'm just deciding on a matrix at present.

Then again, I dont mind chugging along at 2-3fps ;)

[S¡nTë£]

Up until now, every custom mtx I tried gave worse results than the best default settings, M4G High Detail came closest, but didn't beat it. Am now retrying HRM with the extra information mp4guy gave me.

It would be interesting to compare your output to mine, if you are willing to do so. The part I uploaded in particular, if you could do a test run of your target bitrate and also a trim(0,5500) with the settings you decide upon. The 30% difference in bitrate should have almost no impact since my test runs have 30% more pixels (horizontal resolution).

If yours is visually better, I may switch ^^

[DarkZell666]

Man, you forgot to read the chapter about "why PSNR and matrices don't mix" ;)

There are tons of threads out there explaining why, but here's the bottom line: all custom matrices almost always yield lesser PSNR than the standard flat one, but they almost always *look* better on the sources they are designed for. You find matrices for anime, others for grainy material, others for keeping all the details but that need a crazy bitrate, others for getting voluntarily rid of details to squeeze the most out of a small bitrate, etc.

So don't trust PSNR too much, and if your eyes can't really see a difference, maybe you should just not use any at all =)

Have some reading about HVS (Human Visual System), you'll learn a lot :)

[S¡nTë£]

ha, I've been waiting for this kind of reply for days, finally :)

I have been doing screenshot comparisons yeah. And now that I have the reference settings encoded again, I remember one result with different mtx being superior to it. Now have to find which one again, since most of the test files have been lost, bleh.

[HisDivineShadow]

From my testing, M4G's High Detail V3.1 seems to be the best at that particular scene, though still not perfect.

Just out of curiosity I did a Lossless encode of that scene and got a nice ~120mb file, and that blocky area around his face was still fairly noticeable, though the blocking was kind of replaced by banding. So I have a feeling its going to be a 'not-fixable' thing. All my other test scenes so far have been excellent, so its probably just a once-off for the movie.

I dont mind upping my samples, though I trimmed it a bit more just to cover those two areas with a buffer either side.

[S¡nTë£]

Well adaptive quantization at 0.5 all but solved the noticeable blocks in that face area.

The briefing still had some noticeable ones tho, M4G HRM (with all the deadzone, no b-rdo, -2;-4 deblocking settings) made it look the best, but seems inferior on the sharp parts to my eyes. Ringing is the biggest problem atm, but the source already has some ringing too, so dunno if that's solvable.

Have switched to anamorphic encoding as well, since comparing shots of my upscaling and the anamorphic did show better detail for the latter. What irritates me are the artifacts produced by the bilinear resizing for the overlay tho. I may be going overboard with this but can clearly see them :) That's not x264's fault tho

Am now down to selecting my best sample out of 4, kinda hard :) Got the speed up to around 7.5kbps, good enough for me.

[Mug Funky]

bear in mind that quite often blocking only becomes visible because of rounding errors in the YV12 to RGB conversion that happens on output - monitors are RGB after all, and LCD TVs necessitate a digital conversion that may be suboptimal. CRTs tend to do this conversion in analog, which of course is free from rounding error (it does have it's own problems of noise and sharpness, but they're far far less noticable).

so.. perhaps you could add a small amount of noise on playback to disguise the rounding errors and make the blocks disappear. you're using a HTPC, so i assume you have the option to put playback filters in. ffdshow will add noise nicely, and you don't need much at all. there's also the gradfun filter, but that can cause some highlights to be softened.

another advantage of this is you can filter out the ringing in the source.