Improve quality

[Viechal]

HI guys
As probably some of you know i am making gamemovies of the game enemy territory and in my opinion i have a very nice quality atm.
But I still want to know if there are any things in my encoding batch file that should be changed for a better result.
I am using x264 and encoding it with this batch and the following settings:
echo AVISource("%mypath%\%mymovie%.avi") > "%mypath%\%mymovie%.avs"
echo Crop(0,4,-0,-4) >> "%mypath%\%mymovie%.avs"
echo ConvertToYV12() >> "%mypath%\%mymovie%.avs"
echo Lanczos4Resize(1280,800) # Lanczo4 (Sharp) >> "%mypath%\%mymovie%.avs"

echo ======= x264 pass 1/3 ========
"bin\x264.exe" -p 1 --bitrate %myvbitrate% --ref 6 --trellis 2 --direct temporal --bframes 5 --filter -2,-2 --mixed-refs --no-fast-pskip --subme 7 --scenecut 40 --weightb --b-pyramid --b-rdo --bime --analyse all --vbv-maxrate 3500 --ratetol 5.0 --8x8dct --qpmin 4 --qpstep 6 --merange 32 --threads 2 --thread-input --cqm "jvt" --me esa --progress --no-psnr --stats "temp\x264.stats" --output NUL "%mymovie%.avs"
echo ==============================
echo.

echo ======= x264 pass 2/3 ========
"bin\x264.exe" -p 3 --bitrate %myvbitrate% --ref 6 --trellis 2 --direct temporal --bframes 5 --filter -2,-2 --mixed-refs --no-fast-pskip --subme 7 --scenecut 40 --weightb --b-pyramid --b-rdo --bime --analyse all --vbv-maxrate 3500 --ratetol 5.0 --8x8dct --qpmin 4 --qpstep 6 --merange 32 --threads 2 --thread-input --cqm "jvt" --me esa --progress --no-psnr --stats "temp\x264.stats" --output NUL "%mymovie%.avs"
echo ==============================
echo.

echo ======= x264 pass 3/3 ========
"bin\x264.exe" -p 2 --bitrate %myvbitrate% --ref 6 --trellis 2 --direct temporal --bframes 5 --filter -2,-2 --mixed-refs --no-fast-pskip --subme 7 --scenecut 40 --weightb --b-pyramid --b-rdo --bime --analyse all --vbv-maxrate 3500 --ratetol 5.0 --8x8dct --qpmin 4 --qpstep 6 --merange 32 --threads 2 --thread-input --cqm "jvt" --me esa --progress --no-psnr --stats "temp\x264.stats" --output "temp\%mymovie%.264" "%mymovie%.avs"
echo ==============================
echo.

So this is a short part of my batch.
Well i am capturing the scene with 1440*900 and resizing it to 1280*800. I use 30 fps and my bitrate is 3000..
well... anything more you need to know?
Would be nice if some of you knew sth to improve this.
Thx a lot guys

Mfg

[pinkie_1]

Umm.... are you being serious ?
--me esa, --subme 7 and --trellis 2 on the 1st pass (out of three) ? You gotta be kidding...

Please explain the reasoning beyond these :
--vbv-maxrate 3500 (for an avg bitrate of 3k)
--qpmin 4
--qpstep 6
--cqm "jvt"
Also you should avoid using --merange 32. Stick to 24.

Another thing to consider : Lanczos resizing is supposed to produce a 'better looking' output when upscaling. This isn't your case, so switch to bicubic.

BTW, you've posted in the wrong place... ;)

[Sharktooth]

half of your settigs are plain wrong. limiting qpmin to 4 and vbv-maxrate to 3500 (also it has no effect at all if other vbv options arent set), --qpstep 6 ?!? etc...
x264 defaults could give a better result...

[Viechal]

Well as you can see i am not that familiar wit all those settings.
So i took this preset batch file i found anywere.
I modified something because I downloaded a movie from Shaolinproductions. They put the codec settings into a textfile inside the movie.rar.
This movie had an impressive quality and they have long years of experience. So I trusted most of those settings.
I just modified a little bit.
Now i encoded and the result was not that bad as you say it should be.
But there is space to improve it and thats why i turn to you who know every little option and result.
So what exactly would you change to what values?

Thx for your help

/edit: Well now i decided to put the complete file into my first post...
I have the same settings all 3 passes. As you said this is a pritty poor batch...
Would be very nice if you could help me improving this one.
;)

[foxyshadis]

Their latest x264 article is very sensible, using these settings:
http://www.shaolinproductions.org/images/stories/fei/MeGUisetup.jpg
Aside from using raw instead of mp4/mkv, and using JVT matrix, which is a bizarre decision - but will work better with 0,0 deblock. The m4g matrices would be better.

If you're targeting low end machines with maxrate, consider using CAVLC instead. It plays much faster, and may not cause as much quality degradation in hard scenes. (But overall will be a bit worse.) If you're not, don't use maxrate at all.

[Viechal]

http://rapidshare.com/files/12223904/Viechal.mp4.html
i now uploade this small scene so that you can see it is not that bad =)
Dont worry about the color... i just played a little bit with some new plugin filters on Vegas =)

btw foxyshadis =) i know this tutorial on the site but i took the settings of one of their latest movies of Quake3 called Unity.
It is a package which contains of the movie and much extra stuff like for example the encoding settings.
And that was the basis i took for my batch file.

So you until now i set the merange back to 24, i deleted the maxrate and i set the deblock to 0,0.
Thx so far. =)


edit:
Probably i need to tell you why i am asking this.
As i am from Austria and only talk Schoolenglish some problems with some special words and functions of the codec appear and i was working on my codec settings kind of blind.
I changed something and then watched the result. Always the same.
I dont have a clue what many of the commands mean just because i am that bad in english.
I tried to study every guid and tutorial i could find but many things are just too hard for a noob like me =)
So i m looking for help here.
You are familiar with all the settings :>
So i m sorry for the mistakes =) i m just looking for a nice quality. =)
ty

[Viechal]

echo AVISource("%mypath%\%mymovie%.avi") > "%mypath%\%mymovie%.avs"
echo Crop(0,4,-0,-4) >> "%mypath%\%mymovie%.avs"
echo ConvertToYV12() >> "%mypath%\%mymovie%.avs"
echo BicubicResize(1280,800) # Bicubic (Sharp) >> "%mypath%\%mymovie%.avs"



echo ======Encoding AAC audio======
"bin\faac.exe" -b %myabitrate% -o "%mypath%\temp\%mymovie%.m4a" "%mypath%\%mymovie%.wav"
echo ==============================
echo.

echo ======= x264 pass 1/3 ========
"bin\x264.exe" -p 1 --bitrate %myvbitrate% --ref 16 --trellis 2 --direct auto --bframes 3 --filter -2,-1 --mixed-refs --no-fast-pskip --scenecut 40 --weightb --b-pyramid --b-rdo --bime --analyse all --ratetol 5.0 --8x8dct --merange 24 --threads 2 --thread-input --me umh --progress --no-psnr --stats "temp\x264.stats" --output NUL "%mymovie%.avs"

echo ==============================
echo.

echo ======= x264 pass 2/3 ========
"bin\x264.exe" -p 3 --bitrate %myvbitrate% --ref 16 --trellis 2 --direct auto --bframes 5 --filter -2,-1 --mixed-refs --no-fast-pskip --scenecut 40 --weightb --b-pyramid --b-rdo --bime --analyse all --ratetol 5.0 --8x8dct --merange 24 --threads 2 --thread-input --me umh --progress --no-psnr --stats "temp\x264.stats" --output NUL "%mymovie%.avs"
echo ==============================
echo.

echo ======= x264 pass 3/3 ========
"bin\x264.exe" -p 2 --bitrate %myvbitrate% --ref 16 --trellis 2 --direct auto --bframes 5 --filter -2,-1 --mixed-refs --no-fast-pskip --scenecut 40 --weightb --b-pyramid --b-rdo --bime --analyse all --ratetol 5.0 --8x8dct --merange 24 --threads 2 --thread-input --me umh --progress --no-psnr --stats "temp\x264.stats" --output "temp\%mymovie%.264" "%mymovie%.avs"
echo ==============================
echo.

Thats what it looks like now.
What do you think?
Anything "bizarr" left?
And are there some things that still can improve it?
Ty for the feedback so far

[pinkie_1]

Try this progressive approach :
echo ======= x264 pass 1/3 ========
"bin\x264.exe" -p 1 --bitrate %myvbitrate% --ref 2 --direct auto --bframes 5 --filter -2,-1 --mixed-refs --no-fast-pskip --subme 5 --weightb --b-pyramid --bime --analyse all --ratetol 5.0 --8x8dct --merange 24 --threads 2 --thread-input --me hex --progress --no-psnr --stats "temp\x264.stats" --output NUL "%mymovie%.avs"
echo ==============================
echo.

echo ======= x264 pass 2/3 ========
"bin\x264.exe" -p 3 --bitrate %myvbitrate% --ref 5 --trellis 1 --direct auto --bframes 5 --filter -2,-1 --mixed-refs --no-fast-pskip --subme 6 --weightb --b-pyramid --b-rdo --bime --analyse all --ratetol 5.0 --8x8dct --merange 24 --threads 2 --thread-input --me umh --progress --no-psnr --stats "temp\x264.stats" --output NUL "%mymovie%.avs"
echo ==============================
echo.

echo ======= x264 pass 3/3 ========
"bin\x264.exe" -p 2 --bitrate %myvbitrate% --ref 8 --trellis 2 --direct auto --bframes 5 --filter -2,-1 --mixed-refs --no-fast-pskip --subme 7 --weightb --b-pyramid --b-rdo --bime --analyse all --ratetol 5.0 --8x8dct --merange 24 --threads 2 --thread-input --me umh --progress --no-psnr --stats "temp\x264.stats" --output "temp\%mymovie%.264" "%mymovie%.avs"
Also have a look at *.mp4 guy' s CQMs (keep in mind that you should tweak --filter and --direct accordingly).

@ mods :
Please move this to AVC section.

[Viechal]

very nice thx a lot man!

i m rendering atm
what exactly are the effects of all those changes?
mfg

[pinkie_1]

what exactly are the effects of all those changes?
Speed optimizations. Your last attempt pretty much maxed out what x264 can give when it comes to quality... but some of the commands were simply overkill for the first two passes.

I've proposed a more reasonable approach, emphasizing what changes from one pass to another.

One last thing : give it a try with the default flat QM. If you'll run into problems with banding then consider using CQMs.

[Viechal]

Well the quality is very good now

Hmm how exactly can i use a custom matrix?
is there one matrix you would suggest me for the best result in my kind of video?