Psy RDO: Official testing thread (version 0.6 out!)

[LoRd_MuldeR]

--keyint 240 --min-keyint 24 Why????????

Well, I think the "rule of thumb" says: min-keyint = fps, max-keyint = 10 * fps.

But I also think you can safely raise max-keyint much higher and trust in the scene-cut detection...

[survivant001]

Question.

when I mux into m2ts and I press ff and the PS3.. it takes few sec (could take 1-2 min) before the video start fast forwarding. is it because we use the default min-keyint value ?

ps. I don't have this problem when I mux into mp4 (I can take the same output for the both tests)

i'm using the default setting for ripbot.

[LoRd_MuldeR]

Question.

when I mux into m2ts and I press ff and the PS3.. it takes few sec (could take 1-2 min) before the video start fast forwarding. is it because we use the default min-keyint value ?

ps. I don't have this problem when I mux into mp4 (I can take the same output for the both tests)

i'm using the default setting for ripbot.

This question is off-topic :rolleyes:

But I think although a large key-frame interval might slowdown seeking, the problem is the m2ts (transport stream) container here.
As the name implies, a "transport" stream is intended for broadcast/streaming, not for local playback. Therefore seeking is not easily possible in those streams.
Maybe the player has to start at the very beginning and hunt trough the stream frame-by-frame until the desired position is reached...

[MasterNobody]

Dark Shikari
I think there is bug in ssd_plane function of Psy RDO patch. First, dc_coefs for sa8d must be equal not sad/2 but sad/4 (otherwise (sa8d - dc_coefs) would be most of the time negative). Second, abs function must be used in ADD_ABS_SATD macros and not only in result calculation of ssd_plane function because signs of satd and sa8d differences may be opposite.
Here is fixed (in my opinion) patch: http://stashbox.org/144987/x264_psy_rdo.r889.diff

[bkman]

Why????????


whyyyyy?????

1.) As Mulder says, it's a common rule of thumb. I've had no reason to deviate from it as yet and its good for consistent seeking through the movie.

2.) It looks good. Higher deblocking strengths kill detail too much.

[LoRd_MuldeR]

1.) As Mulder says, it's a common rule of thumb. I've had no reason to deviate from it as yet and its good for consistent seeking through the movie.

2.) It looks good. Higher deblocking strengths kill detail too much.

to 1) As said before, raise max-keyint in order to give x264 more freedom for deciding the optimal I-Frame position.

to 2) Using such extreme deblocking filter settings provokes artifacts! VAQ + Psy-RDO will already keep more details, so you shouldn't need to touch the deblocking settings.

[Dark Shikari]

Dark Shikari
I think there is bug in ssd_plane function of Psy RDO patch. First, dc_coefs for sa8d must be equal not sad/2 but sad/4 (otherwise (sa8d - dc_coefs) would be most of the time negative). Second, abs function must be used in ADD_ABS_SATD macros and not only in result calculation of ssd_plane function because signs of satd and sa8d differences may be opposite.
Here is fixed (in my opinion) patch: http://stashbox.org/144987/x264_psy_rdo.r889.diffHmm, good point. I'll look over this in a bit.

I'm not sure about the SATD/SA8D absolute value issue; its possible that if one is higher and the other is lower we want psy RDO to have them cancel. I think this may have actually been my original intent though I'm not 100% sure.

[shae]

An ideal AVC encoder would vary the quant matrix per frame to completely maximize RD. ;)Is this, unlike in ASP, compliant to the standard?

Perhaps something less radical would be possible, like using 2-3 QM sets for closeup/(mid/)longshot.

[Mr VacBob]

I don't know if optimizing the QM is really useful when you have AQ+trellis. Especially if you add QNS on top of that, there's already more than enough ways to mess with coefficients.

[akupenguin]

Is this, unlike in ASP, compliant to the standard?
According to the H.264 standard alone: Yes.
According to the H.264 standard plus restrictions on the headers required by certain containers such as mkv and mp4: Yes with limitations (no more than 256 different sets of QMs over the course of the movie).
It takes a certain number of bits to write down a new QM, so you'd probably want to limit yourself to a finite number even if it weren't for the container limits.
... which is still far more than ASP. Even non-compliant XviD only switched between 2 QMs.

[Blue_MiSfit]

Problem here...

I encoded L4yer Cake last night - it's a ~18GB (video only) MPEG-2 BluRay Source. I encoded to CRF20 using Psy-RDO:


--[NoImage] Job commandline: "C:\Program Files (x86)\megui\tools\x264\x264.877.modified.exe" --crf 19 --level 4.1 --ref 4 --mixed-refs
--bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -1,-1 --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8
--8x8dct --vbv-maxrate 1500 --me umh --threads auto --thread-input --sar 1:1 --progress --no-psnr --no-ssim
--output "I:\BluRay Rips\L4yer Cake\l4yer.mp4" "I:\BluRay Rips\L4yer Cake\l4yer.avs"
--[Information] [6/26/2008 2:41:16 AM] Encoding started
--[NoImage] Standard output stream
--[NoImage] Standard error stream
---[NoImage] avis [info]: 1920x1080 @ 23.98 fps (151604 frames)
---[NoImage] x264 [info]: using SAR=1/1
---[NoImage] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
---[NoImage] x264 [warning]: VBV maxrate specified, but no bufsize.
---[NoImage] mp4 [info]: initial delay 2002 (scale 24000)
---[NoImage] x264 [info]: slice I:1380 Avg QP:14.83 size:156543
---[NoImage] x264 [info]: slice P:61988 Avg QP:18.49 size: 83390
---[NoImage] x264 [info]: slice B:88236 Avg QP:20.50 size: 38383
---[NoImage] x264 [info]: mb I I16..4: 25.0% 66.8% 8.2%
---[NoImage] x264 [info]: mb P I16..4: 0.7% 4.3% 0.4% P16..4: 31.4% 21.0% 14.1% 0.0% 0.0% skip:28.0%
---[NoImage] x264 [info]: mb B I16..4: 0.1% 0.2% 0.0% B16..8: 48.1% 1.0% 4.4% direct: 8.2% skip:38.0%
---[NoImage] x264 [info]: 8x8 transform intra:75.5% inter:64.4%
---[NoImage] x264 [info]: direct mvs spatial:98.9% temporal:1.1%
---[NoImage] x264 [info]: ref P 54.7% 21.8% 14.5% 9.0%
---[NoImage] x264 [info]: ref B 66.1% 25.4% 8.5%
---[NoImage] x264 [info]: kb/s:11098.2
---[NoImage] encoded 151604 frames, 4.32 fps, 11098.94 kb/s
--[Information] Final statistics
---[NoImage] Desired video bitrate: 19 kbit/s
---[NoImage] Obtained video bitrate (approximate: 11101 kbit/s
--[Information] [6/26/2008 12:26:30 PM] Job completed
--[Information] [6/26/2008 12:26:30 PM] Postprocessing


And this is what happened to the intro (the rest of the movie looks _fantastic_)
http://www.mediafire.com/?n2mdbjxjmfn
(2MB)

Fail :( The fade in breaks up horribly and looks quite wrong.

Ideas? The VBV thing was ignored I think - I don't know how that snuck its way into my profile... ;)

Just in case - I will go ahead and re-encode WITHOUT any VBV.


I think it was totally the VBV... According to DGAVCIndex, the average bitrate for the fade in is ~ 500kbit. That's not good! Perhaps VBV dutifully tried, realized its life's work was meaningless, and gave up! :P

~MiSfit

[DarkZell666]

---[NoImage] Desired video bitrate: 19 kbit/s
... what the ... !? :confused:

[Avenger007]

---[NoImage] Desired video bitrate: 19 kbit/s

... what the ... !? :confused:
It's the CRF value, quite normal. :)

[DarkZell666]

It's the CRF value, quite normal. :)

It does say this doesn't it ? :
Desired video bitrate: 19 kbit/s

Never mind the "19" ... XD
It's not an x264-related quirk though, I admit :)

The thing is, I first though it was x264 misinterpreting the crf as bitrate (which would have surely been found out by now anyway), so I supposed that could have been the cause of Blue_Misfit's problem, but reading the log a second time allowed me to understand it was MeGUI playing tricks on me again :P

[Ranguvar]

There's a setting that can affect the quality of the beginning of the video in x264 :)

VBV Initial Buffer (--vbv-init), it defaults to 0.9. Higher values cause more bitrate in the beginning of the video, lower causes lower bitrate in the beginning.

[PlazzTT]

Problem here...

I encoded L4yer Cake last night - it's a ~18GB (video only) MPEG-2 BluRay Source. I encoded to CRF20 using Psy-RDO:


--[NoImage] Job commandline: "C:\Program Files (x86)\megui\tools\x264\x264.877.modified.exe" --crf 19 --level 4.1 --ref 4 --mixed-refs
--bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -1,-1 --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8
--8x8dct --vbv-maxrate 1500 --me umh --threads auto --thread-input --sar 1:1 --progress --no-psnr --no-ssim
--output "I:\BluRay Rips\L4yer Cake\l4yer.mp4" "I:\BluRay Rips\L4yer Cake\l4yer.avs"
--[Information] [6/26/2008 2:41:16 AM] Encoding started
--[NoImage] Standard output stream
--[NoImage] Standard error stream
---[NoImage] avis [info]: 1920x1080 @ 23.98 fps (151604 frames)
---[NoImage] x264 [info]: using SAR=1/1
---[NoImage] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
---[NoImage] x264 [warning]: VBV maxrate specified, but no bufsize.
---[NoImage] mp4 [info]: initial delay 2002 (scale 24000)
---[NoImage] x264 [info]: slice I:1380 Avg QP:14.83 size:156543
---[NoImage] x264 [info]: slice P:61988 Avg QP:18.49 size: 83390
---[NoImage] x264 [info]: slice B:88236 Avg QP:20.50 size: 38383
---[NoImage] x264 [info]: mb I I16..4: 25.0% 66.8% 8.2%
---[NoImage] x264 [info]: mb P I16..4: 0.7% 4.3% 0.4% P16..4: 31.4% 21.0% 14.1% 0.0% 0.0% skip:28.0%
---[NoImage] x264 [info]: mb B I16..4: 0.1% 0.2% 0.0% B16..8: 48.1% 1.0% 4.4% direct: 8.2% skip:38.0%
---[NoImage] x264 [info]: 8x8 transform intra:75.5% inter:64.4%
---[NoImage] x264 [info]: direct mvs spatial:98.9% temporal:1.1%
---[NoImage] x264 [info]: ref P 54.7% 21.8% 14.5% 9.0%
---[NoImage] x264 [info]: ref B 66.1% 25.4% 8.5%
---[NoImage] x264 [info]: kb/s:11098.2
---[NoImage] encoded 151604 frames, 4.32 fps, 11098.94 kb/s
--[Information] Final statistics
---[NoImage] Desired video bitrate: 19 kbit/s
---[NoImage] Obtained video bitrate (approximate: 11101 kbit/s
--[Information] [6/26/2008 12:26:30 PM] Job completed
--[Information] [6/26/2008 12:26:30 PM] Postprocessing


And this is what happened to the intro (the rest of the movie looks _fantastic_)
http://www.mediafire.com/?n2mdbjxjmfn
(2MB)

Fail :( The fade in breaks up horribly and looks quite wrong.

Ideas? The VBV thing was ignored I think - I don't know how that snuck its way into my profile... ;)

Just in case - I will go ahead and re-encode WITHOUT any VBV.


I think it was totally the VBV... According to DGAVCIndex, the average bitrate for the fade in is ~ 500kbit. That's not good! Perhaps VBV dutifully tried, realized its life's work was meaningless, and gave up! :P

~MiSfit

Same thing happened me recently encoding Six Feet Under. I was using VBV too. The HBO intro at the start (with all the noise) didn't look great, but it's very hard to encode I'd imagine anyway. The maybe 5 or 10 seconds after that was encoded with a very low bitrate (again, around 450kbps I think), and looked quite bad compared with the source. I'll try to cut up a sample.

[Blue_MiSfit]

Thing is - I didnt even want VBV - since I'm doing a CRF encode :)

I think somehow my profiles got goofed up (copied a profile with VBV and modified it for CRF without looking at the VBV settings). At any rate, I'm re-encoding the file as we speak, and im 99% sure the problem will go away then ;)

~MiSfit

[qyqgpower]

I have also noticed the I frame flashing issue with Psy RDO turned on, especially at low bitrate. Because frames before the I frame was so badly encoded.
It seems that P/B frames keep losing details when there's no good I frame to reference, e.g. a fade-in scene.
In my opinion. "referencing" to some already badly encoded (washed out "details") P/B frames is meaningless or even would lead to worse result.

http://www.mediafire.com/?z3j5dmuj2nm

[elguaxo]

could you encode and share the same sample at the same bitrate without PSY RDO?

[menlvd]

could you encode and share the same sample at the same bitrate without PSY RDO?
and putting cmd line for both encoded files

[bkman]

I have also noticed the I frame flashing issue with Psy RDO turned on, especially at low bitrate. Because frames before the I frame was so badly encoded.
It seems that P/B frames keep losing details when there's no good I frame to reference, e.g. a fade-in scene.
In my opinion. "referencing" to some already badly encoded (washed out "details") P/B frames is meaningless or even would lead to worse result.

http://www.mediafire.com/?z3j5dmuj2nm

That's it exactly. The problem is that frames prior to the I-frame degrade much more than they should, lowering quality and making the I-frame very noticeable.

[burfadel]

Maybe optimising the way changes by AQ and PSY go on top of other changes will resolve that issue... So when you have one P?B frame then the next, does the second p/B-frame build AQ and psy on top of the first? What you could do is keep track of a certain number of P/B frames, then for the following p-frame make changes based on the original first and not the aq/psy modded frame.

[Blue_MiSfit]

I'll do that this evening - sorry for the delay. I re-encoded without VBV, and the problem persisted (I think) :(!

I will do some comparisons with and without PRDO and see if the issue goes away.. Details forthcoming...

~MiSfit

[Blue_MiSfit]

Uh oh...

http://www.mediafire.com/?zjzzeytcwto

PRDO enabled - artifacts.
PRDO disabled - no artifacts.

Command line with PRDO:

Job commandline: "C:\Program Files (x86)\megui\tools\x264\x264.877.modified.exe" --crf 19 --level 4.1 --ref 4 --mixed-refs --bframes 16 --b-pyramid
--b-rdo --bime --weightb --direct auto --filter -1,-1 --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto
--thread-input --sar 1:1 --progress --no-psnr --no-ssim --output "I:\BluRay Rips\L4yer Cake\PRDO Tests\PRDO Enabled (Sub-ME 6).mp4" "I:\BluRay Rips\L4yer Cake\l4yer.avs"


Command line WITHOUT PRDO

"C:\Program Files (x86)\megui\tools\x264\x264.877.modified.exe" --crf 19 --level 4.1 --ref 4 --mixed-refs --bframes 16 --b-pyramid
--bime --weightb --direct auto --filter -1,-1 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto
--thread-input --sar 1:1 --progress --no-psnr --no-ssim --output "I:\BluRay Rips\L4yer Cake\VAQ Tests\No VAQ.mp4" "I:\BluRay Rips\L4yer Cake\l4yer.avs"


:confused:

~MiSfit

[Dark Shikari]

Looks like you're using a build from before the bugfix (http://git.videolan.org/?p=x264.git;a=commit;h=2389de25ff7fe6f84c9c885578c0fbaa6b656f4a) that dealt with that. Its not a bug in Psy RDO--rather its a bug that Psy RDO tends to reveal since its extremely unlikely otherwise.

[J_Darnley]

Well it might not appear in the good clip because RDO has been disabled completely in it (--subme 6 and --b-rdo have been removed) . If you want to disable psyRDO then use --rdcmp ssd.

I need to find a good clip to test psyRDO, FGO and VAQ2 on. Problem is I can't think of a scene that might show a clear benefit.

[ToS_Maverick]

check this (from dark shikari's blog):
http://x264dev.multimedia.cx/?p=14

[Blue_MiSfit]

Interesting. Is there a more recent patched build with PsyRDO that I can use?

edit:

NM - I saw that the build posted on the OP has been updated :) I will double check tonight that this resolves the issue ;)

~MiSfit

[mahsah]

Anyone know what settings/bitrate would be good for retaining dither in anime? I'm currently using the AE-maxquality profile in megui, but it doesn't keep as much dither as is needed to hide all of the banding/dct blocks...

[wyti]

it's impossible to say what bitrate because like in real time content, the bitrate needed varie between animes...
try crf mode if you don't care about the final size of your encode (try crf 20 and use lower or higher values to get what you want).
for the settings you can do better but you will need to increase merange or use esa (or tesa) wich ar really slow, so try to increase bitrate.

[mahsah]

Well even with CRF 18 it is throwing out way too much dither...

[wyti]

have you try to lower deblocking value ? AE-maxquality use 1,1 (stronger than normal) because there is no very fine détails on anime, try to lower the deblocking value to -2,-1 (maybe you will have blocking, but you may retain more dither)

[mahsah]

With lower deblocking settings (or even higher ones), even at CRF of 1 it still tosses away the dither...

[Dark Shikari]

With lower deblocking settings (or even higher ones), even at CRF of 1 it still tosses away the dither...Are you sure then that the dither even exists upon x264 receiving the input!?

It would be difficult to avoid retaining dither when using psy RDO; I have never seen banding at any sane bitrate with it.

[mahsah]

It does exist. I can post a short lossless huffyuv clip (or whatever Megui makes during its rendering pass) and upload it, if you want.

EDIT:
http://miscstorage.nfshost.com/test.avi

There it is (please don't download it frivolously). Very short clip, but keep an eye on his red cape; it has some bad banding/blocking that gradfun2db covers up fairly well, but x264 does not.

[wyti]

I don't know where do you see a problem.
With this commande line :
program --crf 18.0 --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input"
(it's AE maxquality with crf 18 and deblocking at -2,-1)
have you updated your megui x264 version ? (At this time, it's the jarod patched version 895)

[mahsah]

I was actually using the one in the first post of this thread -- Jarod's patched version 895 doesn't have Psy RDO in it, does it?

EDIT: I can clearly see a problem, but then again I have an LCD. Still, the problem is nearly gone in the original. First is the same clip I uploaded, 2nd is the most recent build on MeGUI's auto updater, 3rd is the one from the start of this thread. All using the above command line.

http://miscstorage.nfshost.com/huff.png
http://miscstorage.nfshost.com/noRDO.png
http://miscstorage.nfshost.com/RDO.png

[ToS_Maverick]

I'm sorry, I cannot see a problem.

I'm sensitive to blocking in x264 but this seems to be a VERY minor problem, but that's just me.

[Dark Shikari]

I don't see any issues there at all... the encode looks basically transparent to the original.

Well, actually, I do see an issue... that you're watching Gurren Lagann!

/hides

[mahsah]

Huh. Guess it must just be my LCD then. Sorry to bug you about it.

And yes, it is Gurren-Lagann :)

[CruNcher]

WOW there im away sometime and what happens Dark brings the next HVS improvement and this time it really blasts (in Motion even more) :)

http://s1.directupload.net/file/d/1483/od2hf88f_png.htm (no RD = 40 fps)
http://s1.directupload.net/file/d/1483/xzu76qjy_png.htm (old RD = 30 fps)
http://s1.directupload.net/file/d/1483/r8vhwmp3_png.htm (Darks Psy RD = 29 fps, it seems deblocking needs to be adapted, AQ useless now non usage could speedup more without Visual difference?)

Since i was away X264 performance improved +6 fps now practicaly Psy RD comes for free (almost) :D (you really gave RD for Film Source a reason to live thx Dark another time superb work)

PS: It's a Live Retranscode of Ben Waggoners heavy compressed (2mbit) VC-1 Encode :) (x264 puts it to shame)

[lexor]

Huh. Guess it must just be my LCD then. Sorry to bug you about it.

And yes, it is Gurren-Lagann :)

Actually I think I see what you mean if I pump the brightness a bit. But I think it shouldn't be visible in normal viewing, so if you do see it I'm inclined to think that the calorimetry of your LCD is wacked, or you have one the cheaper panels, like me, and if you not looking at it from the "correct" angle everything is uber bright (or dark, depending on the angle).

[burfadel]

Lower the brightness and gamma and up the contrast through the control panel for your graphics driver, you generally don't have to do it much to make them look a lot better!

[lexor]

Lower the brightness and gamma and up the contrast through the control panel for your graphics driver, you generally don't have to do it much to make them look a lot better!

Doing it in drivers is not a good idea, using levels (either in ffdshow, mpc shader or avisynth) to do the 0-255 -> 16-235 conversion is the cleaner way.

[burfadel]

Thats true! I did think of that, but I thought since the problem is with the monitor and not the player that having a blanket correction may be better for picture quality in general, not just the video :) since the adjustments through the driver have different results than the adjustments on the monitor itself.

[Inventive Software]

So hows the VAQ2+PsyRDO combo going Dark? Any thoughts on how this particular patch (PsyRDO) could be improved?

[Blue_MiSfit]

Sorry to keep anyone waiting...

I re-encoded the beginning of L4yer Cake with bob0r's patched 901 build (including PsyRDO), and the issue with the fade-in to the sony logo has been resolved.

:)

~MiSfit

[Dark Shikari]

Updated to version 0.4, considerable changes. See post for details. Psy RD strength is now adjustable!

[elguaxo]

4. Automatically lower AQ strength slightly when psy RDO is enabled.
by how much is it lowered? Does this still apply if AQ strength is in the commandline, let's say --aq-strength 1.2.

Thanks!

[Dark Shikari]

by how much is it lowered? Does this still apply if AQ strength is in the commandline, let's say --aq-strength 1.2.Yes, its still lowered if you specify it explicitly, much like AQ raises qcomp even if you specify qcomp explicitly.

Its lowered by log(psy RDO strength+1)/3.0. So for the default strength, it lowers AQ by 0.25 or so. This formula is completely arbitrary and just meant to scale the amount by which AQ is lowered slightly based on psy RDO strength.