Psy RDO: Official testing thread (version 0.6 out!)

[Ranguvar]

Are you using Trellis 2 ?

Why? Does it help, or hurt PsyRDO?

[Razorholt]

Trellis 1 probably is not the best idea with this because that means the quantization that x264 chooses is not the quantization that psy RDO looked at in mode decision. Therefore, unless its demonstrated that this isn't a real issue, deadzones or trellis 2 are probably a better idea.
Trellis 2 is not a bad thing at all with psy RD; the problem before was that it encouraged blurring because RD would often prefer blurrier modes, and trellis would generate blurrier modes. But now, RD prefers *less* blurry modes.
Trellis 2 sharpen the picture. What settings you had? Trellis 1 or were you using deadzones?

[gav1577]

I find low deadzones usually inter 6 intra 6 work great for me :)

[Blue_MiSfit]

Yes I am using Trellis 2, and SubME 7, when I can afford the speed hit.

I did Princess Mononoke the other day, a nice high quality soft-telecined DVD9 source.

I did a very light script with light fft3dgpu(sigma=1), Toon() and gradfun2db(1.8). Then I encoded into Psy-RDO'd CRF19 with Multi-Hex SubME 7, Trellis 2, 16 refs, all the usual bells and whistles. It came out to 1.2 GB. That's really impressive to me :)

Muxing the original english and japanese 5.1ch AC3s, plus an english SRT netted a file size of just under 2gb. 9gb -> 2gb with utterly transparent quality (I actually preferred my version due to the denoise / warpsharp / line darkening / dithering). Now THAT's impressive.

Heck, the encoding only took a few hours. I think I was getting ~15fps on my Q6600 @ 3 GHz :D

~MiSfit

[TheRyuu]

Why? Does it help, or hurt PsyRDO?

Trellis 2 generally provides the best result of the different possible options you can use. I consider using trellis 2 to just be 'smarter' then using deadzones with it which does make sense.

I think people need to break out of the mentality that trellis is bad for keeping grain when isn't anymore with psy RDO. (no idea of anyone has this mentality to begin with... just making comments)

[LoRd_MuldeR]

Question: I see a lot of people talking about "using deadzones" :confused:

Aren't deazones always there? If you don't set them explicitly, they will simply default to "deadzone=21,11", right?

[Dark Shikari]

Question: I see a lot of people talking about "using deadzones" :confused:

Aren't deazones always there? If you don't set them explicitly, they will simply default to "deadzone=21,11", right?"Using deadzones" means not using trellis.

[LoRd_MuldeR]

"Using deadzones" means not using trellis.

I see. thx.

[MasterNobody]

Probably deadzones are used even with trellis=2 because trellis is used only for luma plane compression, and not used for chroma planes.
/* no trellis; it doesn't seem to help chroma noticeably */

[Dark Shikari]

Probably deadzones are used even with trellis=2 because trellis is used only for luma plane compression, and not used for chroma planes.This is correct.

Also note that deadzones are used for RDO when trellis=1, and even in trellis=2, deadzones are used for B-frames unless b-rdo is enabled. Finally, even if RDO is on all frametypes, deadzones are still used for fast P-skip decision.

[gav1577]

Are there any advantages in using trelis 2 over low deadzones Qualitywise with bitrates up to and above 4000kbps ? :)

[TheRyuu]

Are there any advantages in using trelis 2 over low deadzones Qualitywise with bitrates up to and above 4000kbps ? :)

In my opinion, using trellis 2 will always provide a better looking result then using deadzones simply because it is smarter then deadzones. More so at lower bitrates, but I've seen it seems to consistantly look better any a number of bitrates I have tried.

Although you should always do the testing and decide for yourself as I am no expert on the matter. :p

[bokonon]

This is correct.

Also note that deadzones are used for RDO when trellis=1, and even in trellis=2, deadzones are used for B-frames unless b-rdo is enabled. Finally, even if RDO is on all frametypes, deadzones are still used for fast P-skip decision.

Would a combination of --trellis 2 and low deadzones (6, 4) work favorably? OR would it work better with default deadzones (21, 11)

[Irakli]

Would a combination of --trellis 2 and low deadzones (6, 4) work favorably? OR would it work better with default deadzones (21, 11)

No, modify deadzones if and only if trellis is off. So, either use trellis 2 with default deadzones or modify deadzones if trellis is off.

[gav1577]

In my opinion, using trellis 2 will always provide a better looking result then using deadzones simply because it is smarter then deadzones. More so at lower bitrates, but I've seen it seems to consistantly look better any a number of bitrates I have tried.

Although you should always do the testing and decide for yourself as I am no expert on the matter. :p

Thanks for the reply will try some clips trellis 2 and see if there is any increase in quality the only thing about trellis 2 is i find it slows things down quite a bit :p

[Dark Shikari]

Here's an encode of part of the original clip that inspired psy RDO in the first place (http://mirror05.x264.nl/Dark/Flash/concertclip.html). The flashing lights, the flat but gradient-filled background areas--all of them contrive to make encoding a nightmare for x264.

[yesgrey]

I want to try this patch. I am thinking in using Megui with the DXVA-HD-HQ profile with trellis 2 and subme 7. The bitrate used will be 16000 or 22000 kbps. Will it be worth it? or due to the high bitrate the effect will be negligible... even using trellis 2 and subme 7 the encoded file will be DXVA compatible?
For using this is enough substituting the x264.exe by the patched one in megui directory, right?

[Blue_MiSfit]

I want to try this patch. I am thinking in using Megui with the DXVA-HD-HQ profile with trellis 2 and subme 7. The bitrate used will be 16000 or 22000 kbps. Will it be worth it? or due to the high bitrate the effect will be negligible... even using trellis 2 and subme 7 the encoded file will be DXVA compatible?
For using this is enough substituting the x264.exe by the patched one in megui directory, right?

Yep :) All you have to do is enable Sub ME >=6, and use MeGUI's preferences to select the patched .exe.

At that bitrate, subme7 will likely be totally unnecessary (guessing). It boosts the effectiveness of Psy-RDO, but is a lot slower as well.

You might try plain old FGO too, instead of Psy-RDO.

Just out of curiousity, why target a specific bitrate? CRF mode is wonderful :D

~MiSfit

[yesgrey]

Just out of curiousity, why target a specific bitrate? CRF mode is wonderful :D

Because I want a specific target file size to put in dvds.

:thanks:

[bkman]

Hi Dark Shikari,

A couple of observations from testing the psyRDO.

1.) It works noticeably better if employed in the first pass also.
2.) The picture seems to degrade much quicker between I-frames, making for a much more noticeable transition on a mid-scene keyframe.

Any ideas?

[Dark Shikari]

2.) The picture seems to degrade much quicker between I-frames, making for a much more noticeable transition on a mid-scene keyframe. This is an interesting claim, because psy RDO as a whole is much much more effective on non-I-frames, because there it can use previous frames to get detail/texture, while on I-frames it can't. From my testing I've noticed that I-frames tend to benefit less than P-frames, rather than vice versa.

[Razorholt]

Does it mean that by adding more P-frames it will improve the quality?

[bkman]

To describe what I'm seeing, it's like the scene gets grainier and grainier, and then BAM I-frame, and its clear again. Doesn't happen all of the time though.

I haven't compared to an encode without psyRDO as of yet, but I've never seen x264 exhibit this sort of behaviour previously at reasonably high bitrates. It's like old DivX or something.

[Dark Shikari]

To describe what I'm seeing, it's like the scene gets grainier and grainier, and then BAM I-frame, and its clear again. Doesn't happen all of the time though.Hmm, that is not at all surprising; the I-frame can't nearly express the kind of fine detail/graininess the psy RDO retains in P-frames.

[bkman]

I don't think you quite understand... in some scenes it becomes increasingly inaccurate (loses coherence) the longer it goes without an I-frame. Then the I-frame seems to restore accuracy for another while. It makes for noticeable and unwanted "jumps" in quality during the same scene. Seems like a bug of some sort.

[Dark Shikari]

I don't think you quite understand... in some scenes it becomes increasingly inaccurate (loses coherence) the longer it goes without an I-frame. Then the I-frame seems to restore accuracy for another while.If by "accuracy" you mean "smoothness," yes. Psy RDO tends to promote graininess, fine detail, and at low bitrates a bit of "lumpiness" rather than a pure attempt to approximate the source as best as possible. The I-frame makes things "smooth" again.
It makes for noticeable and unwanted "jumps" in quality during the same scene. Seems like a bug of some sort.A bug is a mistake in implementing an algorithm; this is not a mistake (but obviously a potential consequence of the chosen algorithm).

This is the problem when people ask for x264 to be more like Xvid; they get the exact same problems Xvid had.

Possible ideas on how to solve this--maybe try using --deadzone-intra 0 --deadzone-inter 20 or something like that? Or use a CQM.

[bkman]

If by "accuracy" you mean "smoothness," yes. Psy RDO tends to promote graininess, fine detail, and at low bitrates a bit of "lumpiness" rather than a pure attempt to approximate the source as best as possible. The I-frame makes things "smooth" again.
A bug is a mistake in implementing an algorithm; this is not a mistake (but obviously a potential consequence of the chosen algorithm).

This is the problem when people ask for x264 to be more like Xvid; they get the exact same problems Xvid had.

If the source frames are smooth, then yes accuracy = smoothness. I notice that in general psyRDO promotes grainyness that wasn't in the original source-- an effect that I don't mind for the most part-- but when it causes these noticeable jumps mid-scene then it becomes unusable. Which is a real shame because I think it has a lot of advantages in detail retention.

So if it is not a mistake in the implementation, would it be possible to modify the algorithm to alleviate this issue?

Edit: Here's a sample to illustrate what I'm talking about. Pay attention to the hand.

http://www.mediafire.com/?jqrhgdlw3de

[akupenguin]

To describe what I'm seeing, it's like the scene gets grainier and grainier, and then BAM I-frame, and its clear again.
So don't put I-frames in the middle of a scene.

[bkman]

So don't put I-frames in the middle of a scene.

Um.. I don't decide where to place the I-frames... x264's (your) rate control does.

If there are some settings I should tweak, then let me know.

[Dark Shikari]

Potential way to resolve this (and many other related issues known collectively as "I-frame flashing"):

1. Encode I-frame as P-frame.
2. Re-encode the frame as an I-frame, except targeting the appearance of the encoded P-frame as the "target image" rather than the original source.

This would probably eliminate all forms of I-frame flashing in cases in which I-frames are placed in the middle of a scene.

And if you think creatively, this can actually be done without breaking threading.
Um.. I don't decide where to place the I-frames... x264's (your) rate control does.

If there are some settings I should tweak, then let me know.Raise --keyint.

[Sagittaire]

Um.. I don't decide where to place the I-frames... x264's (your) rate control does.

If there are some settings I should tweak, then let me know.


- Change the sensivity for scene cut
or
- change the max key frame interval
or
- change the ratio for Iframe.

Short GOP like BD/HDDVD profil mean more Iframe without scene-cut ... I use ratio at 1.10 for better temporal quality (keyframe popup) and better efficiency.

[Sharktooth]

DS solution is a general solution that will work on any GOP size.
However those i frames on the middle of a scene can be "detected" (consecutive frames exceed the key-frame interval) and re-encoded as D_S said. while I frames placed for scene change can be encoded as usual.

[Dark Shikari]

DS solution is a general solution that will work on any GOP size.
However those i frames on the middle of a scene can be "detected" (consecutive frames exceed the key-frame interval) and re-encoded as D_S said. while I frames placed for scene change can be encoded as usual.Exactly, it would only be used on non-scenecut I-frames.

[bkman]

Potential way to resolve this (and many other related issues known collectively as "I-frame flashing"):

1. Encode I-frame as P-frame.
2. Re-encode the frame as an I-frame, except targeting the appearance of the encoded P-frame as the "target image" rather than the original source.

This would probably eliminate all forms of I-frame flashing in cases in which I-frames are placed in the middle of a scene.

And if you think creatively, this can actually be done without breaking threading.
Raise --keyint.

But, and just judging by the sample I posted, isn't the main problem that in some scenes psyRDO deviates from the source too much over time? And this is why the I-frame infusion looks so out of place--because as you say the psy optimisation affects I-frames less.

Ideally what we'd want is a psy optimisation that stays closer in look to the source-- smooth when it is smooth, and grainy/detailed when its detailed.

[Underground78]

isn't the main problem that in some scenes psyRDO deviates from the source too much over time?

Maybe you should post the source too ... :p

[Dark Shikari]

Ideally what we'd want is a psy optimisation that stays closer in look to the source-- smooth when it is smooth, and grainy/detailed when its detailed.But that's what psy RDO does; it penalizes both more complexity than the source and less complexity than the source.

One potential issue may be that it weights high frequencies no different from low frequencies, which may be a problem.

[bkman]

But that's what psy RDO does; it penalizes both more complexity than the source and less complexity than the source.

One potential issue may be that it weights high frequencies no different from low frequencies, which may be a problem.

Well it doesn't quite seem to be working as expected.

Hang on, I'll see what I can do about getting a small sample of that scene in the source up, though I'm afraid it won't be too helpful for replicating the problem without the full source.

[Dark Shikari]

Hang on, I'll see what I can do about getting a small sample of that scene in the source up, though I'm afraid it won't be too helpful for replicating the problem without the full source.A few-hundred frame sample is plenty (just the source version of what you gave me for the encoded clip).

[bkman]

75MB ffv1.
Might take about an hour, heh.

One thing to note is that I'm not exactly using a low bitrate for this encode. The metrics, QP (about 17 avg), and overall look indicate that it is sufficient.


Edit:
Here we go: http://www.mediafire.com/?yc9ttqc1m9d
Sorry its not exactly the same frames.

For some reason this level of discontinuity only happens a few times in the encode.

[Razorholt]

@DS: Do you recommend we still use VAQ2.0 with PsyRDO? Is it causing any conflicts anywhere?

Thanks
- Dan

[desta]

I can actually appreciate what bkman is saying, to an extent. Although I think psy rdo is brilliant, I have had encodes that have suffered because of it - mainly anime, where it introduces mosquito-like noise and some other artifacts, which were too prominent to ignore. I tried lowering the aq, but it didn't really help. Infact I tried lots of things to eliminate the problem, but the only thing that actually put a stop to it was to turn psy rdo off. Unfortunately that meant I traded in one bunch of artifacts for another (mosquito noise & ringing - to - banding & blocking). Catch 22.

[Dark Shikari]

I'll do some testing on that source this week; one major question I have to resolve is whether its simply psy RDO being weighted too highly or whether the algorithm itself needs tweaking.

By the way, have you tried without such a low deblocking strength? Its possible that might be part of the problem.

[MfA]

Would it be possible to force a couple of B-frames which use the next I-frame as a reference? It would gradually smooth out the difference.

PS. low frequency components (especially the ones with high anisotropy) don't encode fine detail.

[Dark Shikari]

Would it be possible to force a couple of B-frames which use the next I-frame as a reference? It would gradually smooth out the difference.Yes, that's called openGOP, and its one possibility for helping avoid this sort of issue.

[Sagittaire]

Would it be possible to force a couple of B-frames which use the next I-frame as a reference? It would gradually smooth out the difference.

PS. low frequency components (especially the ones with high anisotropy) don't encode fine detail.

Will default setting bframe are really overquantized. Moreover with longgop Iframe are generaly scene cut.

[Gabriel_Bouvigne]

Potential way to resolve this (and many other related issues known collectively as "I-frame flashing"):

1. Encode I-frame as P-frame.
2. Re-encode the frame as an I-frame, except targeting the appearance of the encoded P-frame as the "target image" rather than the original source.

Or better: run the fast P skip heuristic on I frames macroblocks.

case A: fast p skip is not triggered => encode intra MB as usual

case B: fast p skip is triggered => within metrics, use (a*L0+b*ref) instead of only ref

In my opinion this is way more elegant than the encode-as-p-then-as-I hack, but unfortunately requires more code changes.

[Dark Shikari]

Now here's something interesting: I just encoded that test sample that was posted earlier at 1500kbps--and there was no I-frame flash at all, nor the texture smudging. It looked basically flawless.

What settings did you use for encoding it? Since you gave me a clip of a longer h264 file, the x264 header wasn't included.

[bkman]

Settings used were:

--pass 2 --bitrate 1800 --stats "G:\Encoding\Dune\hfyu_dune.stats" --level 4.1 --keyint 240 --min-keyint 24 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,0 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 24 --threads auto --thread-input --sar 1:1 --progress --no-dct-decimate

First pass used the same settings. Anything else you need to determine the issue, let me know.

[LoRd_MuldeR]

"--filter -3,0" sounds a bit harsh to me. What happens if you remove "--filter" ???

[Atak_Snajpera]

--keyint 240 --min-keyint 24
Why????????

--filter -3,0
whyyyyy?????