Psy RDO: Official testing thread (version 0.6 out!)

[Razorholt]

After size analyse I think that it's simply a different complexity analysis. For really grainy picture (with really fine grain) fgo mode will use really more bit than psy mode.

http://jfl1974.free.fr/upload/1080p_3.264_020491.jpg (http://jfl1974.free.fr/upload/1080p_3.264_020491.png) vs http://jfl1974.free.fr/upload/1080p_Q1.264_020491.jpg (http://jfl1974.free.fr/upload/1080p_Q1.264_020491.png) vs http://jfl1974.free.fr/upload/Casino_MCCLI_BD9.264_020489.jpg (http://jfl1974.free.fr/upload/Casino_MCCLI_BD9.264_020489.png)

PSY vs FGO vs Elecard

31 832 vs 64 267 vs 63 751 bits

And here FGO produce by far the best quality ...


How about the original frame? How many bits?

[Adub]

I agree. FGO does seem to come out better here.

[Dark Shikari]

I agree. FGO does seem to come out better here.Given that its frame size is double that of psy RDO, how am I not surprised? :p

If its a B-frame, you can compensate for that by setting pbratio lower... that's what FGO does anyways and its probably wrong to put anything into mainline x264 that changes settings like that when the user should be in control of it.

If it isn't, maybe you didn't use the same bitrate for the two clips...?

Also, note that here, the Elecard frame isn't even the same frame of the video...

[Sagittaire]

How about the original frame? How many bits?

Source is FFV1 master ...

[Sagittaire]

Given that its frame size is double that of psy RDO, how am I not surprised? :p

Yes not surprising for quality. The surprise come from decision. Here FGO seem make better choice than PSY mode. It's a really complexe frame with very fine but visible grain (grain on blue sky)

If its a B-frame, you can compensate for that by setting pbratio lower... that's what FGO does anyways and its probably wrong to put anything into mainline x264 that changes settings like that when the user should be in control of it.

Only Pframe here.

If it isn't, maybe you didn't use the same bitrate for the two clips...?

6900 Kbps for overall bitrate (more than 200 000 frames).


Also, note that here, the Elecard frame isn't even the same frame of the video...

yes because it's bframe for Elecard. Anyway all the Pframe for the encoded sequence are the same size with equivalent grain complexity for picture source. So the comparison is correct here.

[Razorholt]

Source is FFV1 master ...

Thanks Sagittaire. I just wanted to find out whether PSY was actually increasing the size of the video, as FGO does.

[Dark Shikari]

Yes not surprising for quality. The surprise come from decision. Here FGO seem make better choice than PSY mode. It's a really complexe frame with very fine but visible grain (grain on blue sky)I'm proposing that FGO has absolutely nothing to do with the double-size frame and that there has been user error here, such as comparing B-frames with different pbratio or encoding with different bitrates. Especially since the clips you posted earlier were not even of the same video clip, I don't exactly trust that you've done everything right here...

[Sagittaire]

I'm proposing that FGO has absolutely nothing to do with the double-size frame and that there has been user error here, such as comparing B-frames with different pbratio or encoding with different bitrates. Especially since the clips you posted earlier were not even of the same video clip, I don't exactly trust that you've done everything right here...

Well it's just fast size partition with mkvmerge without range selection. Anyway the overall bitrate for the complete encoding are exactly the same: 6.9 Mbps. Sample come from complete encoding (more than 200 000 frame). It's just local difference for this frame. For the Pframe 47 499 I have another size: 122 889 vs 130 734 vs 99 707, and for the 64 190 frame another size: 82 008 vs 44 114 vs 74 374.


Source is Casino Royal with really different grain level.

[ToS_Maverick]

about the frequencies and the quantization...

that sounds like Psy RDO works somehow like a custom matrix? I understand that a matrix uses fixed values, but it goes into the same direction, be more "kind" to the lower frequencies, to avoid blocking, preserving a good "look"...

[Sagittaire]

I'm proposing that FGO has absolutely nothing to do with the double-size frame and that there has been user error here, such as comparing B-frames with different pbratio or encoding with different bitrates. Especially since the clips you posted earlier were not even of the same video clip, I don't exactly trust that you've done everything right here...

Well I find the problem ... it's a Rate Control problem between first and second pass. With the same final bitrate crf mode produce really better result.


2 pass mode with fast first pass:

x264.exe --threads 3 --thread-input --keyint 24 --min-keyint 2 --vbv-maxrate 20000 --vbv-bufsize 30000 --mvrange 511 --level 4.1 --bframe 3 --b-pyramid --bime --weightb --ref 1 --mixed-refs --direct auto --deblock -2:-2 --bitrate 6900 --pass 1 --stats "casino.log" --ipratio 1.10 --pbratio 1.10 --partitions "all" --8x8dct --me "hex" --subme 5 --trellis 0 --aq-mode 0 --aq-strength 0.00 --aud --nal-hrd --sar 1:1 --cqmfile Sagittaire.cfg --progress -o NUL Lossless.avs

And after I use always the same casino.log stat for all the encoding.

x264.exe --threads 3 --thread-input --keyint 24 --min-keyint 1 --vbv-maxrate 20000 --vbv-bufsize 30000 --mvrange 511 --level 4.1 --bframe 3 --b-pyramid --b-rdo --bime --weightb --ref 3 --mixed-refs --direct auto --deblock -2:-2 --bitrate 6900 --pass 2 --stats "casino.log" --ipratio 1.10 --pbratio 1.10 --partitions "all" --8x8dct --me "umh" --subme 7 --trellis 2 --aq-mode 2 --aq-strength 1.00 --no-fast-pskip --no-dct-decimate --aud --nal-hrd --sar 1:1 --cqmfile Sagittaire.cfg --zone 201560,207442,b=0.33 --progress -o 1080p_3.264 Lossless.avs


Quality mode:

x264.exe --threads auto --thread-input --keyint 24 --min-keyint 1 --crf %E_BR% --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --mvrange 511 --level 4.1 --bframe 3 --b-pyramid --b-rdo --bime --weightb --ref 3 --mixed-refs --direct auto --deblock -2:-2 --ipratio 1.10 --pbratio 1.10 --partitions "all" --8x8dct --me "umh" --subme 7 --trellis 2 --no-fast-pskip --no-dct-decimate --aud --nal-hrd --sar 1:1 --cqmfile Sagittaire.cfg --aq-strength 1.00 --aq-mode 2 --zone %CRE_FR%,%END_FR%,b=0.33 --progress -o 1080p_Q1.264 %E_SRC%


With 2 pass encoding and exactly the same setting than crf mode (and same final bitrate) I have completely different Rate Control (for example 31 832 bits vs 70 926 bits for frame 20 491). VAQ + PSY seem imply major modification in RC between first and second pass. It's perhaps necessary to use VAQ and PSY in first pass too ... ???

[Dark Shikari]

With 2 pass encoding and exactly the same setting than crf mode (and same final bitrate) I have completely different Rate Control (for example 31 832 bits vs 70 926 bits for frame 20 491). VAQ + PSY seem imply major modification in RC between first and second pass. It's perhaps necessary to use VAQ and PSY in first pass too ... ???No, its really mainly VAQ. Its always been the case that AQ should be on in the first pass.

[elguaxo]

thanks Dark Shikari! this looks very promising. :)

[Dark Shikari]

Here's a really dramatic test, Touhou as always: Linkage (http://mirror05.x264.nl/Dark/psy/)

And another one, from World in Conflict, with FGO vs Psy RDO: Linkage (http://imk.cx/junk/a2a_3mbit.html)

[gav1577]

New patch. Updates:

3. Now available as a commandline option (--rdcmp).

Does this mean the patch now disabled by default and the switch above will activate it ? :confused:

[Dark Shikari]

Does this mean the patch now disabled by default and the switch above will activate it ? :confused:Nope, still on by default. It can now be switched on and off.

[TheRyuu]

http://forum.doom9.org/showthread.php?p=1145012#post1145012

[mahsah]

Tried using Psy RDO on a movie that I had processed with MVdegrain3, limitedsharpenfaster, and gradfun2db to see how it did with the dithering (the original grain was very distracting). Encoded at 2 pass 1500 kbps, base cmdline:


--pass 2 --bitrate 1500 --stats ".stats" --deadzone-inter 10 --deadzone-intra 10 --ref 5 --mixed-refs --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh

Original:
http://img88.imageshack.us/img88/4760/origfo3.png
Default Command line:
http://img88.imageshack.us/img88/2306/defaultqj2.png
--fgo 10
http://img508.imageshack.us/img508/5979/fgocj7.png
Psy RDO
http://img209.imageshack.us/img209/4286/psyru0.png

[gav1577]

Thanks Dark Shikari & Wizboy11

[desta]

Regarding trellis; I was led to believe that trellis=2 was generally a bad thing. Has that changed now because of the way psy rdo interacts with it, or is it still advisable to avoid it if possible?

Obviously with trellis=1 being advised against with psy rdo, would that mean deadzones are the safest bet?

[Dark Shikari]

Regarding trellis; I was led to believe that trellis=2 was generally a bad thing. Has that changed now because of the way psy rdo interacts with it, or is it still advisable to avoid it if possible?Trellis 2 is not a bad thing at all with psy RD; the problem before was that it encouraged blurring because RD would often prefer blurrier modes, and trellis would generate blurrier modes. But now, RD prefers *less* blurry modes.

[desta]

That's good to know, thanks. :)

[techouse]

So was anyone able to fix the me-prepass patch so that it can work with psy_rdo 0.22?

[Sagittaire]

So was anyone able to fix the me-prepass patch so that it can work with psy_rdo 0.22?

This patch is not really usefull for the quality. Use too many patch is not very good too.

[DeathTheSheep]

"This patch is not really useful for the quality." --
I'd have to disagree with you there, Sagittaire. ;)

Anyway, for myself, visually speaking, this psy RDO seems to do a fairly decent job at knocking the blurriness down and retaining detail, even if this detail isn't theoretically "exact." :p I can live with the scars on Neo, too--the motion of subsequent frames minimizes that effect anyway. In the context of real viewing, one is not likely to pause, zoom in, and analyze Neo's face carefully for evidence of distortion, but rather to watch the video at normal speed and notice instead the obvious decrease in blurring and ugly blocking.

[Episodio1]

"...and notice instead the obvious decrease in blurring and ugly blocking."

That's it. Thanks a lot. :)

[SquallMX]

Dark Shikari you´re the best, the decrease in blurring is amazing, especially for low/mid bitrates:

Original:
http://www.imagenchile.com/img/f4cd0243a7c650a489cb7a8f8940faf4/tOriginal2.png (http://www.imagenchile.com/view.php?img=f4cd0243a7c650a489cb7a8f8940faf4)
http://www.imagenchile.com/img/df7ac70a0dbb75d5d320667cdf414385/tOriginal3.png (http://www.imagenchile.com/view.php?img=df7ac70a0dbb75d5d320667cdf414385)

No PSY:
1-Pass:
Job commandline: "D:\Archivos de programa\megui\tools\x264\x264.exe" --pass 1 --bitrate 3308 --stats "F:\Archivos de Programa\WatchHDTV\Ok" --level 4.1 --keyint 240 --min-keyint 24 --bframes 2 --b-pyramid --filter -2,-1 --subme 1 --partitions none --qpmax 40 --vbv-maxrate 24000 --me dia --merange 12 --threads auto --thread-input --sar 1:1 --progress --no-psnr --output NUL "F:\Archivos de Programa\WatchHDTV\Pantera.avs"
2-Pass:
Job commandline: "D:\Archivos de programa\megui\tools\x264\x264.exe" --pass 2 --bitrate 3308 --stats "F:\Archivos de Programa\WatchHDTV\Ok" --level 4.1 --keyint 240 --min-keyint 24 --ref 2 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-1 --trellis 1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --qpmax 40 --vbv-maxrate 24000 --merange 12 --threads auto --thread-input --sar 1:1 --progress --no-psnr --output "F:\Archivos de Programa\WatchHDTV\Pantera.mp4" "F:\Archivos de Programa\WatchHDTV\Pantera.avs"

http://www.imagenchile.com/img/01f206edf0afe92996e13e05915f908f/tNoPSY2.png (http://www.imagenchile.com/view.php?img=01f206edf0afe92996e13e05915f908f)
http://www.imagenchile.com/img/10c3bf4f88c1189b0baee4a43535ffe2/tNoPSY3.png (http://www.imagenchile.com/view.php?img=10c3bf4f88c1189b0baee4a43535ffe2)

SSIM Mean Y:0.9884279

PSY:
1-Pass:
Job commandline: "D:\Archivos de programa\megui\tools\x264\x264.exe" --pass 1 --bitrate 3308 --stats "F:\Archivos de Programa\WatchHDTV\Ok" --level 4.1 --keyint 240 --min-keyint 24 --bframes 2 --b-pyramid --filter -2,-1 --subme 1 --partitions none --qpmax 40 --vbv-maxrate 24000 --me dia --merange 12 --threads auto --thread-input --sar 1:1 --progress --no-psnr --output NUL "F:\Archivos de Programa\WatchHDTV\Pantera.avs"

2-Pass:
Job commandline: "D:\Archivos de programa\megui\tools\x264\x264.exe" --pass 2 --bitrate 3308 --stats "F:\Archivos de Programa\WatchHDTV\Ok" --level 4.1 --keyint 240 --min-keyint 24 --ref 2 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --qpmax 40 --vbv-maxrate 24000 --merange 12 --threads auto --thread-input --sar 1:1 --progress --no-psnr --output "F:\Archivos de Programa\WatchHDTV\Panterapsy.mp4" "F:\Archivos de Programa\WatchHDTV\Pantera.avs"

SSIM Mean Y:0.9867381

http://www.imagenchile.com/img/9ee4bb79390adbba92e8feb5e09b05d9/tPSY2.png (http://www.imagenchile.com/view.php?img=9ee4bb79390adbba92e8feb5e09b05d9)
http://www.imagenchile.com/img/a1afb43073ac41cbb252630dd456de16/tPSY3.png (http://www.imagenchile.com/view.php?img=a1afb43073ac41cbb252630dd456de16)

:thanks:

[bokonon]

@SquallMX

Please let me know what build you are using to test with....

is there a switch to turn it off even when using --subme 7 and -b-rdo ?

nevermind:
x264.869.modified.exe (http://www.mediafire.com/?xb4ittdi0nz)
(alt DL link (http://www.zshare.net/download/130689194e58f8bb/))

x264 Rev. 869 Modified Patch Build
Avis Input, mp4 output, pthreads config.

Built with GCC 3.4.5, make fprofiled, and statically linked pthreads. (means no pthreads.dll required in x264 directory)

Patches:
x264_gxRC.diff
x264.gaussian.cplxblur.01.diff
x264_hrd_pulldown.04_interlace.diff
x264_me-prepass_DeathTheSheep.01.diff
x264_progress.diff
x264_psyRDO_0.22.diff

The x264_fix_win_stdin.diff patch has been committed into the git so that has been left out.

Notes:
See the psyRDO thread (http://forum.doom9.org/showthread.php?t=138293) on usage of it although I briefly explained it in an above post.
Option: --rdcmp
Usage:
--rdcmp psy (psy rdo)
--rdcmp ssd (regular)
Psy RDO is on by default. at subme levels 6 and 7. 7 + b-rdo is best.
Read psyRDO thread for more info regarding it.

me-prepass:
Use --me-prepass flag to activate, off by default
Runs a half-pel search at a slight speed cost to improve quality with any --me mode.

x264_gxRC.diff:
Activated at subme levels 6 and 7, basically increases quality at little to no speed reduction.

x264_progress.diff:
Adds a little progress thingy at the top of the x264 window. Neat feature.

[techouse]

This patch is not really usefull for the quality. Use too many patch is not very good too.

I fixed the me-prepass patch! However, has to be applied after the psy_rdo patch.

DOWNLOAD: http://pastebin.com/f62aea32f

[Razorholt]

I fixed the me-prepass patch!

Is it REALLY worth it? Maybe Wizboy or Bobor can try and compile an x264 version with the fixed me-prepass patch for us to test? :D

Thanks,
- Dan

[Stakiman]

After various tests I came to a conclusion that this patch is extremely useful for grain keeping. Thanks DarkShikari for the creation of it! Here are some screens from my tests. The B-frame is not very good but the P-frames are amazing. Without PSY_RDO or FGO this would look without grain.

Source vs Encode
http://achumpatoxford.com/u/files/55/91f5e19f819733ed60b9a1ca0afbf85d_thumb.png (http://achumpatoxford.com/u/files/55/91f5e19f819733ed60b9a1ca0afbf85d.png) http://achumpatoxford.com/u/files/55/f98863cbe31c1c5746d499a6829c6a6b_thumb.png (http://achumpatoxford.com/u/files/55/f98863cbe31c1c5746d499a6829c6a6b.png)
http://achumpatoxford.com/u/files/55/Hooligans/1e58e93cf20dc267827da6e35825406b_thumb.png (http://achumpatoxford.com/u/files/55/Hooligans/1e58e93cf20dc267827da6e35825406b.png) http://achumpatoxford.com/u/files/55/Hooligans/9f640acd8ef5d6bf40174a73db308fad_thumb.png (http://achumpatoxford.com/u/files/55/Hooligans/9f640acd8ef5d6bf40174a73db308fad.png)
http://achumpatoxford.com/u/files/55/Hooligans/2359e169adb5b49f13e7338767688f4a_thumb.png (http://achumpatoxford.com/u/files/55/Hooligans/2359e169adb5b49f13e7338767688f4a.png) http://achumpatoxford.com/u/files/55/Hooligans/fba38548db1cae551d3f8f1740647ccc_thumb.png (http://achumpatoxford.com/u/files/55/Hooligans/fba38548db1cae551d3f8f1740647ccc.png)
http://achumpatoxford.com/u/files/55/Hooligans/7cfac15056f601d220f80662875580ac_thumb.png (http://achumpatoxford.com/u/files/55/Hooligans/7cfac15056f601d220f80662875580ac.png) http://achumpatoxford.com/u/files/55/Hooligans/a5a283e3ee9e086bf95e96946444378e_thumb.png (http://achumpatoxford.com/u/files/55/Hooligans/a5a283e3ee9e086bf95e96946444378e.png)
http://achumpatoxford.com/u/files/55/Hooligans/225571282676a5ae36fc3d7411fa977b_thumb.png (http://achumpatoxford.com/u/files/55/Hooligans/225571282676a5ae36fc3d7411fa977b.png) http://achumpatoxford.com/u/files/55/Hooligans/9091d938bb984850b4fab48fdd63e980_thumb.png (http://achumpatoxford.com/u/files/55/Hooligans/9091d938bb984850b4fab48fdd63e980.png)

[mahsah]

How do you think this would work on anime? Could it be used with gradfun2db to hide DCT blocks?

[Dark Shikari]

How do you think this would work on anime? Could it be used with gradfun2db to hide DCT blocks?Yes, in my experience so far it is incredibly good at keeping the dither that gradbun2db introduces as long as you give it enough bitrate (while without, regular x264 will generally butcher banding even at absurdly high bitrates).

[desta]

I've tried it on quite a bit of animated material now, and have got to say I'm very impressed.

Would I be right in assuming that although psy rdo is activated by default at >subme 6, it's optimised to process an image based on how grainy/noisy it is, including fine texture/detail, etc (in other words good on clean, grainy, or mixed footage)... whereas fgo by nature assumes a source is grainy/noisy, and could therefore be detrimental to a source that has both clean and grainy scenes?

[Dark Shikari]

I've tried it on quite a bit of animated material now, and have got to say I'm very impressed.

Would I be right in assuming that although psy rdo is activated by default at >subme 6, it's optimised to process an image based on how grainy/noisy it is, including fine texture/detail, etc (in other words good on clean, grainy, or mixed footage)... whereas fgo by nature assumes a source is grainy/noisy, and could therefore be detrimental to a source that has both clean and grainy scenes?Both psy RDO and FGO attempt to make the output have the same <metric> as the input. But you are sort of correct, the metric used by psy RDO will generally be more friendly towards all types of complexity, not merely grain.

[desta]

Brilliant, thanks. I know you said the object of psy rdo was to eventually replace fgo. Is that likely soon? Would you consider it to be more efficient than fgo already, or do you think they both still have their place?

[Dark Shikari]

Brilliant, thanks. I know you said the object of psy rdo was to eventually replace fgo. Is that likely soon? Would you consider it to be more efficient than fgo already, or do you think they both still have their place?I think its more efficient, but I haven't done enough testing.

One thing I may see is if I can combine the two; Psy RD uses a 4x4 and 8x8 transform for comparison, while FGO uses a 2x2. Maybe I could combine all three transforms? :p

[gav1577]

I think its more efficient, but I haven't done enough testing.

One thing I may see is if I can combine the two; Psy RD uses a 4x4 and 8x8 transform for comparison, while FGO uses a 2x2. Maybe I could combine all three transforms? :p

Wow that would be neat would love to test that one out :p

[Yoshiyuki Blade]

With these visual improvements (VAQ and Psy RDO), I wonder if CQMs still have a large impact on image quality. I always have Sharktooth's AVC CQM on, even with animated material, but I haven't tried without it yet. Encoding a full length animated episode is incredibly slow on my lappy. You think CQMs will still remain significant at this stage (given a reasonably high bitrate/resolution)?

[Dark Shikari]

With these visual improvements (VAQ and Psy RDO), I wonder if CQMs still have a large impact on image quality. I always have Sharktooth's AVC CQM on, even with animated material, but I haven't tried without it yet. Encoding a full length animated episode is incredibly slow on my lappy. You think CQMs will still remain significant at this stage (given a reasonably high bitrate/resolution)?I've never really personally liked CQMs, for many reasons.

If you want to make comparisons, short samples should probably be sufficient.

[shae]

I've never really personally liked CQMs, for many reasons.Like what?

[Dark Shikari]

Like what?
1) When you're actually trying to keep fine detail, they butcher it.
2) They skew x264's lambda values.
3) They confuse RDO.
4) They confuse trellis even more than they confuse RDO.
5) Prestige, especially, does some really bizarre things.

[ToS_Maverick]

sry for hijacking this thread, but why is x264 so not-suited for CQMs? do other encoders work better with them?

[Dark Shikari]

sry for hijacking this thread, but why is x264 so not-suited for CQMs? do other encoders work better with them?It isn't that x264 isn't suited for use with CQMs; nothing of the sort.

First, on the topic of RDO, you can intentionally make CQMs so that RD-wise, the inter matrix will always be favorable (for example), and so inter blocks are nearly always used when in RD mode.

However, the creator of the CQM might not have intended this; for example, the CQM might have been intended for an encoder that didn't use RDO, and therefore would not have taken into account the CQM when making its decision. I strongly suspect Prestige is this sort of CQM.

With trellis, trellis simply does not take into account CQM weights when working. I've found doing so actually reduces visual quality, unless I made a mistake in my patch--go figure.

[techouse]

Is it REALLY worth it? Maybe Wizboy or Bobor can try and compile an x264 version with the fixed me-prepass patch for us to test? :D

Thanks,
- Dan
Just Check out my site http://x264.tk :rolleyes:

x264_x86_r871_techouse (http://techouse.project357.com/builds/x264_x86_r871_techouse.7z)

Source: x264 r871 GIT (git://git.videolan.org/x264.git)

Applied patches (current versions):

x264_hrd_pulldown.04_interlace.diff

x264_me-prepass_DeathTheSheep_techouse_fix.diff (http://pastebin.com/f62aea32f)

x264_progress.diff

x264_psy_rdo_0.22.diff


Please check http://forum.doom9.org/showthread.php?t=130364 and http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog for more info

Compiled by techouse on June 6th 2008, 03:13:38 CEST with GCC-4.3.0 on Windows Vista Ultimate SP-1 32-bit.

Commandline used: ./configure&&make fprofiled

Platform: X86
System: MINGW
avis input: yes
mp4 output: yes
pthread: yes
gtk: no
debug: no
gprof: no
PIC: no
shared: no
visualize: no

[desta]

Just Check out my site http://x264.tk :rolleyes:

x264_x86_r871_techouse (http://techouse.project357.com/builds/x264_x86_r871_techouse.7z)
http://forum.doom9.org/showthread.php?p=1146886#post1146886

[shae]

1) When you're actually trying to keep fine detail, they butcher it.Well, doesn't that depend on the QM and is exactly what CQM are for, to allow you to fine tune what's kept and what's not?

2) They skew x264's lambda values.
3) They confuse RDO.
4) They confuse trellis even more than they confuse RDO.Wouldn't that be implementation specific? It may conflict with other features currently, but in an ideal AVC encoder it should just add flexibility.

5) Prestige, especially, does some really bizarre things.Just extreme values leading to extreme results, no?

[LoRd_MuldeR]

Wouldn't that be implementation specific? It may conflict with other features currently, but in an ideal AVC encoder it should just add flexibility.

I'd say an ideal AVC encoder wouldn't need any special CQM to workaround it's visual problems :p

Also I think features like VAQ, FGO and Psy RDO do exactly what people tried to achieve with CQM's before, only in a more sophisticated and more general way.
IMO it's always better to resolve the cause of the problem itself instead of fighting the symptom of the problem...

[foxyshadis]

An ideal AVC encoder would vary the quant matrix per frame to completely maximize RD. ;) That's one of those exponentially hard problems though, unless someone comes up with a good way to quickly test whether a CQM benefit outweighs its size every frame.

[Blue_MiSfit]

I had an idea for something similar back in the Xvid days, but it would have been very time consuming, and required a lot of user interaction. Here's how I saw it:

1) Take a movie, and split it into comp-test style chunks
2) Encode this multiple times with different CQMs
3) Show the compressionist the individual chunks, with the ability to flip between versions easily, stackhorizontal, interleave, whatever.
4) Allow the compressionist to pick a "top 3" or something for each chunk
5) Tally the votes, and pick a CQM for the movie.

I wish I could code, or I would have cobbled this together just out of curiosity.

But back on topic - I'm a huge fan of PsyRDO. It's improved almost every single thing I've thrown at it.

~MiSfit

[Razorholt]

I'm a huge fan of PsyRDO. It's improved almost every single thing I've thrown at it.
Are you using Trellis 2 ?