Spatial mode for B-frames?

[Caroliano]

Sorry for the long post, but you can skip most of the testing part. :o

I decided to play a bit with the spatial mode instead of temporal mode for b-frames. I searched a bit but I coudn't find anybody that have done tests with that, nor much thing explaning its diferences. So I done some tests:

First with noir opening. A rather high motion clean anime sample:

Job is a video job. encoder commandline:
--bitrate 800 --ref 8 --mixed-refs --bframes 3 --b-pyramid --weightb --analyse all --8x8dct --progress --output "E:\Testes\Azumanga\noirtest.mkv" "E:\Testes\Azumanga\noirtest.avs"

x264 [info]: PSNR Mean Y:44.350 U:45.998 V:46.438 Avg:44.733 Global:42.533 kb/s:796.08Job is a video job. encoder commandline:
--bitrate 800 --ref 8 --mixed-refs --bframes 3 --b-pyramid --weightb --analyse all --8x8dct --direct spatial --progress --output "E:\Testes\Azumanga\noirtest.mkv" "E:\Testes\Azumanga\noirtest.avs"

x264 [info]: PSNR Mean Y:44.380 U:46.011 V:46.463 Avg:44.760 Global:42.563 kb/s:795.80

I couldn't spot any diferences between them. But the one with --direct spatial has a slight better PNSR and an slight lower bitrate. I don't benchmarked, but both shoud have aproximatedly the same encoding time.


Then an test with an very small sample (396 frames) of low motion conversation with a zoom of the Sekai no Monshou anime:

Job is a video job. encoder commandline:
--qp 30 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse all --8x8dct --progress --output "E:\Testes\Azumanga\Sekai.mkv" "E:\Testes\Azumanga\Sekai test.avs"

x264 [info]: PSNR Mean Y:38.636 U:44.240 V:44.170 Avg:39.801 Global:39.589 kb/s:54.94Job is a video job. encoder commandline:
--qp 30 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse all --8x8dct --direct spatial --progress --output "E:\Testes\Azumanga\Sekai.mkv" "E:\Testes\Azumanga\Sekai test.avs"

x264 [info]: PSNR Mean Y:38.628 U:44.212 V:44.147 Avg:39.792 Global:39.580 kb/s:51.57

Now the bitrate diference is more expressive, and the PSNR is almost the same, but a bit lower for spatial mode. But for my eyes the spatial mode one has the best quality, because it has more temporal stability than the other and no detail loss in relation to temporal for me, even though the lower bitrate and PNSR.

The fist is an high quality and high motion anime test. The second one an low quality, low motion, low resolution (400x304) anime test. The setings I used for the first encode are realy strange, and I will not explain this non-sense, but they shoud not afect the comparison (I hope!).


Anyone is welcome to make more test with diferent seting and diferent sources.


And maybe somebody could explain the technical diferences between these two modes. Pengvado? Manao? I aways read and bookmark yours explanations!

What happens with the b-frames if I select "none" as the mode for them? How the others features (b-rdo, trellis, references and mixed ref, n° of consecutive b-frames, etc) are related to the modes. The efficience of them changes in some way when I change between the modes? Or it is not related at all to some or all them?

Cheers

[Sirber]

I heard long ago that --direct spatial wasn't working and I haven't read it has been added. Am I wrong?

[akupenguin]

How it works: If a macroblock in a B-frame is coded as Skip or Direct, then no motion vector is coded, and instead a predicted mv is used. This takes fewer bits compared to the other inter types. The --direct option controls how that mv is computed.
Temporal: uses the mv from the future P-frame, scaled appropriately since delta_time(P0,B1) is less than delta_time (P0,P2). Note that if the P-frame used partitions, then the Direct block uses those same partitions, and scales each mv.
Spatial: uses the median mv of neighboring blocks in the current frame. No partitions (always 16x16) except in one special case.
None: forbids Skip and Direct types. This is always worse quality than either of the above, and is included only for my use in development.

Since Direct takes fewer bits, obviously the best mode is whichever allows Direct to be used more often, i.e. when the predicted mv matches the real mv most often.
When there is coherent motion, temporal tends to be the better predictor, since motion may vary more rapidly across space than it does from one frame to the next, and also due to the use of partitions.
But in anime, there is often no real motion, and instead you have objects that sit in one place for 2-3 frames, then move large distances. There, spatial may work better.

There's also the psychovisual criterion: Since Direct mode takes fewer bits, there will be a bias towards using it even if it doesn't quite match the real mv. So which mode produces uglier artifacts at low bitrates? imho temporal looks better in such cases, because adding motion coherency when it wasn't in the source is less noticeable than the converse. This can be a strong effect, and is why I have default=temporal.

Effects from other options:
Multiref & mixed ref: Slightly reduces the effectiveness of temporal mode. If P2 wasn't predicted from P0, but rather from P-8, then its motion vector is less likely to be related to real motion. This effect is somewhat mitigated by the fact that multiref is most used in regions where Skip types wouldn't be useful anyway.
B-pyramid: Slightly reduces the effectiveness of temporal mode. Consider [P0 b1 B2 b3 P4]. In blocks where B2 used a backwards mv only, there is nothing for b1 to use for temporal interpolation.
No-fast-pskip: B-skip has artifacts in the same places as P-skip. Fixing the P tends to fix both.

[Sirber]

:goodpost:

Is Spatial implemented and ready to use? If so, I will :D

[akupenguin]

Both modes have always worked (since we first got B-frames)

[Sirber]

nice :D thanks!

[foxyshadis]

I don't suppose there's a possibility of choosing spacial or temporal depending on which is closer? Are they mutually exclusive, so that the stream only supports one interpretation of the value?

[akupenguin]

It is possible to switch per-frame. I haven't written it yet.

[DarkZell666]

what about doing several frame partitions and switching per partition ? ... xD

[akupenguin]

Even if there were reason to expect different mv types in different pieces of the frame (and note that you can only cut frames into horizontal bands, not arbitrary shapes), I think slices have too much overhead to ever improve compression. Though if you are slicing anyway in order to multithread, then yes you could add this for free.

[Caroliano]

Some extrapolations based in what I understanded:

- In higher qualitys (around q22) I can chose between one mode or another only seeing the bitrate and PSNR diference because the bitrate is suficiently high to stop the artifacts that they can cause, therefore the only diference will be in the efficiency.

- Using an lower number of B-frames shoud also help with the temporal mode, because the references are closer. But probably will degrade the quality compared to the same configurations with more B-frames...

Are my assumptions correct?

There's also the psychovisual criterion: Since Direct mode takes fewer bits, there will be a bias towards using it even if it doesn't quite match the real mv. So which mode produces uglier artifacts at low bitrates? imho temporal looks better in such cases, because adding motion coherency when it wasn't in the source is less noticeable than the converse. This can be a strong effect, and is why I have default=temporal.
In the sample I did of Sekai no Monshou the situation is the reverse: spatial mode has more temporal stability. But highering the number of references frames with mixed ref I can see some temporal instability, of a diferent type, in spatial mode q30 too. I don't know if the bitrate gain in an normal situation compense this.

I'm ataching the two coded samples (spatial and temporal). 396 frames only for testing purposes. Look around her eyes in the temporal sample. Only wait the mods aprove this...

[Caroliano]

Files aproved. What then, people?

[DarkZell666]

without looking too hard, I can't see any difference myself :)