xvid's chroma optimizer

[plugh]

Is there a way to impliment this in a script so I can 'see' what it does?image_chroma_optimize(IMAGE * img, int width, int height, int edged_width)
{
int x,y;

for (y = 1; y < height/2 - 1; y++)
for (x = 1; x < width/2 - 1; x++)
{
#define IS_PURE(a) ((a)<=16||(a)>=235)
#define IMG_Y(Y,X) img->y[(Y)*edged_width + (X)]
#define IMG_U(Y,X) img->u[(Y)*edged_width/2 + (X)]
#define IMG_V(Y,X) img->v[(Y)*edged_width/2 + (X)]

if (IS_PURE(IMG_Y(y*2 ,x*2 )) &&
IS_PURE(IMG_Y(y*2 ,x*2+1)) &&
IS_PURE(IMG_Y(y*2+1,x*2 )) &&
IS_PURE(IMG_Y(y*2+1,x*2+1)))
{
IMG_U(y,x) = (IMG_U(y,x-1) + IMG_U(y-1, x) + IMG_U(y, x+1) + IMG_U(y+1, x)) / 4;
IMG_V(y,x) = (IMG_V(y,x-1) + IMG_V(y-1, x) + IMG_V(y, x+1) + IMG_V(y+1, x)) / 4;
}
}
('edged_width' is width rounded up to mod 16 plus padding...)

It doesn't have to be efficient, as I don't intend to actually run a full movie through such a script. I just want to compare before and after on some frames...

[Didée]

No problem to do that with MaskTools. Basic operation is just this:

mt_luts(last,last,mode="average",pixels="0 -1 -1 0 0 1 1 0",expr="y",Y=2,U=3,V=3)

To complete cloning of Xvid's operation, one would additionally need to build a mask for Y values Y<16 + Y>235, expand it with mt_expand(mode="both"), and use that for masking. Not sure if that's really needed for your tests, though.

[plugh]

Thanks for the reply. Is that - hmmm what term to use - 'self referential' like the xvid code snippet?

The calc for the (I)th pixel uses pixel (I+1). When it then goes to calc the (I+1)th pixel, it uses the value it just calculated - ie the (I-1)th pixel.

Ditto for the vertical direction - when calc'ing pixel in row J, it uses pixel in row (J+1). When it gets around to calc'ing that pixel in row (J+1) it uses the value in the previous row that it calc'ed from this pixel.

So all references to the left and up use previously calc'ed pixel values, those to the right and down use actual pixel values. That looks fishy to me, which is why I want to visualize the results...

[Didée]

Oops, my bad. Too many years of not using a programming language ... now that you mention it, I see that loopback effect.

But now, I don't see a (sane) way to do that with an Avisynth script, except you really want to run a chain of 360 times (mt_luts + mt_merge) for a 720*y frame. It should be easier to make a plugin that does the filtering in such a back-injecting way.

When looking at the resulting weights, my guess is that the effect isn't too bad. 50% of the samples used for filtering are already filtered. Those filtered samples contain 25% weight of the actual pixel ... no hard science, but since chroma filtering is quite forgiving anyway, probably it just works out. But nonetheless, a slight color shift towards upper-left is to be expected, roughly in the range of 1/4 chroma pel.

[plugh]

ok, that's what I thought. I've never looked at the avisynth sdk, so don't know how involved it might be...

Glad you concur that it seems a bit odd. I never use the 'Chroma optimizer' but happenned to look at the code the other day and said WTF? Hence my desire to visualize the result...

[sixide]

I did a single frame test in MATLAB. I found a JPEG screenshot of No Country For Old Men with Google.

I am assuming that images are indexed the same way, from top left to bottom right.

I used a bilinear resize for upsizing/downsizing the chroma planes, and the rgb2ycbcr/ycbcr2rgb functions for conversions. I know that Cb/Cr are not U/V, but the effect on those planes will be the same. The effect on the final color is probably not.

Original:
http://img186.imageshack.us/my.php?image=nocountryforoldmeniq6.jpg
Optimized:
http://img149.imageshack.us/my.php?image=ncfomopthh1.png
Scaled Cb difference:
http://img183.imageshack.us/my.php?image=ncdiffuhg3.png
Scaled Cr difference:
http://img338.imageshack.us/my.php?image=ncdiffvws8.png
Original Cb:
http://img339.imageshack.us/my.php?image=ncumw5.png
Original Cr:
http://img214.imageshack.us/my.php?image=ncvwn5.png
Optimized Cb:
http://img214.imageshack.us/my.php?image=ncoull2.png
Optimized Cr:
http://img297.imageshack.us/my.php?image=ncovbe0.png

The max diff for luma was +/-0, Cb was +9/-39, Cr was +5/-8.

As to why this is "optimized"? I don't know, I'm curious. Isn't this pretty much a chroma smoothing operation?

[Dark Shikari]

I would assume it makes the chroma easier to encode for Xvid, maybe.

[gzarkadas]

What the algorithm does (note that x,y range as [1..width/2) and [1..height/2) respectively) is (in pseudo-code):


ForEach interior chroma pixel (ie for all interior 2x2 luma pixel blocks)
If all luma pixels in the 2x2 pixel block are pure black or white Then
chroma pixel value = Average of adjacent (ie +/-1) left,top,right,bottom chroma pixels


I havent studied xvid's code so I can only guess that the intention of the code is to balance somehow the chroma of image areas that exhibit limiting luma values.

[plugh]

I did a single frame test in MATLAB. [...]Isn't this pretty much a chroma smoothing operation?
wow, thanks!... Looking at the images, nothing jumps out at me, though.

ForEach interior chroma pixel (ie for all interior 2x2 luma pixel blocks)
If all luma pixels in the 2x2 pixel block are pure black or white Then
chroma pixel value = Average of adjacent (ie +/-1) left,top,right,bottom chroma pixels

What you describe would be the case if the result of each pixel calc was stored in a seperate (ie a new) image array. But since each calc'd pixel is stored back into the original image, pixels down and to right make use of modified pixels to left and up (which were calc'd from virgin pixels to right and down).

edit: example - take for example chroma at [x=2,y=2], here's where portions of its value gets spread to...
1 2 3 4 5

1 0 1/4 1/16 1/64 1/256
2 1/4 1/8 3/64 1/64 5/1024
3 1/16 3/64 3/128 5/512
4 1/64 1/64 5/512 5/1024
5 1/256 5/1024

[sixide]

It seems to me that it should work on a copy of the original frame, too.

It is probable that not many frames have huge areas of luma outside (16,135), so the propagation may be minimal.

[plugh]

I agree, and also considering the magnitudes involved (ie 8 bit) propogation is probably at most three cells. But if the intent, for example, was for [2,2] to use the original [2,1] and [1,2], they've (potentially) been clobberred by the time it gets to [2,2].

Anyway, Thanks all...

[gzarkadas]

Well, no need to guess anymore. After reviewing my understanding of the chroma optimisation algorithm thanks to plugh's correction, I gave up my attempts to create a script-based solution and made a plugin for this.

The plugin is attached in the message; it contains the source, a debug-dll and three test-case avs scripts. It is a fresh alpha, so please be kind with me :) ; any comments are welcomed of course.

[plugh]

Wow, thanks. I'll play with it a bit, but at this point it seems pretty certain that the xvid code is borked - I sent an message to the mailing list...

[gzarkadas]

I agree with you. I downloaded the xvid source (1.1.3) and from the comments just above the function body it seems to me to be the case that the intention was to produce a (non-recursive) closest -neighbourhood average but the fact of the in-place substitution was not taken in account.

I have made an updated version of the plugin that supports both modes of averaging, recursive (like xvid's current) and non-recursive (as I pressume it should be). It also fixes some bugs discovered. Attached.

From the tests I have made so far the second, non-recursive, mode produces a more calm effect located only at the frontier of black/white regions with other colors. I upload an image as example, produced by the following script:


LoadPlugin("chroma_optimize.dll")
LoadPackage("avslib", "array")
LoadModule("avslib", "filters", "stack")

# test clip generation

bottom = BlankClip(width=320, height=240, pixel_type="YV12", color=color_orange)
top_w = bottom.BlankClip(width=60, height=60, color=color_white)
top_b = bottom.BlankClip(width=60, height=60, color=color_black)

Overlay(bottom, top_w, x=60, y=60)
Overlay(top_w, x=180, y=30)
Overlay(top_b, x=180, y=140)
Overlay(top_b, x=130, y=80)
Overlay(bottom.BilinearResize(120, 40), x=132, y=90)

# filter, diff, scale diffs, add borders & stack

src = ArrayCreate(last, last)
dst = ArrayOpArrayFunc(src, "true,false", "ChromaOptimize") # [ old (xvid) , new ] style
diff = ArrayOpArrayFunc(dst, src, "Subtract").ArrayOpFunc("Levels", "124,1,132,0,255")
Stack(ArrayPlex(src, dst, diff).ArrayOpFunc("AddBorders", "2,2,2,2"), 2, 3)

# output frame

ConvertToRGB24()
ImageWriter("comp", 1, 1, "png", false)


It also results in slightly more compressible encoded videos compared to unfiltered ones at least when encoding to MPEG4 (with xvid) and also to MOV (with ffmpeg); this was not observed when encoding to MPEG2 or WMV (with ffmpeg). However my test examples are very few and thus these results should be considered as preliminary only and possibly subject to future revising.

Thus a possible use of the plugin is as a final step in processing to squize a few more bytes from the encoded clip's size if the target format allows it. And of course as a replacement to the xvid's "optimize chroma" option for all those that agree with the view that xvid's implementation is not correct.

[plugh]

That image really drives it home!

I'm amazed that it seems the ENTIRE 'is pure' region is affected by the current algorithm.

Thanks again!

[henryho_hk]

Since we have interlaced encoding for XviD, will this plugin have an interlace mode too? ^_^

[henryho_hk]

And... will it be ported back to XviD?

Edit: If we cache the original values of the last row and the "left" previous chroma pixels, there seems no need to work on an whole new frame buffer.