VAQ: Making Xvid's Adaptive Quantization Not Suck

[Dark Shikari]

I was bored, so I spent 15 minutes and wrote this patch to port x264's new VAQ to Xvid.

I haven't tested it much (only on this video), but it seems to work surprisingly well, despite MPEG-4 ASP's drastic limitations on quantizer deltas.

Patch [updated with minor fixes]

Xvidcore.dll [updated with latest patch]

To use it with Virtualdub or any VfW interface, just overwrite the XvidCore.dll in your /Windows/System32 or similar with the one I have provided.

To activate VAQ, just check off the Adaptive Quantization box; it replaces the normal luminance masking.

Note that I could not get it working correctly with mencoder; for some reason libavcodec seems to have its own lumimasking library that's overriding my patch. It works with xvid_encraw and VfW though...

Have fun!

[squid_80]

If people want, I can add it to my xvid_encraw as a new option. Unfortunately I have no time to do my own tests at the moment.

[Dark Shikari]

Originally Posted by squid_80

If people want, I can add it to my xvid_encraw as a new option. Unfortunately I have no time to do my own tests at the moment.

In the patch's current form it overwrites lumimasking, so one would need a separate version (it uses the same commandline as lumimasking).

[Zep]

Originally Posted by Dark Shikari

I was bored, so I spent 15 minutes and wrote this patch to port x264's new VAQ to Xvid.

I haven't tested it much (only on this video), but it seems to work surprisingly well, despite MPEG-4 ASP's drastic limitations on quantizer deltas.

can you link to the without patch encode for comparison?

[elguaxo]

[Dark Shikari]

Originally Posted by Zep

can you link to the without patch encode for comparison?

That is the x264 version of the video

Do Xvid test encodes yourself.

[Ranguvar]

Awesome!

But I SO wish you had done this before I backed up a bunch of stuff to Xvid

Squid_80, please do. I will test if I get any time, which is most likely as I have this week off.

Dark Shikari, can this Xvidcore be used against any version of the Xvid codec? I have a recent build of 1.2.-127.

[Dark Shikari]

Originally Posted by Ranguvar

Awesome!

But I SO wish you had done this before I backed up a bunch of stuff to Xvid

Squid_80, please do. I will test if I get any time, which is most likely as I have this week off.

Dark Shikari, can this Xvidcore be used against any version of the Xvid codec? I have a recent build of 1.2.-127.

This is the latest CVS snapshot of Xvid + my patch.

[squid_80]

Originally Posted by Dark Shikari

In the patch's current form it overwrites lumimasking, so one would need a separate version (it uses the same commandline as lumimasking).

The beauty of the xvid's plugin system - I can add however many plugins I want to encraw and xvidcore.dll doesn't need to be updated.

[G_M_C]

Originally Posted by Dark Shikari

I was bored, so I spent 15 minutes and wrote this patch to port x264's new VAQ to Xvid[...]

OMG

I'd allmost wish you'd be bored more often.

.... But on the other hand ...

If you do things like these VAQ-patches for X264 & Xvid while bored ... i'd hate to think of things you could do while really interested



Anyway,just a question; Could this same patch / idea be ported to Hank's HCEnc MPEG2 encoder ? I use that one quite often to make (author) DVD's.

I use Hanks luminance-adative code, wich works surprisingly well. But that code works only on "darker" scenes, not on flat(ter) scenes like the VAQ does. I would be really interested in such a feature

[Dark Shikari]

Originally Posted by G_M_C

Anyway,just a question; Could this same patch / idea be ported to Hank's HCEnc MPEG2 encoder ? I use that one quite often to make (author) DVD's.

I don't see why not...

All I need is a link to the source.

Originally Posted by G_M_C

If you do things like these VAQ-patches for X264 & Xvid while bored ... i'd hate to think of things you could do while really interested

Well x264's took quite a while--but once I had it working I can easily port it to any other similar encoder with very little work.

[hank315]

I'm already working on it for some time, the general idea is about the same, variance based.
Will be in the next HCenc release.

Just had a look at the implementation in Xvid, well, I'm a C n00b but when I do something like this in Fortran it will overflow on high luminance values:

  for (k = 0; k < 16; k++)
      for (l = 0; l < 16; l++)
         {
             int val = ptr[k*data->current.stride[0] + l];
             sum += val;
             sum_of_squares += val * val;
         }

  /* Variance = SSD - SAD^2 / (numpixels) */
  int variance = sum_of_squares - sum * sum / 256;

[elguaxo]

Originally Posted by Dark Shikari

Edit: I accidentally left a debug statement in there, so if you're using xvid_encraw... well... you might notice "THISISATEST" constantly printing

I'm using xvid_encraw. I have no idea how to compile this myself. Could you or somone else post a binary without the debug text? Thanks in advance!

[TheRyuu]

I think that patch isn't really proper (at least it didn't work for me).
After some slight editing of it though I managed to get it to work (with a fuzz with the last hunk, but I checked the luminance_masking.c and everything was patched correctly)

Now off to test. I think the patch doesn't have that debug line Dark was talking about so I'll post a build in a bit I guess and see if that helps (it's only a debug line though so does it really matter?)

Edit:
Build:
http://www.fileducky.com/qRILYtae/

It's built with pthreads enabled.

[Dark Shikari]

Originally Posted by wizboy11

I think that patch isn't really proper (at least it didn't work for me).
After some slight editing of it though I managed to get it to work (with a fuzz with the last hunk, but I checked the luminance_masking.c and everything was patched correctly)

Now off to test. I think the patch doesn't have that debug line Dark was talking about so I'll post a build in a bit I guess and see if that helps (it's only a debug line though so does it really matter?)

Yes, I removed the debug line in the patch so that people didn't accidentally include it Only my build has it.

[Dark Shikari]

Originally Posted by hank315

I'm already working on it for some time, the general idea is about the same, variance based.
Will be in the next HCenc release.

Just had a look at the implementation in Xvid, well, I'm a C n00b but when I do something like this in Fortran it will overflow on high luminance values:

  for (k = 0; k < 16; k++)
      for (l = 0; l < 16; l++)
         {
             int val = ptr[k*data->current.stride[0] + l];
             sum += val;
             sum_of_squares += val * val;
         }

  /* Variance = SSD - SAD^2 / (numpixels) */
  int variance = sum_of_squares - sum * sum / 256;

The maximum value that sum_of_squares can reach is (255 * 255 * 256), or a bit under 2^24, so it is not a problem.

The maximum value that sum can reach is 255*256... so yes, you are right; sum*sum could overflow a signed integer.

The correction would be to make sum and ssd unsigned integers.

[squid_80]

If the product is going to be divided by 256, why not bitshift before the multiply?

[Dark Shikari]

Originally Posted by squid_80

If the product is going to be divided by 256, why not bitshift before the multiply?

Rounding error. Same problem occurred in x264; that was the fix between 0.46 and 0.47.

[squid_80]

Divide by 256, with rounding:
(x+128)>>8

No good?

[Dark Shikari]

Originally Posted by squid_80

Divide by 256, rounding upwards:
(x+128)>>8

No good?

It won't give the same result as multiplying before shifting, IIRC.

Patch updated with unsigned fix.