VAQ: Making Xvid's Adaptive Quantization Not Suck

[Dark Shikari]

I was bored, so I spent 15 minutes and wrote this patch to port x264's new VAQ to Xvid.

I haven't tested it much (only on this video), but it seems to work surprisingly well, despite MPEG-4 ASP's drastic limitations on quantizer deltas.

Patch [updated with minor fixes]

Xvidcore.dll [updated with latest patch]

To use it with Virtualdub or any VfW interface, just overwrite the XvidCore.dll in your /Windows/System32 or similar with the one I have provided.

To activate VAQ, just check off the Adaptive Quantization box; it replaces the normal luminance masking.

Note that I could not get it working correctly with mencoder; for some reason libavcodec seems to have its own lumimasking library that's overriding my patch. It works with xvid_encraw and VfW though...

Have fun!

[squid_80]

If people want, I can add it to my xvid_encraw as a new option. Unfortunately I have no time to do my own tests at the moment.

[Dark Shikari]

Originally Posted by squid_80

If people want, I can add it to my xvid_encraw as a new option. Unfortunately I have no time to do my own tests at the moment.

In the patch's current form it overwrites lumimasking, so one would need a separate version (it uses the same commandline as lumimasking).

[squid_80]

Originally Posted by Dark Shikari

In the patch's current form it overwrites lumimasking, so one would need a separate version (it uses the same commandline as lumimasking).

The beauty of the xvid's plugin system - I can add however many plugins I want to encraw and xvidcore.dll doesn't need to be updated.

[kypec]

Originally Posted by squid_80

If people want, I can add it to my xvid_encraw as a new option. Unfortunately I have no time to do my own tests at the moment.

Yes, I'd really appreciate that xvid_encraw version with separate VAQ option. The current state where it overrides lumimasking option is not a good solution IMHO. One could easily forget about it and be VERY confused with outcoming encodes later

[Brother John]

@Dark Shikari
I see. So VAQ basically is about as close to CRF as Xvid can get. Those bad quant restrictions indeed are one of the reason why I’d like to abandon Xvid. There are more important ones making me stay, though. And VAQ is definitely one more of those.

Originally Posted by Ranguvar

Same thing with QPel. Not enough benefit

QPel usually doesn’t even improve compression but lowers it! However it makes the picture look a bit grittier. It’s a kind of pseudo-detail and if you like the effect, QPel is a good thing.

Originally Posted by Prettz

ASP I don't think was ever intended for the kind of high-quality videos Xvid initially allowed to be produced.

When reading the MPEG-4 Visual specs I get the impression that MPEG was more concerned with the whole non-rectangle video/compositing thing than plain old movie encoding. Maybe the restrictions make more sense from that point of view.

[Zep]

Originally Posted by Dark Shikari

I was bored, so I spent 15 minutes and wrote this patch to port x264's new VAQ to Xvid.

I haven't tested it much (only on this video), but it seems to work surprisingly well, despite MPEG-4 ASP's drastic limitations on quantizer deltas.

can you link to the without patch encode for comparison?

[elguaxo]

[Dark Shikari]

Originally Posted by Zep

can you link to the without patch encode for comparison?

That is the x264 version of the video

Do Xvid test encodes yourself.

[Ranguvar]

Awesome!

But I SO wish you had done this before I backed up a bunch of stuff to Xvid

Squid_80, please do. I will test if I get any time, which is most likely as I have this week off.

Dark Shikari, can this Xvidcore be used against any version of the Xvid codec? I have a recent build of 1.2.-127.

[Dark Shikari]

Originally Posted by Ranguvar

Awesome!

But I SO wish you had done this before I backed up a bunch of stuff to Xvid

Squid_80, please do. I will test if I get any time, which is most likely as I have this week off.

Dark Shikari, can this Xvidcore be used against any version of the Xvid codec? I have a recent build of 1.2.-127.

This is the latest CVS snapshot of Xvid + my patch.

[G_M_C]

Originally Posted by Dark Shikari

I was bored, so I spent 15 minutes and wrote this patch to port x264's new VAQ to Xvid[...]

OMG

I'd allmost wish you'd be bored more often.

.... But on the other hand ...

If you do things like these VAQ-patches for X264 & Xvid while bored ... i'd hate to think of things you could do while really interested



Anyway,just a question; Could this same patch / idea be ported to Hank's HCEnc MPEG2 encoder ? I use that one quite often to make (author) DVD's.

I use Hanks luminance-adative code, wich works surprisingly well. But that code works only on "darker" scenes, not on flat(ter) scenes like the VAQ does. I would be really interested in such a feature

[Dark Shikari]

Originally Posted by G_M_C

Anyway,just a question; Could this same patch / idea be ported to Hank's HCEnc MPEG2 encoder ? I use that one quite often to make (author) DVD's.

I don't see why not...

All I need is a link to the source.

Originally Posted by G_M_C

If you do things like these VAQ-patches for X264 & Xvid while bored ... i'd hate to think of things you could do while really interested

Well x264's took quite a while--but once I had it working I can easily port it to any other similar encoder with very little work.

[hank315]

I'm already working on it for some time, the general idea is about the same, variance based.
Will be in the next HCenc release.

Just had a look at the implementation in Xvid, well, I'm a C n00b but when I do something like this in Fortran it will overflow on high luminance values:

  for (k = 0; k < 16; k++)
      for (l = 0; l < 16; l++)
         {
             int val = ptr[k*data->current.stride[0] + l];
             sum += val;
             sum_of_squares += val * val;
         }

  /* Variance = SSD - SAD^2 / (numpixels) */
  int variance = sum_of_squares - sum * sum / 256;

[Dark Shikari]

Originally Posted by hank315

I'm already working on it for some time, the general idea is about the same, variance based.
Will be in the next HCenc release.

Just had a look at the implementation in Xvid, well, I'm a C n00b but when I do something like this in Fortran it will overflow on high luminance values:

  for (k = 0; k < 16; k++)
      for (l = 0; l < 16; l++)
         {
             int val = ptr[k*data->current.stride[0] + l];
             sum += val;
             sum_of_squares += val * val;
         }

  /* Variance = SSD - SAD^2 / (numpixels) */
  int variance = sum_of_squares - sum * sum / 256;

The maximum value that sum_of_squares can reach is (255 * 255 * 256), or a bit under 2^24, so it is not a problem.

The maximum value that sum can reach is 255*256... so yes, you are right; sum*sum could overflow a signed integer.

The correction would be to make sum and ssd unsigned integers.

[squid_80]

If the product is going to be divided by 256, why not bitshift before the multiply?

[elguaxo]

Originally Posted by Dark Shikari

Edit: I accidentally left a debug statement in there, so if you're using xvid_encraw... well... you might notice "THISISATEST" constantly printing

I'm using xvid_encraw. I have no idea how to compile this myself. Could you or somone else post a binary without the debug text? Thanks in advance!

[TheRyuu]

I think that patch isn't really proper (at least it didn't work for me).
After some slight editing of it though I managed to get it to work (with a fuzz with the last hunk, but I checked the luminance_masking.c and everything was patched correctly)

Now off to test. I think the patch doesn't have that debug line Dark was talking about so I'll post a build in a bit I guess and see if that helps (it's only a debug line though so does it really matter?)

Edit:
Build:
http://www.fileducky.com/qRILYtae/

It's built with pthreads enabled.

[Dark Shikari]

Originally Posted by wizboy11

I think that patch isn't really proper (at least it didn't work for me).
After some slight editing of it though I managed to get it to work (with a fuzz with the last hunk, but I checked the luminance_masking.c and everything was patched correctly)

Now off to test. I think the patch doesn't have that debug line Dark was talking about so I'll post a build in a bit I guess and see if that helps (it's only a debug line though so does it really matter?)

Yes, I removed the debug line in the patch so that people didn't accidentally include it Only my build has it.

[kypec]

Originally Posted by wizboy11

Build:
http://www.fileducky.com/qRILYtae/

It's built with pthreads enabled.

Does that mean your build enables SMP (multithreaded) encoding?
If yes could you please upload it somewhere else?
The link doesn't work for me at all.