VHQ manual

[sysKin]

: this info is absolete. VHQ always improves quality, works with GMC and in fact also with b-frames, and it has no known bugs

Hey.
I welcome every xvid alpha tester here.

As you might have noticed, the newest uManiac's build contains a new feature called VHQ.

VHQ stands or whatever you want, but Koepi said it's 'vastly hyped quality'. As far as I'm concerned it might mean 'very handsome qaligosaur'.

Important is what it does.
It does two things - first, it is able to compute the number of bits a macroblock would take in different scenarios and choose the scenario which shows the smallest number. Second, it can do some limited motion vector seach to minimize the bits for a given scenario. The 'scenarios' are macroblock modes - there can be one vector (we can look for it using this tool), four vectors (we can also look for them) or the macroblock can be intra (nothing to look for here).
As you can see (I hope?) it doesn't change quality at fixed quantizer - it reduces number of bits at fixed quantizer.

VHQ is only used in P-frames. In I-frames it doesn't have anything to do. It is possible in B-frames, but it's not there (yet? dunno, don't look at me).

It works with bframes, qpel, chroma, lumimasking etc.
It doesn't work with GMC yet. <-- until further notice. please remember that. I mean it.

There are several possible levels of additional search. Koepi and I decided to make four levels in the configuration - first is mode decision alone (without any search), next include additional search patters and are much slower.

OK, what have I forgotten?
Ah yes.
This is the first release and it's obviously buggy Current number of known bugs is: 0. Number of expected bugs is about 7649321 and a half.

If you feel like testing, you're welcome. Please share your comments and suggestions.
If you see a bug, do the following:
1. make sure GMC is disabled
2. check if it hasn't been reported before
3. report it here

And most importantly: Have fun testing.

Best regards,
Radek 'sysKin' Czyz

[drebel]

Reporting black(dark luma?)flashing macroblocks with version 0.2 1st pass(quant2),b-fr(3,150,100),cromaME,s.effort4.Filesize seems smaller compared to other buils with the same settings but quality isnt the same.But one should have a pretty sharp eye to tell the difference(at 25 fps)
Speed drop from 7-8 fps to 5 fps

[MaTTeR]

Testing a full 2-pass on two different machines. I'm anxious to see the results:)

Is there any room for speed improvement sometime in the future, in regards to VHQ? On both of the dual CPU rigs here, the CPU usage dropped from to about a fourth of what was being used previously. Just from what I read above I would have expected full CPU usage but then again I don't understand coding;) Thx for the new feature.

[Mango Madness]

Is it just me or is it nearly impossible to DL binaries from his site? Any mirrors available? I know this has been brought up before, if my memory serves me right. I shall commence testing once it finishes the infinitely (limit of x->0, oh wait, that's calculus crap) long download. Alpha tester, i like that title.

[angelyote]

Sadly this build (12022003 15:00) just crashes vdubmod and avs2avi no matter which options I selected. It crashed in 1-pass quant, 2-pass 1st pass.
*Did not crash during Null Speed Test but crashed when it finished.*

My most barebone attempt was:
loadplugin(mpeg2dec3.dll)
mpeg2source(mymovie.d2v)

and then ultra-high, h.263, b-frames off, all xvid options off (yes, gmc off) and VHQ at 0.

my system is a p4-2ghz, 512meg of memory and lots of gig of drive space free.

Dave

P.S. I will not be defeated. Gonna keep altering things until this works :)

[gamr]

nice results so far, about to try psnr tests to check that what im seeing isnt just me being optimistic but here's something to help those out there wondering if its useful

mode0 41084k
mode1 40048k
mode2 39622k
mode3 39378k
mode4 39298k

once i do the psnr ill report back again. nice compressability gain there. good work! :D

[seewen]

I just launched the 2-pass encode now, with VHQ mode set on "4 - Wide Search".
Other options : B-frames 3/150/100, ChromaME, MCP 6, DX50 B-VOP Compatibility = NO, Alt Curve = Medium ( default setting for the rest).

And the speed is quite the same compared with 2.2.2003 Koepi build ( same options, without VHQ ).

It's a very good surpise ;) ( i was sure that the speed would be 1/10, or something like that ;) )

AvScript

Mpeg2Source("F:\EOTS_D3_JAMES\XVID\Ennemi_Etat.d2v",cpu=6)
Crop(8,72,-8,-72)
Lumafilter(-2)
Undot()
SimpleResize(688,320)
Unfilter(10,10)
Limiter()


About 10-15 fps on an AMD ATHLON XP ( + Avisynth 2.5).


Bye, and thanks to all the dev !!


P.S.
Just made a short compressibility test, and it seems that "VHQ mode 4" make a gain of about 6-7% of compressibilty, maybe more.. But I'm not really sure of it ( I didn't remember exactly the result of the previous test )

[MaTTeR]

I'm seeing about a 4% compressibility change with mode 2 on a full encode two pass encode. Maybe it's just me but the visual quality looks great particularly the stable backgrounds such as walls. I'm noticing any flickering or moving blocks. Time to start another encode with mode 3.

BTW- The speed is more than acceptable here.

[ookzDVD]

I still have problem with uManiac's XviD.Alpha.12.02.2003.1500.exe build. The VirtualDubMod (31012003) suddently crash on the 2pass-1st pass. Avisynth 2.5 (dated 28/01/2003). MarcFD's MPEG2DEC3 (version 1.0).

Chroma, B-Frame(3/150/100), VHQ=4.

Even I've already clicked the "load default" button.

PS: Win2k SP3 on Intel P4 1.7Ghz.

[Bulletproof]

I just tried a Mode 1 encode w/ constant quantizer 2 and it gave me about 6% off the file. I was able to notice a degrade in quality, however it's not that bad and it's pretty acceptable. This should help alot in those long movies. All the new options have made XviD come a long way, although each option takes off a small amount of the size, when combined they offer great compression.

[vinks]

Originally posted by ookzDVD
I still have problem with uManiac's XviD.Alpha.12.02.2003.1500.exe build. The VirtualDubMod (31012003) suddently crash on the 2pass-1st pass. Avisynth 2.5 (dated 28/01/2003). MarcFD's MPEG2DEC3 (version 1.0).

Chroma, B-Frame(3/150/100), VHQ=4.

Even I've already clicked the "load default" button.

PS: Win2k SP3 on Intel P4 1.7Ghz.


i had a similar problem on my p4 as well, disabling sse2 sorts it out and all is well.

(running xp sp1 on a p4 2.4ghz (northwood) )

[ookzDVD]

@vinks,

Thank you for the reply, I just turn off the SSE2,
and everything is working well now :)
The Intel is office PC. :)

I just not try with my home PC yet, it's Athlon! :)

[Teegedeck]

4.5% filesize reduction with NO visible reduction of quality, here.

Encoded at quants 2 & 5, qpel and B-frames, with wide search range. Speed drop about 1/3. Now that's what I call an acceptable speed drop for the improvement, especially if I compare it to something called 'nth pass'...

Gain without pain. :D Thank you, sysKin! F-A-B-U-L-O-U-S.

[NuclearFusi0n]

Originally posted by Teegedeck
...I compare it to something called 'nth pass'...
ahahahahahah - it's so fun to chill in the DivX forums and laugh at them :D

[OUTPinged_]

to compute the number of bits a macroblock would take in different scenarios and choose the scenario which shows the smallest number. Second, it can do some limited motion vector seach to minimize the bits for a given scenario. The 'scenarios' are macroblock modes - there can be one vector (we can look for it using this tool), four vectors (we can also look for them) or the macroblock can be intra (nothing to look for here).

Doesn't that approach break compatibility with iso mpeg4(ie standalones)?

Did a quick check at q2 - it seems static scenes dont benefit alot from that - 3% for a clip with static pictures, 5% for himo-only clip at quant 2. I can see differences when i really look for them but i am doubt that would be noticable in dynamic.

(And IMO "VHQ" was a bad way to name it. Let me guess, 101% of idiots will enable it to 4th settings because they will think it means "very high quality".)

[Teegedeck]

Originally posted by OUTPinged_

Doesn't that approach break compatibility with iso mpeg4(ie standalones)?
No, as this is a more precise/economical mode-decision and search, nothing more. The resulting bitstream should be indistinguishable from an 'ordinary' encode.

(And IMO "VHQ" was a bad way to name it. Let me guess, 101% of idiots will enable it to 4th settings because they will think it means "very high quality".)
Something like 'very economic' would of course be more true to what it actually does. But on the other hand, if you go for a certain filesize it is a more 'HQ' way of reaching that filesize. Without VHQ, some higher quantizers would be used to reach that filesize, and this would be clearly noticeable.

[Didée]

Two questions:

First:
VHQ does some motion vector search, okay.
But what about the "normal" setting for ME precision? Is that related somehow, is there any influence, or does the setting for conventional ME not apply at all when VHQ is activated? Or works VHQ as a kind of pre-calculator for the normal ME?
Call me dumb, but this is not clear to me from syskin's explanation above ...

Second:
I noticed that when encoding a 3000frames-sequence with VHQ=4, final size ~ 30'000 kB, window's explorer showed me a filesize > 200'000 kB (!) during encoding.
How big may the file get during conversion? Could be of importance in low diskspace scenarios ...


Apart from that: hat off again, XviD team!!!

Regards

Didée


P.S.

"nth pass encoding" , huhuhuuu :D good joke, really ...

[OUTPinged_]

Ok, i wanted to make more tests. For some strange reason i had an impression that VHQ would work best on "hard to ME" material like shaky VHS captures with rotations, zooms and shakes. I picked up new year celebration DVD i had at hand (the cameraman was drunk so it was full of k3wl zooms and shakes :-) and checked with it.

VHQ=0: 6,50fps 5,552,128b
VHQ=1: 4.92fps 5,412,352b
VHQ=2: 3.58fps 5,251,584b
VHQ=3: 2.78fps 5,196,800b
VHQ=4: 2.20fps 5,067,776b

There. At quant2 filesize difference between 0 and 4 was 9.5%. Increasing quantizer to 5 it increased to 10.9% but when futher increasing quantizer to 10 it suddenly decreased to 4.7% - I dont know yet why did that happen.




About quality:

Comparing VHQ0 to VHQ4 PSNR (I know it's lame but anyway :-) was:
q2: 46.01-47.24 range
q5: 42.86-43.46 range
q10:39.90-41.87 range

Here are how the modes decrease PSNR (average for 10 frames):

Quant=2
VHQ0: 45.946239
VHQ1: 45.976748 - hehe. VHQ1 mode was better than VHQ0 here =)
VHQ2: 45.721406
VHQ3: 45.667344
VHQ4: 45.663510

Quant=5
VHQ0: 42.234175
VHQ1: 42.228793
VHQ2: 41.824608
VHQ3: 41.800283
VHQ4: 41.779857

Quant=10
VHQ0: 39.770136
VHQ1: 39.582636
VHQ2: 39.237580
VHQ3: 39.190024
VHQ4: 39.118455

At quant2 I couldnt see difference watching clip, and i could clearly point at differences only when zoomed it to 2x :/

At quant10 it was already crappy and VHQ4 didnt made it any worse :-) Actually i couldnt see it in motion, too.

My verdict: feature has great potential (Up to 10% bitrate savings!), but I should point everyone that some difference is present and it is possible that in some scenes there will be visible quality degradation. Some seroius testing is required here. It is slow, too. I dont see how people manage to get away with only 30% slowdown, since i am getting 6->2 fps drop here :-( I hope that futher optimisation is possible for VHQ (expecially mode4).

[symonjfox]

Originally posted by OUTPinged_
(And IMO "VHQ" was a bad way to name it. Let me guess, 101% of idiots will enable it to 4th settings because they will think it means "very high quality".)

So 4th setting doesn't always give the best quality? I didn't have any time to test it, yet.

Isn't it possible to add an option called 5 - AUTOMATIC where it performs all 4 possible alghoritms and then just uses the one that gave the best quality in that particular scene? It will slow down the encoding (I know), but it should improve the quality.

Another idea is doing something where, performing the first pass of a 2 pass encoding, it saves a VHQ.STAT where there are the results. When performing the 2nd pass, it will speed up because the Xvid already knows the best way to perform the VHQ algorithm, so ...

I studied a little bit of programming, but I'm very very far from being a programmer ... just the idea should be nice (don't you think)? :)

[sysKin]

We've been doing some tests as well. The results are similar:

Level 1 gives both better filesize and better quality (psnr).

All next levels decrese filesize more, but also decrese quality. This is all true.

At this moment I'd say that level 1 is a winner. But, as I said, it's work in progress :)

Radek

[seewen]

I juste made a 2CD encode ( B-frames 5/150/100, DX50 B-VOP : No, ChromaME, VHQ - 4-Wide, alt curve "medium" , and the rest default ).

And the result is really extremely good.

I don't see any quality loss between "VHQ -4" & "VHQ - 0". But the file size was about 8% smaller for the first pass ( = Quant 2 ).

And really, I cannot see any differences loss.
The speed loss is about 20-25%.. Very acceptable for such a compressibility gain.

[OUTPinged_]

@sysKin:

Results for a very first build are real good. And it is Simple Profile compatible too, which makes it expecially valuable.

Good luck at improving it.

[easyfab]

Originally posted by sysKin
We've been doing some tests as well. The results are similar:

Level 1 gives both better filesize and better quality (psnr).

All next levels decrese filesize more, but also decrese quality. This is all true.

At this moment I'd say that level 1 is a winner. But, as I said, it's work in progress :)

Radek

I agree with all that but i have a question. You compare quality (psnr) @ different files size but if I do a 2pass encode @ 650mo for exemple (for all modes) what mode will have the best quality?
mode 1 or mode 4 ? Can someone do a test. I think mode 4 will be the winner.

[OUTPinged_]

VHQ isnt magic, it is a quality/cpu time/filesize tradeoff. ATM, as sysKin mentioned, use it only for testing purposes.

If you feel lucky, then keep in mind that it is supposed to decrease average quantizer value for your movie(it will depend on your skill if that will increase quality or not), at the cost of increased processing time (15-20% on static, up to 200% on dynamic scenes).

Go test it with VHQ=4, why should you hold back?

If we take average gain with VHQ=4, which is 7%, then your average quantizer would decrease, ie, from 5.00(40% 1/2pass ratio) to 4.65(43% 1/2pass ratio). It may look like a little, considering processing time will almost double. We still dont know if it is completely safe to use. But it is a real thing, and not some of those "preprocessing" filters.

[Teegedeck]

sysKin,

could the brute-force approach also do some good for adaptive quantization? (When I read about those MPEG2-transcoders which re-use motion vectors and do nothing but requantization on a macroblock-level with quantization-error-checking - that thought came to my mind.)

[trbarry]

I haven't tried it yet but does anybody notice if VHQ makes Xvid handle fades better? I'd predict it might.

- Tom

[ookzDVD]

Just test full movie, B-Frame:3/100/200, Chroma Enabled,
without VHQ (VHQ=0) the 1st-pass size : 727MB
and with VHQ=4 the 1st-pass size : 680MB,
it's about 6.5% smaller.
The 2nd-pass target size : 385MB, the result is good enough for my eyes :)

And it's my first attempt to make 1CD (700MB) with AC3 sound,
and I'm quite happy with the result.

Salute for XviD Development team!

[OUTPinged_]

@trbarry: sorry but it doesn't. same 10% max at VHQ=4.

[kilg0r3]

@ookzDVD : what res have you used?

[Tommy Carrot]

I did just quick test with quantizer 6 (the quality difference is more obvious at lower bitrates). VHQ4 amplifies somehow the qpel noise quite badly. The speed is dropped from ~15 fps to 5-7. I haven't checked the other VHQ modes yet.

[HarryM]

I tested VHQ too (at level 4, off course:D). Bad idea, use it with qpel feature. Smearing effect (mainly at background) is much bigger.

But for simple profile (no q-pel, no b-frames) is VHQ really good.

[Koepi]

If you use VHQ at level 1, you won't get any smaring or similar artefacts.

Those artefacts can be introduced by VHQ mode 2-4, but they're less noticable in full 2pass scenarios - the overall quality is still better.

Regards
Koepi

[Gazza]

Hi all,
I've completed some initial tests with VHQ and can confirm (not surprisingly) that the final file size can decrease when going from VHQ=0 to VHQ=4. By the way and IMHO, the quality is very very good, especially when coupled with qpel.

However, if VHQ is going to remain (and no reason I can see why it shouldn't) then what advice is available for determining the final file size for different settings of VHQ?

Note, I've compared two different video sources with VHQ=4. The first hit about the right filesize prediction given by Gknot but the second test resulted in a file that was about 8% smaller than what Gknot advised me. 8% doesn't seem unusual when using VHQ=4 (don't ask me why the first test wasn't smaller..... maybe I made a mistake and kept VHQ=0 and didn't realise)?

I would welcome any advice for calculating the filesize when using VHQ=1,2,3,4.... Also, will VHQ affect the filesize for all video sources or is it going to be trial and error with some video sources affected but others not?

Gazza

[Blueseb]

I've done three 2 pass-1st passes on a 5900 frames clip to test compressibility

VHQ -> Video Size
0 -> 36517K
1 -> 35884K
4 -> 34664K

average I-frame and B-frame size is almost the same, vhq seems to shrink principally P-frame size :o
on the quality side, I noticed a small artifact in a single frame (the same with both vhq1 an vhq4) apart of that it looks great to me

just an humble question: does 'unstable' mean that future decoder version may not read correctly these encodings? (I guess not, but I'm unsure since I'm not a programmer :p )

[Animaniac]

Is it possible to change the VHQ setting from the first pass to the second? My gut instinct would say no, but it doesn't hurt to ask. ^_^ Thanks.

[gamr]

no because that changes motion estimations

[kilg0r3]

Originally posted by Blueseb
... average I-frame and B-frame size is almost the same, vhq seems to shrink principally P-frame size :o There is answer to this in the 08032003-build thread in this forum and on my site
on the quality side, I noticed a small artifact in a single frame (the same with both vhq1 an vhq4) apart of that it looks great to meartifacting of vhq is also discussed in the above mentioned thread

just an humble question: does 'unstable' mean that future decoder version may not read correctly these encodings? (I guess not, but I'm unsure since I'm not a programmer :p ) yes and no. generally every feature in deveolpment can of course break compatibility with later builds. with vhq, this is improbable, however, which can be deduced from the way it works. It does not code the frames in a new way but looks for the best possible way to use the present capabilites of the codec.
cheers :)

[crusty]

OUTPinged_:
(And IMO "VHQ" was a bad way to name it. Let me guess, 101% of idiots will enable it to 4th settings because they will think it means "very high quality".)

May I suggest "MDQ"?
MDQ for Mode Decision Quality.
Less suggestive, more to the point of what it actually does.

trbarry:
I haven't tried it yet but does anybody notice if VHQ makes Xvid handle fades better? I'd predict it might.

I'll try it out with some fade material as soon as possible. Why do you think it handles fades better? The most notable artifacts with fades are blocks, not stuff like smearing or posterizing (IMO).

OUTPinged_:
@trbarry: sorry but it doesn't. same 10% max at VHQ=4.
That's purely the filesize decrease, not the noticeable quality.

HarryM:
Bad idea, use it with qpel feature. Smearing effect
Tommy Carrot:
VHQ4 amplifies somehow the qpel noise quite badly.
Gazza:
By the way and IMHO, the quality is very very good, especially when coupled with qpel.

Somehow, this is completely contradictory.
Could you three post what type of material you were testing on? Maybe it's good for some materials combined with Qpel, and bad for some others.
Also, have you tried the same with the latest builds and gotten the same effects?

blueseb:
average I-frame and B-frame size is almost the same, vhq seems to shrink principally P-frame size

As SysKin wrote in the opening post:
VHQ is only used in P-frames.
SysKin also mentioned in another thread that VHQ could probably be made to work with B-frames as well, but I don't know if there's any work done on that, but if the effect would be equivalent with the effect on P-frames, we may see a shrink in filesize of up to 15% total. :cool:

[Gazza]

Crusty,
On following the various threads discussing qpel, vhq, etc I have dropped using these for the moment and gone back to testing with just the defaults.
The reason is that I noticed that encodes with all the bells (qpel, b-frames, etc) had a dramatic effect during replay. The long encode period of ~24 hours was acceptable but during replay I noticed that the video was stopping and starting due to 100% processor usage. And this was without any Post Processing. I also had the final encoded resolution wound right up (720x***) which would not have helped anyway. I have a 1GHz, 512MB P3.
Therefore I have gone back to basics to see what settings suits me personally at this time. I have also decided not to use qpel at all pending better reports from others in this forum. The testing I am doing is to get maximum fps during encoding and then compare the visual quality with an encode that uses b-frames, vhq, etc.
My first impressions are that if you go for a 2 cd encode all the time, then the basic default settings are good enough (keeping resolution sensible, etc). I'm not sure if the gain in final quality is at all noticeable when you start adding b-frames, etc for a 2cd encode. Therefore you can get good encode fps for reasonable quality.
However, if you want to increase your chances of good quality for a 1 cd encode then you need to consider using vhq, b-frames which impacts on your encoding speed?

[cipher]

I also had the final encoded resolution wound right up (720x***) which would not have helped anyway
Gazza, still yet a lotta videocards out there CANNOT handle clips with resol'n higher than 640pixels due to incompatibility of overlay conversions.
:) I've got this experience with my stupid computer, tnt2 + p3 1g + 256m, when playbacking xvid encoded frames with high bitrate (1400kb/s or higher), resol'n higher than 640(like720*), it certainly will be jerking. But my roommate has a geforce4 videocard, 1.7g p4 cpu and 512ddr mem, nothing jerking has ever happened to his machine.
Playing b-frame-and/or-qpel-encoded clips will certainly load more to the cpu and the video card. In addition, resol'n higher than 640 pixels may actually make it incompatible with the vedio card, and the clip may jerk more.(but new, fast video card may handle it well) Your machine is better than mine, for you've got twice as much of your memory as mine, you should be confident that your machine will playback video smoothly one day, for mine now usually do. :)
One last thing, imo, 720 x ** is NOT a correct framesize, because of different pixels used in DVD making and tv or computer monitor(DVD pixels are not squares but strickly rectangles while our screens are perfect squares), and that's the major reason why we should crop and resize it back to 640 or lower.
For more about resolution stuff, please check this out (http://forum.doom9.org/showthread.php?s=&threadid=42708)
Good luck with playback! :)

[Gazza]

Thanks cipher. The article is well known to me and a great resource for information. In this forum you often see people trying resolutions greater than 640. Also (in my experience) when using gknot and following Doom9's excellant guide I tend to get resolutions greater than 640 as the preferred for a particular video source. Therefore there is a tendency to 'push the envelope'. However, the danger is that you forget the basics, end up with crappy results and then start to question what xvid is capable of.

For me (and until I can get my hands on some decent hardware), I have gone back to basics to refresh my understanding of whats possible and determine how to get the best from xvid with the hardware I have at the moment.

I suppose that one rule of thumb is to keep in mind that if a new feature is released it could put an additional strain on your existing hardware which means that your current xvid settings are no longer valid. The trap I fell into was not reviewing my whole setup (e.g. maybe lowering the resolution) before saying there was a bug in the software (when there wasn't).

Thanks for your advise and help.

I'm hoping that my experiences will help others who have seen similar problems.

Cheers & I hope this all makes sense.

[crusty]

I don't want to be picky, but a tnt2 is hardly a state of the art videocard anymore....so you can hardly expect people to try to remain compatible with a 4 year old videocard.

For correcting AR errors, I find picking a part of the movie with some true round objects in them , like cannon barrels or lamps, works like a charm.

[AlienTiger]

woot i can finally post :)

sorry to revive such an old topic, but since so many guides still link here and the last post was made nearly 2 years, I was just curious how accurate this information is now

[Manao]

Now VHQ is usable in P and B frames ( bVHQ in 1.1 beta builds ). It's also usable with GMC and with in fact any xvid's options. And VHQ's quality at fixed filesize raises with VHQ level ( hence, VHQ4 + bVHQ will always be the best, but also the slowest ).

[AlienTiger]

thanks, glad to kno all of the former issues have been resolved ;)