bitrate at xvid

[roy]

I have tried latetest version of Xvid, and have problem with bitrate.

I have encoded film, where the first scene was very dynamic, battle scene, and next was very quaiet. In dynamic scene bitrate was very high, too high, but the quality was, of couse, great, but in next scene, bitrate was very low, so I got a lot of macroblocks.

Can anyone tell how to configure codec to avoide problem.

[Teegedeck]

Originally posted by roy
I have tried latetest version of Xvid, and have problem with bitrate.

I have encoded film, where the first scene was very dynamic, battle scene, and next was very quaiet. In dynamic scene bitrate was very high, too high, but the quality was, of couse, great, but in next scene, bitrate was very low, so I got a lot of macroblocks.

Can anyone tell how to configure codec to avoide problem.
Can you describe how you encoded? One-pass, two-pass?

[Teegedeck]

VBR? Lumi-masking, no lumi-masking?

Upside down in a bucket of piranha-fish? 8)

[roy]

I have encoded it 2 pass, first using lumi masking and second withpot it, but for first scene bitrate was still very high and for second I got macroblockes.

[ProfDrMorph]

Luma masking does alter the quantizers for each macroblock based on it's brightness. Doing one pass with luma masking and one pass without it surely makes the ".stats" file wrong in a way since the information the codec gathered differs in a way from what the codec compresses in the second pass.

[-h]

If you're going to use lumi masking with the goal of high-quality output, I'd consider setting the maximum quantizer to 6 or 7 (Advanced tab). This 'reigns in' the somewhat aggressive lumi masking scheme that's currently in place.

This is essential for a 2-CD rip, which can still fall prey to blocky dark areas if lumi masking is left to do whatever it feels like.

-h

[Teegedeck]

Yes, and personally I wouldn't recommend the use of lumi-masking for 2 CDs, either. Perhaps lumi-masking can be helpful in some 1CD-encodings, it the movie is quite bright. I found it interesting that while using lumi-masking in first _and_ second pass yields terrible results, using it only in second pass helps a bit on small bitrates, sometimes.

I think, what I just called 'terrible' results, comes from the fact that activated lumi-masking in first pass already shrinks the bitrate for dark scenes down to a size that is _low_enough_, but GKnot treats all frames alike when it calculates for 2nd pass. That means for example a bright frame where lumi-masking won't kick in will get shrunk by 50% for second pass. OK, that's how it's supposed to work. But a dark frame will be 50% smaller with lumi-masking than without it in the 1st pass already! And GordianKnot cuts it down by another 50%! The result looks no good in the second pass.

On the other hand, lumi-masking only in second pass just makes it easier for the codec to meet the bitrates GKnot calculated correctly and maybe even accumulates bits in the bit-bucked that get re-distributed and results in at least some frames with lower quantizers. Way too vague to be really recommended as a method (actually it messes up the concept of GKnot), but interesting.

[Doom9]

if we're at it.. maybe somebody could give me some pointers for optimal xvid settings for my upcoming codec comparison.. so far I haven't even touched xvid. tomorrow I try to get the 1cd rips in wm8 and vp3 (haven't used that before either) done

[Teegedeck]

Originally posted by Doom9
if we're at it.. maybe somebody could give me some pointers for optimal xvid settings for my upcoming codec comparison.. so far I haven't even touched xvid. tomorrow I try to get the 1cd rips in wm8 and vp3 (haven't used that before either) done

Allright! Let us all throw some settings at ya!

I'd say, deactivate lumi-masking in any case!
No need to restrict the quantizers in debug, if you deactivated lumi-masking.
For motion-search intensity, choose '5' ('highest', I believe); '6' (ultra-high?) just re-checks and can save a bit or two but really not much and is slower.
For a quantizer, choose 'MPEG'.

The first pass stats file can be treated like a Nandub stats file in GKnot and also calculated like DivX 3.11. For second pass you don't have to enter a bitrate, but it's important that in the config dialogue you leave the first pass stats in the '1st pass stats file' line and choose the stats for second pass in the appropriately named '2nd pass stats file' line.

Sorry if I didn't get the exact names of everything in the codec frontend - bad brains #_#

[Teegedeck]

You can't use luma-correction in GordianKnot for XviD (and you shouldn't use motion correction).

I don't know about 'optimized' settings for 'upward/ downward reaction period' and stuff. It looks like in DivX4, so perhaps ChristianHJW has any ideas about it?

[-h]

For 1-CD rips, I've been using:

1-pass quality mode.
85% quality.
Motion search precision as default (not slowest option)
MPEG quantiser
min/max quant = 2 and 7.
Lumi masking enabled.
640x272 resolution, except for (rare) pathological cases.

This surprisingly hits the desired file size 7 or 8 times out of 10 for me (!). Adjust quality percentage if file size doesn't suit..

I'd personally like to see better fade detection, less aggressive thresholding (when quant = 2 or 3) and less aggressive lumi masking before a huge comparison kicks off. But they could take a while..

-h

[BlackSun]

Originally posted by Teegedeck
You can't use luma-correction in GordianKnot for XviD (and you shouldn't use motion correction).

I don't know about 'optimized' settings for 'upward/ downward reaction period' and stuff. It looks like in DivX4, so perhaps ChristianHJW has any ideas about it?

ChristianHJW is in China for two weeks.

@Doom9: I was also planning to write a DivX/XviD comparison, but time is missing :(

[-h]

I don't know about 'optimized' settings for 'upward/ downward reaction period' and stuff. It looks like in DivX4, so perhaps ChristianHJW has any ideas about it?

If you're referring to the Averaging period / Reaction period / Up/Down ratio, they only kick in for 1-pass CBR mode. If they affect any other encoding modes, we have problems.

-h

[Teegedeck]

Originally posted by -h

If you're referring to the Averaging period / Reaction period / Up/Down ratio, they only kick in for 1-pass CBR mode. If they affect any other encoding modes, we have problems.

-h

I never touch upon these settings. Most of the time I do 2-pass encodings, anyway, as GKnot seems a perfect match for XviD; it hits the correct filesize within 2 or 3 megs and I bet this is just the remaing 'debt' in the bit-bucket, like it was in Nanddub before Nando did something about that. (But don't quote me on the accuracy; ATM I'm still experimenting and doing the same 2 DVDs over and over again... ´_` )

[easyfab]

you said:

Originally posted by Teegedeck


For a quantizer, choose 'MPEG'.


@Teegedeck


What's the difference with H.263 ?

I've also tested quantizer mode and quality mode and the result with FrameAnalyzer from DEAD2 is that quantizer (806Kb) is better than quality (847Kb).
35 frames tested, 27 with quantizer better and 8 with quality better.
What do you think about this results ?


Easyfab

[roy]

Compressing in VirtualDub may I use FastRecompress mode?
Or always Full processing mode. Maybe you suggest another program to use with xvid?

[Teegedeck]

@Easyfab: H.263 is an experimental quantizer. All I know about it, is that it is supposed to work better on very low bitrates. It removes more details, so that quantizer '2' in H.263 should look similar to quantizer '3' in MPEG-quantizer. I don't use it, I also had some crashes when I tried to. Anyone knows details about H.263?

Quality mode is designed to switch between two quantizers so you have something in-between if you don't like choosing a fixed quantizer. Like, if you choose quality 99%, it switches between quant 1 and quant 2 from frame to frame (though I'm not sure about the actual quantizers - perhaps it quants 1-1-2-1-1-2 with quality at 99%?). That's all the difference.

@Roy: Fast recompress and Virtual Dub surely are the fastest and most stable solution!

[-h]

The H.263 quantiser is the same as that used by DivX 3.11 and DivX 4.11 - a single value is used to divide all the DCT coefficient results (low-frequency coefficients are divided with the same aggression as high-frequency). The MPEG quantiser uses a 64-value matrix to quantise the low and high frequency coefficients with a bias towards low-frequency quality, ala the matrix you see in TMPGEnc. It is currently much slower than H.263, being C-only at the moment.

The quality mode chooses an 'average' quantiser for the whole clip. The formula is:
((maxQ-minQ)/100) * (100-quality) + minQ
thus 99% quality, when minQ=1 and maxQ=31, will give:
((31-1)/100) * (100-99) + 1 = 1.3

This will resolve to a continuous stream of 1,1,1,2,1,1,2,1,1,2 with the current formula.

-h

[Teegedeck]

Whoa, thanks for the info!

Now if only someone knew as much about VP3, so we could help poor doomie!

[Teegedeck]

Now, can you tell me something about H.261, too? :)

[Doom9]

so.. what's the preference now for the quantize matrix? mpeg or h263? I can't test the two.. it's enough work getting this all done already.. and I still have my vp3 problem.
atm I'm test encoding in 1 pass quality mode just to get a feeling for the codec.. the test will of course be based on 2 pass encoding. at least in this mode xvid doesn't seem to be faster than divx4..

[Teegedeck]

Sorry to hear that you're still choking on that VP3-issue.

For the quantizer I'd prefer MPEG as it looks sharper and I think not worse than H.263 at low bitrates. -h, BlackSun - what would you say?

[Doom9]

another thing.. playback.. I think I read something about changing fourcc codes in the xvid forums.. that's no longer needed, is it? wmp would use the divx4 filter to decode my xvid sample clip

but.. at leats in 1 pass quality mode I definitely have to say xvid is slower.. once I will do the full movie I will compare the logs (too bad gknot doesn't do xvid so I have to time that one by myself). But... I have an athlon xp... and I got quite a speed boost from the sse2 release of divx4 so maybe that makes the difference.

[Teegedeck]

Originally posted by Doom9
another thing.. playback.. I think I read something about changing fourcc codes in the xvid forums.. that's no longer needed, is it? wmp would use the divx4 filter to decode my xvid sample clip.

I believe there now is a DShow-filter for XviD, but uManiac hasn't compiled it, yet. By default the latest compile still gives all XviD-encoded files the DIVX-fourcc, so they get played back by the DivX4-DS-filter. This is gonna change, now.
You can watch the decoding-engine of XviD, though, if you change the fourcc to XVID. It gives a better picture than the DivX4-filter, but it's a BIT too slow on my Celeron466... It's VFW, for whatever that means, I never dived that deep into the matter... But I guess, for a fair comparison you SHOULD use XviD's own decoder!

[-h]

I'd stick to H.263 - the improvements in the MPEG mode are not yet worth the speed hit. It's useful to me because I often use 1-pass quantiser mode set to 3, in which case the MPEG quantiser gives *much* smaller filesizes for the same quality.

As for playback, you could set the FourCC to XVID, but you won't be using any post-processing. Depends if you see this as crucial in the comparison. I'd actually prefer to see an MPEG4 encoder with TMPGEnc's 'soften block noise' feature, instead of a (non-standard) post-processor tacked onto the process - this may bite people in the ass if they try to play their DivX4 rips on a hardware device that lacks post-processing, which isn't part of any MPEG4 video spec.

And speed - on my 1.2 Ghz K7 (and apparently anything older than a P4 / AthlonXP), XviD is marginally faster.

-h

[Teegedeck]

...well, this is a comparison, and I'd go for the quantizer that looks better in the screenshots, not for the quantizer that makes for better speed. ;)

Post-processing is the work of the devil, SSE2 is work of the devil... :devil:

[BlackSun]

Originally posted by Teegedeck
Sorry to hear that you're still choking on that VP3-issue.

For the quantizer I'd prefer MPEG as it looks sharper and I think not worse than H.263 at low bitrates. -h, BlackSun - what would you say?

Like you I prefer MPEG quantizer because it look sharper.

@h, can you add what xiphrom and gknot does for the 2_pass_vbr ? It will be easy to do a two pass encoding.

For thoses who got problems with the 2_pass_encoding, I dunno if Isibaar or h corrected the bug, but with mpeg2avi you'll have to use the RGB mode (-o7) for the first pass...

[Teegedeck]

Originally posted by BlackSun

For thoses who got problems with the 2_pass_encoding, I dunno if Isibaar or h corrected the bug, but with mpeg2avi you'll have to use the RGB mode (-o7) for the first pass...
...I remember from suxen_drol's last changelog that he fixed that bug.

[Doom9]

so.. if anybody can give me a compiled and working version of the playback filter before wednesday I will use it.. otherwise I'll use the divx4 filter... and the test will contain filtering.. the picture is way too ugly without it and the major part of the people use filters for their playback and they would wonder why my screnshots are significantly worse looking than what they get.

[Teegedeck]

Just wondering... Is it 'The Matrix', again?
And I don't think the DShow-filter will feature any post-processing... It's way down on the priority list - pre-processing filters are planned first.

[Doom9]

of course it will be the matrix.. still thinking if I should use another movie for the 2cd comparision (there will no longer be 2 bitrates that are picked with no respect to the size of the final product).. but somehow I don't like to have another movie because if I take a 3+h movie then the bitrate might be so low that it qualifies more as a special case of 2CD encoding than the general 2 CD = twice the size thing and I also have to find new screenshot positions.. anyway.. I will think about that a little more over the week-end as my encoding sessions progress

[-h]

Well to be honest I haven't used the 2-pass modes at all :) There really should be some documentation for it.. I think parts of the code make more sense than the interface at the moment.

Filtering is probably a while off too..

A word of caution with the matrix - due to the large number of dark scenes throughout, lumi masking could run amok unless you set the max quantiser to a fairly low value, i.e. 5 or 6.

-h

[Teegedeck]

Originally posted by Doom9
of course it will be the matrix..
OK, so definitely, please don't use lumi-masking - the movie is too dark for that...

If you use the same movie for 2 CDs, how about encoding it at anamorphous resolution, no resize? Because The Matrix hardly is very challenging for any codec to compress.

[EDIT:] Thanks to TheWEF and other people, we can just use GKnot's compressibility-check to verify this and don't have to encode the whole movie... :) I LOVE GKnot!

[Teegedeck]

I really like that movie - it was the first DVD I ever bought... But I have to say, I don't like it for testing a codec... sorry. It is such a untypical movie, very dark, very low contrasts, all greenish colours... I use my 'Charlie's Angels'-DVD most of the time for testing. It is bright, colourful, lots of 'noise' in it lots of action and also some action-scenes are dark. It's quite hard to compress.

[BlackSun]

Originally posted by Teegedeck

...I remember from suxen_drol's last changelog that he fixed that bug.

Nope, seem that the bug is still here... Will post result tommorow because I didn't try it myself.

Todo list: test last XviD release, re-found the solution for vp3 & Doom9, try to compile the DirectShow filter (if uManiac join the DSF source)

Well to be honest I haven't used the 2-pass modes at all There really should be some documentation for it.. I think parts of the code make more sense than the interface at the moment.

Lol, I know you can do it :D, simply add this part from the Xiphrom code:

while not Eof(F) do //Count Frames
begin
read(f,xvidfile);
TotalBytes:=xvidfile.bytes + TotalBytes;
end;

// Calculate First Average Bitrate
first_bitrate:=round((((TotalBytes / 1024)/i)*StrToInt(ComboFPS.Text))*8);

// Calculate new avg bitrate (user desire)
BitrateCoef := first_bitrate/new_bitrate;

*** You must know the new bitrate ***

for i:=1 to FrameCount do
newbytes[i]:= round(frame[i].bytes / BitrateCoef); // Change the average Bitrate


Just translate this to C in VFW.C and we have the 2 pass VBR working pretty good.

[Teegedeck]

Originally posted by BlackSun

Just translate this to C in VFW.C and we have the 2 pass VBR working pretty good.

Err... I really hope I'm not bugging you with this, too much, but does this code disable the bit-bucket-routine in the codec? Because I remember I got considerably strange codec behaviour when Xiphrom-modulated .stats-files prompted (for a 1CD-encode) quantizers that were so low that they collected huge debts in the bit-bucket, which in turn prompted an insanely high quantizer for the following frame. So I guess, your code forces the quantizers for 2nd pass, directly - you once mentioned something like this?

[Doom9]

bitrate for 1CD matrix is 560-570kbit when I deduct 128kbit for the audio.. that is a really low bitrate case and should be sufficiently challenging for the codecs. As for the 2 CD case.. first I don't have charlies angels and 2nd it's too short.. normally you don't encode that for 2 CDs. Austin Powers would be a good one, too but here again, too short for 2 CDs. And as you can imagine, such a comparision is a major time killer.. encoding alone takes several days (doing the whole movie), taking the screenshots via avisynth is a major pain and the comparison is another very time consuming task. such a comparision can only give you a glimpse at the situation.. you'd have to encode dozens of movies to get really good results but such a thing is beyond me (and probably beyond anyone not doing this for a living)

[Doom9]

well.. I'm going to do spr for the 2cd case.. that should be fairly hard.. it's a long flick and there are many complicated scenes (fog, the beach landing, the church scene, etc) that can push the codec to it's limits.

btw.. why did I chose matrix and that particular scene initially? Once wm7 came out I was thinking about difficult to encode scenes (remember it was the time prior to mm4, sbc and fair use).. and I knew about the lot of blocks on the walls in matrix.. and the lobby shootout is well known by almost anybody.. and it contains a lot of action.. and when I encoded it and watched the actual bitrate used.. it really went to DivX's peak for this scene even at low bitrate settings

[-h]

Re: 2-pass code - you just want a constant ratio of the source bitrate to become the output? I.e. if the first pass hit 3000, and you wanted 500, it'd set the desired frame size to a sixth of the original .stats file? There would be file size issues with that, I'd have to play around for a while.

I'm writing this from a mac, because my 4-year old PC monitor finally died. Annoying that. I've coded up a brute-force inter/intra choice function (should cut bitrates significantly in action scenes) and a *possibly* faster h.263 quantiser, but I've got nothing to test it on #@$!

@doom9 - when are you intending to perform the tests?

-h

[Doom9]

@doom9 - when are you intending to perform the tests?
as soon as possible.. some time next week after having encoded the source in all codecs... I've already done the 1cd matrix rip in wm8, sbc and divx4 and vp3.. now starting xvid

actually.. I'm a bit confused.. there's no bitrate setting anywhere.. what do I do with the stats file once it has been generated?

[-h]

You load up the .stats file from the first pass in GKnot, perform whatever curve mods you feel like doing, then save the edited .stats file to a new file. In XviD, you specify *both* the first pass and second pass .stats files when doing the second pass.

Or so I'm led to believe.

And no it isn't too friendly yet :)

-h

[Doom9]

ok.. that's pretty much how I thought it would work. that means tomorrow I have my 1 cd case done and can move on to the 2cd stuff

[gldblade]

And as has been said before, don't use motion and luma correction in GKnot because XviD doesn't support them.

[BlackSun]

@h: I didn't got any problems with filesize and xiphrom

[Doom9]

2 things: xhiphrom: set the framerate to 23.976, load the stats file and it comes up with an error saying 23.976 is not valid integer.. that's of course correct but unless you do pal you have to enter floats

gknot: gknot does the calculations assuming 1k = 1024.. does that apply for xvid or does it use 1k = 1000? In the latter case it can't be used to properly calculate the 2nd pass curve.. and that may explain undersized files

[Teegedeck]

I calculate it as I do with DivX 3.11 and it's pretty precise: I got 2.8 MB filesize overshoot on my last 2-CD, and that's OK, I think. About the same with 1-CD.

[Doom9]

wohoo... xvid is the only codec to create a 700mb file as desired.. though I have to admit that I used the wrong interleave settings for sbc.. that's why it got undersized.. time to restart the sbc 2nd pass

[-h]

XviD is assuming that one Kilobit = 1000 bits, as per the MPEG4 spec. See CONFIG_KBPS in vfw/src/config.h

suxen_drol's bit bucket should adhere very closely to the desired file size..

-h

[Teegedeck]

Originally posted by -h
XviD is assuming that one Kilobit = 1000 bits, as per the MPEG4 spec. See CONFIG_KBPS in vfw/src/config.h

...so has it been adjusted to process the GKnot-modified .stats-files correctly? I would guess so.

suxen_drol's bit bucket should adhere very closely to the desired file size..[/B]
BlackSun, have you changed anything in your 2-pass-formula, since you released Ximorph?

[BlackSun]

@Doom9: thanks, as I only have PAL Movie, I didn't tested with Float value :p



BlackSun, have you changed anything in your 2-pass-formula, since you released Ximorph?

Nothing, not enough time :(