beta2 slower and filesizes not as accurate as beta1?

[Zep]

i have run many tests and beta2 for me is slower and the final file size is smaller than
beta1 encodes using the same files and settings.

I did clean Xvid installs and load defaults when going back and forth between versions.
Here are the times and sizes of a HDTV encode i just did for testing purposes


profile=unresrticted
Adaptive Quant = on
BVOP = on --> 2/1/2
BVOP sensitivity = 40
Q range 2 to 31 for i/p/b
overflow treatment = 20/20/20
Motion Search = Ultra
VHQ = mode decision
Chroma motion=off
(everything else default)


beta1

1stpass.avi
Time=16.39

2ndpass.avi
Time=17.52
Size= 367,249,408



beta2

1stpass.avi
Time=16.01

2ndpass.avi
Time=25.53
Size=366,794,092


I could see the 2nd pass code fixes adding some time to pass 2 but that much time? We
are talking about what a 40% slow down on this test.

yes pass 1 is a little faster in all my tests but not enough to make up for the rather large slow
down of pass 2. So far the slow down is from 20% to 50% and all files sizes are 100K
to 1.2 megs smaller on 41 to 44 minute encodes compared to beta1.


Can others here confirm this?


thx


NOTE: running an AMD 64 3500+ @ 2.7Ghz with PC4000 ram and Prime95 runs without error.




.

[Kagura]

My guess is that VHQ was fixed (logarithmic scaling?) to yield better quality. This would make sense because that was a fix in the changelog and also would explain the lower filesize. It's either that or someone tweaking VHQ to be more aggressive on Mode Decision. (more like Wide Search?)

[yaz]

@zep
- the filesize difference is about 0.15%! i would accept it even if it were random error only ;) but it's not that. amongst others the scaling algo was refined as well.
- i always use 'wide search' (time is not an issue for me) and i don't notice any significant changes in the encoding time. i did not measure that accurately but i would have noticed such a big difference for sure.

btw, what that extreme b-vop sensitivity for ?

the bests
y

[squid_80]

Whose builds were you testing?

[Zep]

Originally posted by Kagura
It's either that or someone tweaking VHQ to be more aggressive on Mode Decision. (more like Wide Search?)

well @#$% if that is the case that is not good. I use mode
Decision for the speed. Well balance of speed/quality.

IMHO if someone tweaked the code like that we should have been told.
Did i miss that in the change logs?

[Zep]

Originally posted by yaz
@zep
- the filesize difference is about 0.15%! i would accept it even if it were random error only ;) but it's not that. amongst others the scaling algo was refined as well.
- i always use 'wide search' (time is not an issue for me) and i don't notice any significant changes in the encoding time. i did not measure that accurately but i would have noticed such a big difference for sure.

btw, what that extreme b-vop sensitivity for ?

the bests
y


True the file size diff is small BUT there should be none because
now using the xvid calc evertyhing comes in undersized and I
can't just add an extra 200K to compensate because it is random per
source input. beta1 hits the target no matter what the source
input.

can you please try and encode with my settings? or at the very
least time one with your settings between the beta1 and beta2?

if beta1 mode decision is just that but beta2 is really wide search
then we have a problem because then what the heck is mode decision
in beta2?


b-vop sensitivity is 40 that is not extreme. It is just enough
to get a IPBBPBB... since I want 2 b frames. If you leave it
at anything less you may only get 1 bframe. If you leave it at
O sensitivity you may get large runs of no bframes and that
is a huge bit waster since the P frames do not grow much at
all in those runs when i jump to 40 sensitivity. So you save
a lot more bits than you lose to P frame size increase. IMHO
the sensitivity default of 40 should really be what the 0 default
is internally. i.e make 0 really = 40 internally.

BTW - even at 40 sometimes you still get IPBPBB but it is rare
and why i feel 40 is by no means extreme.



thx

[Zep]

Originally posted by squid_80
Whose builds were you testing?

Koepi. In fact i only have ever used his builds. :)

[Zep]

Originally posted by Kagura
My guess is that VHQ was fixed (logarithmic scaling?) to yield better quality.

figured i would comment on this too.


The increase in quality at low bitrates is very small. I can't tell
at 1000 bitrate stuff i do when watching in real time. A frame
by frame i can spot a few details that are better but even so
very VERY small improvement.

IMHO a 40% speed hit for that is not worth it. A 5% speed hit
ok fine but not 40%. My point in the first post was why such
a huge hit for those code tweaks?

On the other hand, i would be fine with a 40% speed hit if the
quality improved by 40% :D

[Kagura]

1000 kbps is not low. Try ~600 or 800. The quality improvement is significant and evident. In the end, it's your choice. You can always choose to revert to beta1 for your encoding needs. These builds ARE betas, after all, and not official, so your results are understandable.

That said, I thank you in advance if your discovery leads to any fixes.

[yaz]

Originally posted by Zep
True the file size diff is small BUT there should be none because
now using the xvid calc evertyhing comes in undersized and I
can't just add an extra 200K to compensate because it is random per source input.it's just the way it is. it is the case when u can (would) never rely on a calculator. ;)
... from xvid changelog
10.01.2005
... fixed overflow in bitrate calculator ...are u aware of it ?
anyway, expecting correct target size on the byte-scale is just a dream that never comes true. the real question is how close can the codec get to the target. in this meaning xvid is the best i've ever tried. however, if the difference shocks u just make a 3rd round w/corrected target size and u'll get it.
Originally posted by Zep
beta1 hits the target no matter what the source input.hum ... it's hard to believe (considering your expectations on accuracy ;) ) how many sources have u tested ?
Originally posted by Zep
can you please try and encode with my settings?sorry, but not. see, comments above ;) i've noticed some filesize discrepancies when installed b2 but running full 2pass encodes get as close to the target as b1 did.
Originally posted by Zep
if beta1 mode decision is just that but beta2 is really wide search
then we have a problem because then what the heck is mode decision
in beta2?hum ... would that enigmatic sentence of kagura mean sg like that ? i don't think so, but a devel should drop the final words about it.
Originally posted by Zep
b-vop sensitivity is 40 that is not extreme. It is just enough
to get a IPBBPBB... since I want 2 b frames.of course, you're free to tease xvid in any way u just want but ...
- b-vop sens=40 is extreme in all meanings
- why to force xvid to do sg which it's unwilling to do? why do you think a strict vop pattern is better than that xvid figures out? why do u think 'bb' is better in a certain place (or everywhere!)than 'pb' ? all these questions have the same meaning; why do u think u know what and how to do better than xvid in this instance ? :confused:
Originally posted by Kagura
1000 kbps is not low. Try ~600 or 800.yep ... or try sg 250-500 ;)
Originally posted by Kagura
In the end, it's your choice. You can always choose to revert to beta1 for your encoding needs.that's it. but remember, beta1 was(is) buggy. at least 3 different bugs were discovered and been debugged so far.

the bests
y

[bond]

hm i also did two encodes with beta2 recently and also noticed that my encode was one mb smaller than i wanted (700mb backup), with earlier versions i didnt notice that

[Zep]

Originally posted by Kagura
1000 kbps is not low. Try ~600 or 800. The quality improvement is significant and evident. In the end, it's your choice. You can always choose to revert to beta1 for your encoding needs. These builds ARE betas, after all, and not official, so your results are understandable.

That said, I thank you in advance if your discovery leads to any fixes.

1000 is VERY LOW.

bit rate being considered high or low depends on the source and
other factors like rez. 600 to 800 is VERY HIGH bit rate for low
motion, low rez, lower FPS stuff like web cam feeds and animation.

1000 is VERY low rate for my source. it only gets me a Q average
of 8.5 on this source content with no quality improvement
to speak of. My guess is the Q average is so bad already the
tweaks are not gonna help much and that appears to be exactly
the case.

Anyway yes it is my choice and of course i can go back and i
already said i did but that has NOTHING to do with why i posted.
I posted because i feel a 40% speed hit is HUGE and i want
to find out if the minor code tweaks really should cause that
kind of speed hit or if there is a bug etc... or if we can have
the option to turn that stuff off when it doesn't improve
quality that much at all, like in my case.


thx

[Zep]

Originally posted by yaz
it's just the way it is. it is the case when u can (would) never rely on a calculator. ;)
are u aware of it ?


yes I am and the calcs are not done via xvid built in they are
done in a script. a bat file. The calcs were dead on until
beta2. Beta2 is the first Xvid version i have used that the
target may not hit Close enough depending on the source content.
I boosted the overflows and it helps but as said many times
in other threads that throws the Quants off if you go too high.


Originally posted by yaz
anyway, expecting correct target size on the byte-scale is just a dream that never comes true.


well it used to prior to beta2. i will be more than happy
to point you to the proof in a pm. I hit to the byte
more oftern than not :)


Originally posted by yaz
the real question is how close can the codec get to the target. in this meaning xvid is the best i've ever tried. however, if the difference shocks u just make a 3rd round w/corrected target size and u'll get it.


heck i was talking about how 2nd pass was so much slower and
how i did all these tests and went back to beta1.
now you want me to do a third pass? my guess it that
is even slower than 2 passes :D


Originally posted by yaz
hum ... it's hard to believe (considering your expectations on accuracy ;) ) how many sources have u tested ?

wow i lost count. these days maybe dozen a week.
total i would guess a few thousand over the last
couple of years. Though for 2 years now mainly HDTV
streams.


Originally posted by yaz
sorry, but not. see, comments above ;) i've noticed some filesize discrepancies when installed b2 but running full 2pass encodes get as close to the target as b1 did.

not for me. i need it within 500K no matter what and up until
beta2 i could count on that

Originally posted by yaz
- b-vop sens=40 is extreme in all meanings

IMHO it is not. It gets you *almost* 2 bframes in row.
That is not extreme. The default is 2 and it doesn't
even get you that at 40.


Originally posted by yaz
- why to force xvid to do sg which it's unwilling to do? why do you think a strict vop pattern is better than that xvid figures out?
why do u think 'bb' is better in a certain place (or everywhere!)than 'pb' ?
wow i'm shocked you would even say that.
Xvid has options for a reason. if Xvid was perfect for all
sources there would be no options. no need for them.
I save a TON in bits over letting xvid decide and quality
improves greatly. Good enough reason for you? lol


Originally posted by yaz
all these questions have the same meaning; why do u think u know what and how to do better than xvid in this instance ? :confused:

testing. lots of testing. and like I said above thousands of encodes over the years.


Originally posted by yaz
yep ... or try sg 250-500 ;)

Now you are just being silly as well as not reading my post
above where i said why that is bad. :D


Originally posted by yaz
that's it. but remember, beta1 was(is) buggy. at least 3 different bugs were discovered and been debugged so far.

3 bugs found maybe 4 bugs added who knows. i do know
there is a rather large slow down on pass 2 though. Is that a bug?
or is that the hit we take for fixing those bugs. That is
why i posted and what i want to find out. Have you lost sight
of that? You seemed to want to have it out with me more
than find the answer to the main question and why i posted
to begin with.

Anyway, i still feel a 40% hit is more than it should be for such
bug fixes but only the devs can truly answer that. So lets see
what happens in beta3. fix a few more bugs lose another 40% in
speed. Maybe then you will start to see it from my point of view :D


.

[Zep]

Originally posted by bond
hm i also did two encodes with beta2 recently and also noticed that my encode was one mb smaller than i wanted (700mb backup), with earlier versions i didnt notice that
yup. seen this myself dozens of times now.

However, I'm more curious as to the speed hit on pass2 and if it
was the cost for fixing those bugs or is it a bug or opt that was
missed in beta2 etc...

Well at least the issue is out there now. So maybe the devs
will read this stuff and take a look and see if they can improve
upon the code for beta3.

[yaz]

@zep
easy, man, just take it easy :) i don't think it's worth risking a heart attack or so :D

we differ a lot in a substantial way as regards our expectations and (mis)beliefs on encoding.
you get anxious if u loose 0.2M from 365M. i don't. (amof, i wouldn't bother about 1M from 700M either. i bet, i wouldn't even notice it :D )
you think regaining this 0.2M would mean sg significant. i don't.
you think forcing a strict vop scheme is beneficial. i don't.
... and i would list it endlessly. but that'd be quite pointless. we have our standpoints and preferences and they seem to be pretty steady. so what to do ? dunno, but i'm sure there's no point of starting an endless debate. ;)

anyway, i've started a full movie (~130min) encodes in the morning, exclusively for you. not finished yet (it's not a 3.5M+ cpu), but i don't see such a big difference in encoding speed u mentioned. maybe u should test it on a longer stream too. as regards final size, we'll see, but i don't expect more than the usual undersize about 0.1-0.2%.

if u felt me off-handing u it could have been because of my poor english. got no doubt about your skills, experience and knowledge. of course, u can pm me whatever results u just want, but i think it would be more beneficial to do it public. if your results are so convincing why not let others learn from it?

however, whatever we find, the final words would be told about it by a devel, imho.

the bests
y

[Didée]

Two days before I did a 2-pass encoding of a shorter movie, ~ 80 min. in length:
with XviD beta2, ofcourse

Requested filesize: 600'000 kB

Resulting filesize: 599'930 kB

70 kB undersize is no reason for complaining :)

[Sharktooth]

I encoded "Exorcist - The beginning" with xvid 1.1b2 and the following settings:
Bitrate: 812kbps
Resolution: Lanczos resized at 720x304 (anamorphic at that bitrate is a suicide :cool: )
Avisynth filters used: RemoveGrain(mode=2), LRemoveDust(2,3)
Codec settings: DEFAULTS + AQ + Trellis + VHQ=4 + BVHQ + Chroma OPT + CQM = EQM V3ULR
Final derised file size 700Mb
Final effective file size was off of about 90Kbytes (perfectly normal).

Screenshots (lossless PNGs): *removed*
Screenshots (100% quality jpegs): *removed*

[AssassiNBG]

Dunno, but with beta 2 it's so hard for me to get to a 175mb file. Using the calc it ends up 171 mb. Inout source is the same. But 1.0.3's calc worked just fine. Hope this is fixed in the new binary Koepi is preparing. :)

[Leak]

Dunno, but with beta 2 it's so hard for me to get to a 175mb file. Using the calc it ends up 171 mb. Inout source is the same. But 1.0.3's calc worked just fine. Hope this is fixed in the new binary Koepi is preparing. :)
If it's really the calc, would you mind posting the values both versions produce? Unless there's a bug, they should be quite (if not totally) similar...

And are you sure it's not the rate control code, which would be likely if the values produced by both calcs don't differ?

[Koepi]

Maybe it's the difference between opendml and "normal" avi files. You should be aware about the difference when choosing the container in xvid's calc.

opendml has less overhead and thus saves space compared with the "traditional" avi-muxing engines.