undersize problem

[steelman]

Hi there.
Currently I'm encoding 24 Season 1, 25fps pal source and I'm always getting undersize about 30-50 MB without audio. My settings:

Quantization type: H.23
Quarter Pixel: On
B-VOPS: 2,1.50,1
Closed GOV: On
Packed Bitstream: Off
Mode Search Precission: 6
VHQ Mode: 1
Use Chroma Motion: On
Turbo On
Fram Drop Ratio: 0
Maximum I-Frame Interval: 300
Quantization: 2,31,2,31,2,31
I-Frame closer...:1
...are reduced..: 20
Overflow treatment: 10,60,60

I'm using rc3 and I want to fit to 350Mb per episode (2 eps for one cd).

Avisynth source:
LoadPlugin("C:\Program Files\Gordian Knot\mpeg2dec3.dll")
LoadPlugin("C:\Program Files\Gordian Knot\undot.dll")
mpeg2source("F:\pokus\stripped\diel5-6\projekt\projekt.d2v")
crop(6,0,706,572)
LanczosResize(576,320)
Undot()

Thanks for any help and suggestions.

[celtic_druid]

Increase the resolution to 640x352.

[Gaia]

I did quick search and found 100 topics about "undersized AND xvid". Most of them would give answers you are looking for.

[steelman]

I did search (undersize in xvid section), and I didn't found what was I looking for. I've tried settings from few posts but nothing helps. Maybe I missed some post, in that case I'm sorry for new post. Now I'm trying higher resolution, I hope that it will help...

[Taurus]

@ steelman

As I can see from your posted Xvid encoder settings, there's nothing unusual.
Except the ''overflow treatment'' settings are not defaults, the ''quantization'' values are set to ''2, 31'', this should not make the encode so much undersized.
You posted the values of the ''second pass''.
Maybe you have just used different settings for the ''first pass''?
What audio settings are used to calculate?
Sometimes it's just a ''thinking ''error (I've made some myself).
Or maybe the source is getting compressed so well?
Try ''default'' settings on xvid.
The most actual release of the GKnot(0.28.8)RipPack is recommended.
And, of course the most the actual VDubMod, that comes bundled with the RipPack should be used.
Post your Logfiles, if you need further assistance.

Taurus

[Koepi]

Oh hell, don't use codec packs! They can terribly mess up things and lead to errors.

Uninstall any codec pack - if that doesn't work correctly you need to fresh install your whole system :( - and install a "pure" binary.

Then, just to be sure, hit "load defaults". Make sure the quant ranges are from 1-31.
Overflow values should be 20,20 and 20%.

This should help it already.

Koepi

[steelman]

Thanks guys. I've just finished re-encoding with higher resolution (640x352) with my settings and it seems that it was realy problem with low res. I'm not using codec packs and I don't trust gordian knot.
Koepi, thanks for suggestion, I'll try overflow on some other pc at the work :cool:

[lordadmira]

Sometimes, a source really is "just that compressible". Meaning that even at max quality it compresses very small. You can watch the output with ffdshow's overlay visualization to see the frame quantizers. If your undersized output has lots of quants 1, 2, and 3's then it is pointless to try to increase the size. Now if it's undersized and there are mainly quants 4, 5, & 6, or higher, then maybe there's a rate control problem. Increasing the resolution just pads your file with a lot of chaffe. The only time you have to worry about undersized outputs is if it leads to buffer underruns in stand alone players, as mentioned in other threads.

[steelman]

Well, well.
Right now I'm doing 4th time encoding of another eppisode and there is still undersize. Now I'm using default quant ranges and koepi's overflow vlaues. I don't think that it will be now good, becouse stats reader says that target size will be 261mB, but who nows (at 18% vdubmod shows target size 288mB)...
I think I should stop today, and go to sleep or I'll shrink dvds and burn them :rolleyes:

[lordadmira]

Don't worry about VDub's expected target size. It's meaningless for 2-pass encoding. And even for CBR it's only marginally accurate. But I have some other observations. Your source file seems to be 720 x 576 and you are cropping some junk off the edges right? But your crop settings don't make sense considering your resize operation. You're cropping to a 5:4 aspect ratio but then resizing to 16.2:9?? Is the source file encoded as squished or something? In that situation, what I would do is keep the full horizontal resolution and change the vertical to bring it into the proper aspect ratio. e.g. 720 x 400. Otherwise I would try it at the full cropped resolution of 706 x 572. With 2-pass the codec might do a better job at 350MB than u might think. If the quality isn't acceptable then try incrementally decreasing the size. Resizing to an integer factor of 572 will give u the best quality. (i.e. x mod 572 = 0, where x is the decimation factor)

[steelman]

DVD is definitely 16:9 (as dvd2avi says). There are no Y black tracks.
Only way I trust GK is to make avisynth script and using autocrop is for me right choice. I'm lazy to determine cropping settings myself :) I could try disable cropping but I'm not sure that it will help...
I've just finished another episode and there was a little undersize (my first settings and 640x352 resolution), only 10MB wich is acceptable. Maybe the source is really so compresable. Quality in the undersized file isn't bad, but 350MB per episode looks better ;)

Half of the 24 show is in the night, so maybe that is the problem and xvid is simply "saving space" on dark places...

[lordadmira]

Originally posted by steelman
Half of the 24 show is in the night, so maybe that is the problem and xvid is simply "saving space" on dark places...

Yeah, that explains it. Even at quants 1 and 2 a black area will go to almost nothing.

[mikeX]

becouse stats reader says that target size will be 261mB
Well, there is your answer right there!
The two pass encoding mechanism roughly works like this:
- first pass is made at fixed quantizer (2 for I,P and 4 for B frames @ default settings) which is also the best achievable quality with current settings
- second pass lowers the quantizers in order to achieve the desired bitrate/size

Now if you open a .stats file in Stats Reader you will get an estimate filesize, which is the size of the file you would get if you encoded your video at the best possible quality!
If your .stats file reports a 260MB file, don't expect the file that's produced by the 2nd pass to be bigger than that, it simply can't!

Now in order to avoid questions of the type: My files come out oversized, XviD developers decided to use a default quantizer range of 1-31 so that if the desired size is bigger than the one at quantizer 2, quantizer 1 will be used in order to compensate for that.
Note however that quantizer 1 is an overkill! Quality is insignifficantly better than quant 2, while resulting size is quite bigger!

Of course you can only gain that much in size using quantizers of 1.

A better solution for your problem is using a higher quality Quantization Matrix (like MPEG or HVS Best) instead of H.263 and/or disabling B-VOPs and/or increasing your resolution (like Iordadmira suggested)

[lordadmira]

Originally posted by mikeX and/or increasing your resolution (like Iordadmira suggested)
       :stupid: That's an ell. :D

Heh, but anyway quant 1 is the raw value of the DCT transform IIRC. For any encodes that would fall in the "high quality" category I would totally disable the B frame quant offsets so that they receive full quality. Like 8/1/0.

LA

[steelman]

Thanks guys.
I encoded few episodes in 640x352 then i had to decrese the resolution to 576x320 (one episode) and now I've got serious oversize, something about 120mb without audio. Settings are same as posted in first post. I already tried to go with resolution lower but it helps minimaly.. :mad: Xvid status show whole first pass encoding witch average bitrate 1200-1400 kbps...
These dvds are worst of all dvds that I had home.

[yingx2]

Originally posted by Koepi

Overflow values should be 20,20 and 20%.

Koepi
Are these default values?
Becuase everytime I hit the "Load Defaults", they just become 5, 5 and 5%, and I've been getting undersized files forever back when I started using RC 2(or earlier).

Modifying them to 20 makes perfect sized files again in my case. You would never know how happy I am now.

[steelman]

Hi, guys.
I hope that this will be my last post. I "found" that problem may by on dvd side. Source dvd has really big bitrate 6+ mbps.
I lowered data stream on one episode and I got still big oversize but after lowering bitrate it was better. Now, I have to find right value of lowering bitrate...

[mikeX]

yingx2:
no 20,20,20 isn't the default, 5,5,5 is, but if you find it suits your need that great you should go ahead and use it, that's why the option is there in the first place!

steelman:
huh? is it me or are you not making any sense at all?

[steelman]

Originally posted by mikeX

steelman:
huh? is it me or are you not making any sense at all?

No it's my english :(

I was simply saing that source dvd has really big bitrate above 6Mbps and I think that it may be my problem with oversize...

[mikeX]

I was simply saing that source dvd has really big bitrate above 6Mbps and I think that it may be my problem with oversize...

I don't think this should be a problem...
You think you could post some screenshots of the 'XviD Status Window' for your first and second pass?

[TiaoMacaleh]

Now in order to avoid questions of the type: My files come out oversized, XviD developers decided to use a default quantizer range of 1-31 so that if the desired size is bigger than the one at quantizer 2, quantizer 1 will be used in order to compensate for that.
Note however that quantizer 1 is an overkill! Quality is insignifficantly better than quant 2, while resulting size is quite bigger!~

Im getting 400-300mb undersize file need some help :)

Question i was encoding with 2 min, now im trying for min 1.... is that possible that this settings give me a worse quality encode but bigger? :confused:

A better solution for your problem is using a higher quality Quantization Matrix (like MPEG or HVS Best) instead of H.263 and/or disabling B-VOPs and/or increasing your resolution (like Iordadmira suggested)

Which one you recommend? :devil:

[Taurus]

Originally posted by TiaoMacaleh

I'm getting 400-300mb undersize file need some help :)
Without log files and corresponding xvid settings, it's like searching for the famous needle in a hay stack.
A compressiblity check with GK or Enc 1.03 by jonny will give you some hints about the bitrate needed for your source.
Minus 400-300 Mb is almost impossible without extrem wrong settings....

Cheers

Taurus

[jon.schaffer]

Originally posted by Taurus
Minus 400-300 Mb is almost impossible without extrem wrong settings...
It depends on the desired final size. Example: you can wait for 1400 MB and end up with less than 700 MB ! (it happened to me 2 or 3 times) The settings are not to blame. The video is too compressable (and could we complain about that?:p)

Originally posted by TiaoMacaleh
now im trying for min 1.... is that possible that this settings give me a worse quality encode but bigger?
I would say... yes! It's possible, by the use of too many quantizer-1 frames in some scenes to fatten the file.
I think - in order to achieve max quality - that we should consider Q1 as a "trick" to reassure the newbies. When you know how it works, you should fatten your file by increasing its quality (higher resolution, more sharpen, matrix). About the matrix: I don't know them enough. Search the forum: plenty of post are about matrices producing higher bitrate videos.

[TiaoMacaleh]

How do i get log files? you mean that graph when its encoding?

I changed min quantizations from 2 to 1 that gave the extra mb i needed :) Now i got a perfect 1,36 file...

Dont think theres a big problem in settings i used it for other encodes with sucess, the problem is the high compressability movie :(

Was there any better solution?

I thinked on increase resolution but didnt found any nice ar bc the resolution was already high 624 x 336.... Then i checked i could use a better matriz but too late :devil:

Any suggestions? Reecode whould give me any big improve right? because the movie was already good even with 400 less mb than set (hence its over 2 hours).... now its pretty good imo even better than the dvd :devil:

[TiaoMacaleh]

http://img235.echo.cx/img235/7724/stats3sr.jpg


Is that usefull?

[Blue_MiSfit]

I thinked on increase resolution but didnt found any nice ar bc the resolution was already high 624 x 336.... Then i checked i could use a better matriz but too late



624x336 is NOT a high resolution :) Try running your encode with no cropping or resizing. IE 720x480 (NTSC) or 720x576(PAL). This should produce a much more even quantizer distrobution. You will have no undersize, and your movie will be sharper as a result, paritally due to increased resolution and partially due to the fact that you can now do a proper 16:9 anamorphic rip!

High compresibility is not a problem at all! In fact it's awesome because you can use a higher bitrate matrix, which will preserve more detail and sharpness!

Notice that your total average quantizer is 2.43 - This is EXTREMELY low. An average quantizer of around 4 or so usually results in transparent quality.

If you are shooting for 1400mb+ total filesize, try a higher bitrate matrix, like Sharktooth's EQM-V3HR (my personal favorite, and default). Otherwise stick to h263 or maybe try v3lr or hvs-better.

Essentially the reason you are getting undersize is that your source compresses really well. The encoder can't make it look any better with your current settings so you are getting the "best case" scenario out. Q1 frames just pad the filesize (terrible IMO) so that newbies don't complain about undersize. They do NOT increase quality as many have said already.

If you get undersize or ANY q1 frames, INCREASE RESOLUTION OR USE A HIGHER BITRATE MATRIX. Cannot stress this enough :) Do yourself a favor and limit the quants to 2-31 (as they should always be IMHO). Q1 is a waste of space.

Good luck, tell us how it goes! :D

~misfit

[SharekhaN]

i have been encoding with xvid for some time now although i must admit that i do not have indepth understanding of how somethings work.

coming to the problem .

i am using gordianknot 0.35.0
and using xvid 1.1 beta 2

i am try and rip big movies with runtime say around 2hrs 30 mins or 3 hr + ones to approximately 700 mb .

now i am also using the custom quant matrix by sharktooth ie v3ulr_rev3
resolutions are also 512 * xxx +

with this i am getting undersized movies which are off mark by 10 mb or so , consistently . however if i do rips 2 cd size ... they are usually fine size wise but i also use v3lr as a cust quant in that case.

i do know that sharktooth recommends a bitrate of atlest 700 kbps or more for v3ulr . my rips will usually have 475 or around 500 kbps as their video bitrate. but could cqm's be a reason for undersizing of the files

i have trellis on , gmc on though i dont think this could be an issue.

please do help

[TiaoMacaleh]

Gonna reecode now.. the load defaults is really important when installing new versions?:confused:

[jon.schaffer]

Originally posted by TiaoMacaleh
the load defaults is really important when installing new versions?:confused:

Yes. To be sure not to apply old registry settings to the new version. It's a safe behavior.

[TiaoMacaleh]

Reecoded with EQM-V3HR matrix, increased resolution to 720 x 384....

Got a 1160 bitrate and 0.175 qf...

But one problem, now i got 20mb undersize :(

Any suggestions to push this extra 20mb? maybe decrese the b-vop sensitivity? i used 15 how much i need to get extra 20mb? 10? 5? -10?

Help please :D

[TiaoMacaleh]

The source dvd has some noise. The encode with the different matrix you sayed that give a more sharpen and detailed image but in this case as the dvd isnt perfect seems to give a more grainy image :(

Previus rip, higher bitrate and qf, 0.0% error (0.001)
Last rip, lower bitrate and qf, higher res, 0.1% error (0.002)

Last rip seems to be a bit more sharpen but a bit more grainy is high movement scenes (not that bad since isnt an action movie) and its 20mb undersized so ill have to re-rip,

Which one keep? the first or go for a re-rip?

thanks :devil:

[Didée]

Originally posted by TiaoMacaleh
maybe decrese the b-vop sensitivity? i used 15 how much i need to get extra 20mb? 10? 5? -10?
Default. Let B-frame sensitivity at DEFAULT.

The source dvd has some noise. The encode with the different matrix you sayed that give a more sharpen and detailed image but in this case as the dvd isnt perfect seems to give a more grainy image.
No surprise. Your 2nd encoding did a better job in "reproducing the source as it is". After all, that#s exactly the job the encoder is trying to do. If grain is removed (and the overall impression is "more smooth") after encoding, then this is - basically - a sign of too strong compression.

So, even without having seen anything of it, I'm pretty sure your 2nd encoding with the 720ish resolution is the better one.
If the reproduction of the source's grain is not to your liking, I'd rather suggest to do some *slight* filtering to the source.

[yaz]

Originally posted by TiaoMacaleh
Any suggestions to push this extra 20mb? maybe decrese the b-vop sensitivity? i used 15 how much i need to get extra 20mb? 10? 5? -10? geee, man, do u really know what are you doing ? where from did you get that 'extra-sense' b-frame settings ? piling up b-frames always increases compressibility (unless forcing them to use q1/2 exclusively) why not use (try) 'defaults' before tampering w/extra settings ?

btw, what is 'qf' and how to get it ?

the bests
y

[Teegedeck]

TiaoMacaleh, if you don't like noise you could try removegrain(mode=8). Get the AviSynth plugin removegrain, put it into your AviSynth plugins directory, add that line as a final line to your avs, open the avs in your player and decide whether you like the image better than the look of the original DVD - before you encode with that avs.

Also, try searching the forum on 'anamorphic'.

[TiaoMacaleh]

Both rips looks almost perfectly :D

Question what max resolution i can go for stand alone compatibility?

[Shinigami-Sama]

Originally posted by TiaoMacaleh
Both rips looks almost perfectly :D

Question what max resolution i can go for stand alone compatibility?
depends your player mate ;0

[ChronoCross]

Originally posted by yaz
geee, man, do u really know what are you doing ? where from did you get that 'extra-sense' b-frame settings ? piling up b-frames always increases compressibility (unless forcing them to use q1/2 exclusively) why not use (try) 'defaults' before tampering w/extra settings ?

btw, what is 'qf' and how to get it ?

the bests
y


I have to disagree. In some cases the more bvops there are the less compression it gets. when using 3 consecutive B frames there is a noticeable increase in filesize. it has to do witht he way the quants of I and P frames are calculated, I can't remember what it is but it was explained by koepi in another thread why this might occur.

[lordadmira]

Originally posted by ChronoCross
In some cases the more bvops there are the less compression it gets. when using 3 consecutive B frames there is a noticeable increase in filesize.
There is no "magic number" of B frames that is "best". For any given scene there is a sweet spot on the number of B frames but that number is only valid for that scene. Xvid is somewhat smart but not *that* smart in deciding where to put B frames. Maybe someday Xvid will have brute force BVOP decision where the optimum sequence can be deterministically arrived at. Then the Bsens and Bmax setting will be obsolete.

Having many BVOPs can make the file bigger when u set settings that force the codec into making bad B/P decisions.


LA

[Shinigami-Sama]

Originally posted by lordadmira
maybe someday Xvid will have brute force BVOP decision where the optimum sequence can be deterministically arrived at. Then the Bsens and Bmax setting will be obsolete.
LA
maybe try and put that into first pass coding?
make it detect low motion scenes and put a flag for it to use more b-frames there
of course that could have hte potinal drop filesizes another 10-20% so moer people may start complaing about undersize:rolleyes:

[Laffer]

I use GK and xvid (latest version I think) I have been going through encoding all my DVD's for eventual use with M$ media centre.

I want these encodes to be good quality. I always encode with the AC3 soundtrack (not converted to MP3)

Anyway I have encoded over 100 films with great results, more to the point GK/Xvid have 'obeyed' my bitrate setting. I have come to 2 films however that no matter what I do, come out incredibly undersized with a bitrate of around 750ish (I tend to go for a minimum bitrate of 1024 OR 0.250 bits/(pixel*frame, whichever gives the lowest file size)

The larger of the two films is 2 hours 48 minutes and the AC3 alone was over half a gig. I specced 2400MB for the total file size (inc audio) The actual file produced is 1500MB - 900MB difference!!!!
This is at AS@L5, quant set to H263 and 640x352 res.

I don't think this is a coincidence, something somewhere has changed (is there a setting somewhere in xvid to IGNORE passed settings OR does xvid always try to optimise the size? if the latter, how can the size ever be accurate?)

Anyways, after doing so many encodes with the expected results, then to get two in a row that are no where near (and I have re-enc's 3 times on each now)..... I just think something may be borked (although my system has not changed at all what so ever).

Anyways, any guidance would be appreciated. By the way I have read most of the undersize topics etc.