Basic Xvid questions

[Richard Berg]

Getting back into Xvid, got some simple questions that I don't see in the FAQ or Crusty/Koepi/etc. websites or in the last few months of discussion here.

- What are some ways people use to choose a target filesize? I've heard one rule of thumb is to take the "recommended size" from your 1st-pass stats file and divide by 2, but figure you guys have better ideas.

- Does Xvid expect PC (0-255) or CCIR (16-235) YUV values?

- The 1st pass runs about the same speed no matter how many advanced options I turn on, is this normal? Not complaining :)

[Manao]

- What are some ways people use to choose a target filesize? I've heard one rule of thumb is to take the "recommended size" from your 1st-pass stats file and divide by 2, but figure you guys have better ideas. It depends of the custom matrice used, but for h263 / hvs best, it's usually the ratio I aiming for - Does Xvid expect PC (0-255) or CCIR (16-235) YUV values?PC i think- The 1st pass runs about the same speed no matter how many advanced options I turn on, is this normal?Yep, it's called fast first pass : basically, in the first pass, you only needs an estimation of the frame sizes at quant 2, so you don't have to bother with vhq, trellis & qpel, gmc and the like.

[Koepi]

Choosing a target file size:

1) Do a first pass. Look at the resulting file size.
2) Depending on that, develop some good guessing from guts whether this should be a 1CD, 1XCD, 2CD,... encode. (Rule of thumb: compression ratios from 1:1 up to 2.5:1 [first pass size vs. desired 2nd pass size] can look good)
3) Encode your audio.
4) Use a bitrate calculator to subtract the audio size from your desired size (including container overhead). That way you'll hit your desired size within +- 500-100kb.

Used YUV scale:

I don't have the sources around to look at them, but _I think_ to remember that we played with TV range for a little time and switched back to full PC scale. Can't say for sure without looking at the sources unfortunately.

Constant 1st pass speed:
Agreed with Manao here. It's the fast firstpass option which causes this behaviour. If you'd used a different sized video you'd notice a speed difference though ;)

I hope this helps

Cheers
Koepi

[Teegedeck]

For an alternative way to determin target filesize that I tend to use recently:

Choose the matrix that you think would give your encode a good look, depending on the content (for example: film grain would require SixOfNine; clean HDTV could do with SixOfNine-HVS) and the level of compression you want to reach over the original file (for example SixOfNine should probably not be used to reach more than about 40% compression over MPEG-2; but that it does fabulously, no visual quality-loss, I would argue). No resizing at this stage of course, only cropping.

Make a size-prediction with Enc at the quantizer you regard the highest that should be used in your encoding (for example; for high quality I would use SixOfNine @ quant=3, for non-hq stuff SixOfNine @ quant=4 or 5). If the result (predicted size) comes out, say, less than 15% smaller or bigger than what seems a sane choice of target filesize (for example "half a DVD minus soundtrack", there you have it. If not, repeat with a higher quant, choose a different matrix, resize etc.

It is a very good idea to make the first pass at the quantizer that gave you a good result in the Enc filesize prediction test. 2nd pass then should deliver something as close to a constant quantizer encode as possible (IF you turn off VBV).

Hope that might be helpful.

Tee

Edit: On how to configure XviD for a fast first pass at a quantizer other than 2, I think you'll find all information here (http://forum.doom9.org/showthread.php?s=&threadid=86174)

[pelle412]

- What are some ways people use to choose a target filesize? I've heard one rule of thumb is to take the "recommended size" from your 1st-pass stats file and divide by 2, but figure you guys have better ideas.

I personally use avs2avi to encode my XviDs, and primarily for TV viewing. I start with a full quality first pass (h263, Turbo ON, VHQ 1). Once this encode is complete I run a home-grown tool that takes the size of the first pass file and multiplies with a compression scale (between 60 - 80% depending on my mood, mostly I pick 75%) and the tool spits out a new .reg file which my avs2avi script picks up and then I start the 2nd pass (Turbo OFF, VHQ 4). I guess this always gives me a similar quality level between encodes and I don't care so much if one encode becomes 400MB and another 900MB since I burn them all to DVD-R anyway.

[Poutnik]

[i]Originally posted by Teegedeck : SixOfNine

BTW. Does sixOfNine stands for something special, or is it just name ?

[Boulder]

Use the search! It's a custom quant matrix.

[Poutnik]

No need for exclamation. I know it.

[Richard Berg]

@Teegedeck: what do you mean by "Make a size-prediction with Enc"?

These videos aren't terribly high quality; I'm just concerned about getting the right bitrate because they are very hard to compress (sports w/ animated graphics overlaid). I think I'll try the const-first-pass trick @ quant 4.

Related question: does the minimum quant setting affect the behavior of fast first passes? I heard from someone that leaving it at 1 (default) could cause my 1pass size to be way too big. Likewise, if that were true, then I could get your suggested effect without messing with zones.

PS if it's not obvious yet, these don't need to be multiples of 700MB or anything like that...I don't even have a burner :)

[Teegedeck]

Nah; first pass is always done at quant=2 except if you specify otherwise by using a zone with constant quantizer. So setting min.quant to 1 doesn't do a thing in 1st pass. In 2nd pass though it can needlessly eat bits. I would set it to '2'; the '1' is just there to prevent the the question 'why do I get undersized files?'.

Enc (http://forum.doom9.org/showthread.php?s=&threadid=50714) is a very good tool for quite accurate filesize prediction and compressibility tests. As heavy quantizer fluctuations in 2nd pass are not a good thing, I try to guess a good first pass quantizer with Enc (size prediction with a constant quantizer encode) which I then use for first pass in XviD by using the 'zone' feature.

Edit: BTW, for horribly overcompressed DVB-T stuff like music videos I actually like the look you get with those 'low bitrate' matrices (Sharktooth's ULR, Javor's 1CD).

Edit: As I've been experimenting with DVB-T a bit recently, I've found doing a first pass with SixOfNine-HVS at quant=5 quite satisfactory lately. I aim for about 50-60% size-reduction without much visible loss.

That is, together with some denoising; I use a script like that one, perhaps it could use improvement:


mpeg2source("dvb-t.d2v",idct=7,cpu2="xxxxoo",moderate_h=25,moderate_v=40)
colormatrix()
removegrain()
tdeint(order=1,mode=1,mtnmode=1,type=2)
fdecimate(25.00)
removedirt(mode=2)

[Richard Berg]

The Enc hosts appear to be broken. Anyone have the latest version?

[jeffhuge]

y,cannot be linked

[Boulder]

http://www.saunalahti.fi/sam08/enc_1_03.rar

Don't know if it's the latest one though.

[pwh04]

For me, only quants lower than 4 will do. I like to use bitrates under 1500 to as low as 1200. I like 1 pass only. After much testing, I seem to have found a very consistent style that will give me quality comparable to one pass constant quant 3 (but with size control), the speed of 1 pass, and good brate control (thus file size prediction). I use Gknot as my front end setup for:

Gknot:
dvd2avi
Brate calculator
Audio set up/transcode using besweet (to constant bitrate mp3)
Resolution: usually 528/ 544
Cropping: usually 4-4-4-4)
resizing: lanczos
deinterlace: tomsmocomp (thus multiples of 4 seen above)
Avs setup: No Filters used!

XviD settings:
one pass
quant zones: 2-4 2-4 2-4
trellis
profile: as @l5
custom quant matrix: hvs best (saves detail/nice compression needed for such low quant range)
target bitrate (not quant):1200 - 1500
adaptive quant (for better compression)
vhq: 2
vhq for brames
bvops
chroma motion
chroma optimizer

I mux the audio in nandub after the encode finishes (approx 3 min and always in sync) and I have a very high quality (near constant quant 3) file usually just under my desired file size. To me, I feel I have found a great combination of one pass speed, quality (one pass constant quant 2 or 3), and file size prediction (normally only achieved in 2 pass).

If you must have a file size at or just under, just take the bitrate from gnot calculator and input slightly less in xvid target bitrate.

[Richard Berg]

Ok. I tried a test encode (10k frames from the middle of a game) with these settings:
- one pass @ quant 4
- zone @ quant 4 over whole file
- SixofNine-HVS matrix w/ trellis
- VHQ 1
- all other settings (bvop, qpel, gmc, adaptive, turbo) off

This gave a file of 231MB, or 11Mbps. I know 60fps sports will need a lot of bits, but that seems excessive especially for quant 4. Stepping down to H.263 / quant 5 still gives 6Mbps (12MB / 1k frames).

encoder: XviD 1.1 beta via AVS2AVI 1.39

[Boulder]

What kind of a script did you use? It might be useful to do some motion adaptive denoising.

[Richard Berg]

Here's the script. We can discuss the details in the Avisynth forum if needed...

SetMemoryMax(200)

SetWorkingDir("r:\")
src=Avisource("capture (2005-01-05 at 19-00-11).avi",
\ "capture (2005-01-05 at 19-02-09).avi",
\ "capture (2005-01-05 at 19-10-08).avi")
src.AssumeTFF # in DG filters, order=1 is TFF

#VCF2AVS output - paste cuts below
Trim(1151,13161)+Trim(15205,30500)+Trim(34096,46107)+Trim(49718,65697)+
\Trim(69302,88115)+Trim(114893,128629)+Trim(132232,141615)+Trim(145218,150892)+
\Trim(154505,168282)+Trim(170984,187836)+Trim(191439,206378)
cap = last

# TODO: nice automated framework for cropping/bordercontrol
#Crop(0,2,0,0) # for Compro Ultra
crop = last

MvBob()
decomb = last

#TemporalSoftenPreset(preset = "sh0dan-soft").PixieDust(2) # IIP's method
#PixieDust(2)
#DeGrainMedian(limitY=4,limitUV=7,mode=1)
FFT3DFilter(sigma=2)
clean = last

LimitedSharpen(dest_x = 640, dest_y = 480, ss_y = 1.8, soft=false)
resize = last

ConvertToYV12.YLevels(gamma=0.9, input_low=6)
#Histogram(mode = "levels")
gamma = last

#BoxCompare(decomb, clean, resize, gamma, "decomb", "clean", "resize", "gamma")
gamma
Trim(10000,11000)

I previously did a compressibility test of the postprocessing. 1000 frames at xvid quant 2 gives these filesizes:
No postprocessing: 26.9MB
LimitedSharpen: 29.2MB
FFT3D + LS: 23.2MB

So obviously I left in both filters since they look good to me and actually give a more compressible file. As you can see I also tried a couple other denoisers but they gave too many artifacts for a sports video. Now I just need to know what I'm doing wrong that makes xvid quant 4 output 11000 Kbps.

Here is a sample frame:
http://richardberg.net/bin/temp/duke-before.jpg
http://richardberg.net/bin/temp/duke-after.jpg

Here (http://richardberg.net/bin/temp/xvid-quant4-6of9hvs.avs2avi) is my avs2avi config file if anyone wants to double-check the settings.

[Boulder]

The result looks very good indeed, but the processing speed must be really low:D

See this thread, particularly scharfis_brain's post : http://forum.doom9.org/showthread.php?s=&threadid=81241&highlight=depan . You can denoise quite heavily in temporal dimension with a little help from depan. It might help you with a high-action video.

I don't wonder why you got 11Mbps, high action at 60fps is quite demanding;) Wouldn't it be possible to bob the video with ffdshow during playback?