how to best use xvid

[Rasi]

i have done the following now:

ripped dvd, made an dvd2avi file and converted it with vfapi...
then did the 1st pass in VD and read the stats file in nandub...
and do the 2nd pass then...

could i also just add the dvd2avi file in GK and do the 1st pass there?

then using the avs file and fast recompress in VD?

[Killah]

I completely missed this Xvid thing... But how do I use it in combination with Gordian Knot?

Is this correct?


Do a first pass with virtual dub, using the Xvid first-pass, 2-pass encoding. And write down where the .stats file is kept
Read the createt stat file in Gordian knot, and re-calculate
Do the second pass, with the original stats file and the gordian knot stats file

[Teegedeck]

That's correct, Killah, just don't forget NOT to use luminance-correction in GKnot.

[Killah]

Woooow Ultra fast response!!!
Thanx anyway... Gonna do my first Xvid tonight...

Any tips for settings? cause I see a lot of people ask about it over here the forum, but they get all kinds of political answers :sly:

[Rasi]

still my question... can i only do the 1st pass with vfapi?

[-h]

You don't have to use vfapi - reading the .d2v file you created via avisynth will be both faster and probably look a bit better too (you can use fast recompress in virtualdub, and avoid unnecessary conversions).

-h

[Teegedeck]

You can always try, but use the same resize/cropping parameters. Also, I don't know whether the internal YUV/RGB-conversion that I think VFAPI does could have produced a 1st pass that might be considerably different from the 2nd pass with an .avs-file. Basically avisynth allows you to do the whole encoding in YUV-colourspace, while VFAPI - as far as I remember - does an internal conversion to RGB and back to YUV. But I might be wrong there because I haven't touched upon VFAPI for a very, very long time. Anyone with better memory/knowledge?

[EDIT:] Perhaps I got you wrong. Was this what you intended to do?:

You have already done the first pass using VFAPI, but now you want to do the second pass with avisynth only?

[Rasi]

yes i have done a 1st pass with vfapi, but i dont want to ;)

cause its slow...

i want to use avisynth for the 1st pass.. but if i do that in GK the stats file will be produced using the divx3 codec.. if the stats would include the same data there, there shouldnt be a problem, but i doubt divx3 creates a smiliar stats file to XviD....

[-h]

The .stats file format is the same for Nandub and XviD - the decision was a while back made that people would rather work with .stats than DivX4's .log format.

All people have been doing for XviD 2-pass encoding is performing the first pass with Avisynth, loading the .stats file in GKnot (same as you would with Nandub), disabling luma/motion correction, then doing XviD's second pass with the GKnotted .stats file added.

Heh I still haven't tried it myself though. Teegedeck started a promising guide that I hope to polish up tomorrow - should clear up some of the confusion.

-h

[Rasi]

perfect, thats what i wanted to hear ;)

thx

[Teegedeck]

...I hope. But there was something in one of Rasi's posts that startled me and makes me think I probably still didn't get him right:

... but if i do that in GK the stats file will be produced using the divx3 codec

That indicates that you want to use the automated jobcreation in GKnot? In that case I'd have to say, you can't do that. GKnot always creates jobs for Nandub, not for VirtualDub and hence will always use the 3.11 codec. But perhaps I still get something wrong, pardon me.

[Rasi]

oh dear... here is what i am doing.. it seems to work...

opening d2v file in GK... doing all the settings, like cropping and final size and so on...

then save the first pass... which creates a avs and a stats file...

then i open the stats file in gk again and recalculate...

finally do the 2nd pass in virtual dub using the new created stats file...

correct me if something coulnt work there....

[Teegedeck]

Bear with me :) You're all doing it manually, that's finde. I'm just an old codger that never uses anything automated like the jobcreation, so I really like the way you work. ^_^

[Killah]

Okee so to make things clear (mostly for myself ;) )


Rip DVD (smartripper)
Create a project file using DVD2AVI
open the .d2v project file in Gordian Knot
Do some resizing cropping...
Save (do not encode!!!) the .avs file
open the created .avs file in Virtual Dub
Do the first pass (with full processing) to create the .stat file (this file is created by the .xvid codec)
load the stat file in Gordian knot, do you're nasty funky stuff in Gordian Knot, and make sure not to have the luminance-correction on
Save the stats file (this will be the Gordian Knot stats file)
Go to Virtual dub for the second pass, use the .stats file created by the codec, and the stats file created by Gordian knot (in that order) in the codec settings screen... Set fast recompress...
Do the second pass

[Killah]

Originally posted by Rasi
oh dear... here is what i am doing.. it seems to work...

opening d2v file in GK... doing all the settings, like cropping and final size and so on...

then save the first pass... which creates a avs and a stats file...


correct me if something coulnt work there....

But this will be a .stats file created by Nandub using the divx 3.11 codec. Am I right?

[sierrafoxtrot]

just wondering if using xiphrom to generate the 2nd stats file would work ... i'm not exactly sure how to go about it in gordian knot. <stupid question warning> how does one change the bitrate of a 1st pass stats file in GKnot? i've always used automated 2-pass so i'm at a bit of a loss. i know how to open the stats file, and to mess with motion correction, but where do you go to set new bitrate?

thanks for any help anyone can offer.

cheers

SF

[Ripe73]

HI!
What about the other settings in the codec?default?
Quantization type?
Motion search precision?
CBR constants is it only for a first pass CBR?
Debug?
and last Luminance masking?

thanks

[Rasi]

@killah:

Do the first pass (with full processing) to create the .stat file (this file is created by the .xvid codec)

why do you want to do it in full processing mode? makes no sense to me, as it only takes longer...

[Killah]

@Rasi you're right... but I read somewhere that the first pass needed to be done with Full processing...

By the way is someone working on a Howto?

[Ripe73]

Originally posted by Killah
@Rasi you're right... but I read somewhere that the first pass needed to be done with Full processing...

By the way is someone working on a Howto?
HI!¨

Fullprocessing was needed in the first pass when the lumacorrection in Nandub not work but there is a lumafix for Nandub now.Hope you mean that :)

[Killah]

@Rasi

yes you're completely right...
Do you use Nandub or Virtualdub for the first pass?

[Rasi]

virtual dub... couldnt find a way to get other codecs than divx3 working in nandub...

[Teegedeck]

... just wanted to say that a how-to by -h and me is in the works.

[Killah]

oke this is my slightly corrected "workflow"


Rip DVD (smartripper)
Create a project file using DVD2AVI
open the .d2v project file in Gordian Knot
Do some resizing cropping...
Save (do not encode!!!) the .avs file
open the created .avs file in Virtual Dub
Do the first pass (use fast recompress for speedup) to create the .stat file (this file is created by the Xvid codec)
load the stat file in Gordian knot, do you're nasty funky stuff in Gordian Knot, and make sure to turn off the luminance-correction and motion correction
Save the stats file, give it a .gkstats extension (this will be the Gordian Knot stats file used in the second pass)
Go to Virtual dub for the second pass, use the .stats file created by the codec, and the stats file created by Gordian knot (in that order) in the codec settings screen... Set fast recompress...
Do the second pass
Be Happy... and forget about the "Why My Video takes 8 times longer to encode now" format ;)

[Doom9]

a few hints:

1CD: motion search precision: 6, quantization type: H263, luminance masking on for the 2nd pass ONLY.
2CD: motion search precision: 5, quantization type: mpeg, luminance masking off for both passes.

directly from one of the programmers ;)

and.. you can use fast recompression in both passes. In Gknot load your stats file, uncheck both motion correction and luma correction, press calculate and save the stats file again.

[SleepEXE]

1CD: motion search precision: 6, quantization type: H263, luminance masking on for the 2nd pass ONLY.
...
directly from one of the programmers ;)

I had followed this advice (in bold text) when testing the Jan 7 build but the bitrate control seemed to get skewed. I was only able to hit the desired file size if the luminance treatment was consistent between the two passes. Using the recommendation, the file ended up ~950MB vs the desired 1125MB.

Can anyone confirm or refute this behavior?

Best regards,
SleepEXE

[Rasi]

ahh, crap!

the short clips worked perfect.. now i tried to do a whole movie...
load the avs in virtual dub and chose the codec.... do the settings
and after 1 minute i already have a 400MB file... and that in the 1st pass....

...anyone knows what went wrong there?

[-h]

I had followed this advice (in bold text) when testing the Jan 7 build but the bitrate control seemed to get skewed. I was only able to hit the desired file size if the luminance treatment was consistent between the two passes. Using the recommendation, the file ended up ~950MB vs the desired 1125MB.

I can only suggest trying the newer builds with Isibaar's new lumi masking code. The situation you described is very possible, depending on how dark the movie is and how high a target bitrate you have.

-h

[Doom9]

hehe.. I forgot that.. not sure when the changes were made but try a build from code dated 21/1 or later... what I posted only applies to the latest codebase

[-h]

the short clips worked perfect.. now i tried to do a whole movie...
load the avs in virtual dub and chose the codec.... do the settings
and after 1 minute i already have a 400MB file... and that in the 1st pass....

Sounds like it's saving the 1st pass bitstream - go to the Advanced tab, and there should be a checkbox for "Discard first pass bitstream" - make sure it's checked, or your first pass will dump a huge file.

-h

[SleepEXE]

Hi Doom9 and -h, actually I have moved on to the newer builds as they become available. However, I was curious if anyone has tested the Jan 12 or Jan 20 builds to see if file size predictability is accurate when luma masking is enabled in the second pass...

I'll probably test it tonight...

Best regards,
SleepEXE

[-h]

If you have lumi masking enabled for both passes, the end result will be more accurate size-wise, but look far worse. This is because during the first pass, dark frames are being quantised heavily, then the quantisation is stepped up *again* in the second pass, leaving the frame completely savaged.

If you only enable it in the second pass, the codec will have to depend on its bit bucket (which is usually very good), and when a dark frame is encountered which uses way fewer bits than was expected, the subsequent frames will probably be upped in quality. If you're aiming for a 2-CD rip, I would recommend setting the min quantiser to 1, and the max to 6 or 7 to reign it in.

-h

[Teegedeck]

I'm happy to hear that activating lumi-masking in 2nd pass only seems to be a method that everyone agrees on. Not perfect, but well... We just should still restrict quantizers whenever using it, right? But I still don't like lumi-masking when I encode at high bitrates, anyway - sorry. :b

[Killah]

@Doom9

I'm doing a test right now with Xvid and the settings you recommended... I'll post as soon as I know what came out of it...

[Doom9]

I encoded matrix for 1 CD 3 times, once with the 12/1 build, and two times with private builds isibar sent me (though it's cvs code)... each time the size was perfectly right. The 2nd and the 3rd time I turned on lumi masking in the 2nd pass and didn't restrict the quantizers and the size was still accurate. isibar told me that they should've fixed pretty much all the issues people were having with lumi masking earlier. but as I said.. only enable it during the 2nd pass

as for higher bitrates (2CDs...).. there lumi masking shouldn't be used.. I'm still encoding spr (there was a bug in the mmx optimized quantizers.. just got a fixed release this morning)

[Teegedeck]

Boy! That's what I call endurance! :)

I'm looking forward to those 'lumi-masking fixed' builds...

[Dali Lama]

@Doom

How is the quality of those xvid rips of Matrix?

Also, where can I download the latest build...or any build for that matter.

Can't wait to start experimenting,

Dali

[Doom9]

quality: you have to wait for the results of my comparison ;)

binary downloads: try nic's thread a little below this one in this very forum.. it has the most up-to-date binary atm. The ones I have cannot be downloaded anywhere.. but you can get the source off the CVS and compile it for yourself. I asked the developers and they think it's best that I wait until they have a milestone release before putting any binaries on my page

haven't tested the SPR rip.. but the size was on target again.. what a great start for the new codec.. I wished divx4 would be as accurate..

[Dali Lama]

Gotcha Doom...

But one thing, I am using the 20/1 build. Is that ok for using the settings you reiterated.

Side Note: Is it hard to compile the latest build? I have no coding experience.

[Doom9]

I've never compiled a build myself.. in fact I don't even have a C compiler on my box and all my CDs with coding tools are at my parents' place.

as for the settings.. I wrote it in this very thread ;) They only apply to builds based on code that was put to the CVS on January 21th.. Nic's binary is from that day so this might work.. maybe you can ask him to compile a new binary from the latest sources.. then you'll have what I'm using now.

[Killah]

@Doom9 I hope you will also mention the encoding speed in your test. I'm using a AMD TB 1000@1200 Mhz, and it sux that Divx 4.11 doesn't support 3dnow optimisations...

[Killah]

Originally posted by Dali Lama

Side Note: Is it hard to compile the latest build? I have no coding experience. [/B]

It compiles quite easy, but I get a .lib file... And don't know what to do with it... longtime since I hacked c/c++ on the windows platform

[Doom9]

sure.. fps will also be part of the comparision.. you'll get pretty much anything you want.. and I will be as mean as usual selecting the thoughest scenes to encode.. found a couple of very good ones in Saving Private Ryan.. splitting files for later reviewing now.

[Ripe73]

HI!
I have done a first pass with Xvid and calculate in GKnot then load the 2 stats file and choose "2 pass second pass" but i cant set the bitrate,its greyed out,whats wrong?Or will the bitrate be calculted in the GKnot statsfile?
I can only set the bitrate in 1 pass CBR

[-h]

It compiles quite easy, but I get a .lib file... And don't know what to do with it... longtime since I hacked c/c++ on the windows platform

The core branch generates core.lib, which contains the codec's encoding/decoding functionality. To create the vfw dll or dshow ax files, just compile the vfw or dshow branches, making sure the compiler can find core.lib when it needs to link to it. By default it looks to C:\xvid\core\build\win32\bin\core.lib , so you might want to change it in the vfw project settings.

-h

[Doom9]

you can't set any bitrate in 2 pass encoding... after the first pass you load your stats file in nandub, set an audio size and final output size in the bitrate tab, load the stats file, uncheck luma and motion correction, press calculate and save the file.. this new stats file will contain the scaled curve so xvid knows much bits to give each scene.. all my test so far were right to the point sizewise.

[Ripe73]

Originally posted by Doom9
you can't set any bitrate in 2 pass encoding... after the first pass you load your stats file in nandub, set an audio size and final output size in the bitrate tab, load the stats file, uncheck luma and motion correction, press calculate and save the file.. this new stats file will contain the scaled curve so xvid knows much bits to give each scene.. all my test so far were right to the point sizewise.
HI!
You mean load statsfilen in GKnot i hope?Im using Vitualdub.So Xvid cant be used without GKnot with 2pass?

[Killah]

Originally posted by Ripe73

HI!
You mean load statsfilen in GKnot i hope?Im using Vitualdub.So Xvid cant be used without GKnot with 2pass?
Yep that's correct... One of the TODO's in the CVS source, is to make a statsfile editor so Gordian Knot is not needed.

[Dali Lama]

Question

1. Is Doom going to use bilinear or bicubic resize to do the testing?
2. Why hasn't the quantizer values been emphasized in 2 pass encoding? Is it best to leave it at its default min=1 max=31?

Thanks.

Dali

[Doom9]

sorry for the confusion.. of course you load the statsfile in gknot.. not nandub.

other than that.. neutral bicubic it is for all tests..

[Koepi]

I did some tests with xvid as well and i don't like the need for GKnot. A very basic curve treatment (with giving an average bitrate) shouldn't be too hard to implement? Maybe i'll find some time in 3 weeks and code something like that :P

Btw., is it normal that GKnot pops up with an error message like "the frames in the stats aren't the same as in the calculator..." and that only after typing in manually the number of frames (down next to the dvd2avi field) it is possible to calculate a new stats file?

This is all unneccessary trouble with a nice codec and confusing me like hell ;)

All this is IMHO, nobody should feel offended!

Regards,
Koepi

[Ripe73]

HI!
I encoded a movie with Xvid and used 2pass encoding for the movie and 1pass CBR for the endcredits but i got a error when i append the two files"Cannot append segment: The video streams have different data forms".I have the same resolution on both streams,anyone with the same problem?

[-h]

I did some tests with xvid as well and i don't like the need for GKnot. A very basic curve treatment (with giving an average bitrate) shouldn't be too hard to implement? Maybe i'll find some time in 3 weeks and code something like that :P

We finally got a long weekend over here. I hate not being able to code - travelling 4 hours a day kinda kills the urge :)

I'm going to implement a curve adjustment this weekend (along with finishing the quicktime stuff, add some pointless eye candy to the frontend, and some other stuff that I probably won't finish anyway), but I'm at a loss as to how to still allow GKnot modifications to the first-pass .stats file. Perhaps a toggle that preserves the current behaviour?

-h

[Teegedeck]

@Koepi: Hummm, I think that you could perhaps have chosen a wrong framerate?

@Ripe73: I get that error, too. Try merging video-streams with AviUtil and mux the audio lateron.

@-h: That would be fine with me... I believe manual editing still has a future with many of the people around here.

[Ripe73]

Originally posted by Teegedeck
[B
@Ripe73: I get that error, too. Try merging video-streams with AviUtil and mux the audio lateron.

[/B]
I cant get a error "unsupported format"when i try to open the first avi:(

[Teegedeck]

Originally posted by Ripe73

I cant get a error "unsupported format"when i try to open the first avi:(

Try this in AviUtil:
file >> avi-operation >> concatenate

In the dialogue that follows, add the files you want to merge, type in where you want your output-file and how you want to call it (don't forget that the name should end in '.avi') and press 'OK'.

[Ripe73]

Originally posted by Teegedeck


Try this in AviUtil:
file >> avi-operation >> concatenate

In the dialogue that follows, add the files you want to merge, type in where you want your output-file and how you want to call it (don't forget that the name should end in '.avi') and press 'OK'.

I only get a 2kb avifile nothing more and i cant open a DivX 3.11 in AviUtil either

[Teegedeck]

Originally posted by Ripe73


I only get a 2kb avifile nothing more and i cant open a DivX 3.11 in AviUtil either

I'm a bit at my wits end right now - AviUtil never let me down before. Even when it displays a warning, something like 'can't recompress, so output file may be abnormous' (or similar), the end result always works for me. Maybe if you give some more details about how you proceed, someone will detect a mistake or have an idea. Otherwise, you can alyways try the file-merging utilities on Doom9's download-site.

[Ripe73]

I will do a new first pass without trim endcredits because it seems to be impossible to append video for me with XviD :mad:

1.How will Xvid work with bits/pixel when i set the resolution in GKnot should follow the DivX 3.11 rule?

[Teegedeck]

Generally you can expect XviD to compress a little better at high bitrates (2 CDs), but it all depends on personal taste.

[Ripe73]

HI!
Good news its now possible for me to append with Xvid,i download the AviUt on Dooms downloadsite and then i download aviutl97d on this http://ruriruri.zone.ne.jp/aviutl/ site and execute the *.exe file with the new.Then open the movie.avi->append credits.avi->save avi:)