Near lossless quality with xvid from dvd?

[SiKKo]

Hopefully someone can enlighten me on this subject. Basically, what I want to do is take my TV show DVDs, such as Seinfeld, Simpsons, South Park, Arrested Development, etc and convert all the episodes to xvid using AutoGK. I have actually done this for a couple of seasons so far, with seemingly great success.

The question I have however is that I have been using a specific file size (say 400MB for example, so I can fit 11 episodes per single layer DVD). So far I have just done 4:3 aspect ratio shows, and thus I have been using a fixed resolution of 640x480 in AutoGK. I've been keeping the original AC3 files which typically range around 30mb for a 22 minute program. The quality seems to be very good, at least watching on my regular 32" Sony tv (non-HD) via my attached home theatre PC (Windows Media Center 2005 based).

What I am realizing however, is that since these are basically all going to be saved on a hard drive and played through the HTPC (not a set top player), is that I should be more concerned about quality and not necessarily file size. Basically, I do not ever want to have to re-encode these episodes if I get an HDTV in the next couple of years, and want to make sure I have the highest quality possible.

So assuming that file size is not a factor, should I be using 'target quality' in AGK as opposed to 'custom size'? It seems to me that using the 'custom size' - which uses 2 passes - would produce better quality than the single pass 'target quality' option. Does that make sense at all? Let me give an example to clarify. Say I run two batches in AutoGK...first I do 'target quality' of 85, which then produces a file that is 455MB. If then I run (using the same IFO source of course), the 'Custom size' option in AutoGK using a size of 455MB, it does a 2-pass encode of the episode. Either way, both files end up at 455MB. What I am wondering is will the one that used the 'Custom size' option be of better quality since it did 2-pass instead of single pass?

To sum up, basically I guess what I want is to have xvid files which are as close to humanly possible as the source DVD - so what settings should I use? Just put 'target quality' at 100? Like I said, I want to do this right the first time so I don't have to redo all of this again 2 years down the road when I have an HDTV.

Thanks in advance!

[jggimi]

Hello, and welcome to the forum.

Single pass encodings use fixed (or effectively fixed) quantizers. Size is unpredictable, but quality will be consistant. 100%, with bi-directional encoding disabled, will usually render the best possible results with any MPEG-4 ASP codec, such as DivX or XviD.

Multi-pass encodings analyze the video in the first pass, and make decisions based upon your desired average bitrate (or filesize, same thing). As such, they will have scenes with less bits, and scenes with more.

[check]

xvid constant quantizer 2 sounds to be what you're looking for.

[SiKKo]

Thank you for the responses. Am I able to disable bi-directional encoding, or to use constant quantizer 2, within AutoGK, or do I have to use regular Gordian Knot?

I hope this is not bad to do, but I am going to post one of the logs from AGK I ran last night, is there a way to tell what quantizer it is using from this? Or would you guys, obviously without seeing the actual video, be able to tell from this log whether this video "should" look good on an HDTV? I know it is always a subjective thing, but just generally speaking.

[7/27/2006 7:39:55 PM] AutoGK 2.27
[7/27/2006 7:39:55 PM] OS: WinXP (5.1.2600).2
[7/27/2006 7:39:55 PM] Job started.
[7/27/2006 7:39:55 PM] Input file: C:\SEINFELD\SEASON_02\EPISODE_201-THE_EX-GIRLFRIEND\VTS_02_0.IFO
[7/27/2006 7:39:55 PM] Output file: C:\SEINFELD\SEASON_02\EPISODE_201-THE_EX-GIRLFRIEND\EPISODE_201-THE_EX-GIRLFRIEND.avi
[7/27/2006 7:39:55 PM] Output codec: XviD
[7/27/2006 7:39:55 PM] Audio 1: English AC3 2ch
[7/27/2006 7:39:55 PM] Subtitles: none
[7/27/2006 7:39:55 PM] Format: .AVI
[7/27/2006 7:39:55 PM] Target size: 415Mb
[7/27/2006 7:39:55 PM] Custom resolution settings: fixed width of 640 pixels
[7/27/2006 7:39:55 PM] Custom audio settings: Original
[7/27/2006 7:39:55 PM] Started encoding.
[7/27/2006 7:39:55 PM] Demuxing and indexing.
[7/27/2006 7:40:39 PM] Processing file: C:\SEINFELD\SEASON_02\EPISODE_201-THE_EX-GIRLFRIEND\VTS_02_PGC_01_1.VOB
[7/27/2006 7:40:39 PM] Source resolution: 720x480
[7/27/2006 7:40:39 PM] Found NTSC source.
[7/27/2006 7:40:39 PM] Source aspect ratio: 4:3
[7/27/2006 7:40:39 PM] Source seems to be pure FILM.
[7/27/2006 7:40:39 PM] Color correction enabled.
[7/27/2006 7:40:39 PM] Output will contain 33072 frames
[7/27/2006 7:40:39 PM] Audio1 size: 33,100,800 bytes (31.57 Mb)
[7/27/2006 7:40:39 PM] Overhead: 229,888 bytes (0.22 Mb)
[7/27/2006 7:40:39 PM] Video size: 401,828,352 bytes (383.21 Mb)
[7/27/2006 7:40:39 PM] Running compressibility test.
[7/27/2006 7:40:39 PM] Writing the following script to C:\SEINFELD\SEASON_02\EPISODE_201-THE_EX-GIRLFRIEND\agk_tmp\EPISODE_201-THE_EX-GIRLFRIEND_comptest.avs
===========================================================
LoadPlugin("C:\PROGRA~1\AutoGK\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\AutoGK\filters\autocrop.dll")
LoadPlugin("C:\PROGRA~1\AutoGK\filters\ColorMatrix.dll")
LoadPlugin("C:\PROGRA~1\AutoGK\filters\RemoveGrainSSE3.dll")

movie = mpeg2source("C:\SEINFELD\SEASON_02\EPISODE_201-THE_EX-GIRLFRIEND\agk_tmp\EPISODE_201-THE_EX-GIRLFRIEND.d2v")
cropclip = autocrop(movie,mode=0,wmultof=4,hmultof=4,samples=10,aspect=0,threshold=34,samplestartframe=0,leftadd=0,rightadd=0,topadd=0,bottomadd=0)
fixed_aspect = 0.888888888888889
c_width = width(cropclip)
c_height = round(height(cropclip) / fixed_aspect)
input_par = float(c_width)/float(c_height)
input_par = (input_par > 1.4) || (input_par < 1.25) ? input_par : (4.0/3.0)
out_width = 640
out_height = round(float(out_width) / input_par)
hmod = out_height - (floor(out_height / 16 ) * 16)
out_height = (hmod > 4) ? (out_height + (16 - hmod)) : (out_height - hmod)
new_aspect = (float(out_width) / float(out_height)) / fixed_aspect
autocrop(movie,mode=0,wmultof=4,hmultof=4,samples=10,aspect=new_aspect,threshold=34,samplestartframe=0,leftadd=0,rightadd=0,topadd=0,bottomadd=0)
LanczosResize(out_width,out_height)
RemoveGrain(mode=2)
ColorMatrix("Rec.709->Rec.601",opt=0,hints=false)
SelectRangeEvery(214,15)
===========================================================
[7/27/2006 7:41:32 PM] Duration was: 53 seconds
[7/27/2006 7:41:32 PM] Speed was: 43.38 fps.
[7/27/2006 7:41:32 PM] Compressibility percentage is: 111.68
[7/27/2006 7:41:32 PM] Using sharper matrix
[7/27/2006 7:41:32 PM] Chosen resolution is: 640x480 ( AR: 1.33 )
[7/27/2006 7:41:32 PM] Predicted comptest value is: 83.87%
[7/27/2006 7:41:32 PM] Running first pass.
[7/27/2006 7:41:32 PM] Writing the following script to C:\SEINFELD\SEASON_02\EPISODE_201-THE_EX-GIRLFRIEND\agk_tmp\EPISODE_201-THE_EX-GIRLFRIEND_movie.avs
===========================================================
LoadPlugin("C:\PROGRA~1\AutoGK\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\AutoGK\filters\autocrop.dll")
LoadPlugin("C:\PROGRA~1\AutoGK\filters\ColorMatrix.dll")
LoadPlugin("C:\PROGRA~1\AutoGK\filters\RemoveGrainSSE3.dll")

movie = mpeg2source("C:\SEINFELD\SEASON_02\EPISODE_201-THE_EX-GIRLFRIEND\agk_tmp\EPISODE_201-THE_EX-GIRLFRIEND.d2v")
cropclip = autocrop(movie,mode=0,wmultof=4,hmultof=4,samples=10,aspect=0,threshold=34,samplestartframe=0,leftadd=0,rightadd=0,topadd=0,bottomadd=0)
fixed_aspect = 0.888888888888889
c_width = width(cropclip)
c_height = round(height(cropclip) / fixed_aspect)
input_par = float(c_width)/float(c_height)
input_par = (input_par > 1.4) || (input_par < 1.25) ? input_par : (4.0/3.0)
out_width = 640
out_height = round(float(out_width) / input_par)
hmod = out_height - (floor(out_height / 16 ) * 16)
out_height = (hmod > 4) ? (out_height + (16 - hmod)) : (out_height - hmod)
new_aspect = (float(out_width) / float(out_height)) / fixed_aspect
autocrop(movie,mode=0,wmultof=4,hmultof=4,samples=10,aspect=new_aspect,threshold=34,samplestartframe=0,leftadd=0,rightadd=0,topadd=0,bottomadd=0)
LanczosResize(out_width,out_height)
RemoveGrain(mode=2)
ColorMatrix("Rec.709->Rec.601",opt=0,hints=false)
===========================================================
[7/27/2006 7:48:03 PM] Duration was: 6 minutes 31 seconds
[7/27/2006 7:48:03 PM] Speed was: 84.54 fps.
[7/27/2006 7:48:03 PM] Expected quality of first pass size: 82.02%
[7/27/2006 7:48:03 PM] Running second pass.
[7/27/2006 7:59:43 PM] Duration was: 11 minutes 39 seconds
[7/27/2006 7:59:43 PM] Speed was: 47.30 fps.
[7/27/2006 7:59:43 PM] Job finished. Total time: 19 minutes 47 seconds

[unskinnyboy]

[7/27/2006 7:39:55 PM] Target size: 415Mb
This is a 2-pass encoding and will use variable quantizers. Means different quantizers for different frames, based on the complexity (or simplicity) of the source. You should be doing a quality based encode here. At one point (maybe at 100%), AutoGK will use quant 2 and disable b-frames. Quant 3 is somewhere around 67-68%. It never uses quant 1 though.

[SiKKo]

Right on, thanks. I reckon I will start using 100% quality from now on then. For the meantime, I will just hope the seasons I have already finished will look good in that mystical day when I finally can afford to buy a nice HDTV.

[BigDid]

Right on, thanks. I reckon I will start using 100% quality from now on then...
Hi,
You will still not get B-frames disabled. For that you will need (2 pass mode) to saturate the codec and AGK will try to get under the 100% quality and disable B-frames in the process.
Pick your 100% size multiply by 1.5 ( no big deal, you will get an undersized file anyway) and go 2 pass full auto, that should get you with the maximum quality for AGK (search a thread related to King-kong if you want some other testimony).

That being said you may attain a visually lossless encode (bitrate above 2000kbps) but not a real near lossless one. For that you will need an encode (1 or 2 pass) without B-frames and a near lossless CQM like the V3EHR from Sharktooth. It is not directly doable in AGK and best done with VdubMod or Arcalculator; the trick to begin is just to re-use the AGK movie.avs and feed that in VDM or ARCalc; You will just have to learn to set the params for encode, not the avisynth or all the Xvid params and that makes a big difference in the beginning :)

Did

[Pulp Catalyst]

i'm stretching out on a limb here so please don't shoot me down,

but i always thought that a encoder can't pick up all the information possible on the 1st (also sometimes called analysis) so on the 2nd pass it will correct any errors it made on the 1st pass.

logic would suggest (unless the codec in question was 100% accurate, which considering Xvid's development i don't think it is) that the 2nd pass would do what you said above but also correct any mistakes floating in the 1st pass.

it's just that i find it hard to believe that a codec can can make all the right decisions on just 1 single pass unless the codec was 100% error free?

again i don't mean to split hairs, and please don't shoot me down, but a analysis comes with all encoders and i would of thought that this helps in the decission making of the codec for the 2nd pass that will increase the quallity overall.

if i'm talking s**t please be nice hehehehe...

[jggimi]

You misunderstand, but we won't shoot you down.

MPEG Quantization is the technology used to compress video. In a quality based single pass, there is no analysis. Instead, there is a fixed quantizer used -- granted, with codecs that use percentages, some percentages will cause cause cycling of quantizer values. But at 100% quality, you will have a fixed quantizer value used for every frame.

For a terrific guide on MPEG -- granted, written for DivX 5, but valuable for understanding how any MPEG codec works -- grab a copy of the User Guide for DivX 5. Its helpful for all types of MPEG, but particularly helpful with MPEG-4 ASP codecs like XviD and DivX. And, it is available in five different languages.

http://www.divx.com/divx/windows/codec/guides/

[Pulp Catalyst]

read some of that guide, thankyou for that, very interesting guide, should have fun reading it from time to time,

did some other research, and your absolutely right, i stand corrected, i appologize,

one thing i did find interesting though "1-pass quality based mode is not explicitly unsupported by divx certified devices, however the video buffer verifier is disabled in this mode and there is no guarantee that the video stream produced will not exceed the capabilities of the certified device."

but no doubt you knew that and maybe that is no longer a issue with the more updated codecs.

thanks for putting me right, and thanks for not shooting me down, as some others would of,

respect, live long and prosper my friend.

[jggimi]

Because the bitrate in complex scenes can reach maximum, this can be well beyond the limited capability of "Home Theatre" profile used with set top boxes. But as you are using a PC for playback, I didn't consider this a limitation you needed to worry about.

[BigDid]

Because the bitrate in complex scenes can reach maximum, this can be well beyond the limited capability of "Home Theatre" profile used with set top boxes. But as you are using a PC for playback, I didn't consider this a limitation you needed to worry about.
Yes, the statement (or should I say concept?) is interesting, I mean limited bitrate of home theatre profile (4854kbp/s) vs unrestricted profile ( no max bitrate) or 1 pass vs 2 pass. A few things:

Before posting earlier in this thread I made tests (85%) on a small clip for one pass and re-used the same size for 2pass, with the unrestricted profile.
I had quant from 1 to 4 for 1 pass and quant from 1 to 5 for 2 pass and a lower DRF for 1 pass ...
I doubled the size to try the 2 pass-without-B-frames trick just to verify my assumption and looked at the quite different bit (or bitrate) distribution with or without B-frames. I used DRF analyser for that.
I even made a 1 pass encode with the V3-EHR CQM (with B-frames) to try to find differences between the EHR and V2 (Agk sharpmatrix). Result: no visual difference, hence my statement concerning visual near lossless encodes.

Did

[SiKKo]

Dang, you guys mind as well be speaking Arabic or something, because I understood about 1/10th of those last few posts! :) I guess the discussion got a bit over my head....

To sum it up, in laymans terms, will the video I get using a 100% target quality profile look extremely close to the original DVD source when viewing on an HDTV? I am not a complete videophile or anything, but would like it to look as good as possible, you know? I'll probably be using FFDshow to upscale the videos in the future, as I've seen some examples of that and it looks pretty sweet.

I guess the main reason I ask this is because back in the day, I encoded all of my 100s of CDs into crappy 128kbs mp3's before I knew any better, then later realized that sucked, and re-ripped and encoded using EAC/LAME -APS, and finally a 3rd time settled on lossless files now that hard drives are so large and cheap. I just don't want to have to take the time to do that same thing with video now - I'd like to do it right the first time. That being said, I don't necessarily have 100's of hours to spend learning a million different programs - so I was hoping AGK would suit my needs since its very easy to use. Any thoughts?

[CWR03]

If you use 100% target quality, you might as well leave it in its original format since the file size will be about the same, but the quality will still be inferior.

[Pulp Catalyst]

i also agree, if 100% (or aka 1:1) to the dvd source, than perhaps looking at Mpeg4 ASP codecs is not the way to go,

also if using a PC to playback these files only then again, this gives you a lot more options to play with,

i would recommend you use Mpeg4 AVC Codecs these are designed for higher compression in mind, but not always necessarily, and besides the quality of these codecs are very very impressive, only problems with these codecs is,

the resources needed to do the encode (You will need a powerful rig)
and there isn't much Set top Box support (Again due to resources)

but if you plan on playing back through a Pc, and your Pc is a powerful beast, then i would have to recommend Mpeg4 AVC,

and i believe (but don't hold me to it) Nero AVC does hold the crown at the moment,

done some early test a while back, with small chapters, and i was very impressed with the quality, but the codec was very slow for me encoding and decoding (my athlon XP 2400+ was not up for the job),

you will need a 64bit chip to keep up, and even better 2 CPU's on the job.

but as for getting 100% (1:1) matching the dvd source, you will not achieve this with Mpeg ASP codecs, these codecs was not designed for that, they were design to compress, not for Quality (meaning compression as a higher priority than quality) with ASP codecs the codecs have there limits,

just try some tests your self with some small clips,

ASP Vs AVC (nero being the leader, although x264 is coming along nicely)

Live long and prosper, and i wish you luck.

[SiKKo]

I will have to research the difference between AVC vs ASP. I do have a powerful dual core Opteron to encode, although the playback would be on a Athlon 64 3500+ Venice. I would assume that would be enough to playback the files.

Thanks for all of your recommendations, I really appreciate the feedback.

[HymnToLife]

Just out of curiosity (before I encode my DVDs into XviD because my current laptop can't handle AVC decoding smoothly), how would AVC decoding run on the Dual Core Centrino 1,66 GHz I'm getting in a few weeks ?

[Pulp Catalyst]

decoding is not the main problem, granted if you were using all the encoding features this would put a strain on the decoder, but i have heard that people with a p3 1Ghz system has been able to play back the files,

although for this eficiency stay away from nero showtime, nero showtime may look good (interface wise) but trust me it's all for show and no for blow (meaning not efficient at all)

the people out there mostly recommend CoreAVC, and a few others,

again i would try to stay away from codec packs, but i would try and get the film to play through Media Player Classic.

also for you lucky people out there with a latest Nvidia Graphics Card, you could try Pure Video Decoder from nvidia, which claims to support most codecs & put most of the strain through the Nvidia chipset rather than the CPU (you will need a latest GPU though for this, the list is on nvidia site).

[HymnToLife]

Well, on my 2,8 GHz Celeron in VLC, playback was quite horrible. I'll try CoreAVC with a DS player, thanks for the tip :)

[Pulp Catalyst]

cleron based chips can be missing many features, if you have a celeron D, then you should be ok, but celeron based chips usually are quite bad, not to sure on the actual reasons (reasons can be quite technical and beyond the scop of this discussion), however i to have seen even pentium 4 top range CPU's struggle, but this is usually down to bad codecs being installed, and sometimes even worse, down to the way media player classic is set up,

which is the main reason why i gave up with codec packs a long time ago, just install what you need, and windows will appreciate it,

but saying that though, a celeron 2.8 should be equivilant to a athlon XP 1600+, which should be ok for low resolution (dvd's), but will fall over when it comes to HD deffinition.

i done some tests with latest nero 7 using AVC (took ages), but like 6 months ago, very impressive, i do believe this is the way, and the final size was smaller than a Xvid File,

Xvid Max quality 2.4GB

Nero AVC Max Quality (All Features Enabled) 1.7GB and the quality was amazing, and using CoreAVC playback was loverly, smooth and no frame dropping (although multi tasking was out of the question hehehehe).

i'm using Athlon XP 2400+
Windows XP SP2 latest Updates
1 GB of DDR 333 RAM

Latest VIA Drivers
Latest BIOS Flash (With my own optimal Settings)
Latest Graphics Drivers

And i always stay away from Windows XP generic Drivers (there called generic for a reason), if there is a driver out there for any hardware i have got, i always use it,

P.S. i still use XVID 1.2 all the time though, so efficient and does the job for me brilliant, i can't believe that support has nearly been dropped, Xvid is Wicked in so many ways, and considering it's free, kicks DivX's Arse (sorry divX lovers, but respect has to be given, Xvid is free)

sorry, getting of topic, good luck with you AVC, trust me, the quality is brilliant,

[HymnToLife]

Tried to encode some AVC, plays pretty smoothly in TCMP with ffdshow (which is good because it will save me $10 for CoreAVC), but it's relatively low resolution (640x352 @ 1100 kbps), I'll try some bigger stuff later :p

[Scarpad]

I also find keeping the resolution of 720 fix width is importan if you want close to the DVD Original, I keep the quality based # at 75% I find it's good without overkill