h264, qpel, presets and speeding up 1st pass, and xvid bug and development (mayb

[justin]

Hi everyone. Is there a significant difference between what the new h264 codecs will be like and the way xvid is right now?

The VSS h264 codec is only in the beta stage, but encodes done with it look like xvid with slightly better motion estimation. Do you figure h264 encodes will replace the divx/xvid encodes in the next year or so?

I was also wondering if there are any plans for xvid to implement an h264 component once mainstream computers are fast enough to playback a 640x480 clip. Because right now, it takes roughly 800MHz to playback a 320x240 clip at 29.97fps.

Also, I was wondering if there's anything that can be done to improve qpel on xvid on lower bitrate/high motion scenes. I don't know but in these cases, the noise blocks don't seem to match the adjacent blocks too well and the blocks don't seem to look as visually pleasing as encodes done without qpel. I don't know if it's actually possible to improve on this because I don't know enough about qpel motion estimation.

food

And, this is for newbies, could a preset bar be made on the main tab of xvid with the optimal quality/speediest settings that you guys have found for an xvid encode? And even saving presets too, because that way you could just set the encoding bitrate and then select a preset to use with it. But I don't know how much programming that would involve. But it would be really cool to koepi's recomendations and everyone else's here set as presets in the builds so that you could just click on them and be all set eh.

This is my first post. I hope these aren't too many questions eh, it's just that these are the ones that have been on my mind for 5 days now. Oh and by the way, Xvid rocks! Thank you very much to the developers of it and all the many free utilities that make everything a complete package for re-encoding dvd's.

Oh, and another question, I just did an xvid encode, and what happened in a nutshell was, I set the output filesize to be 657MB and it came out to be around 480MB after a 2-pass encode. What happened was I did the 1st pass at 640x272 with no qpel (to make the 1st pass take less time, I thought this should work sort of ok). And for the 2nd pass, I used 704x300 with qpel (did you know that the movie "about a boy" compresses really well to fit on 1 CD if you use 62kbps (Max 92kbps, Min 0kbps) vbr ogg).

What happened was, I checked a stats file for one of my encodes with the stats reader program that came with koepi's xvid build, and it said the filesize was 480MB from the 1st pass. So does that mean you have to scale the size recomendation of the 1st pass to whatever size you want to encode for before you do the 2nd pass? (I was a little tired with dvd ripping after that because it took 14 hours of encoding just to find out that I have to do it all over again because the filesize came out to be 480MB instead of 615MB. So after I tried encoding it again 2 times I got mad.

Alright, that's all the questions from me.


Ok, more questions, this is for further development of video compression (if it could possibly work), Would it be possible when doing a two pass, to store the keyframes at the beginning of the file, and then cache the keyframes to the hard disk when loading the clip? And have every other frame in the stream reference those keyframes. I'm not too good at explaining, but the reason for this is that frames are identical to the ones before it right, but sometimes the same keyframes are used many times throughout the stream. So if that were the case (especially for cartoons), then you wouldn't have to store redundant keyframes over and over again in the stream (and if there are alot of redendant frames - which is rare, but it sometimes happens, then you would save a considerable amount of space for your encodes).

On second thought, I guess this probably wouldn't work, or it might, but it would make the clip non-iso mpeg4 compliant. Which sucks, because xvid is the coolest codec and I don't like rv9 and especially not wmv9. I wonder if you can sue them for annoyances! Real video 9 is NOT real video... those codecs should be called blurry vision 1 and blurry vision 2.

Oh, and I want to put a happy face in here too... :D <- That's my big teeth showing there.

[justin]

whoops, I forgot to ask about speeding up the 1st pass eh. Would encodes done in xvid come out alright if you do the 1st pass exactly the same as the second pass but without Quarter Pixel Motion Estimation? Just that the reason is that it makes the 1st pass go twice as fast if you do it that way.

[Assault]

I think you have saturated the codec. Choose a higher resolution, don't use b-frames if you used them and encode with the mpeg matrix. Then you should get your desired file size.

Use always the same settings for both your first and your second pass. Otherwise your video won't have the desired size.

Regards
Assault

[Wormz]

Originally posted by justin
whoops, I forgot to ask about speeding up the 1st pass eh. Would encodes done in xvid come out alright if you do the 1st pass exactly the same as the second pass but without Quarter Pixel Motion Estimation? Just that the reason is that it makes the 1st pass go twice as fast if you do it that way.

There's a reason that it takes longer: it HAS to

The fact it's taking longer should tell you that something extra is going on, and it's probably important.

Anytime you use different settings between the first and second passes, don't expect to hit your desired filesize.

[Koepi]

Damn, with that weird different settings between first and second pass it should be impossible to get a decent quality at all.

Justin, please feel free to use the search function extensively. You should read all the guides starting with nandub/SBC guides to understand the nature of 2pass encoding. Most of your questions would be answered by that.

I'd recommend: first read Nandub Options Explained, then XviD Options Explained, then all the guides on doom9, then the FAQs and stickies of this forum.

Amazingly you had 5 days to do so, but it seems you didn't use the time properly ;)

Welcome aboard.

Regards
Koepi

[Teegedeck]

Originally posted by justin

food

And, this is for newbies, could a preset bar be made on the main tab of xvid with the optimal quality/speediest settings that you guys have found for an xvid encode? And even saving presets too, because that way you could just set the encoding bitrate and then select a preset to use with it. But I don't know how much programming that would involve. But it would be really cool to koepi's recomendations and everyone else's here set as presets in the builds so that you could just click on them and be all set eh.



food? :D

Just wanted to say that presets will come inevitably, and I think it will be soon when I read the devel-mailing-list archives.

For H.264, it really is such a change from 'old' MPEG4 that one XviD-developer commented not much of XviD's current codebase could be re-used for H.264.

[sysKin]

Originally posted by justin
I was also wondering if there are any plans for xvid to implement an h264 component [...]No, xvid will not produce h264. H246 is too different than 'normal' mpeg4. It's easier to start another project.
Oh, and another question, I just did an xvid encode, and what happened in a nutshell was, I set the output filesize to be 657MB and it came out to be around 480MB after a 2-pass encode. What happened was I did the 1st pass at 640x272 with no qpel (to make the 1st pass take less time, I thought this should work sort of ok). And for the 2nd pass, I used 704x300 with qpelDifferent setting, different resolution... you really can't do that.
What happened was, I checked a stats file for one of my encodes with the stats reader program that came with koepi's xvid build, and it said the filesize was 480MB from the 1st pass. So does that mean you have to scale the size recomendation of the 1st pass to whatever size you want to encode for before you do the 2nd pass?If first pass is 480MB, than you will not achive bigger second pass, ever. Second pass can only be smaller. Try higher resolution, sharper picture, reduce (or remove) bframes...
Ok, more questions, this is for further development of video compression (if it could possibly work), Would it be possible when doing a two pass, to store the keyframes at the beginning of the file, and then cache the keyframes to the hard disk when loading the clip? And have every other frame in the stream reference those keyframes.
No, that wouldn't work. Of course it wouldn't be mpeg4 compatible, but the bigger problem is: how slow would that be to check all keyframes every time? Slow...
whoops, I forgot to ask about speeding up the 1st pass eh. Would encodes done in xvid come out alright if you do the 1st pass exactly the same as the second pass but without Quarter Pixel Motion Estimation? Just that the reason is that it makes the 1st pass go twice as fast if you do it that way. Official answer is: no, your second pass must use the same settings as the first.
Now, unofficial answer: you can safely use VHQ1 for first pass and VHQ4 for the second. The difference in size is small, codec can compensate it without even noticing. You could try with chroma motion too. But remember, it's all sub-optimal so don't go too far :)
However, settings which visibly affect filesize - like Bframes, Qpel or resolution - really have to remain unchanged.

Regards,
Radek

[justin]

Hi Everybody,

Thank you all for the feedback. I'll answer everyone's post in order... It was a long five days :eek:

Is the MPEG quantizer supposed to help a lot with the sharpness? I've heard of this before in other posts but I've always been afraid to use it because of the name MPEG which makes it sound like MPEG1 which makes me scared to use it becuase MPEG1 doesn't have the quality of MPEG4. That was my fear with that.

I have to go out right now, I'll get back to everyone else's as soon as I get back (It's 21 degrees C out now and a great day for kayaking).

[justin]

ok, I'm back again, excellent kayaking weather too eh.

yeah I guess I will have to use the same settings for both passes, some of the tries came out ok, but I really should have done smaller tests first instead of testing on a 2 hour movie over and over.

And I should have included the settings I used too (they were setup by what koepi mentioned as good settings before):

Motion Search Precision: 6
Quantization Type: H.263
FourCC Used: XVid
VHQ Mode: 4 - Wide Search

(I couldn't find the best results for I-frames but I made a guess of about every 30 seconds - just so that keyframes would not be placed in spots they didn't need to be at, and to keep seek time less than 1/2 a minute for spots that have keyframes that far apart).

Maximum I-Frame Interval: 720 (at 23.976 frames per second)
Minimum I-Frame Interval: 1 (I don't know if this should be higher or not but encodes looked better with 1 than 2, 5 and 10).

Use Chroma Motion Checked
Quarter Pel Checked

Maximum B-Frames: 2
B-Frame Quantizer Ratio: 150%
B-Frame Quantizer Offset: 100
B-Frame Threshhold: 255
DX50 B-VOP compatability checked

Quantization Matrix set at default
Two Pass settings set at default except Playback with Porportionality Checked.
Alt. Curve set at the defaults

Credits:
Credits at end of movie begin and end at Frame: 138620/145430 (just for length purposes if useful).
Encode Credits in Greyscale Checked
Desired Percent Rate: 9%

Debug set at defaults.

and I have a AMD Duron 700MHz overclocked to 911.44MHz for speed references (but when I was encoding I had it clocked back down to 700MHz just in case it froze).

Those settings seem to yield pretty good results. And if there's anyone who doesn't know who reads this, any movies over 2:15 hours should probably be on two CD's if you really don't want to step down the resolution below 512x218. For me, personally, I try to fit movies on 1 CD though instead of two (even long movies (except the 10 commandments - it's about 4 and 1/2 hours long)).

Koepi is right, I haven't read many of the guides on two pass encoding. When I was reading through the guides before to learn how to dvd rip, I didn't understand most of it at the time. I thought some of the software was pretty crazy. Mainly AVISynth and frame-serving... I thought "frame-serving"? Why would anyone want to use that to send video clips to friends, you could just use Kazaa. And I thought DVD2AVI was the worst VirtualDub clone ever. But I've gained a lot more knowledge about encoding after the first few encodes. I'm pretty sure I've read most of the guides, but not to the extent that I knew what a lot of them meant. So I will go back to them. I will search the board more extensively, becuase there's no point repeating what's already been said.

ok, I have to get off for the night but I will be back on in the morning to get back to everyone elses' posts (and to think of a good alias).

And to everyone: thank you for the replys

Justin

[justin]

Hello again,

Yeah, presets would be very handy when encoding often, I hope they do come out soon.

Wow, that's wierd that Xvid's codebase is so different from H.264. Up until now, I've always thought that H.264 was just better algorithms for motion search estimation, because they said that H.264 is compatible with MPEG2 and MPEG4 (but there's no sense in using it to encode DVD's in the future becuase all the current DVD players wouldn't be compatible with it).

Man, 1st passes suck. They just take so long. I guess you really can't ignore qpel on the 1st pass and then enable it for the second, because the scenes do get blocky in some spots for no apparent reason. Fuck. It's good to know that some of the time-consuming settings can be removed (or lowered) when encoding the 1st pass.

So it really is that the maximum filesize is determined by the 1st pass. But what if the DVD is 720x480 and your using 720x308 (at an aspect ratio of 2.35:1 - after cropping the video), and you're using Quarter Pel, 2 Bframes et al. and your movie is still coming out to be less than 640 MB? Would the best plan of action be to reduce b-frames, jump to the MPEG quantizer, try and tweak the 1st pass framesize settings with stats reader, or try and add multiple audio streams and subtitiles?

[Assault]

@ Justin
In that case you could also try some mpeg-custom quantization matrices like HVS-Best or even Andreas_78er which delivers great results with high bitrates. See also this thread (http://forum.doom9.org/showthread.php?s=&threadid=53594).

Assault

[justin]

How do you get these matricies into Xvid?

Haven't found any results searching the forum, and the only ones available in xvid are h.263, mpeg, mpeg custom, modultated, and new modulated hq.

[bond]

Originally posted by justin
How do you get these matricies into Xvid?xvid configuration -> advanced options -> quantization -> edit quantizer matrix -> load matrix

[justin]

I hope this doesn't turn out to be in someplace silly, but I can't find any of those matricies in the xvid install directory either.

[Assault]

NiTroGen published a link for a collection of several matrices in this thread (http://forum.doom9.org/showthread.php?threadid=34493&perpage=20&pagenumber=2). If you wanted to use one of the HVS-matrices you should download them separately because some guys (including me ;))experienced crashs with the versions that are included in this collection. I think manono posted an archive with HVS-matrices which didn't crash on my PC.

Assault

[justin]

Thanks Assault. Those matricies are pretty useful, if Doom9 wouldn't mind, it'd be handy to have those two custom matrix linked in the Xvid sticky FAQ too. And also the new Trellis quantizer thread (http://forum.doom9.org/showthread.php?s=&threadid=53383) would be handy too.

[Prettz]

Originally posted by justin
Man, 1st passes suck. They just take so long. I guess you really can't ignore qpel on the 1st pass and then enable it for the second, because the scenes do get blocky in some spots for no apparent reason. Fuck. It's good to know that some of the time-consuming settings
I've never heard of someone wanting to change the first pass settings just to make it go faster. Just keep in mind that if you use different settings in between passes that it will technically be impossible for you to get the best quality for the resulting file size. Quality > Speed, always.

[justin]

I know, but it takes about 4 or 5 1st passes just to get the 1st pass just right before you can encode the second pass.

From what I understand, the best way to go about 1st passes is to get an approximately good resolution for the size of media (I aim for 1CD just about all the time), then encode the 1st pass with the HVS better matrix (or H.263). If the filesize (found with statsreader) comes out to be too large, try it again with HVS good matrix. If the 1st pass turns out to be too small, increse the resolution. And keep doing this over again until you're using the matrix you want at a high enough resolution. So how do you guys avoid pulling your hair out? :angry:

[Gaia]

Originally posted by justin
I know, but it takes about 4 or 5 1st passes just to get the 1st pass just right before you can encode the second pass.

From what I understand, the best way to go about 1st passes is to get an approximately good resolution for the size of media (I aim for 1CD just about all the time), then encode the 1st pass with the HVS better matrix (or H.263). If the filesize (found with statsreader) comes out to be too large, try it again with HVS good matrix. If the 1st pass turns out to be too small, increse the resolution. And keep doing this over again until you're using the matrix you want at a high enough resolution. So how do you guys avoid pulling your hair out? :angry:

It looks like you still don't quite understand 2 pass encoding. What do mean to get 1st pass right? I recommend you to read some guides again. What are you trying to encode?

[pandv]

I think "get 1st pass right" implies configure the resolution, filters chain and xvid (bframe settings) to get a correct sized first pass, to allow a good 1cd rip.

But, you don't need to do a complet first pass each time. Search the forum for Gordian Knot or Enc (in this zone), and the compressibility test concept.

pandv

[Koepi]

pandv,

your intention might be nice, but you're not helping here.

We're dealing with a newbie who doesn't care & wants to read the guides and as such all hints _have_ to fail.

Koepi

[Andrey]

@justin
Once more:
1. DVD resolution is 704xXXX, not 720. (you can use 720, if you wish, of course :) )
2. Use values that can be divided by 16 - i.e. 304, not 308.
So, for 2.35 aspect DVD resolution will be equal to 704x304.