New curve treatment

[Koepi]

Hi,

yesterday I chatted with foxer some hours and he started implementing an improved curve treatment. With this code, if you set iframe boost "too high", you don't have a few frames with very high quantizers, but the overflow gets distributed over the whole scene resulting in a way more consistent and nicer overall image quality!

Until now, the code does harm the encoding when consecutive iframes occur, we have to work on that, and after that, I'd say we have a perfect curve treatment :)

I just wanted to inform you about some real amazing development :)

Regards,
Koepi

[Koepi]

Ok, I did a quick-hack for consecutive keyframes and everything looks fine.

Before we publish the code I will do some testing though....

Regards,
Koepi

[rui]

Great! Can’t wait to try it. :)

After reading your post, i got the impression that with the new code, it would be best to use the default cc rather than the alt cc.

I know that i-frame bost works for both (right?), but lately i have been using the default cc, and like the results i am having.
In my tests, for one to experience some real differences between default cc and alt cc, i think that we have to increment the strength.
But, after incrementing the strength, if the alt cc. Parameters aren’t chosen wisely, it could ruin a encode.

[Koepi]

Well, with this code I don't think that you have much to influence by choosing different curve compression algorithms...
I'm testing with default alt cc now and the result looks very promising, have to look at the final encoding (in about 2hours from now) before I know if it went bad...

The movie has some scenes with consecutive keyframes (my hack looks for keyframe interval <10) and those scenes themselves may look a little worse, but it doesn't punish the whole 5 scenes after these keyframe sequences with quantizing them with 8,9,10... but quantizes them with 3,4,2,... :)
I have to check that this code doesn't break size predictability, it may well be that i did a bad thing to a global buffer ;)

But it's amazing to see curves that are smooth after a keyframe, and not producing high quantizers which lower by small amounts after a few frames... it looks like it should now!

Regards,
Koepi

[Koepi]

A small screenshot of the curve as it is now... :)

[Koepi]

...and another one...

[Koepi]

...and the treatment of consecutive keyframes...

[Koepi]

Strange, noone has a comment for this?

Usually these graphs look really bad with such an iframe boost + quantizer restrictions ;)

Regards,
Koepi

[Koepi]

[228] Quantizer distribution for 2nd pass:
[228] Q:2:4226
[228] Q:3:67773
[228] Q:4:71646
[228] Q:5:3139
[228] Q:6:208
[228] Q:7:158
[228] Q:8:12

Those quantizers >= 6 only derive from scenes where there were many consecutive keyframes...
Btw., it is the last Jean Reno movie ("Die purpurnen Flüsse", dunno the english title) in 720x304, heading for 1 CD...

Sadly it came out 12 MB oversized :( So my code definatly breaks predictability, I have to work with Foxer on this (get online man! ;) )

The visual quality is overwhleming (?hope this is the right english word).

I'll look into redistributing the overflow correctly now and do another test encode (3 hours :( )

Regards,
Koepi

[Doom9]

I think you mean overwhelming ? (at least that passes the spell checker in dreamweaver).
sounds very interesting though.. it gets harder and harder to decide whether to do dvd-r or xvid on every new day ;)

[Marc FD]

Hi koepi :)
Just a little advice : if you're tuning the compression curve. I think you should _really_ take a look on Alt-Dbg : If you want to see Quantizer/stats1/stats2/kf graphs with an emf-export option so you could use snapshots more easily than with VDub :))
(I wrote this program to test the curve treatement but i'm too busy in coding the last 2 months so i couldn't really use it.)

it should be your best friend for this task :cool:

PS : Diese neue CC ist total-voll-crass-cool :)
(In french, the title of this movie is : Les rivières pourpres. the VO title is always the better ! )

[Koepi]

Thx Marc, but doesn't the couple "vdub status" + debugview do the same? I'll give it a shot, I'm usually very minimalistic when it comes to these things.... and I like debugview (that's why I never even looked at anything else) :)

Damn, I can't wait to see if i fixed the code correctly...

What I did was:

If consecutive keyframes (fixed treshold 10 frames distance make them consecutive) occur, just divide the "scaled bitrate" by 2 and let the iframe quantizer range take care of not destructing quality (may iquant 5).

But there is another piece of code: Foxer set a treshold of 1 (so only "real" consecutive keyframes are affected) in a routine where the redistibution of the overflow bits was made partially (if no consecutive frames) or full (if consecutive iframes are detected).
I just set my keyframe_treshold var with it's value of 10 there, this might help it, but it _mustn't_ be the case.

We'll know in 2 hours if it's prime time for the scene based curve treatment ;)

Regards,
Koepi

*searching for AltDbg*
EDIT: ok, found it... but I never log to file, and my history depth is only 200 lines... this isn't very useful then ;)

[Marc FD]

Just try it with a full encode.
Debugview logging will not slow down the process :)
(i offered it exactly for the task you're doing, it's why i would really enjoy it can help you :) )
(still a buggy util from a lame coder :D )

[droolian01]

Wow! This smoothing algorithm sounds briliant!
Does this mean that 'min keyframe interval' can be disabled (set to a value of 1), because if does it is great news in that you can allow the codec to place keyframes where it sees fit. As i feel that the occasional inappropriate keyframe placing may be due to the fact that the codec may want to put a keyframe in before the minimum keyframe interval is reached.

If what i say is stupid, i appologise as i am just a committed amateur.
Great work guys!

[Koepi]

droolian,

you can do that now, but i believe this is still a bad idea because it still can lead to undersize (!), at least in my "hard test case" those consecutive killer frames produced an undersize of ~1.5mb now (which is very acceptable IMHO)

@all:

new results:
[228] Quantizer distribution for 2nd pass:
[228] Q:2:984
[228] Q:3:67658
[228] Q:4:74842
[228] Q:5:3261
[228] Q:6:225
[228] Q:7:174
[228] Q:8:18

So slightly higher quantizers, but this might be due to the size difference of 14 MB (656MB targeted, 1st was 668MB (way off), corrected now is 654 mb (rounded)).

I'll put up a new binary now with this new curve treatment and I hope foxer will commit the code as-is.

Regards,
Koepi

[Marc FD]

Logged the XviD stats ???

You could make a little PSNR comparison between before and after too :D

(I can't resit myself to make ad. for my filters/progs sorry :( )

(but psnr.dll is :cool: + :devil: + :angry: )

Good luck Koepi ! We're all with you :) you will make it !

[Didée]

Koepi:

Perhaps it´s because I´m still suffering from a summer cold, but I don´t quite understand:

In your initial post, you wrote that in case of having a "too high" I-frame-boost and too many consecutive I-frames, the NCT (New curve treatment ;) ) will spread the overflow across the scene ...

Does that mean one has to set an extra-high I-frame-boost, to get the new treatment kicking in? Or is that nonsense, and it works always?

Additionally, how are CC and alt-CC affected? You wrote: "...I don't think that you have much to influence by choosing different curve compression algorithms..."

A little more information about how the new algo works would help a lot.
Or is it in fact not clear what exactly is going on inside, and how CC/alt-CC is influenced by NCT?

Thanks, and keep on!

[Koepi]

Uh, sorry, I spoke to many people and just forgot that you can't know the backgrounds:

the idea is to spread the overflow generated by iframes (or the overflow as is + the new overflow by the iframe) to be spread over the whole _scene_ (means: from this iframe until the next) instead of trying to compensate for over/underflow in just the next few frames... so this code is more-or-less scene-based bitrate distributioin (and it DOES result in improved image quality in general cases).

[cabal]

if i get this right , this is some sort of smoothing ?! will it finally slow down the encoding ?? can i do something wrong if i use certain parameters ??

greetings cabal

[Koepi]

Well, you can do as much harm with wrong parameters as before, but the effect will be less visible.

And yes, it's smoothing an effect where p-frames after i-frames got too high quantized, making the codec look bad ;)

[sneeky]

@Koepi

I think you are on to something. I find certain quant fluctuations
within a scene to be very annoying.

Note: the following observations based on p-frames

For example, using a sequence of numbers to represent quants for a
movie stream in a single scene:

3,4,3,4,3,4,3,4,3,4,3,4,3,4,3,4,3,4 .. not annoying
3,3,3,3,5,5,5,5,5,5,3,3,3,3,3,3,5,5 .. annoying, textures wink
3,3,3,3,6,6,6,6,6,3,3,3,3,3,3,3,6,6 .. more annoying
3,3,3,3,4,4,4,4,4,4,4,4,3,3,3,3,3,3 .. annoying, a little better

I really can't create the sequence as follows for good observation,
but it might look pretty decent as well.

3,5,3,5,3,5,3,5,3,5,3,5,3,5,3,5,3,5, ..

I suspect that if you have scenes change where the are pretty
constant quants within each scene, you will not notice the winking
textures. For example, (s) denotes scene change:

3,4,3,4,3,4,3,4,(s)5,5,5,5,5,5,5,5,5,5,5,5, ... 5,5,(s)3,3,3,4,3,3,4 ..


Ok, I will admit I hate quants winking in larger than 6. I reserve
such quants for credits only.

[droolian01]

Hi Koepi and everyone.
I used the new build to encode a Buffy cap from digital cable. I used min keyframe=1 max=125 (normally 8 - 125) and found no macroblocks visible at all following the keyframes. In previous builds any run of successive keyframes would always produce ugly high quant p-frames with keyframe min=1. Very good so far!
I normally have keyframe boost at 10%, and am wondering what the effect of the new smoothing algorithm is on this variable. It could either become redundant (the 10% boost is redistributed back to where it was 'borrowed') or may act to make the framesize vs time graph more 'peaked' (which would with large values of keyframe boost negate the effect of the new smoothing algorithm).
Any thoughts/advice on this would be cool.
If i have totally misunderstood the concept (quite possible!) please be kind and forgiving.
Thanks again for a great build.

[Kyo]

I have a bit of time and test the new compilation, i check the forum after my last encode :( but... hey i get good news and this binary is a bit better. The movie was the same that Keopi (mmm... i'm after your essence :devil: ) and i got better still images(more cleaner)
and less or no noticeables blocks

Test
Les Riviérès Pourpres
-Maximun-I=6frames
-i-boost=22%
-quants=mpeg
-alt cc low/200/92
-no credits (I forgot use them :rolleyes:)
final size= 614M

i don't have overheads no blocks good images
Keep the work guys! and excuse me the English

[iago]

Hi Koepi and everybody,

In my tests with Koepi's new build 20072002-1, I got very good results both with alt cc (Aggression: Low, High: 300, Low: 100, Strength: 50, Enabled automatic bonus bias calc.); and without alt cc, using the default values of curve compression in Two Pass (high: 25, low: 10, bitrate payback delay: 240, payback with bias).

The movie was Natural Born Killers -> 2cd

Motion search: 6.Ultra High
Quant: H.263 (1st pass), Modulated (2nd pass)
Maximum I-frame interval: 250
Minimum I-frame interval: 5
No Lumi masking, no Hinted ME
Quantizer restriction: Min-Max I-frame: 2-5 / Min-Max P-frame: 2-12
I-frame boost: 20%

However, the result, in both cases, came out 8-10 mb undersized!?
And with almost no macroblocks with resolution 640x352 :)

Just wanted to share my test results.

Best regards...

[Marc FD]

it's just an advice but when you say 2 CD, please say the size off each cutted XviD part, because we can't know the size of you audio part : you could use a 100 Mbytes big mp3 or a 600 Mbytes multilanguage ac3. (but saying your resolution was a very good thing : it's often forgot too :( )

If you are more precise your work will be more usefull for everybody :cool:

PS @Kyo : you don't need to be sorry for you english : mine is _very_ bad too so if people can _just_ understood us, i think it's "okay" :D
(and i think almost of us don't have english <<"`motherthongue'">>)
Thx.

[iago]

@ Marc FD

Oh, sorry pal. You're right. I realize I've skipped to mention the audio and video size , which are essential to make a good judgement of course :).

It was 128mb mp3 - roughly 106,000 KB.
( I usually use mp3 in my 2cd rips, while I absolutely prefer ogg for 1 cd rips :) )

The aimed video size for two CDs was roughly 1,315,000 KB, however it came out approximately 1,305,000 KB with alt cc, and sth like 1,307,000 KB with default two pass.

Any comments on the alt cc and two pass settings would also be apreciated.

Regards.

[Didée]

iago,

don´t get me wrong, but ...
with a 128kbit mp3, and a movie <= 2h, no magic has to take place to do a pretty decent 2CD-rip.
`NoFace319´ enthusiastically mentioned something similar about a movie of 80min :rolleyes:

I suppose your undersizing of 10megs comes from the codec getting `saturated´ (especially since your encoding was `only´ 640*xxx):
quant2 throughout the movie, it could not get any better!

Squeezing a moving of 120min(+) onto 1CD, with decent quality, that´s where the advanced and improved features are needed.

Get me right: I´m not scolding. Running 100m flat in 15sec is really OK. But it´s far away from world record ... ;)

[iago]

@ Didée,

I absolutely agree with you, but anyway I never claimed "world records" :) -though there are many around-, and I am only a tester of a new build and codec which I'm newly getting familiar with, and I just tried to share my experience here :)

However, I doubt that the reason was the codec getting saturated, since the quantizers used during the encoding process of course were not only 2, and the movie mentioned above is not a piece of cake imho :). (The Gordian Knot compressibility value was hardly over 60 only with a resolution of 640*xxx - even going for two CDs).

Anyway, thanks a lot for your reply and comments, but i guess there was a slight misunderstanding in your post :)

Best regards.

[The People's Elbow]

with a 128kbit mp3, and a movie <= 2h, no magic has to take place to do a pretty decent 2CD-rip
- Plz don't state that for general... there is no fixed relation between Film length and compressability. Once I tried to rip a damn 100 minutes movie on 1 CD ( the movie is called "Heartbreakers" ) and it looked like crap...I used 2 CDs and still I wasn't satisfied!
The first pass size was about 2500000KB, so this bitchofamovie was really hard to get for the codec ( probably because of the very noisy picture )

greetz, Elbow!

[Didée]

As I feared: you got me wrong ...

Of course every movie is different, and there are lots of extremes in compressability to both sides of the scale.
And of course the lenght alone is not the one and only variable.
Every single clip is individual, and needs individual settings ... perhaps it´s because of this, that I have done not so many rips, if you count their bare number ... ;)

But there *are* general outlines (<- correct word???)
And if I read messages "hey, I´ve done a 2CD-rip, and - guys, it looks really good!" - - well, then I´m not exactly surprised.

Wait a few hundred more posts of mine, you will get to understand me. Even many people in "real life" fail in this. I´m not a bad guy, really ;) Only some wicked humor ...

[Marc FD]

I agree with didee. When i asked for audio size, i just wanted to know if you were making a really hard encode or just a "cool" 2 CD encode. I think XviD is actually very good on < 1h40 encodes and the "nct" (tm) is a way to make really good 2h rips, and then with B-frames/0-frames(for anime)), i think every movie will be 1 CD encodable as soon as you use highly compressed audio (like mp3/ogg)
but i don't blame 2 CD rippers : they want to keep the ac3 or they want to make an "easy" ripp. no problem. but i think XviD improvement wouldn't help them anymore..
Personnaly i don't like to use 2 CD, i don't have any Dolby 5.1 hardware and i don't want to make it "easy". I think it's some of the 1 CD rippers reasons :D

PS : I think every movie under 1h30 can be encoded with a really big quality with the current XviD. if you have noise, avisynth filters are always here :devil:. so i think i you really _want_ to make a 1CD encode, you can.

Just my toughts.

[The People's Elbow]

Hey, we're already OT - again ;)
...but that's ok, who cares about that :P

Didée wrote:
And if I read messages "hey, I´ve done a 2CD-rip, and - guys, it looks really good!" - - well, then I´m not exactly surprised.
- No doubt about that, I think almost the same way!

Marc FD wrote:
I think every movie under 1h30 can be encoded with a really big quality with the current XviD.
- No!

if you have noise, avisynth filters are always here :devil: .
- And in most cases the picture gets even more ugly, but at least artificial!

so i think i you really _want_ to make a 1CD encode, you can.
- Absolutely true, but how the result will come out is another story!

PS: Don't get me wrong, I'm a "1CD encoder", either, and I don't use AC3, because I watch my rips @ my computer with 2 speakers or headset!
There are some "horrible" movies out there ( horrible in the way they compress ) which are just "out of bounds" ;)

greets, Elbow!

PPS: I don't want to offend anyone, I just like trashtalk ;)

[primitive]

Originally posted by Koepi
[img-amazing]

...and the treatment of consecutive keyframes...

I admit that I know very little about how bits are distributed, but if those images mean what I think they mean we are in for a treat.

-p

*edit*

Let's see if I understand what's happening correctly. Feel free to correct me if I'm wrong.

Before this, in some cases it was desirable for the encoder to manually place keyframes. This was because a difficult-to-encode scene would cause the codec to "overflow"; that is to say, it would cause the codec to use more bits than was initially budgeted to encode said scene. This "overflow" condition was undesirable because the codec tried to win those bits back almost immediately, creating a "downward slope" in the bitrate curve (the graph with frame# on the x axis and quantizer, or compression strength, on the y axis). This downward slope in the bitrate curve would cause the scene following the spike in the curve to look inconsistent in quality; directly after the scene change, the video would look bad because the codec would compress the frames highly, looking to win back the overflow bits. After the overflow bits were eliminated, however, the codec would resume the use of more appropriate quantizers, thus producing a percieved change in quality.

Frankly, I had noticed this problem before, but hadn't realized it was a problem until now. Specifically, I have seen this in the Adventures of Mini-Goddess; "rampant" is not extreme enough a word to describe that instance of this phenomenon when I look back at the encodes I made with past XviD builds.

Hope to try the new build soon :)

-p

[Koepi]

in _where_? mind-bending institutions? ;)

Well, the consecutive keyframes are handled differently now, it's better to set "min keyframe distance" to 4 or something to help the new algos to work fine. (I hope so, I'll see in some hours when I can test the second pass, if it really works better then. distance of 1 didn't work very well, so maybe raising the distance helps it. Still better than what I did first because it doesn't need any hacks with blindly chossen tresholds for what means consecutive keyframes... :) )

Regards,
Koepi

[iago]

@Marc FD

...i think every movie will be 1 CD encodable as soon as you use highly compressed audio (like mp3/ogg)... / ...but i don't blame 2 CD rippers : they want to keep the ac3 or they want to make an "easy" ripp. no problem. but i think XviD improvement wouldn't help them anymore...


Well, of course I respect your views, but i must say that the last thing I want to go into in this forum is a "1CD or 2CD rip" discussion :).

So I want to end my words regarding this issue with only a few remarks:

Deciding on going for a 1CD rip or shooting for another depends on only one thing: the movie itself; in other words, the compressibility of the movie, on the calculation of which Gnot offers great assistance imho :).

Considering only the "lenght" of the movie regardless of its "compressibility" and without choosing a "movie-specific" proper resolution might most of the time mislead you, resulting in a far-from-decent rip :).

So, in some cases, it stands clear that it will be almost impossible to achieve decent quality on 1CD (considering you take into account what Gnot tells you after the first compressibility check, or even better, after you load the first pass stats file), and going for 2CDs becomes a "must", not an "easy rip preference", for you :).

Finally, imho, Xvid improvement will help _anybody_ who put enough effort into the ripping business, spending hundreds of hours trying different settings (and codecs) to find out what is most satisfactory for their needs. But of course, and foremost, basing their efforts not on some preoccupations, but on the basic principles of encoding, such as choosing the correct resolution and bitrate (or the "required" number of CDs) considering the "compressibility" of a movie together with its "length".

And I guess... that's all I can say... :).

Regards and my best wishes.
Iago

[Marc FD]

It's your point of view. I respect it. ;)

But XviD/Z-frames is really too powerfull :devil:
the problem is that even a 1h40 HQ anime in 700x400 could saturate (const quant2) and get undersized. :D
My problem will soon be : how to fill my CD ??? :cool:

(ps : i'm just a little euphoric now, but it'll pass...)

[Foxer]

Part of the 2pass 2nd pass modifications commited.

It now takes the overflow generated by keyframes and spreads it out over the frames inbetween instead of bumping up the overflow upon keyframes which caused a jump in quantizers.

[spyder]

Great job guys. This new CC works great. This is one of the best encodes I have ever made.

PS: I am now convinced that 1CD rips can look good. I have just never minded using 2CDs and keeping full quality at 640x480 with AC3 audio since I have two drives and CDs are dirt cheap. BTW, Marc FD, how would you encode a movie like Pearl Harbor?

[Marc FD]

CD are cheap, but they take too much place (i prefer search a film in 100 CDs than in 200 CDs) But if you like 2CD encodes and you don't have any organisation pbs (i have :D ) 2CD is great too.
2CDs encodes are just really easyier than 1CD encodes. (right ?)

Pearl Harbour ?? dunno. a long action film ??
sure that for films length > 150 min and with lots of action, it's not possible to make very good 1CD encodes with the current XviD.
But XviD is going better and better ...
Maybe a day you could reencode your 2CD on 1 (for fun :cool: )

[spyder]

I very seldom put a movie on 1CD but if it is under 100 minutes and is low action I will unless it is one I really like. I am waiting for a DVD burner so I can fit 3 of my 2CD rips per DVD. ;) I feel there is no point in throwing away as much quality as you could retain with 2CD and AC3 or high bitrate VBR MP3. But that's just my opinion and I respect yours. With the new curve treatment, I can't see any difference in my rips and the original DVD. Also, I wouldn't really say 2CD rips are easier, You use the same methods but aim for a total of 1400MB in the end. Plus you do have to split these files which you do not with 1CD.

[MaTTeR]

Originally posted by spyder
Also, I wouldn't really say 2CD rips are easier, You use the same methods but aim for a total of 1400MB in the end. Plus you do have to split these files which you do not with 1CD.

I might have to agree with you on that issue. Don't forget all the problems people have with trying to find the "golden" AC3 interlave settings. Like you I mostly do 2CD rips because I watch a lot of action type movies and I refuse to throw away a nice 5.1 soundtrack:)

[tomy601k]

@spyder

I totally agree with you. It is not always that easy to make 2 CD rips, though that I was a 1 CD ripper before. I even tried to put Armageddon into 1 CD :D . I was satisfied with the results of my 1 CD rips at that time, but now ... I don't think so. I wish that I have DVD burner also, but not untill that I can do 1:1 DVD copy :)

If you guys don't have the hardware that can play 5.1 AC3 audio ... I think that you have to think about the future. You might get it later and then regret that you didn't rip the movie with 5.1 AC3 audio. Don't say that I didn't warn you ;)