New Version of Avi.net

[iNFO-DVD]

Well I've implemented a comptest into avi.NET, it doesn't do much as of now apart from tell you a percentage. Because of working with b-frames don't expect it to give the same values as AutoGK. I suppose after a little time we will discover it's own thresholds. I'll try and get a version out later today on this forum. At the moment it will be for 2-pass XVID encodes only.

One thing I would like to ask......
The term compression test and it's value get refered to when people talk about how much a movie will compress. It's also gets refered to when people talk about the quality. Now if after a comptest we have, for example, a value of 30%. How do we know if that is 30% because the movie just happens to compress very well or 30% because our settings are to low, like low bitrate, resolution too high?

[mod]

Now if after a comptest we have, for example, a value of 30%. How do we know if that is 30% because the movie just happens to compress very well or 30% because our settings are to low, like low bitrate, resolution too high?
IMO the answer is:
"With target size of (ie) 1400 MB, you're trying to compress this source at the XX% of the "full quality" size. If for you it's too low give me more bits, if it's too much use less bits, if it's ok just encode." :) Usually a value between 0.60 and 0.75 gives good results.

[iNFO-DVD]

"With target size of (ie) 1400 MB, you're trying to compress this source at the XX% of the "full quality" size. If for you it's too low give me more bits, if it's too much use less bits, if it's ok just encode." :) Usually a value between 0.60 and 0.75 gives good results.Yes I understand that but what if a test was 30% because it just compress really well, not because the settings are too low? How do you know the difference? Or do we not.

BTW, thanks for that link above you provided, was very good.

[MarkCurly]

Two points, using the source resolution completely threw the value. It seemed to work much better using the encoding resolution.
This might indicate a problem (bug) with your calculation.

Suppose you have source video with framesize 640x480.
Comptest this using the source framesize, and then again with a 320x240 framesize. If the two results differ by a factor of ~ 4, then you are not taking the source framesize into account. The two values should be roughly the same, regardless of the framesize used in the comptest.

Point 2, HDTV sources seem to have a large comptest value, 70% to 80% using the standard QF of 0.20.
Same as point 1 perhaps? Source framerate not used correctly?

One thing I would like to ask......
The term compression test and it's value get refered to when people talk about how much a movie will compress. It's also gets refered to when people talk about the quality. Now if after a comptest we have, for example, a value of 30%. How do we know if that is 30% because the movie just happens to compress very well or 30% because our settings are to low, like low bitrate, resolution too high?
What mod said...

(30% sounds like a source that is *not* compressible; or more correctly, needs more target bits than the user has chosen.
What you said would make more sense if you had said:
"is it 30% because the movie just happens to NOT compress very well or 30% because our settings are too low?")

But you are right, ONCE it has been converted into a %'age, you DON'T know if the %'age is high/low because the source is/is not compressible OR because the settings are too high/low.
It could be VERY compressible, but if the chosen settings are still too low, then the %'age will be too low.
It could be NOT VERY compressible, but the chosen settings are very very high, then the %'age will be high.

What you DO KNOW is, are you allocating enough bits to get a good encode?
A small %'age says you don't have enough bits.
A large %'age says you might have too many bits.

I'll just make sure I'm on the same page as you.
The comptest %'age is = Qf(target) / Qf(comptest)
(Or some equivalent calculation.)

All other things being equal (such as source length/framerate/framesize, target filesize):

- A less compressible source will have a larger comptest "100%" number, and then the %'age value will be smaller.
(%'age = target/source.)

- A more compressible source will have a smaller comptest "100%" number, and then the %'age value will be larger.

- If the user choosen bitrate is smaller, the %'age value will be smaller.

- If the user choosen bitrate is larger, the %'age value will be larger.

(Sorry to labour this, I'm just making sure I'm on the same page...)

[iNFO-DVD]

This might indicate a problem (bug) with your calculation.Was just my initial though, made correct amendments, all seems OK now. Yes I use source resolution.

But you are right, ONCE it has been converted into a %'age, you DON'T know if the %'age is high/low because the source is/is not compressible OR because the settings are too high/low.That's fine, that's what I wanted to hear. The logic of it made me think that. Just wanted some verification.

I'll just make sure I'm on the same page as you. The comptest %'age is = Qf(target) / Qf(comptest)
(Or some equivalent calculation.)Yes, we're on the same page. :)

- If the user choosen bitrate is smaller, the %'age value will be smaller. - If the user choosen bitrate is larger, the %'age value will be larger.Yes, my tests incicate this.

My HDTV tests I did that were worrying me I'll forget about for now simply because the largest HDTV clip I have is only one minute long so it's hardly a good source to work with. I'll get some better sources for them.

[mod]

I'm happy the link helped you :)
It would be nice to share a short clip with the same script, to check if the results are the same. Tonight I've implemented the B-frames part and I'm testing it right now, a comparison of final Ratios would save a lot of testing time :)

[iNFO-DVD]

It would be nice to share a short clip with the same script, to check if the results are the same. Tonight I've implemented the B-frames part and I'm testing it right now, a comparison of final Ratios would save a lot of testing time :)Yes I completely agree.

I thought something was wrong as I used a 45min HDTV episode, ran the comptest and got a high 90% or something figure which I thought was far too high. Ran the same thing through AutoGK and it gave an even bigger value (well over 100%) then brought the prediction down to about 90%. This obviously made me feel a lot better, it seems I might be on the right track.

[iNFO-DVD]

I have to go out for a few hours now so I'll post the BETA version, other bugs may have been introduced......


comptest will only work if you select the default 2-pass xvid.
log will say [5%] though for speed it's really only doing 2.5% for now. Don't expect same values as AutoGK, it's the 'trend' of the comptest that I'm interested in.

DOWNLOAD >> HERE (http://www.dvduk.pwp.blueyonder.co.uk/__files/avi.NET_2185_BETA1.zip) <<

(put in installation folder)

[audioman]

Hi INFO

Thanks again for your work

Maybe I'm blind but I don't see any comptest
Is it completely internal to avi.net ?
Because there is nothing in my log


avi.NET v2.1.8.5
Intel(R) Pentium(R) M processor 1400MHz (1399MHz)

IN : C:\--------\VIDEO_TS\VTS_01_1.VOB
OUT: C:\------\VIDEO_TS\Test.avi

ORIGINAL MPEG FILE
MPEG TYPE : 2
RESOLUTION : 720x576
ASPECT RATIO: 16:9
FIELD ORDER : TFF

CONVERTED AVI FILE
RESOLUTION : 640x272
DURATION : 0:05:41
FPS : 25
CODEC : XVID
PASSES : 2
METHOD : SIZE BASED
SIZE : 80MB
BITRATE : 1580Kbps
BITS/PIXEL : 0.36
AUDIO : AC3
SUBTITLES : NONE

SELECTED OPTIONS
DE-INTERLACE: NO
DE-GRAINED : NO
HIGHER CROP : NO

TIME TAKEN [AUDIO]: N/A
TIME TAKEN [VIDEO]: 00:10:55
ENCODE RATE 1/2 : 33 FPS
ENCODE RATE 2/2 : 22 FPS

TIME TAKEN [TOTAL]: 00:11:07

[iNFO-DVD]

Maybe I'm blind but I don't see any comptest You have to select (tick) the compression test option.

BTW, 640x270, 1580 bitrate 0.36 b/p is a little excessive....... you don't need that much bitrate......

[audioman]

Hi info

Thanks For the answer , I have found the Tick comptest

I ran a job on a movie and I got this error message at the end of the 5% compression test

http://www.badongo.com/pic/277797

I 'm trying with another short clip

Same issue : http://www.badongo.com/pic/277801

Hope that helps INFO :)

By the way , at the end of the comptest , What happens ? Will the user have the possibility ( according to the values specified by comptest ) to choose a special resolution or QF to use ?

[ricardo.santos]

I got no error messages the value i got was:
5441236%

and then it carrried on on converting automatically.

my version doesnt say BETA1 at the top

[audioman]

This is very weird

Look at this
Do you have the same version ?

http://www.badongo.com/pic/277817

Can you make a screenshot of your version ?

[iNFO-DVD]

I got this error message at the endThat's the same error we used to get before with the FPS, I'll take a look, it's because I'm trying to return a value and do a type conversion at the same time, never errors on mine though, typical.

at the end of the comptest , What happens ?One thing at a time.

Anyway, since I've been back in I managed to have another little test and am quite pleased so far with the results.

DEEP BLUE SEA [R2] 10 MINS 92MB 0.20QF
AVI.NET 43% AUTOGK 46%

EIGHT LEGGED FREAKS [R2] 10 MINS 96MB 0.20QF
AVI.NET 76% AUTOGK 72%

THE OTHERS [R2] 10 MINS 112MB 0.20QF
AVI.NET 177% AUTOGK 176%

(I like testing 'The Others' movie as it's that damn dark it seems to throw all calculations e.t.c.)

As you can see the trend of the comptest in avi.NET seems to be working even when it went very high with 'The Others', AutoGK displayed similar behaviour. As a BETA 1 it seems to be working well.

Going to try some other treaks e.t.c.

[iNFO-DVD]

and then it carrried on converting automatically.Yes it will, it's supposed to, just click stop.

my version doesnt say BETA1 at the topHow can it not??????? that's weird.

[ricardo.santos]

This is very weird

Look at this
Do you have the same version ?

http://www.badongo.com/pic/277817

Can you make a screenshot of your version ?

well it could be my eyes playing tricks but its the same version and the BETA1 is there now.

but on my first and only try it wasnt there because i compared it with your screenshot twice.

[audioman]

never errors on mine though, typical.


What should I do then ?

Ricardo does not seem to have errors on his version because he did have a %
I really don't understand :rolleyes:

[ricardo.santos]

Yes it will, it's supposed to, just click stop.


ok, i was under the impression that a small "portion" of the film would undergo a comptest and then the user would be presented with the playback of that portion and decide wether to continue or not with the encoding.

[ricardo.santos]

What should I do then ?

Ricardo does not seem to have errors on his version ?
I really don't understand :rolleyes:

but im converting audio to mp3 youre keeping ac3? Could that be it?

[audioman]

No it is not :(

I just ran a test and the error message did show again

I WILL WAIT

[iNFO-DVD]

Download again now

different version

and it's nothin gto do with AC3. The % error is my fault.

[audioman]

Thanks

Do you mean to follow that link ?

http://forum.doom9.org/showthread.php?p=870960#post870960

[ricardo.santos]

Anyway, since I've been back in I managed to have another little test and am quite pleased so far with the results.

DEEP BLUE SEA [R2] 10 MINS 92MB 0.20QF
AVI.NET 43% AUTOGK 46%

EIGHT LEGGED FREAKS [R2] 10 MINS 96MB 0.20QF
AVI.NET 76% AUTOGK 72%

THE OTHERS [R2] 10 MINS 112MB 0.20QF
AVI.NET 177% AUTOGK 176%



Does those preliminar results mean that by selecting a 0.20QF the result will be satisfactory? no need for comptest?

[iNFO-DVD]

Do you mean to follow that link ?
http://forum.doom9.org/showthread.php?p=870960#post870960Yes, same link and filename as before, still BETA 1, only the time has changed in the HELP panel.

Does those preliminar results mean that by selecting a 0.20QF the result will be satisfactory? no need for comptest?No, just means the comptest seems to be working.

I don't really care about what I'll do about the comptest values at the moment, just want to get the comptest working as it should.

Does your % value work now with new version?

[ricardo.santos]

.

Does your % value work now with new version?

yes on 2 minute vob 17MG with 0.19QF i got 49%

gonna try on a 10 minute vob

[iNFO-DVD]

yes on 2 minute vob 17MG with 0.19QF i got 49%. gonna try on a 10 minute vobExcellent, how about you audioman? Does you % work now?

[audioman]

Excellent the comptest works , Bravo INFO

Here is my LOG

avi.NET v2.1.8.5
Intel(R) Pentium(R) M processor 1400MHz (1399MHz)

IN : C:\HARRY_POTTER_III\VIDEO_TS\VTS_01_1.VOB
OUT: C:\HARRY_POTTER_III\VIDEO_TS\VTS_01_1.avi

ORIGINAL MPEG FILE
MPEG TYPE : 2
RESOLUTION : 720x576
ASPECT RATIO: 16:9
FIELD ORDER : TFF

CONVERTED AVI FILE
RESOLUTION : 640x272
DURATION : 0:05:41
FPS : 25
CODEC : XVID
PASSES : 2
METHOD : SIZE BASED
SIZE : 70MB
BITRATE : 1334Kbps
BITS/PIXEL : 0.31
AUDIO : AC3
SUBTITLES : NONE

SELECTED OPTIONS
DE-INTERLACE: NO
DE-GRAINED : NO
HIGHER CROP : NO

TIME TAKEN [AUDIO]: N/A
TIME TAKEN [VIDEO]: 00:11:29
ENCODE RATE 1/2 : 31 FPS
ENCODE RATE 2/2 : 21 FPS

TIME TAKEN [TOTAL]: 00:11:39

[ricardo.santos]

10 minute vob, final size 93 0.20QF i got a value of 46%

PS: very fast the comptest, i mainly use avi.Net for episodic dvd conversions and if i multiply that time by 4 its not that much time.

[iNFO-DVD]

Excellent the comptest works , Bravo INFOGreat news :)

Here is my LOGThe log doesn't contain any comptest info yet, just the screen.

very fast the comptestThat's good, I bumped that second BETA1 back up to 5% as well.

Thank you both for at least running it for me and giving me feedback that it actually works, or at least does something.

[audioman]

The log doesn't contain any comptest info yet, just the screen.


Never mind , The comptest works :)

[audioman]

Thank you both for at least running it for me and giving me feedback that it actually works, or at least does something.

You're welcome !!!
Keep up the good work and inform us about the values and how to deal with them !!!!!!!

[ricardo.santos]

Thank you both for at least running it for me and giving me feedback that it actually works, or at least does something.

No problem, im the one who should be thanking you for developing avi.Net.

Does it only work with Xvid? xvid on my pc is slow the encoding is still running well beyond 20 minutes now with divx it would something like 15/16 minutes.

Would it be a good idea to disable "remove temp files" when doing a comptest? if the user decides to stop the conversion he wouldnt have to go through the indexing phase again?

Sorry just trying to understand, be "gentle" if im wrong as im not that "bright" with this stuff :)

[iNFO-DVD]

Would it be a good idea to disable "remove temp files" when doing a comptest? if the user decides to stop the conversion he wouldnt have to go through the indexing phase again?You don't have to anyway. If you stop it or there's an error the .TEMP files don't get removed. They only get removed if you have selected that option AND the conversion takes place as normal.

[ricardo.santos]

thanks for the clarification

[iNFO-DVD]

It obviously now goes without saying that once the comptest has ran I now need to do something about the results.

After reading THIS (http://forum.doom9.org/showthread.php?t=107897) thread about comptest presets I've decided, at least for now, to use this as a starting point. I can't at the moment use custom matrices, well, until I find some that are more standalone compatible.

I was able to discover that I can change the HomeTheatre profile for the CinemaPlus profile which has the same VBV e.t.c. constraints but allows me to alter the 'Quantization Type', H263, MPEG, MPEG-Custom, which is good news. I've altered the original presets, at least for now, as I think they were a bit too harsh and I'm not using the CQM's listed.

COMPRESSION TEST COUNTER MEASURES [avi.NET]
[key: comptest%|quant type|b-frames|quant restrictions|curve compression|Adaptive quant (lum masking)]

25-44 H263 B1,1.5,1 Q2,31 CC30,15 AQ?
45-57 H263 B1,1.6,0 Q2,15 CC15,05
58-89 MPEG B1,1.6.0 Q2,07(I6) CC10,03
90+ MPEG B1,1.0,1 Q1,05(I4) CC=NONE

Any thoughts?

Things that spring to mind.....
What to do below 25?
Using AQ in first preset
Thoughts concerning CC

I know the MPEG matrix is not everyone's cup of tea but it helps with SAP compliance whilst I'm not using CQM's and helps bring the value down when things are going too high.

Again, the Q1 in the 90+ presets is not everyone's idea as ideal but it does work well with movies like 'The Others' e.t.c. that are very dark. Some people *DO* want to reach there target size.

Just off to do some more reading on CC.

[iNFO-DVD]

Now updated to BETA 2, it will actually do something now depending on the value.

Download >> HERE (http://www.dvduk.pwp.blueyonder.co.uk/__files/avi.NET_2185_BETA2.zip) <<

Current Settings: (opinions sought and welcomed)

05/09/06 06:49 BETA 2

XX-39 [PROFILE LQ] H263 | 1,1.62,0 | 10,15,20 | 20,10 | 2,15
40-59 [PROFILE GQ] H263 | 1,1.62,0 | 05,10,15 | 10,05 | 2,09
60-89 [PROFILE HQ] MPEG | 1,1.62,0 | 05,10,15 | 05,05 | 2,07(I6)
90-XX [PROFILE UHQ] MPEG | 1,1.00,1 | 00,05,05 | 00,00 | 1,05(I4)

KEY: COMPTEST|PROFILE|QUANT|B-FRAMES|OVERFLOW|COMP CURVE|QUANTIZERS

One last thing.... Does anyone use the 352 screen size option? Shall I drop it and use something between 480 and 640, like 576?

[audioman]

Hi INFO

I will have a test tonight :)

I used to select a 0.30 - 0.33 QF for my movies

What kind of value should I use now before the movie encodes ? Carrying on using the same value ?

Will the comptest adapt its profiles to the selected value ?

Thanks for your explanations INFO :)

[ricardo.santos]

...opinions sought and welcomed....


ive tried it and choose a final size half of what i normally use, that gave a bits/pixel value of 0.10, after the compression test(17%)avi.Net choose the LQ profile and comparing the "no cp test " version and the version that used the LQ profile theres no doubt that the profile atenuated some effects caused by low bitrate encodings(ringing,blocks)

final video sizes were "respected" on both versions

Keep up the good work!

[audioman]

Hi ricardo

Do you usually use a 0.20 bits/pixel ? is this not too low ?

Did you test on a short clip or an entire movie ?

[ricardo.santos]

No, i have a sort of rule for my backups:

for every minute of the film i "give it" 10 mg on my final video, for example 90 minute movie would have a size of 900 mg.

there are exceptions(full screen movies and movies wih lots of CGI/fx(King Kong for example didnt look so good in the CGI scenes with that rule, i had to increase the final size).

for the tests i encoded a 15 minute clip from one stargate sg1 episode. i choose 0.10 because thats half the value that avi.Net recommends without comptest.

[iNFO-DVD]

ive tried it and choose a final size half of what i normally use, that gave a bits/pixel value of 0.10, after the compression test(17%)avi.Net choose the LQ profile and comparing the "no cp test " version and the version that used the LQ profile theres no doubt that the profile atenuated some effects caused by low bitrate encodings(ringing,blocks). Final video sizes were "respected" on both versions. Keep up the good work!That's good news, thanks.

Do you usually use a 0.20 bits/pixel ? is this not too low ? Did you test on a short clip or an entire movie ?No, 0.20 is NOT too low though in saying that it's hard to say, hence the comptest.

It varies from clip to clip, or movie to movie. For instance, at the moment whilst programming I using a test clip from the movie 'Eight Legged Freaks', if I were to use your starting point QF it gives a comptest of 154%, far too high. If I use a starting point of 0.20 it gives 82%. The clip from 'Clarlies Angels' at 0.20 only gives me 35%. It just shows you the importance of the comptest.

Infact, concerning the comptest I have actually come to really appreciate it now. How could I have not had one....... and I would like to say it would never have been implimented had it not been for various people in this thread going on and on about. So I thank you all.

[iNFO-DVD]

Talking of the comptest I suppose what I'm trying to achieve with the four profiles is this:

LQ: Too low, not enough bitrate, have to take counter measures.
GQ: Within reasonable thresholds. H263 Good Quality.
HQ: Within reasonable thresholds. MPEG High Quality.
UHQ: Too high, too much bitrate, have to take counter measures.

The GQ and HQ will just use, I suppose, the best quality setting, within reason, that I can do whilst keeping SAP happy.

With LQ and UHQ things get more tricky. The UHQ I would say is easier to deal with, the MPEG quant type will help sharpen and the slight change in codec settings will help bring the % down. The LQ setting would probably be the hardest to deal with, it's just too low, I may have to change the encoder performance with the LQ setting. Here's an extract from the DIVX guide when talking about the 'SLOW' Encoder Performance mode: Slow performance/quality mode enables the rate-distortion algorithm, designed to dramatically improve the video quality of low-bitrate encoding.

So for example, if a comptest determines Profile LQ because the value is so low it isn't saying oh, you want a low quality encode, hence LQ, it's saying more like, it *is* going to be low quality so lets set everything up to make it better, a counter measure, does that make sense. It could end up and LQ and UHQ encode could take a longer.

I know I'm just ranting on, just speaking out loud really, just keeping you all informed of my thoughts.

The LQ and UHQ could also invoke some different synth filters to help, any ideas?

[iNFO-DVD]

Adding the divx comptest support to avi.NET is giving me a right headache. Using the same methods as I did with XVID I'm getting totally different results, (usually much lower). I verified this with AutoGK, DIVX % is usually much lower. Reading some threads I have came across one or two that suggest problems with doing comptests with DIVX. Hmmmm, what to do.... what to do......

1. So do I just go with and accept its different value?
2. Accept its different value but take that into account when looking at what counter measures to take? (like change the counter measure thresholds)
3. Do the comptest using XVID, even for DIVX, to get the correct, more widely accepted, or known, values? (though this does seems weird)

Anyone got any input on this?

[Sharktooth]

Well, when changing codec parameters the compressibility will change.
If you're going to repeat the comptest with the new settings, remember there are few other "counter measures" to add (like lowering resolution or heavy filtering).
IMHO lowering the resolution is the way to go so you can mantain a good encoding speed (even with slower codec settings) and a good overall quality.
You can even do it in a "single pass", maximize the codec settings and lower the res by a notch.
But remember to:
a) Inform the user that quality is too low and avi.net is reducing the res
b) Keep a SAP compatible resolution
c) Offer an option in the program settings to override the resize (and go on with the user selected resolution) or an option to disable the comptest.

[audioman]

for the tests i encoded a 15 minute clip from one stargate sg1 episode. i choose 0.10 because thats half the value that avi.Net recommends without comptest.

Ricardo , you are talking about a value avi.NET recommends

Where the hell is that value ? is this the value below in the pic ?

http://pix.nofrag.com/6d/16/05cbe220732eebedd7c7db208e5bt.jpg (http://pix.nofrag.com/6d/16/05cbe220732eebedd7c7db208e5b.html)

Second question

I am completely newbie to the comptest
What should I do ? Keep the 896 value and then run the encoding ?
The comptest will give me a value and determine the proper profile to use , this is it ? :eek:

Thanks for your answers guys

[weaver4]

That's good news, thanks.

No, 0.20 is NOT too low though in saying that it's hard to say, hence the comptest.

It varies from clip to clip, or movie to movie. For instance, at the moment whilst programming I using a test clip from the movie 'Eight Legged Freaks', if I were to use your starting point QF it gives a comptest of 154%, far too high. If I use a starting point of 0.20 it gives 82%. The clip from 'Clarlies Angels' at 0.20 only gives me 35%. It just shows you the importance of the comptest.

Infact, concerning the comptest I have actually come to really appreciate it now. How could I have not had one....... and I would like to say it would never have been implimented had it not been for various people in this thread going on and on about. So I thank you all.

Or use single pass, constant quality!

[audioman]

Could I have answers please cause I gotta encode a bunch of movies

Thanks

[ricardo.santos]

Ricardo , you are talking about a value avi.NET recommends

Where the hell is that value ? is this the value below in the pic ?

http://pix.nofrag.com/6d/16/05cbe220732eebedd7c7db208e5bt.jpg (http://pix.nofrag.com/6d/16/05cbe220732eebedd7c7db208e5b.html)

here it is:
http://img423.imageshack.us/img423/1830/untitledfm8.jpg

Second question

I am completely newbie to the comptest
What should I do ? Keep the 896 value and then run the encoding ?
The comptest will give me a value and determine the proper profile to use , this is it ? :eek:

Thanks for your answers guys

exactly. i think the default value that avi.Net gives us is a good starting point, with time you´ll be able by looking/watching the movie what the final size will be, for a 640 encoding like i said before the "1 min=10 mg rule" is a good starting point.

[audioman]

Thanks ricardo

This is pretty weird

How can a 896 Mo movie give a very good quality ?

I am quite surprised

[ricardo.santos]

Thanks ricardo

This is pretty weird

How can a 896 Mo movie give a very good quality ?

I am quite surprised

how long is your movie? what aspect ratio?