divx5comptest

[hubereevez]

hi jonny,
I made few encodings and had no errors and a perfect predictible size with your comptest. but now with an other movie I have 50 Mo less. ( Used 60%, psycho -> light, all mpeg4 tools). I always used these tools and there was no problem at all (playback was good with QP). Any ideas? perhaps I need a 3third pass?
thanks
PS : thank you (again) for your tools.....

[DJ Bobo]

Another question that has been already answered thousands of times: if you get less size than you expected, your movie has already reached maximum quality and can't get any bigger.

And just a remark: don't ever use quarter pixel

[proletarian]

Originally posted by hubereevez
hi jonny,
I made few encodings and had no errors and a perfect predictible size with your comptest. but now with an other movie I have 50 Mo less.


Comptest has nothing to do with getting the filesize you want, rather with getting a suitable resolution for the filesize you have chosen, so you get a high quality rip.

Prole

[jonny]

What % of comp. test you have used?

[JimmyBarnes]

Originally posted by jonny
What % of comp. test you have used?

40-50 % with DivX Pro 5.0

prole

[jonny]

Sorry for being not clear :p

@hubereevez:
What % of comp. test you have activated when you have saved the .avs file (example: 1% or 2% or 5% ...)?

[hubereevez]

Thanks for your replying,

to bobotns and proletarian : I red many posts and guides from doom9 (I don't want to be rtfm'ed :) ). My question was, why is comptest saying , " between 50% and 60% with a bitrate of 544" : so for me I can expect the desire filesize. Comptest in gordian or comptest is "Compressibility Test", cause of the non predictible compressibility of the movie, so I don't understain why you're saying that it is non involve with filesize. Why not using QP?

To jonny : I always used 5%, should I increase the %? And I make a new one when I'm changing the resolution) (ex : 384*208 @0.20 comptest ---> 31% ; 352*192 new comptest --> 55%)

Thanks and bye....

[jonny]

It's a bit strange you get undersized with 60%... i usually get undersized after 80%

[theReal]

Increase your resolution! You're not going to get a good video with such a low resolution.

Then, have you turned on all the divx5 pro features when you do the comptest? I don't know if you can do this with the utility by Johnny(?), I always make the comptest manually and I use all the features in the comptest as well (b-frames, that is, nothing else for me). If you get 60% then, it should be ok. If you're getting 60% without all the features and then you use all features for the encode you might end up with an undersized file.

btw. as soon as you set your resolution to something like 512xnnn, you will not get an undersized file, I'm sure!

You shouldn't use Quarter Pel because:
-nobody has seen any improvements from it so far (the improvements are only there in theory, I guess)
-it might cause artefacting
-it uses much more processor power on playback.

[jonny]

Then, have you turned on all the divx5 pro features when you do the comptest? I don't know if you can do this with the utility by Johnny(?)


Yes you can :), if you are useing the manual method i described some threads ago, useing my util is the same (my util simply automate the process i have described)

jonny

[hubereevez]

Yep, I used mpeg4 tools with compcheck then I press the load button like descriving in the thread from jonny. But why should I -technikli- used an higher resolution if the comptest tells me that with 580*xxx i'm with 30% (enevn if I can lower the audio bitrate).

[jonny]

How long is the movie?
And what is your video target size (you can read this with gknot)?

[hubereevez]

Nothong unusual, about 2 hours on 700 Mo.
I try to make another comptest with another movie...Perhaps i've done something false or perhaps the QP option has modify the comptest....
thanks for replying jonny....:)

[theReal]

@jonny, I have a very bad memory for names, that's why I put the '?' But hey, it only had one letter too much, that's really good in my case ;)

I don't know why I never used your tool - maybe because it's also rather easy to do manually and I can impress my friends with it *LOL*
No, I think because I'm encoding a lot of avi's (almost no dvd's) and so I have to mess around with the avisynth scripts anyways.

@hubereevez, sorry, I thought you said you reached 60% - at 30%, getting an undersized result is weird (even at 60%, yet there might be a small chance - but really not at 30%)

[jonny]

I can impress my friends with it *LOL* :D:D:D

[jonny]

@theReal:

Ops don't forgive this: if you are useing b-frames and divx 5.0.1, you must delete the second frame line (there is a double 0 frame)

[theReal]

I don't use 5.01 yet because it doesn't work correctly on my sys (inexplainable "-2" errors occured).
But thanks for the tip, I guess it will be the same in 5.02 (which I really hope will come soon and fix the rest of the bugs)

[jonny]

inexplainable "-2" errors occured


This error came out when you encode in virtualdub?
(if you encode useing .vcf files saved with divx5.0, this can be the cause of the error, divx5.0.1 have changed a bit the internal data format)

[theReal]

hmmm, first I got the -2 error when I tried to capture, and capture settings had been saved with Divx5.0 before.

But then I tried to encode many different videos in normal mode, using no .vcf files, no filters and all different re-processing methods, but still I got this -2 error in 3 out of 4 cases.

[jonny]

hmmm, first I got the -2 error when I tried to capture, and capture settings had been saved with Divx5.0 before.


This is normal.


But then I tried to encode many different videos in normal mode, using no .vcf files, no filters and all different re-processing methods, but still I got this -2 error in 3 out of 4 cases.


This is really strange. Have you tryed to use the "restore codec defaults" function after useing the .vcf files?

[hubereevez]

:confused:
hem, did I wrote 30% soory, I'm always encoding between 50% and 60% but I have certainly messed up something with my last encoding....
With the second zero frame, I red some posts but never had problem with divx 5.0.1 what are the results if I don't delete the second frame line?

[jonny]

If you use CompTest, you don't have to worry about this, this is done in automatic
If you use the manual test, i think GKnot goes mad because it's not able to discard the correct frame every 13 frames.

[theReal]

@jonny, I'm making a fresh install this weekend (set up my harddrives as Stripe-RAID and install Win2k instead of 98SE - so basically I'm changing very much). I'm going to try 5.01 without installing 5.0 before and I hope it works.
Thanks for your info, you're more and more becoming the godfather of Divx 5 for me :) (mainly because you figured out how to do a comp. test!)

@hubereevez, jonny's tool for Divx 5.01 already deletes that second frame line (given you are using the updated tool for Divx 5.01).
Jonny was only warning me about it because I said I make the comp. test manually.

[jonny]

you're more and more becoming the godfather of Divx 5 for me :D

let me know if 5.0.1 work after the opsys update!

jonny

[hubereevez]

Thanks both of you, you answered all my questions.
Jonny can I add a link on my web page to your page?
Would it be possible to add (just for the name of the comptest result) what type of filter we took. so we don't have to rename the log file from 540*250.log to 540*250_bil.log?
thankssssssssssssssssssssssssssssssssssss

[jonny]

Jonny can I add a link on my web page to your page?


Yes!


Would it be possible to add (just for the name of the comptest result) what type of filter we took. so we don't have to rename the log file from 540*250.log to 540*250_bil.log?


I'm working on a new compressibility test, integrated with DivX 5 Enc, all will be out in this week (if the tests i'm doing will be ok)
CompTest is not 100% accurate (specially if you use b-frames), the new method will be VERY accurate ;)

[hubereevez]

yes!!!!!
i'll wait then
thanks.......

[theReal]

jonny, I guess you were right with the VDub presets and Divx 5.01 not working.

I'm using a shiny new Win2K system now and Divx 5.01 works like a charm for capture and encoding. Only it seems to be a little heavier on the processor usage, I think, than Divx 5.0 was. At least when capturing it seems so... but that might as well be the completely different WinTV drivers...

P.S.: I'm also waiting for your new comp. test tool. Will it be based on 100% quality encoding rather than on a 1st pass?

[jonny]

Happy for you that 5.0.1 now work!

The method for the new comptest will be based on the old good method used by GKnot (and the only working method i think: analysis of the log file), with some modifications, mainly for make this work correctly with b-frames.
The current method have some imperfections with b-frames, and the resulting comptest % value is not real, i think this is why we obtain undersized files with comptest values > 80%.
I'm currently testing it, my poor p3@733 is running for about 3 days now (without shutdown), but the results are very good ;)
I think i'll finish testing all in 3-4 days from now...
After this, it's your turn to test :D

Regards

jonny

[philippas]

Hey jonny do you rember my method for the compressibility test tha i had described ?here (http://forum.doom9.org/showthread.php?s=&threadid=22532) ? Have u tried it ?

[jonny]

Sure i've tried, but doesn't work, the final size prediction is always oversized useing the method you described.

Example:
If you keep 13 frames every 260 frames (better know as 5%) you'll have this situation in the final avi:

frame 0 (KEYFRAME)
frame 1
frame 2
frame 3
frame 4
frame 5
frame 6
frame 7
frame 8
frame 9
frame 10
frame 11
frame 12
frame 13 (KEYFRAME) <--- this keyframe is artificial
and so on...

keyframes need more bits... so if i use the final avi size as you described, i'm takeing in account a lot of artificial keyframes, and this false the test, trust me, try to encode a 5%, then encode the full movie and you'll see it's true.

i've tried the method with the final average quantizer... and doesn't work for the same reason.

The only method that work is cut off this artificial keyframes.

Thank you very much for try to help me, i've taken big ispiration from the method you described, the method i'm testing now is able to predict very well the final video size.

just wait a bit :) i'm testing it a lot to be sure there are no hidden trap

after finish testing the new version will be out (and near the tests i've done)

Cheers

jonny

[philippas]

I'm alway using using 130,13=>10% and i think is very accurate. I think that the artificial keyframes are not included but i'm not 100% sure. Does GKnot take's into account the artificial frames ?
I'll try to find the thread where alot of people where testing the SelectRangeEvery command. Bare in mind that the codecs doesn't scale linearly as the Gnot assumes. I think Steady was trying to improove the command and had conclude to some very usefull findings=> most movies scale with a factor between 0.7-1 thus the formulae for the prediction of the quality becomes:
(big resolution)/(small Resolution)*compTest*scale factor
Thus as you can see if the scale factor is smaller than 1 which in most cases it's true the test is undersized.
So as closer you do the Comp test to the target resolution as accurate will be.
The only way that you can see the above is correct is to do a comp test in a resolution X and then do a whole encoding in the same resolution and compare the results.

[jonny]

Does GKnot take's into account the artificial frames ?


Yes, it discard the frame


The only way that you can see the above is correct is to do a comp test in a resolution X and then do a whole encoding in the same resolution and compare the results.


I'm doing tests in this direction.
Fixing the parameters i want to test, i use 1%, 2%, 3%, 4%, 5%, 6%, 7%, 8%, 9%, 10% and then full 100% encode.
I compare the predicted file size in every case with the 100% encode, no more than a 2% difference in all the cases for now!

PS i have searched the thread you described a week ago, i've found a link to it but the thread seems to no more exists?!?

[rvas18]

the -2 error seems it has to do with you resolution not being in multiples of 4 and even sometimes 16. you should always check this to make sure.

[jonny]

This is not the only reason, if you use .vcf files from 5.0 with 5.0.1, a similar error came up (the internal data format is different for the 2 versions, the codec trust there is a wrong resolution simply because the internal data structure is not set correctly).
If you use wrong combination of DivX 5 - DivX 5 Enc - CompTest, this error came up too (try to encode with DivX 5 Enc v1.00 and DivX 5.0.1, you'll get the same error for the same reason)

[hubereevez]

youhouhouhou!!!
Finally you did it, great, this is a famous and great tool! Even your howwork.html is comprehensible by me!! (yes yes! :D)
But (there is a always a but by me), can you explain me 2 or three little things please?
*First how do I change the resolution in Gknot in order to reach 50 to 60%, is there a formula (for example I have 75% with 576*240 with bilinear). The thing is when I make 3 or 4 comptests, I must note all the settings back in the log files, perhaps you can add something in divx5enc in order to write to the log file which settings (resoltuion and filter) the user used....
*How works the credits encoding by now, personnaly, I write in the gKnot box the end of the movie (for exemple 173143 frames) and then I copy paste this number. Encoding then with 50 kbps. Virtual dub show then taht the bitrate specified is to low for the resolution....Is this technique good, or do I must write the entire frmaes of the movie in Gknot for the comptest, 50 is too low????

Thanks again for your soft!!!!!!
hubhub (I love you :eek: )!

[jonny]

Thanks :D:D:D



First how do I change the resolution in Gknot in order to reach 50 to 60%, is there a formula (for example I have 75% with 576*240 with bilinear).


If you want to calculate the new comptest value, after changing the resolution, i think philippas give a very good explanation on how to do this in this thread: http://forum.doom9.org/showthread.php?s=&threadid=19403
Please note that the best accurate way to do this is to make a new compressibility test with the new .avs file.
I personally use 1% when doing the first tests, than, when i've choosen resolution/filters, i make the last comptest at 5%, i think this is the fastest way.




The thing is when I make 3 or 4 comptests, I must note all the settings back in the log files, perhaps you can add something in divx5enc in order to write to the log file which settings (resoltuion and filter) the user used....


The best method is to save this parameters in the .avs file, example: "704x304_bil.avs".
You can save the job list after a comptest, this will save the status of the job too. After doing another comptest, you can use the append job list function to quick compare all the tests you have done directly in the job list. I think this is enough... let me know what you think about this :)




How works the credits encoding by now, personnaly, I write in the gKnot box the end of the movie (for exemple 173143 frames) and then I copy paste this number.


You must specify the correct number of frames for the .avs file you want to test, so if you want to do a comptest without the credit part, you must save the .avs file enabling the trim option and selecting "movie only".




Encoding then with 50 kbps. Virtual dub show then taht the bitrate specified is to low for the resolution....Is this technique good, or do I must write the entire frmaes of the movie in Gknot for the comptest, 50 is too low????


I'm not experienced in encoding the credits separately, so i can't tell you what is the best way (in the forum there are a lot of opinions, i think you must do some experiments)
If an error popup, telling you that the bitrate you specified is too low, means that the divx codec is not able to encode the clip with the bitrate you have specified.


Cheers
jonny


PS: hubhub (I love you :eek: )! hihihihihiihihiii :D

[jonny]

PS: anyway i think autodetecting resolution and filtering method used is a cool thing (i can put this in the status after the the job end).
Wait for the next ver :)

[theReal]

For the seperate encoding of the end credits, I find this the easiest way:

1. encode the end credits as soon as you have the "credits" avs-file
2. change the number of frames in Gknot to that of the movie file without the end credits (you can take this number from your "movie" avs-file)
3. add the end credits to the Gknot calculation with "Files -> Add"


@jonny, I didn't have time to check out your new tool, but I'll do so as soon as possible. Keep up the good work!

[jonny]

Thanks ;)

[hubereevez]

yopyop,
I'm so dumb that I didn't think to save a resolution_filter.avs stead of mymovie.avs. Ok, hem.....:scared:
I would not comptesting with a formula cos of your great tool and I haved already checked this formula but it's a litlle bit annoying to make such calculs (yes i'm not so hard worker...). I only asked if the result from the comptest of divx5enc is 70% at a given resolution which resolution should I next use (Gknot, cursor and save avs) with the new comptest (my english is very crappy too, hope you understodd).
The add fonction with gordian for credits : could it be difficult, cos of the audio muxing ,to really have the desire file size. Anyway, it's a godd techniqque and the total FPS could be see in d2v window by "set credit start". I planned to apend the credits without including them in the comptest, there are 3 or 5 Mo so overburning going well.....And last, The real could you explain me (if you gotta time) why is VirtualDub saying that 50 Kbps is too low, I want a crappy file for the credits, and Gordianknot is setting this bitrate by default for credit, isn't it? VirtualDub is forcing credits a higher resolution, I don't check this, and by the way it's a brake to a total-automation encoding process, na! :D
thanks both of you
hubhub

[jonny]

I only asked if the result from the comptest of divx5enc is 70% at a given resolution which resolution should I next use (Gknot, cursor and save avs) with the new comptest (my english is very crappy too, hope you understodd).


Are you asking what is the best range of comptest values to use with the new comptest? I think staying between 60%-70% should give very good results (with b-frames)
To find what are the best % values, i encourage all to encode and post opinions about this (someone remember my tests with the manual method? :D )

[hubereevez]

Yep, I though that when you go on higher resolution and still have a reasonable comptest, like 50-55%, then you have a better-quality movie than with a lower resolution and 70% comptest. There can be more artifacts with an higher resolution but not with 50 to 55%. Am I true?
and thanks (again, yes) jonny for your tools.....
hubhub

[JimmyBarnes]

Originally posted by jonny


Are you asking what is the best range of comptest values to use with the new comptest? I think staying between 60%-70% should give very good results (with b-frames)

To find what are the best % values, i encourage all to encode and post opinions about this (someone remember my tests with the manual method?

Jonny

Those are the sort of limits that one had to use with DivX3.11a. DivX5 claims to give the same quality in up to 40 % lower filesize. So far (with 8 DivXPro5.0 DVD rips) that seems a reasonable claim (tho I've just switched from DivX Pro 5.0 to 5.0.1).

Earlier DivX5.0 rips were done with around 40 % 1st Pass %(I determined the 1st pass size manually from a 1-pass 100 % quality quant=2 run ie. 4-6 hr run and not using your comptest) and gave good quality tho I stick to around 50 % at the moment. Only The Mummy Returns gave somewhat disappointing quality at 50 %.

Note that I watch all my rips on a 68 cm Sony TV in a well lit room. Quite different conditions from someone who watches on a PC monitor in a darkened room.

JB

[jonny]

This 40% is with or without b-frames?
Note that is you have done this test without b-frames, the real value with b-frames is around 55%

Anyway what value to use is a really subjective thing, this is because i encourage to post the results :) .

@hubereevez:
I agree with you, i prefer higher res too (but i usually reach at least 60% in any case)

[JimmyBarnes]

Originally posted by jonny
This 40% is with or without b-frames?
With B-frames

Note that is you have done this test without b-frames, the real value with b-frames is around 55%
:confused:
Could you expand on this a bit?
JB

[jonny]

I've only said that bframes increase compressibility by ~15%, so if you do a test without bframes and after you encode with bframes, the real value is 55%. I hope now it's clear what i was saying :)

!!! 40% is low for me... i think this is for the TV out, it should give a totally different perception.

Cheers
jonny

[hubereevez]

Great resultsss with 51% on pc too. I have encode with 720x298, 2 hours movie, b-frames and gmc; I'm suprise with the quality and this high resolution.....
hubhub

[JimmyBarnes]

Originally posted by jonny
I've only said that bframes increase compressibility by ~15%, so if you do a test without bframes and after you encode with bframes, the real value is 55%. I hope now it's clear what i was saying :)

Still not clear to me Jonny

If I encode with B-frames and get a 1st Pass % of 40 %, does this mean the "real value" is 55 %?

What do you mean by "real Value" anyway?

JB

[jonny]

I mean, if you do the 1st pass without bframes and you obtain 40%, the same 1st pass with bframes should give you something around 55%.

Since you have obtained 40% WITH BFRAMES, you can forgive what i've said.

Hope is clear now :)

[jonny]

Encoded 1h 30m movie at 63%... looks very very good (704x384).
(final avg quantizer around 4.0....)

[JimmyBarnes]

Originally posted by jonny
Encoded 1h 30m movie at 63%... looks very very good (704x384).
(final avg quantizer around 4.0....)

That's cool but it might well be possible to do the same using DivX3. For the 30+ DivX movies I've ripped with DivX3.11a I found in general I could encode a movie less than 100 min on a single CD with a 1st Pass % around 67 %. I'm hoping for better compression with the same quality from DivX5.

JB

[jonny]

The only way to compare DivX3.11a and DivX5 is to:

1 - Use the same .avs and the same bitrate (note: 1kbps with DivX3 = 1024 bytes, 1kbps with DivX5 = 1000 bytes), or use the same target size instead of the bitrate.
2 - Calculate the 1st Pass % with DivX3.11a
3 - Calculate the compressibility test value with DivX5 (activating bframes)

bframes make the difference (IMHO)!

[hubereevez]

Originally posted by jonny

(final avg quantizer around 4.0....) looks good too, you made such a great tool, why don't you put some banner that I can click 10000000per day , so you can be bilionar :cool:
can you exlain avg quantizer 4.0?
So waht I can deduce is go on highest resolution but do not go under 50% and taking a look at apect error and zooming....is that a valuable conclusion?
hubhub

[jonny]

can you exlain avg quantizer 4.0?


You can get this info after the encode, reading the "analyse.log" file, it indicates the average quantizer used in the encode (DivX5Enc show this value too). The cool thing is that if you use the similar values of compressibility test for differents movies... you'll end up with very near average quantizer values.


So waht I can deduce is go on highest resolution but do not go under 50% and taking a look at apect error and zooming....is that a valuable conclusion?


I like to use high resolution too, but at least a 60% of comptest value (consider that i don't like to use filters when i play)


looks good too, you made such a great tool, why don't you put some banner that I can click 10000000 per day , so you can be bilionar


:D:D:D
(10000000 per day = 115,7 cps (clicks per second) :O )

[hubereevez]

bouhou, :(
I made a complete guide in french, translation from yours jonny on divx5enc in french, and now gknot is out.....
Such a pitty..........;
c'mon lets make our own divx12proencoder :D
hubhub

[JimmyBarnes]

Originally posted by jonny
This 40% is with or without b-frames?
Note that is you have done this test without b-frames, the real value with b-frames is around 55%

Jonny

I think I boobed. I've been doing 1-pass 100 % quality without
B-frames and the 2-pass runs with. Naturally this has resulted in my getting abnormally low 1st Pass % values.

My 40-50 % results after correction are probably at least 55 %-65 % on your scale.

How did I make such a blunder? I always thought the 1st pass size had to be the maximum possible for that res - leaving out B-frames or any ther MPEG4 Tools sure maximises the result. Now it seems clear that the only difference allowable between 1-pass and the 2-pass runs for a valid comparison is that 1-pass is done with quant=2 (constant).

I hope I have got it right now.

JB

[jonny]

Yes it's right, to make a comparison you must use the same codec parameters you want to use for the encode.