Trying to implement SBC for DivX 4: released

[DmitryR]

Hello all,

I made the new version of my util for DivX 4 log. Now it is more patcher then analyzer.

You are to make first pass with 6000 bps and min/max quantizers 2. Then the util will recommend max quantizer and write new log for the desired bitrate. Features different types of quantizer distribution plus separate settings for DF and KF quantizers.

Available at http://divx4log.narod.ru along with an old version.

[oPerrin]

Can someone explain to us Newbies what this means/is good for? :)

-Perrin

[DmitryR]

This means that we (check the BlackSun's Morphix also) are trying to take control over DivX 4 2-pass compression. It can enhance compression quality because codec reads/writes the log file seqentially, and my utility operates on an entire file, so has full statistics and can make more precise decisions.

It is good for making all keyframes with low compression (improves quality with low bitrate cost) and distribute compression ratio over delta frames with desired algorythm. For example, if we have dynamical movie, we might want to have logarithmic distribution, and for slow motion exponential will be more suitable.

It also is good to forecast quantizer values after first pass, not after the job is done. For example, recently I set quantizers 4-14 for movie and got undersized file because nearly all quantizers went to 4. With this utility I can see that this movie will need settings like 2 for KF and 4-12 for DF.

[Kandor]

I have been playing with the quant settings and I am not so interested in getting a specific size but I am interested to get a clean copy of my movie and I just found out that atleast for me it is very hard to see the diffrence in quant 2 and 3.
but 2 uses a lot more space.

so my question is it possible to use quant 3-3 ? and not 2-2?
and if anyone thinks that you can actually see a diffrence please post and tell me.
because the movie I have tried this on is Aliens and that movie is dark so it can be why it actually looks no diffrence between 2 and 3.

[DmitryR]

For the first pass you are to use quantizers 2-2 so my util can see actual sizes of frames. After that set quantizers in util for KF and DF as desired. I advise to set 2-2 for KF and 4-? for DF.

As for me, I noticed that quantizer 2 helps a lot when you have large areas of similar colours, for example clean blue sky. With q=3 it sometimes is blocky even with postprocessing=6.

[Steady]

Thank you for your fine work Dmitry!

Could you tell me what the ratio's are between quant levels in Divx4 ?
In Divx3 a frame at quant level 3 was close to 2/3 the size of the same frame at quant 2. This does not seem to be true for divx4. It seems the quant 4 is close to 1/3 the size of quant 2 instead of 1/2 the size.

[Kandor]

Thanks man, just saw that it looks much better at Q2 than Q3.
it just have to be a brighter scen to be detected.
I tried my old American Pie and that movie is very colorfull and there it did look a much better att Q2.
I will try your program when I come home from work today.
I started the second pass normal first to be able to compare yours with the normal.

its nice that you have taken your time to do this.
I think Nandub is just amazing and if you could get that with divx 4 it would be very nice.

just one bug I saw when I started your program was that the options didnt look good. text was on text so it was very hard to read but thats just cosmetic so that is not as important as the program itself.

[DmitryR]

Originally posted by Steady


Could you tell me what the ratio's are between quant levels in Divx4 ?


Statistical examination of different log files makes me think that "quantizer" is a synonym for "compression ratio". For if you make one movie twice with bitrate, let say, 600 and 1200, measures of central tendency of quantizers will be exactly double. May be it is not true for large quantizer values - I have examined only up to 15.

[vlad59]

Hi DmitryR,

I just tried your tool and I must admit that I should have miss something.

I made a first pass at 6000 with quantizer 2-2. I loaded it in your tool and play a little with the setting. I set my projected size and then F7, my min an max quantizer changed. I reduced the KF quantizer to 2-2 and verify the total size it seems good so I saved the new log.

But if I want to encode the second pass with the new log, I don't have the bitrate.
Do I have the calculate it with the projected file size and the length of the movie ?????

I should have missed something !!

Thanks in advance

[DmitryR]

Originally posted by vlad59


But if I want to encode the second pass with the new log, I don't have the bitrate.
Do I have the calculate it with the projected file size and the length of the movie ?????



Yes, you does currently. But I have received a number of notifications about this item so probably next version of my util will do it for you.

[b0b0b0b]

Do you know how your quantizer distribution strategies differ from the default behavior of divx 4.02?

Thanks

[DmitryR]

Originally posted by b0b0b0b
Do you know how your quantizer distribution strategies differ from the default behavior of divx 4.02?


Seems that codec uses square distribution by motion. At least this algorythm gives project size close to codec results with same quantizer settings.

[murattttt]

Well I must admit what you have done is quite impressive. However it needs a guide that explains every possible way of using it effectively. Those count as size predictibility and greater quality.

[Klumsy]

Thanks for sharing your tools with us.
I try to hit a certain file size with the best possible quality, so that 2 or 3 CDs are fully used up (I don't like to have only 100MB or something on 1 CD ;) ).
Could you please write a litte guide that explains all the options exactly and which option to chose for the right film?
And a more detailed step by step guide would be appreciated too. :)
Or maybe, if you are too busy, someone else could probably explain that stuff, cause I'm not that good in it :D

Thanks in advance

[Klumsy]

Isn't it possible to do kinda function where you only enter the desired filesize and the tool would automatically do the settings, save the log-file and give you the right bitrate. Would this be possible?

[DmitryR]

I will pay attention to documentation as soon as I will rewrite the util on C++ for some people can not use this clumsy VB setup.

As to automatic settings - it is not designed for. If you want to think less - just use the codec with default settings. This util is designed for advanced work.

[Peters]

Hi DmitryR, 0.32, nice new version !

I got very undersized files using your prog.
In fact, i use low bitrates like 800 kbps and size prediction (6000->800 interpolation) seems to be very hard.

Am i missing something ?

[DmitryR]

I have uploaded stills trom the movie coded with the help of the utility to my site http://divx4log.narod.ru.

Also have an information that C++ version is nearly written and will be uploaded soon. Great performance improovement (approximation and verification will be realtime).

[DiegoFG]

Hi, DmitryR.

Excellent, work! :)

This program is really good, but in the versions "0.23/0.24" you can specify "Start/End" for the Final Credits (directly).

Is this possible with the new versions (i.e.: 0.32)? In other words, we need MODIFY Quantizers (manually) since the Frame "xxxxx" until End frame (for example) of the movie (LOG) for the Final Credits?

Bets regards.

[DmitryR]

Originally posted by DiegoFG
Hi, DmitryR.

... in the versions "0.23/0.24" you can specify "Start/End" for the Final Credits (directly).

Is this possible with the new versions (i.e.: 0.32)?


In newer version, if you, let say, hav credits from the beginning to the frame 5000 and from the frame 150000 up to the end you must create 3 zones: 0-5000, 5001-149999 and 150000-end. Then set quantizers for the first and last zones to high values (I use 6-8 for key frames and 15-16 for delta frames) and relatively low values for the second zone (I recommnd 2-2 for key frames).

BTW Check the new version 0.34 from 23 Nov. 2001.

[Peters]

Hi DmitryR.

I'm not lucky with your versions :)

With the last, trying opening a log (6000 frames) i got an "Invalid pointer operation" In the bottom i read 500 frames read.

So i encode just 1000 frames. This time, i can play with your prog but it can't exit, i get an access violation. I have to kill the prog.

(Tbird 1.33 ghz, WinXp)

Thank's for your efforts

[b0b0b0b]

Dmitry: is there any way you could post "before" pictures along with your "after" pictures of your godzilla encoding?

Thanks

[b0b0b0b]

Hi, it turns out your logfile writer is incompatible with divx4.11. You say "quantizer" instead of "quant"

[b0b0b0b]

First I was hoping to get a question answered. Is the quantizer distribution dependent on the target bitrate or are they independent? I guess I need a definition of quantizers in terms of divx.

Okay, I just spent some time testing divx4log with divx4.11. I noticed something weird about divx4.11's default behavior (I don't know if Divx4.02 does this). If I say my minQ is 2 and my maxQ is 12, the logfile shows all quants of 12. Maybe this is a way for the first pass to tell the second pass to use its best judgement, bow I fail to see how this conveys the minQ information. Bug? At any rate, I have good filesize predictability with the default behavior.

On to divx4log. I've found that filesizes using divx4log are wildly unpredictable. For my 12 second test file, divx4log suggested a file size of 1318570 or about 879kbps. So if I use that as my bitrate for the second pass, I end up with a file that's 2356k, or about 1570kbps. So in an attempt to find the right "bitrate" to attain divx4log's predicted filesize, I did a kind of binary search and ended up here:

kbps filesize
200 1140
210 1142
211 1142
212 1142
213 1586
215 1586
220 1586

As you can see, divx4 does weird things if you happen to tell it some quantizer values other than maxQ in the logfile. The other question is why there's a 440K jump and not a gradual increase.

I'm looking forward to divx 4.5.

[Peters]

Originally posted by b0b0b0b
Okay, I just spent some time testing divx4log with divx4.11. I noticed something weird about divx4.11's default behavior (I don't know if Divx4.02 does this). If I say my minQ is 2 and my maxQ is 12, the logfile shows all quants of 12. Maybe this is a way for the first pass to tell the second pass to use its best judgement, bow I fail to see how this conveys the minQ information. Bug? At any rate, I have good filesize predictability with the default behavior.


Maybe this is a way to tell you make a mistake swapping Max Q and min Q in the Frontend codec :)

[b0b0b0b]

Nope, it's not that.

After looking at analyse.log I realized that the 1st pass's quant values are ignored by design. It's as though the quant column in the logfile is just a placeholder. In analyse.log you can see that the 2nd pass basically ignores the quant you told it, comes up with a new one, and then tells you the ratio between what you suggested and what it came up with, independently.

Assuming the analyse.log doesn't lie, divx ignores the min and max quant settings.

Is this old news? I hadn't heard of this before.

[ProfDrMorph]

Maybe another problem is that you should set both the min and max quantizers to 2 and the bitrate to 6000 for the first pass. Dmitry said this right at the beginning of this thread I think.

[b0b0b0b]

Yep I did that.

My point is that quantizer settings in the divx.log stats file are ignored. I ran two 2nd passes with the same stats file, but the second time I screwed with the quantizers. The filesize was the same. The analyse.logs showed the same quantizers being used regardless of my changes.

So (my guess is that) Dmitry's program works solely by modifying the other fields in an attempt to induce divx4's own quantizer distributor to decide that Dmitry's solution is the best.

After comparing the divx.log and analyse.log however it's apparent that at least with divx4.11 this is unsuccessful. Moreover, filesizes are unpredictable and uncorrelatable with the bitrate you set in the setup panel.

[Peters]

You have a pb because the log of Divx 4.11 is fine for me

Ex:
##version 2
quality 5
Frame 0: intra 1, quant 4, texture 122820, motion 0, total 122820, complexity 1535250
Frame 1: intra 0, quant 4, texture 14993, motion 1530, total 19891, complexity 187412
Frame 2: intra 0, quant 5, texture 11084, motion 1547, total 15868, complexity 221679
Frame 3: intra 0, quant 5, texture 24990, motion 1901, total 30968, complexity 499799
Frame 4: intra 0, quant 5, texture 18580, motion 1128, total 22622, complexity 371599
Frame 5: intra 0, quant 5, texture 21775, motion 1236, total 26105, complexity 435499

And divx ignoring min and max quant is a non sense..
I suggest you to verify again Max and Min Q

[Doom9]

I just checked one of the many divx4.11 logs I have... they all contain quantize levels different of 12.. and changing all the time.

[DmitryR]

Originally posted by b0b0b0b

... I ran two 2nd passes with the same stats file, but the second time I screwed with the quantizers. The filesize was the same.

... So (my guess is that) Dmitry's program works solely by modifying the other fields ...



In order to control the codec's behavior the utility changes quantizer, texture, total and complexity values.

[b0b0b0b]

I get the following. My minQ is 2 and my maxQ is 31. I'm using virtualdub 1.4.7 (same results for fast recompress and full processing), and avisynth 1.05. I'll try reinstalling divx 4.11.


Frame 56: intra 0, quant 31, texture 284, motion 441, total 2254, complexity 141999
Frame 57: intra 0, quant 31, texture 839, motion 783, total 4828, complexity 419499
Frame 58: intra 0, quant 31, texture 1241, motion 1121, total 6676, complexity 620499
Frame 59: intra 0, quant 31, texture 825, motion 1058, total 4452, complexity 412499
Frame 60: intra 0, quant 31, texture 869, motion 924, total 4864, complexity 434499
Frame 61: intra 0, quant 31, texture 1247, motion 1208, total 6173, complexity 623499
Frame 62: intra 0, quant 31, texture 1477, motion 1363, total 7390, complexity 738499
Frame 63: intra 0, quant 31, texture 1329, motion 1403, total 6138, complexity 664499
Frame 64: intra 0, quant 31, texture 1524, motion 1540, total 7625, complexity 761999
Frame 65: intra 0, quant 31, texture 1292, motion 1663, total 7389, complexity 645999


Edit:
Okay I reinstalled divx4.11 and got the same results. I tried encoding an avi (as opposed to avisynth) and got the same results. I tried running an encode through dvd2avi... same thing.

I guess my installation is broken

[michbeck]

Hi DmitryR!
I've read that you want some feedback. here's my report:

I've made a 2cd-rip of 'The Mummy returns (PAL)' with ac3. After a first pass at 6000 kBit/s with quant 2-2 i run your utility version 0.32 . I can't work with 0.35 because vdub gives always a 'invalid log file'-alert when i use the logfile made with 0.35 .
Anyway, the filesize i want to reach was about 1028 MB. I set KF to 2-2 and DF to 3-4. The projected filesize with this settings was 972 MB. After the second pass with 1225 kBit/s the final avi was 1079 MB. I wonder because the times befor when I use the tool the final size of the avi were closer to the predicted size.
The quality is very good! Do you have an idea, what's wrong with my 0.35-version?
Thank you for sharing your tool with us!

greez, michbeck

[DmitryR]

Hi, it turns out your logfile writer is incompatible with divx4.11. You say "quantizer" instead of "quant"...

... Do you have an idea, what's wrong with my 0.35-version? ...


It has a bug :(. You can replace the word "quantizer" in resulting log to "quant". Or download the version 0.36.

[J&J]

Great job!

How to use the trim option in right way.
After 1% increasing the file size seem to increase dramatically.

sorry for my English

Thank you for your divxlog!

[DmitryR]

Originally posted by J&J

How to use the trim option in right way.
After 1% increasing the file size seem to increase dramatically.


Then do not use it. It is useful on long movies, when there are enough frames to calculate quantiles used to trim unwanted peaks. sometimes 1-2% gives smaller file even.

[J&J]

The bitrate derived from divxlog if we set them
above or under

What would happen ?

[DmitryR]

Originally posted by b0b0b0b
Dmitry: is there any way you could post "before" pictures along with your "after" pictures of your godzilla encoding?

Thanks

I did it today.

[anarch1a]

Do you have any plan to port this program to linux?

[J&J]

Zone quantizer Min/Max sometime cannot change to 4 !

bug ?

[legman]

You will see real development and advances when the legal movie releases start taking place. That time is coming very soon. :)

[Ozymandis]

As much as I'd like to see legal divx movie releases, I doubt very much it'll happen :(

Eventually when they release the next "dvd" format they might use mpeg4 compression and increase the resolution quite a bit. Of course, you'd need a very very fast processor to play back divx at hdtv resolutions.

[DmitryR]

2anarch1a: No. I am not a Linux programmer, and the program is written on C++ Builder for faster development.

2J&J May be, I'll check for.

[J&J]

DmitryR:

Found the problem ?

J&J

[ohliuv]

Originally posted by DmitryR
.... For example, if we have dynamical movie, we might want to have logarithmic distribution, and for slow motion exponential will be more suitable....

Is this your recommendation?

Can you please explain in your website(next to the distribution graph) the difference between square and root. Something like 'high-motion scenes get less bits when using root...' etc. Will be more understandable for non-mathematicians, I think.

Also, can you please confirm that:
- logarithmic from the quote above;
- square root (blue) curve from the website; and
- Root DF Rule from the options of Divx4log
refer to the same type of redistribution rule?

[DmitryR]

Originally posted by ohliuv


Also, can you please confirm that:
- logarithmic from the quote above;
- square root (blue) curve from the website; and
- Root DF Rule from the options of Divx4log
refer to the same type of redistribution rule?

Yes, it is so. And I will add some explanation to web site at weekend.

2J&J: No, I did not find anything. Can you post me the source log and comprehensive description of the situation that causes the error?

[SpEeDaMiGo]

Hi
I'm new to divx4 log and I must admit that I'm totally lost.
I have this 2h movie I'd like the encode to a filesize of 1400mb.

I did the first-pass with quants 2-2 and a bitrate of 6000kbps, and I got this divx.log I'm able to load in DmitryR's prog.

I can set the quants to different values but what does actually result and what would I have to set in the codec's 2ndpass settings ?

Thanks for any help

[J&J]

DmitryR

I have two machines AMD 1.2G Me/XP.
The version .38 quantizers's limits KD KF 's MAX/MIN fields
cannot change to single number 2~9

So I back to the verion .37 .

The question don't appear in other people ?

I reinstall it many times ,and download again .

[DJ Bobo]

Well DmitryR, it's kind of you to try to implement advanced SBC encoding for DivX4, but I can't use it now, because I encode movies nights, it takes 8 to 12 hours in 2-pass mode, and I don't want to wake up in the middle of the night to patch the log file ;-)

Can't you automate that procedure?

[dvdyke]

Why not just patch Virtualdub like Nandub was done so that it has your SBCD for DiVX4 in it? The source for Virtualdub is out there ;)

[DmitryR]

2SpEeDaMiGo Projected file size and estimated bit rate for second pass can be found in lower left corner of the program window. KF interval and other settings from the left pane of the DivX second-pass settings should be the same as you used in the firts pass.

2J&J Still can not find the bug ... :(

2bobotns This utility is intended to be used by people who DO NOT want automated encodeing and want to hand-tune their settings. So I will not implement any automation here. Any automatic decision I can provide will not be better then the original DivX 4 work.

2dvdyke Totally have no time to dig in vdub sources. If someone wants to - let me know, I will make the source code downloadable from my site.

[Teegedeck]

It seems to be hard work to get DivX4 to do what we want :(
Unfortunately, both Dmitry's and BlackSun's logfile editors/patchers have crashed on me so far. Perhaps we should just regard it DivX4's fault and abandon this codec alltogether in favor of XviD :D There's still a lot to be done before it will be 'bug-free' (if there is such a thing) but at least the codec leaves options for improvements to two-pass.

[ChristianHJW]

... Dmitry and Isibaar working on XviD .... Hmmmm, what a wonderful suggestion Teegedeck !! I stated more than one time that i am personally convinced that Dmitry could contribute a lot to this interesting project .... hid coding skills are exceptionally as is his mathematical background, and a new video codec should be based on a strong mathematical base ...

[anarch1a]

@DmitryR:
thanks for your answer. keep up the good work!

bye

[EvilFoo]

Err.. could someone give me an overview of what this does? :) It looks interesting but its hard to comprehend with what everyone is talking about...

[EvilFoo]

Okay I read the whole thread and now its making more sense. How do I set min and max Q values using Gknot? How do I just do one (of the two) passes using Gknot? After this, how would I implement the new log file into Gknot to encode the second pass? Thx.

[FactorM]

EvilFoo:

I only use gknot to create the .avs file. Encoding takes place in VirtualDub. There you can set the min and max Q values in the Divx 4.xx Codec Options. Here you can also import your modified .log file

[EvilFoo]

Right. Okay well I made my first pass at 6000 bitrate, slowest, postprocessing=5, and min/max Q = 2. Took quite a while to do this (about 6-7 hours) and ended up with a 2.95 MB file? I loaded it into VDub and it was all black... What's going on?

[J&J]

DmitryR

Why my DV's avi after calculation alway up 3xxx .
And the similar results used ChristianHJW's setting Max=6 Min=4 bitrate=2500 could do.

Were Divxlog not suitable for DV ?

Thank you for your answer .

[DmitryR]

Originally posted by J&J

Why my DV's avi after calculation alway up 3xxx .
And the similar results used ChristianHJW's setting Max=6 Min=4 bitrate=2500 could do.

Were Divxlog not suitable for DV ?


If your DV source is 720X576 then there is no surprise that it needs 2500 kbps. I always reduce the resolution and set quantizers like 3-10.

And there is no matter for divx4log what kind of source do you use. It analyzes the log file after the codec so it can deal with any source that is supported by DivX 4.0 - DivX 4.12.

[everwicked]

Dmitry what are the incompatibility issues you state at your homepage?

Could you give some more info about that?

[DmitryR]

Originally posted by everwicked
Dmitry what are the incompatibility issues you state at your homepage?

Could you give some more info about that?

One persons wrote me that 4.12 logs indicated very small filesize in my util and I thought that the log algorythm changed in 4.12. But after I verified everything myself I found out that it is not so.

[everwicked]

Worked fine for me too!

Is there any proccess yet with DLV?

[DmitryR]

Originally posted by everwicked
Worked fine for me too!

Is there any proccess yet with DLV?

Already decided to implement auto zone split according to motion but have no time totally. Busy with recoding DVD2AVI to SSE2. And my mind do not support frequent task switching :)

[everwicked]

:D Cool m8.

Nice of you to do that for us :)

How about some support for AthlonXP in DVD2AVI?

[DmitryR]

Originally posted by everwicked

How about some support for AthlonXP in DVD2AVI?

Sorry ... SSE and 3dNow AthlonXP supports are useful for floating point optimization, but for video we need integer engine. SSE2 is best for it. Next AMD processor with Hammer core will support SSE2, and now we use Intel to write down and debug the code :) And please use http://rilanparty.com/vbb/showthread.php?s=&threadid=11033 for questions about DVD2AVI.

[DmitryR]

New version of divx4log supports requested automatic zone splitting. Check the http://divx4log.narod.ru

[everwicked]

You're the best!

[J&J]

Interval frames of autosplit couldn't change to other value ?

The kernel's errors was prompted after increasing/decreasing the value.

bugs ?

[KoVaR]

this may be stupid (like all my question :) )
but
hot use this thing ?
i run the program, open firt pass .log and.... ????????????

[FactorM]

@KoVaR:

...read tutorials like on www.everwicked.com

[kartakis]

I 've encoded the DVD Mummy Special Edition and decided to create a 2 disc avi with AC3 sound. I opened the log with the latest DiVX4Log 0.40 and used it to make 2 logs: one with no splitting and one with autosplitting. For both logs I used 2-4 for KF and 2-6 for DF( both in slow motion and High motion parts).
The first log reported that it needed a bitrate of ~1450 and the second ~340. Why this happened? The second bitrate is correct or it shows something else( the bitrate of the first part of the log for example)?
Is it possible for the high motion parts to use the square root and for the slow motion the root rule? Will this give better results?
Thanks,
Nikos

[J&J]

And

1. save as a project file (a txt file)

2. close and reopen it

these values would be different !

waiting for the 4.01

[FactorM]

@J&J:

i have the same 'bug', but i think those are the default values. anyway i haven't tried 4.0 yet.

[KoVaR]

Originally posted by FactorM
@KoVaR:

...read tutorials like on www.everwicked.com
thnx
but i dont understand a word
im stupid :)

[kartakis]

It also happened with the previous versions except ver .31. When I open some logs I get at some point while reading the file "Access violation at address 0040328D in module divx4log.exe. Write of address 009500A4.
I see the error with many logs, mostly pal DVDs,and I can't find a solution. I can't remember if the mem adress was the same in every case or if the error happened in the same frame( 23000 in this case).
But using the 0.31 ver the file loads without any problems.
I know that this isn't a memory problem( I use WinXp with 768MB Ram).
Nikos

[DmitryR]

Hello all,

I fixed the weird bug with the large number of zones. And the "Cancel" button in the "Autosplit averaging" dialog now works.

[kartakis]

I 've downloaded and tried the ver .41 and I 've got again the same problem. Now it is in address 0040318D and 00950080.
The frame that makes the program to crash is:
Frame 23000: intra 0, quant 2, texture 137185, motion 10617, total 160027, complexity 342962
I write below the info from frames 22992 and 23000 if someone can help

Frame 22992: intra 0, quant 2, texture 161821, motion 14682, total 202378, complexity 404552
Frame 22993: intra 0, quant 2, texture 149987, motion 12580, total 183711, complexity 374967
Frame 22994: intra 0, quant 2, texture 154749, motion 12341, total 188785, complexity 386872
Frame 22995: intra 0, quant 2, texture 145921, motion 14069, total 180593, complexity 364802
Frame 22996: intra 0, quant 2, texture 151031, motion 14034, total 184802, complexity 377577
Frame 22997: intra 0, quant 2, texture 149896, motion 13291, total 181626, complexity 374739
Frame 22998: intra 0, quant 2, texture 142593, motion 13282, total 175635, complexity 356482
Frame 22999: intra 0, quant 2, texture 157289, motion 12215, total 190071, complexity 393222
Frame 23000: intra 0, quant 2, texture 137185, motion 10617, total 160027, complexity 342962

Nikos

[DmitryR]

Originally posted by kartakis
I 've downloaded and tried the ver .41 and I 've got again the same problem. Now it is in address 0040318D and 00950080.
The frame that makes the program to crash is:


It will be better to send me the log along with these descriptions ...

[everwicked]

Just for the log, the averaging works ok for me with different values although it gives strange bitrates.

i.e. @ 1500 avg i got ~1000kbps
@1000 avg i got ~350kbps.

[everwicked]

Update: the above was fixed with 0.41

[everwicked]

I also updated my guide with the new version of divx4log for those of you that are having trouble using.

I tried to keep as simple as possible.

Check the new version at http://www.everwicked.com


I apologise for the multiple posts.

[DmitryR]

Originally posted by everwicked
Just for the log, the averaging works ok for me with different values although it gives strange bitrates.

i.e. @ 1500 avg i got ~1000kbps
@1000 avg i got ~350kbps.

If this happens with .41 then you'd probably send me your log too. If you will do it today not very lately I will be able to proceed it along with Kartakis' problem.

[everwicked]

Originally posted by everwicked
Update: the above was fixed with 0.41


It was before i got the new version my friend. Everything works fine now.

[kartakis]

And my problem is fixed. Thanks DmitryR.
it was the result of frames with complexity over 15000. I 'm using DiVX 4.11 build 268 and I had this problem from ver .32.
Nikos

[SpEeDaMiGo]

thanks DmitryR for your nice tool
It's getting more and more complex and nice for fine-tuning .... I like this
However I've got a question about your 0.42 release. When autosplitting it asks me for the averaging interval. What it does is obvious. Is there any disadvantage of choosing a value as low as possible to split the movie into as much zones as possible ?
IMHO there ain't any but correct me if I'm wrong.
Thanks

[DmitryR]

Originally posted by SpEeDaMiGo
Is there any disadvantage of choosing a value as low as possible to split the movie into as much zones as possible ?


Yes, the codec do not use exact values from the log but flattens the bitrate curve so if there will be too much zones it will not have enough time to adapt to it and resulting quantizers will be too far from desired.

[SpEeDaMiGo]

In this case, does 1500 represent your recommended value or what would you as an example suggest for a 2h movie with a bitrate of about 1400kbits.
The newest versions "support up to 256 zones"; so why did you add this if it could have a negative effect? On what number of zones do you think it's getting worse ?
When I lower the value from 1500 below 1000 down to 500 (for my 2h movie with around 1400kbits) I get about a 50kbits lower bitrate, what I can "invest" in lower quants.
Maybe I just don't understand enough to think of an answer myself.
Anyway, gonna play around a little and encode with different settings.

Thanks for any hint

[everwicked]

Possible bug or just me?


When i autosplit, no matter what average i use, the zone before the last (last one is credits so i add it before i split) takes the quantizers' limits values (upper right) instead of the autosplit settings...

Does this make sense?

[DmitryR]

Originally posted by everwicked
Possible bug or just me?


When i autosplit, no matter what average i use, the zone before the last (last one is credits so i add it before i split) takes the quantizers' limits values (upper right) instead of the autosplit settings...

Does this make sense?

No, it does not. It is a bug probably.

[obelix]

@everwicked:

Thank you for your guide. I am surprised how good this works.
I have one suggestion, though. When you set minQuant=maxQuant=2 you make the other settings like bitrate and rate control useless. So you only have to set the quants, the other settings can be left at default.
Why do you reduce the keyframe interval from 300 to 250? Does that improve the result?

@DmitryR:
I confirm the bug with a very high quant in the last autosplit zone

regards, obelix

[everwicked]

Originally posted by obelix
@everwicked:

Thank you for your guide. I am surprised how good this works.
I have one suggestion, though. When you set minQuant=maxQuant=2 you make the other settings like bitrate and rate control useless. So you only have to set the quants, the other settings can be left at default.
Why do you reduce the keyframe interval from 300 to 250? Does that improve the result?

@DmitryR:
I confirm the bug with a very high quant in the last autosplit zone

regards, obelix

You are right on the first one, i was using these settings before divx4log came out :eek:

250 is 10 second for PAL, in NTSC i suggest 300 which is 10 seconds again

[everwicked]

Basically in a second though the other settings DO matter because they are used to calculate the complexity/motion etc so it will make a difference..

Useful input on this appreciated.

[obelix]

I did two runs, one with bitrate 800 and one with bitrate 6000, both quants at 2. The two logs are identical. I did not change the ratecontrol, though. I understand that complexity/motion is not dependent on bitrate. But I will do another test.

[obelix]

I did one pass with Bitrate 900, both quants 2, otherwise default settings. Then I did another pass with bitrate 6000, both quants 2, rate control 200000, rate reaction 50, rate up/down 10.
The logs are identical to the bit.

[everwicked]

Nice input, i will update my guide in the next revision, thanks!


everwicked

[soujir0u]

I was wondering, does using Divx4log improve the quality and/or size predictibility compared to normal 2-pass? If it does, is the improvement a lot or only marginal?

[pale]

>>I was wondering, does using Divx4log improve the quality and/or size predictibility compared to normal 2-pass? If it does, is the improvement a lot or only marginal?

Yes to both. Quality improvements are rather small (and of course, highly dependent on the setting you use). Size predictability on the other hand takes a giant leap, mostly I've been able to reach predictability of +/- 1MB and can not remember an encoding with deviation more than 5MB (I'm not 100% sure though as I don't remember all the encodings).

[Migsan76]

It's possible use de DiX4Log under XMpeg 4.2? Or just in Gordian Knot method.

[FactorM]

@Migsan76:

should work with all methods using 2 pass. but you are unable to do both passes in one job, because you have to edit the log file after the first pass with divx4log.

[Migsan76]

FactorM,

Ok.

After 1-pass i will edit de Log file with DivX4Log. Then with XMpeg4.2 i will make the second pass with the log i have edit.

My question is if the XMpeg will use que definitions of quantatizer in the program or just the definions in the log file i have edit?

If anyone have tried the DivX4log under XMpeg, please post some result...

Sorry about the bad english... :)

[kartakis]

A new ver of Divx4log is out. The new ver. 0.43 supports a new feature in autospliting and fixes the autosplit bug.

[FactorM]

@Migsan76

In the second pass, only the quants of the log file you select in the codec options will be used.

[Acaila]

That is not entirely correct...

Let me quote a piece from this (http://rilanparty.com/vbb/showthread.php?s=&threadid=15994&pagenumber=3) thread:

Originally posted by Moshem
I just finished emailing with Dimitry the author of Divx4Log software.

We tried to force the codec to use only quantizer 2 for all frames. We defined a few zones and ran a 2nd pass.

Sadly, DivX4 ignored the zones and used quantizer 2-6 for the dark scene.

Dimitry did not like the results, but he says that maybe DivX4 isn't programmed to follow the log file *exactly*.

This is bad since there is probably no way to force DivX4 to "go easy" on dark scenes.

[everwicked]

(my) Conclusion:

It's not perfect, but better than the default...

[OUTPinged_]

can there be implemented some nandub features?

like "compression levels" tab, or high/low pass filters.

having possibility to redistribute by both motion and framesize would be nice too.


and one more question to all ppl: is there any tool which would show quantizers for each frame in already encoded avi? like nandub did with divx3 content.

[DmitryR]

In fact all divx4log job is approximation based on statistical data mining, so can not be very precise. Even prooved enthusiasts of the idea consider that it is not perfect. I afraid that future innovations will be insignificant with codec error.

In nearest future I plan to understand the XviD log format. And I am waiting for the DivX 4.5.

[evilhomer]

or did doom9 say there would be no 4.5, that 5.0 would be next? we've been waiting for 4.5 for awhile

[OUTPinged_]

lol, version doesnt matter as long as direct drf control could be attained

[jonny]

What is "direct drf control"?

[Crucio]

basically this is nandub for divx 4 correct?

[Snilly]

Can anyone in this thread tell me if they have taken the time to do a conversion using standard Gnot Divx4 2 pass, and then (with the same filters, size etc...) done the same movie using the Divx4log file technique using Everwicked's guide?

I'd really like to know if anyone has seen any improvements. This for me is the real test for this technique. Does the end movie actually look better?

Snily :sly:

[Fantasma]

Maybe this is not what you are looking for
but, here I go:

I have used NeoDivx and XMPEG both 2-pass process
then I started using Everwicked guide and GKnot, and
I have seen a great difference (Quality improvement)
when encoding my movies based in everwicked guide.

My last encode was "Dances with Wolves", trying to fit
the whole movie in just one CD, I trashed XMPEG version
and kept the newer one because the quality is slightly
better. Of course, 3 hours video in just one CD is too
much to get a decent quality, but at least I have it.

[everwicked]

The author has some screenshots in his page but it compares the DVD with the DivX.

Have a check if you like. http://divx4log.narod.ru and click on Demo.

[madoka]

Originally posted by Snilly
Can anyone in this thread tell me if they have taken the time to do a conversion using standard Gnot Divx4 2 pass, and then (with the same filters, size etc...) done the same movie using the Divx4log file technique using Everwicked's guide?

I'd really like to know if anyone has seen any improvements. This for me is the real test for this technique. Does the end movie actually look better?

Snily :sly:

My personal experience has been that the quality comes out a bit worse when using DivX4log, but then I'm pretty new, and everything I've done so far has been anime. For me the most significant reason why I don't use it anymore has little to do with quality: it simply doesn't streamline well. Unless it offers significant quality improvement (which IMHO it doesn't), I'll stick with vinalla DivX 4 2-pass...

[Snilly]

Thanks! thats the sort of feedback I was curious about. Anyone else? Someone who does think it makes a difference?

Snilly

[everwicked]

My expirience in contrast with madoka, is that divx4log improves quality.

And it gets these extra bits to get used from the codec (divx4 gives always undersized files)...

[jonny]

With divx4log the size is perfect... the quality depends by the settings you use in divx4log...

[kheperi]

my experience is that divx4log doesnt work at all!

Try a clip, do the first pass at 6000/2/2
start divx4log and devide the clip in 2 zones, each half the clip.
set 1 zone kf 2/2 df 2/2
Set the other zone kf 31/31 df 31/31
remember the wanted bitrate (and the frame number of the zonechange)
save the new log
Now run the second pass and watch closely the little window with the progress in Vdub (with the blue and red lines).
If you see a difference in the size of the frames in the first zone and the second zone then you did something different as me.
I dont see any difference.
You can check the analyse.log later and convince yourself that the whole clip has been encoded with roughly the same Quants. Just only depending on movement and so.
The quant setting for different zones HAS NO EFFECT AT ALL

Please test it your self.

[Acaila]

Just looking at the quants doesn't tell you a thing, because DivX4 calculates new quants during encoding.

What really matters is, do the two halves look the same when you watch the video or can you clearly see that one is encoded with 2, and the other with 31?

[kheperi]

looking at the new quants in analyse.log does tell a thing. And watching the encoding window during the proces does too.

I noticed this thing at first when i set much higher Q's for the endcredits and expected, but not got, a bad quality zone. The next time i watched the encoding window and indeed noticed that the frame sizes kept at sizes i would expect for just normal values, while i entered much higher numbers now. Next i used values of 31/31/31/31. But still big frames.

So i did some experiments

Please try it yourself!! I could be doing something wrong, although i have no idea what that could be.

Do what i did. Pick a short (or long. it doesnt matter) fragment of a movie and make 2 zones with divx4log, making the fragment into 2 equal parts. Set 1 at 2/2/2/2 and the other at 31/31/31/31 and encode it using the new log.

See for your self!
I'm sorry to say that as far as i can see the program doesnt do anything.

[Snilly]

Well that is very interesting. Wouldn't that suggest that Divx4 ignores the quant values in the LOG file completely? That doesnt make sense to me.

Is Dimitry still around to comment on this?

Snilly

[DmitryR]

Originally posted by Snilly

Is Dimitry still around to comment on this?


Yes, please E-mail me the log (do not forget to zip) and exact settings as long as the util version used. I will check for and give a conclusion.

[everwicked]

It is already known that the codec doesn't follow the log line by line, what's your point?


2/2/2/2 or 31/31/31/31 doesn't work because the codec probably has some error recovery algorithm if that makes sense.

How about testing when not taking it to the extreme?

And yet again if it does not affect the encoding at all then why we get perfect filesize prediction?

[Doom9]

actually.. I gave divx4log a spin recently.. and I really had to fight to achieve the desired size precition by restricting quantizers... and then the output ended up being like 80mb or so too large (and the regular divx4 got undersized by a couple of megs)

[kheperi]

Originally posted by Snilly
Well that is very interesting. Wouldn't that suggest that Divx4 ignores the quant values in the LOG file completely? That doesnt make sense to me.

Is Dimitry still around to comment on this?

Snilly

Well as i see it the quants are not ignored, but some other values change too, appearently leveling out the q change.

[kheperi]

Originally posted by DmitryR


Yes, please E-mail me the log (do not forget to zip) and exact settings as long as the util version used. I will check for and give a conclusion.


I rather would have you tried it yourself. There still is a chance i do something wrong, or whatever. Its easy to do some experiments with a clip of a few minutes length.

But i'll do another test and i'll send you the logs (zipped)

[kheperi]

Originally posted by everwicked
It is already known that the codec doesn't follow the log line by line, what's your point?


2/2/2/2 or 31/31/31/31 doesn't work because the codec probably has some error recovery algorithm if that makes sense.

How about testing when not taking it to the extreme?

And yet again if it does not affect the encoding at all then why we get perfect filesize prediction?

I noticed this thing NOT using extreme values, when i tried to compress the endcredits at a higher level. In fact i used the values you suggest in your guide...

The filesize prediction is something different. But i have no idea how all the numbers in the log are used, so i cant say anything shure about that. Maybe it IS the fileprediction accuracy that spoils it all.

[kheperi]

ok, new experiment:

Videoclip: The fishermen scene from the Blair Witch Project
(1889 frames, frame 17509 to 19398)
Rather much motion in there, but the clip doesnt change very much, so with constant bitrate the Q wouldnt change very much on average.

Ccodec 2pass, first pass:
slowest, 6000, kf-interval 250, 2/2/2000/10/20, encoding quality 85%, framedropping 0%

Divx4log:
created 1 extra zone (manually added)
frame 0 to 944 : kf 2/4 df 2/6
frame 945 to 1888 : kf 5/12 df 8/18
(so no 'extremes' used here)

All other values defaults. nothing changed.
Bitrate requested: 1574 (size: 14752257)
Statistics:
min total : 310913 / 129384
max total : 378504 / 346943
min motion: 2372
max motion: 5716

Codec: 2 pass, Second pass:
bitrate 1574

Visual estimation while watching Vdub status window/Video-tab
First part: kf +/- 18Kb df +/- 5...9Kb
Secnd part: kf +/- 22Kb df +/- 5...9Kb

Yes in the second half the kf's where BIGGER! not smaller as i would expect. And indeed i can see such in analyse.log

First 3 frames in the divx4log output logfile, used for 2-nd pass
Frame 0: intra 1, quant 2, texture 310913, motion 0, total 310913, complexity 777282
Frame 1: intra 0, quant 4, texture 74046, motion 3217, total 78375, complexity 744446
Frame 2: intra 0, quant 4, texture 84189, motion 3640, total 88119, complexity 1048135

First 3 lines from analyse.log
Frame 0: PRESENT, complexity 777282, quant multiplier 0.750000, texture 152315, total 152315
Progress: expected 46627, achieved 152315, dq 1.000000, new quant 5
Frame 1: PRESENT, complexity 925575, quant multiplier 0.680956, texture 46182, total 53861
Progress: expected 95385, achieved 206176, dq 1.000000, new quant 5
Frame 2: PRESENT, complexity 1052362, quant multiplier 0.807999, texture 62928, total 70924
Progress: expected 149548, achieved 277100, dq 1.000000, new quant 5

3 frames the divx4log output logfile, starting from the last kf
Frame 1750: intra 1, quant 10, texture 69402, motion 0, total 69402, complexity 4769796
Frame 1751: intra 0, quant 12, texture 29556, motion 3155, total 30798, complexity 3045049
Frame 1752: intra 0, quant 13, texture 29121, motion 3447, total 30325, complexity 3401118

3 Frames from analyse.log, starting from the last kf
Frame 1750: PRESENT, complexity 6940200, quant multiplier 0.750000, texture 211809, total 211809
Progress: expected 107349783, achieved 96927043, dq 0.736091, new quant 5
Frame 1751: PRESENT, complexity 4547076, quant multiplier 1.377207, texture 56887, total 64405
Progress: expected 107436646, achieved 96991448, dq 0.735772, new quant 5
Frame 1752: PRESENT, complexity 5824200, quant multiplier 1.455503, texture 62989, total 70844
Progress: expected 107528338, achieved 97062292, dq 0.735500, new quant 5

As you can see from the analyse.log the really used q's here are all 5. There is some variation in the rest of the file but its all really very close to 5. And indeed in the first half i see more 6's then in the second half (which had a higher q setting in the log).
So its clear theres something wrong.

I'll send you the files DmitryR. I'm glad you show interest in this and do not hide behind excuses. I really appreciate what you made to help the divx community and i try to give honest and sound feedback, just hoping that it will end up in a functional extra tool to get best possible results!

(edit)
resulting avi size: 13.367.296
(/edit)

[DmitryR]

DivX starts to follow the logfile after 1000+ frames, so the divx4log is useless for clips but for long movies. And it is useless to make zones shorter that 3000-5000 frames. Codec will have no time to react.

[kheperi]

ahh i see. Interesting.
There's no way of shortening that time (or number of frames)?
3000 frames is ummm 2 minutes of film 5000 is 3.3 minutes. thats a long time, since high motion zones normaly are shorter then that.

But still....
I noticed all of this when trying to lower the bitrates for the endcredits. Normally they take some 4 to 6 minutes or so. but the bitrate didnt lower at all with divx4log. So i'm not really convinced if you are right about this. I'll run some more tests and i will post my findings here. I'll make shure the clip i use is much more then 10,000 frames long.

Another thing:
Morphix does NOT take that many frames to work. I did an experiment with another short clip (of a few minutes) and it works INSTANTLY.
As you probably know, Morphix only changes the complexity values (4 times more means half the bitrate).

[kheperi]

ok i did another test

Seems you are wrong.
I did the same sort of test with a clip of 10500 frames big, but still no effect at all.