Trying to implement SBC for DivX 4: released

[DmitryR]

Hello all,

I made the new version of my util for DivX 4 log. Now it is more patcher then analyzer.

You are to make first pass with 6000 bps and min/max quantizers 2. Then the util will recommend max quantizer and write new log for the desired bitrate. Features different types of quantizer distribution plus separate settings for DF and KF quantizers.

Available at http://divx4log.narod.ru along with an old version.

[oPerrin]

Can someone explain to us Newbies what this means/is good for? :)

-Perrin

[DmitryR]

This means that we (check the BlackSun's Morphix also) are trying to take control over DivX 4 2-pass compression. It can enhance compression quality because codec reads/writes the log file seqentially, and my utility operates on an entire file, so has full statistics and can make more precise decisions.

It is good for making all keyframes with low compression (improves quality with low bitrate cost) and distribute compression ratio over delta frames with desired algorythm. For example, if we have dynamical movie, we might want to have logarithmic distribution, and for slow motion exponential will be more suitable.

It also is good to forecast quantizer values after first pass, not after the job is done. For example, recently I set quantizers 4-14 for movie and got undersized file because nearly all quantizers went to 4. With this utility I can see that this movie will need settings like 2 for KF and 4-12 for DF.

[Kandor]

I have been playing with the quant settings and I am not so interested in getting a specific size but I am interested to get a clean copy of my movie and I just found out that atleast for me it is very hard to see the diffrence in quant 2 and 3.
but 2 uses a lot more space.

so my question is it possible to use quant 3-3 ? and not 2-2?
and if anyone thinks that you can actually see a diffrence please post and tell me.
because the movie I have tried this on is Aliens and that movie is dark so it can be why it actually looks no diffrence between 2 and 3.

[DmitryR]

For the first pass you are to use quantizers 2-2 so my util can see actual sizes of frames. After that set quantizers in util for KF and DF as desired. I advise to set 2-2 for KF and 4-? for DF.

As for me, I noticed that quantizer 2 helps a lot when you have large areas of similar colours, for example clean blue sky. With q=3 it sometimes is blocky even with postprocessing=6.

[Steady]

Thank you for your fine work Dmitry!

Could you tell me what the ratio's are between quant levels in Divx4 ?
In Divx3 a frame at quant level 3 was close to 2/3 the size of the same frame at quant 2. This does not seem to be true for divx4. It seems the quant 4 is close to 1/3 the size of quant 2 instead of 1/2 the size.

[Kandor]

Thanks man, just saw that it looks much better at Q2 than Q3.
it just have to be a brighter scen to be detected.
I tried my old American Pie and that movie is very colorfull and there it did look a much better att Q2.
I will try your program when I come home from work today.
I started the second pass normal first to be able to compare yours with the normal.

its nice that you have taken your time to do this.
I think Nandub is just amazing and if you could get that with divx 4 it would be very nice.

just one bug I saw when I started your program was that the options didnt look good. text was on text so it was very hard to read but thats just cosmetic so that is not as important as the program itself.

[DmitryR]

Originally posted by Steady


Could you tell me what the ratio's are between quant levels in Divx4 ?


Statistical examination of different log files makes me think that "quantizer" is a synonym for "compression ratio". For if you make one movie twice with bitrate, let say, 600 and 1200, measures of central tendency of quantizers will be exactly double. May be it is not true for large quantizer values - I have examined only up to 15.

[vlad59]

Hi DmitryR,

I just tried your tool and I must admit that I should have miss something.

I made a first pass at 6000 with quantizer 2-2. I loaded it in your tool and play a little with the setting. I set my projected size and then F7, my min an max quantizer changed. I reduced the KF quantizer to 2-2 and verify the total size it seems good so I saved the new log.

But if I want to encode the second pass with the new log, I don't have the bitrate.
Do I have the calculate it with the projected file size and the length of the movie ?????

I should have missed something !!

Thanks in advance

[DmitryR]

Originally posted by vlad59


But if I want to encode the second pass with the new log, I don't have the bitrate.
Do I have the calculate it with the projected file size and the length of the movie ?????



Yes, you does currently. But I have received a number of notifications about this item so probably next version of my util will do it for you.

[b0b0b0b]

Do you know how your quantizer distribution strategies differ from the default behavior of divx 4.02?

Thanks

[DmitryR]

Originally posted by b0b0b0b
Do you know how your quantizer distribution strategies differ from the default behavior of divx 4.02?


Seems that codec uses square distribution by motion. At least this algorythm gives project size close to codec results with same quantizer settings.

[murattttt]

Well I must admit what you have done is quite impressive. However it needs a guide that explains every possible way of using it effectively. Those count as size predictibility and greater quality.

[Klumsy]

Thanks for sharing your tools with us.
I try to hit a certain file size with the best possible quality, so that 2 or 3 CDs are fully used up (I don't like to have only 100MB or something on 1 CD ;) ).
Could you please write a litte guide that explains all the options exactly and which option to chose for the right film?
And a more detailed step by step guide would be appreciated too. :)
Or maybe, if you are too busy, someone else could probably explain that stuff, cause I'm not that good in it :D

Thanks in advance

[Klumsy]

Isn't it possible to do kinda function where you only enter the desired filesize and the tool would automatically do the settings, save the log-file and give you the right bitrate. Would this be possible?

[DmitryR]

I will pay attention to documentation as soon as I will rewrite the util on C++ for some people can not use this clumsy VB setup.

As to automatic settings - it is not designed for. If you want to think less - just use the codec with default settings. This util is designed for advanced work.

[Peters]

Hi DmitryR, 0.32, nice new version !

I got very undersized files using your prog.
In fact, i use low bitrates like 800 kbps and size prediction (6000->800 interpolation) seems to be very hard.

Am i missing something ?

[DmitryR]

I have uploaded stills trom the movie coded with the help of the utility to my site http://divx4log.narod.ru.

Also have an information that C++ version is nearly written and will be uploaded soon. Great performance improovement (approximation and verification will be realtime).

[DiegoFG]

Hi, DmitryR.

Excellent, work! :)

This program is really good, but in the versions "0.23/0.24" you can specify "Start/End" for the Final Credits (directly).

Is this possible with the new versions (i.e.: 0.32)? In other words, we need MODIFY Quantizers (manually) since the Frame "xxxxx" until End frame (for example) of the movie (LOG) for the Final Credits?

Bets regards.

[DmitryR]

Originally posted by DiegoFG
Hi, DmitryR.

... in the versions "0.23/0.24" you can specify "Start/End" for the Final Credits (directly).

Is this possible with the new versions (i.e.: 0.32)?


In newer version, if you, let say, hav credits from the beginning to the frame 5000 and from the frame 150000 up to the end you must create 3 zones: 0-5000, 5001-149999 and 150000-end. Then set quantizers for the first and last zones to high values (I use 6-8 for key frames and 15-16 for delta frames) and relatively low values for the second zone (I recommnd 2-2 for key frames).

BTW Check the new version 0.34 from 23 Nov. 2001.

[Peters]

Hi DmitryR.

I'm not lucky with your versions :)

With the last, trying opening a log (6000 frames) i got an "Invalid pointer operation" In the bottom i read 500 frames read.

So i encode just 1000 frames. This time, i can play with your prog but it can't exit, i get an access violation. I have to kill the prog.

(Tbird 1.33 ghz, WinXp)

Thank's for your efforts

[b0b0b0b]

Dmitry: is there any way you could post "before" pictures along with your "after" pictures of your godzilla encoding?

Thanks

[b0b0b0b]

Hi, it turns out your logfile writer is incompatible with divx4.11. You say "quantizer" instead of "quant"

[b0b0b0b]

First I was hoping to get a question answered. Is the quantizer distribution dependent on the target bitrate or are they independent? I guess I need a definition of quantizers in terms of divx.

Okay, I just spent some time testing divx4log with divx4.11. I noticed something weird about divx4.11's default behavior (I don't know if Divx4.02 does this). If I say my minQ is 2 and my maxQ is 12, the logfile shows all quants of 12. Maybe this is a way for the first pass to tell the second pass to use its best judgement, bow I fail to see how this conveys the minQ information. Bug? At any rate, I have good filesize predictability with the default behavior.

On to divx4log. I've found that filesizes using divx4log are wildly unpredictable. For my 12 second test file, divx4log suggested a file size of 1318570 or about 879kbps. So if I use that as my bitrate for the second pass, I end up with a file that's 2356k, or about 1570kbps. So in an attempt to find the right "bitrate" to attain divx4log's predicted filesize, I did a kind of binary search and ended up here:

kbps filesize
200 1140
210 1142
211 1142
212 1142
213 1586
215 1586
220 1586

As you can see, divx4 does weird things if you happen to tell it some quantizer values other than maxQ in the logfile. The other question is why there's a 440K jump and not a gradual increase.

I'm looking forward to divx 4.5.

[Peters]

Originally posted by b0b0b0b
Okay, I just spent some time testing divx4log with divx4.11. I noticed something weird about divx4.11's default behavior (I don't know if Divx4.02 does this). If I say my minQ is 2 and my maxQ is 12, the logfile shows all quants of 12. Maybe this is a way for the first pass to tell the second pass to use its best judgement, bow I fail to see how this conveys the minQ information. Bug? At any rate, I have good filesize predictability with the default behavior.


Maybe this is a way to tell you make a mistake swapping Max Q and min Q in the Frontend codec :)

[b0b0b0b]

Nope, it's not that.

After looking at analyse.log I realized that the 1st pass's quant values are ignored by design. It's as though the quant column in the logfile is just a placeholder. In analyse.log you can see that the 2nd pass basically ignores the quant you told it, comes up with a new one, and then tells you the ratio between what you suggested and what it came up with, independently.

Assuming the analyse.log doesn't lie, divx ignores the min and max quant settings.

Is this old news? I hadn't heard of this before.

[ProfDrMorph]

Maybe another problem is that you should set both the min and max quantizers to 2 and the bitrate to 6000 for the first pass. Dmitry said this right at the beginning of this thread I think.

[b0b0b0b]

Yep I did that.

My point is that quantizer settings in the divx.log stats file are ignored. I ran two 2nd passes with the same stats file, but the second time I screwed with the quantizers. The filesize was the same. The analyse.logs showed the same quantizers being used regardless of my changes.

So (my guess is that) Dmitry's program works solely by modifying the other fields in an attempt to induce divx4's own quantizer distributor to decide that Dmitry's solution is the best.

After comparing the divx.log and analyse.log however it's apparent that at least with divx4.11 this is unsuccessful. Moreover, filesizes are unpredictable and uncorrelatable with the bitrate you set in the setup panel.

[Peters]

You have a pb because the log of Divx 4.11 is fine for me

Ex:
##version 2
quality 5
Frame 0: intra 1, quant 4, texture 122820, motion 0, total 122820, complexity 1535250
Frame 1: intra 0, quant 4, texture 14993, motion 1530, total 19891, complexity 187412
Frame 2: intra 0, quant 5, texture 11084, motion 1547, total 15868, complexity 221679
Frame 3: intra 0, quant 5, texture 24990, motion 1901, total 30968, complexity 499799
Frame 4: intra 0, quant 5, texture 18580, motion 1128, total 22622, complexity 371599
Frame 5: intra 0, quant 5, texture 21775, motion 1236, total 26105, complexity 435499

And divx ignoring min and max quant is a non sense..
I suggest you to verify again Max and Min Q

[Doom9]

I just checked one of the many divx4.11 logs I have... they all contain quantize levels different of 12.. and changing all the time.

[DmitryR]

Originally posted by b0b0b0b

... I ran two 2nd passes with the same stats file, but the second time I screwed with the quantizers. The filesize was the same.

... So (my guess is that) Dmitry's program works solely by modifying the other fields ...



In order to control the codec's behavior the utility changes quantizer, texture, total and complexity values.

[b0b0b0b]

I get the following. My minQ is 2 and my maxQ is 31. I'm using virtualdub 1.4.7 (same results for fast recompress and full processing), and avisynth 1.05. I'll try reinstalling divx 4.11.


Frame 56: intra 0, quant 31, texture 284, motion 441, total 2254, complexity 141999
Frame 57: intra 0, quant 31, texture 839, motion 783, total 4828, complexity 419499
Frame 58: intra 0, quant 31, texture 1241, motion 1121, total 6676, complexity 620499
Frame 59: intra 0, quant 31, texture 825, motion 1058, total 4452, complexity 412499
Frame 60: intra 0, quant 31, texture 869, motion 924, total 4864, complexity 434499
Frame 61: intra 0, quant 31, texture 1247, motion 1208, total 6173, complexity 623499
Frame 62: intra 0, quant 31, texture 1477, motion 1363, total 7390, complexity 738499
Frame 63: intra 0, quant 31, texture 1329, motion 1403, total 6138, complexity 664499
Frame 64: intra 0, quant 31, texture 1524, motion 1540, total 7625, complexity 761999
Frame 65: intra 0, quant 31, texture 1292, motion 1663, total 7389, complexity 645999


Edit:
Okay I reinstalled divx4.11 and got the same results. I tried encoding an avi (as opposed to avisynth) and got the same results. I tried running an encode through dvd2avi... same thing.

I guess my installation is broken

[michbeck]

Hi DmitryR!
I've read that you want some feedback. here's my report:

I've made a 2cd-rip of 'The Mummy returns (PAL)' with ac3. After a first pass at 6000 kBit/s with quant 2-2 i run your utility version 0.32 . I can't work with 0.35 because vdub gives always a 'invalid log file'-alert when i use the logfile made with 0.35 .
Anyway, the filesize i want to reach was about 1028 MB. I set KF to 2-2 and DF to 3-4. The projected filesize with this settings was 972 MB. After the second pass with 1225 kBit/s the final avi was 1079 MB. I wonder because the times befor when I use the tool the final size of the avi were closer to the predicted size.
The quality is very good! Do you have an idea, what's wrong with my 0.35-version?
Thank you for sharing your tool with us!

greez, michbeck

[DmitryR]

Hi, it turns out your logfile writer is incompatible with divx4.11. You say "quantizer" instead of "quant"...

... Do you have an idea, what's wrong with my 0.35-version? ...


It has a bug :(. You can replace the word "quantizer" in resulting log to "quant". Or download the version 0.36.

[J&J]

Great job!

How to use the trim option in right way.
After 1% increasing the file size seem to increase dramatically.

sorry for my English

Thank you for your divxlog!

[DmitryR]

Originally posted by J&J

How to use the trim option in right way.
After 1% increasing the file size seem to increase dramatically.


Then do not use it. It is useful on long movies, when there are enough frames to calculate quantiles used to trim unwanted peaks. sometimes 1-2% gives smaller file even.

[J&J]

The bitrate derived from divxlog if we set them
above or under

What would happen ?

[DmitryR]

Originally posted by b0b0b0b
Dmitry: is there any way you could post "before" pictures along with your "after" pictures of your godzilla encoding?

Thanks

I did it today.

[anarch1a]

Do you have any plan to port this program to linux?

[J&J]

Zone quantizer Min/Max sometime cannot change to 4 !

bug ?

[legman]

You will see real development and advances when the legal movie releases start taking place. That time is coming very soon. :)

[Ozymandis]

As much as I'd like to see legal divx movie releases, I doubt very much it'll happen :(

Eventually when they release the next "dvd" format they might use mpeg4 compression and increase the resolution quite a bit. Of course, you'd need a very very fast processor to play back divx at hdtv resolutions.

[DmitryR]

2anarch1a: No. I am not a Linux programmer, and the program is written on C++ Builder for faster development.

2J&J May be, I'll check for.

[J&J]

DmitryR:

Found the problem ?

J&J

[ohliuv]

Originally posted by DmitryR
.... For example, if we have dynamical movie, we might want to have logarithmic distribution, and for slow motion exponential will be more suitable....

Is this your recommendation?

Can you please explain in your website(next to the distribution graph) the difference between square and root. Something like 'high-motion scenes get less bits when using root...' etc. Will be more understandable for non-mathematicians, I think.

Also, can you please confirm that:
- logarithmic from the quote above;
- square root (blue) curve from the website; and
- Root DF Rule from the options of Divx4log
refer to the same type of redistribution rule?

[DmitryR]

Originally posted by ohliuv


Also, can you please confirm that:
- logarithmic from the quote above;
- square root (blue) curve from the website; and
- Root DF Rule from the options of Divx4log
refer to the same type of redistribution rule?

Yes, it is so. And I will add some explanation to web site at weekend.

2J&J: No, I did not find anything. Can you post me the source log and comprehensive description of the situation that causes the error?

[SpEeDaMiGo]

Hi
I'm new to divx4 log and I must admit that I'm totally lost.
I have this 2h movie I'd like the encode to a filesize of 1400mb.

I did the first-pass with quants 2-2 and a bitrate of 6000kbps, and I got this divx.log I'm able to load in DmitryR's prog.

I can set the quants to different values but what does actually result and what would I have to set in the codec's 2ndpass settings ?

Thanks for any help

[J&J]

DmitryR

I have two machines AMD 1.2G Me/XP.
The version .38 quantizers's limits KD KF 's MAX/MIN fields
cannot change to single number 2~9

So I back to the verion .37 .

The question don't appear in other people ?

I reinstall it many times ,and download again .

[DJ Bobo]

Well DmitryR, it's kind of you to try to implement advanced SBC encoding for DivX4, but I can't use it now, because I encode movies nights, it takes 8 to 12 hours in 2-pass mode, and I don't want to wake up in the middle of the night to patch the log file ;-)

Can't you automate that procedure?

[dvdyke]

Why not just patch Virtualdub like Nandub was done so that it has your SBCD for DiVX4 in it? The source for Virtualdub is out there ;)