DebugView analyzer for XviD codec v0.3

[MoonWalker]

v0.3

I've made some improvents

Write a file with the report call ANALYSED.TXT
1-pass size,scaled size and 2-pass (actual) size are in bytes and Kbytes.Also read the Readme.txt :)


If anyone find my prog usefull I would be glad :o :o I really don't know how usefull this is:p


You can download it from here

http://www.geocities.com/aleksikerayno/

EDIT : Scroll Down for v0.4

MoonWalker

[Franko30]

Nice Tool!

Finally my interest in the average Quantizers of my Voyager encodings was awakened (was too lazy up to now to calculate that "manually").

Thanks!

Two questions, though:

1) Would it be possible to add an analysis of Quant distribution of the Intra frames? For now I search through the Debugview Log in order to find out which Quants were used on the Intra Frames. That's because I want to find out what the different high<->lo settings of the alternative CC do the Intra Frames. Would be great if that was automated, too.

2) Does it matter whether the first pass was written to the log as well (that's important when making overnight encodings, where everything gets written to one huge Debugview Log)?

Cheers

Frank

[-h]

Ok, about having XviD write to a .log file instead of dbgview - I was just thinking of adding the frame number to the start of the line (heh should've done this anyway) and writing to \\machine\currentdrive\x1st.log or x2nd.log.

Because I am lazy :)

Is there more info you'd like?

-h

[Acaila]

Is there more info you'd like?

A report at the end with:
-(Min/Max)/Average Motion
-(Min/Max)/Average Overflow
-Quantizers used by the credits, mentioned seperately or just pooled with the rest of the movie (right now only the quants from the main movie are listed)


Ps.
I'd like to hear from one of the developers if there is any way to measure picture quality somehow? I've heard about PSNR but am not entirely sure what it means and how to measure it.
If you only look at the average quantizer a decrease would not necessarily mean higher quality. Actually what I'm looking for is a way to measure blurring.
Would average motion suffice for this:
-> lower quantizer + equal average motion = higher quality
-> lower quantizer + lower average motion = lower quality
Or doesn't it work like that?
Is there another variable that would be more suitable?

Any insight on this would be highly appreciated.

[MoonWalker]

@Franko30

No it will analyze all the debug file.I will fix tha to analyze only the second pass when you have both 1st and 2nd pass.Possible tommorow it will be ready

@-h

Actually i get the Frames from inter+intra :)..So it will be the same:)

@Acaila
Can these thinks been done by analizing the debug file?? If yes just tell me to inplement these thinks..

MoonWalker

[-h]

-(Min/Max)/Average Motion

What would this measure exactly? The number of intra-coded blocks, the sum of all motion vectors for a frame, the f-code value (max motion vector distance) for each frame, etc.

-(Min/Max)/Average Overflow

Should be simple enough. I'm not sure what motion/overflow would be used for though..

-Quantizers used by the credits, mentioned seperately or just pooled with the rest of the movie (right now only the quants from the main movie are listed)

Fair enough.

I'd like to hear from one of the developers if there is any way to measure picture quality somehow? I've heard about PSNR but am not entirely sure what it means and how to measure it.

It's the "Peak" signal-to-noise ratio of the entire film, and is quite a useless figure if you want to compare one sequence against another - a clip with a PSNR of 37 dB could look worse than one with 30 dB, since PSNR doesn't match our eyes in terms of perceived quality. It's only really useful for comparing a codec against another on an identical sequence, when testing core improvements etc.

If you only look at the average quantizer a decrease would not necessarily mean higher quality. Actually what I'm looking for is a way to measure blurring.

That is asking a lot :)

Would average motion suffice for this:
-> lower quantizer + equal average motion = higher quality
-> lower quantizer + lower average motion = lower quality

I don't know what "motion" is measuring. The kind of values you'd get after encoding an entire film would probably be useless though, they'd get cancelled out again and again as different kinds of scenes are encountered.

Judging the perceived quality of a video sequence is very hard, that's why even the MPEG consortium have settled on PSNR to gauge performance, though it is poorly suited to the task.

-h

[Acaila]

-(Min/Max)/Average Motion

What would this measure exactly? The number of intra-coded blocks, the sum of all motion vectors for a frame, the f-code value (max motion vector distance) for each frame, etc.

I was thinking that when we blur (at least with the temporal kind) the differences between two adjacent frames will decrease. Thus the motion vectors between those two frames will also decrease.
But since motion vectors change once the frame gets quantized, to measure motion vectors after quantization would be useless. If it were possible to report the size of the average (or total?) motion vectors for each frame (before quantization) in the dbglogfile we would have a measure for the amount of blurring that took place (when compared to an un-blurred encoding).

Anyway I could be completely off here, you tell me. I wouldn't want you to waste time coding senseless things just because someone thought it might be a good idea :)

[MoonWalker]

What are the sensible values of quantizers that a movie could take 2-15? or more??And for credits?? 20-31?? I ask this because I want to optimize the code a little...

Thanks,
MoonWAlker

[MoonWalker]

v0.4

-Added Report of Intra-Frames's and Inter-Frames's Quantizers
-If you have both 1st-pass and 2nd-pass in one file it will analyze only the 2nd-pass
-Remove on screen output.It only generates a file.

As usual comments and the relative stuff : s_ilias@gmx.net

You can get it from here :

http://www.geocities.com/aleksikerayno

MoonWalker

[Acaila]

What are the sensible values of quantizers that a movie could take 2-15? or more??And for credits?? 20-31?? I ask this because I want to optimize the code a little...
I don't know what other people use, but I throw away everything that uses quantizers higher than 7. About 99% of my rips only use quantizers 2-5. For credits always 31.

[avih]

Originally posted by -h
Is there more info you'd like?
-h

i join Acaila, end report as mentioned + complete setup of the parameters and the clip name (if exists). this way reports can be kept and logged compared among people.

about 'total motion', it should be quite easy to keep and eventually display global statistics as:
1. sum of absolute MV lengths for the whole film.
2. number of MV for blocks of inter/interV4 modes.
3. average MV length ~(1./2.) (or more correctly if including interv4 as 4 for 2.)
3. INTER/TOTAL blocks ratio: total_INTER_BLOCKS/total_BLOCKS

of course that might extend the encoding time just a bit, but should be controllable from 'extended output log' checkbox.

cheers
avi

[Teegedeck]

Just wanted to say that I go with Acaila here, as far as general quantizers are concerned (humm, I encode credits around quant 25, but then again, I'm a bit superstitious).

[-h]

I was thinking that when we blur (at least with the temporal kind) the differences between two adjacent frames will decrease. Thus the motion vectors between those two frames will also decrease.
But since motion vectors change once the frame gets quantized, to measure motion vectors after quantization would be useless. If it were possible to report the size of the average (or total?) motion vectors for each frame (before quantization) in the dbglogfile we would have a measure for the amount of blurring that took place (when compared to an un-blurred encoding).

Nope, the motion vectors should stay pretty much the same, the blurring will just reduce the number of bits used by the quantized coefficients (since there are fewer details to store). The motion vectors don't change once the frame's quantized though, they're set.

The codec already totals the motion vectors per frame during motion estimation (in order to calculate fcode), it'd just be a matter or reporting it. I've been wanting to stick something in the luma-noise field for a while now..

about 'total motion', it should be quite easy to keep and eventually display global statistics as:
1. sum of absolute MV lengths for the whole film.

Already being calculated, just needs to be reported.

2. number of MV for blocks of inter/interV4 modes.

Not sure I get you.. total count of motion vectors written per frame divided by macroblocks? An inter will always have 1, inter4v 4...

3. average MV length ~(1./2.) (or more correctly if including interv4 as 4 for 2.)

That'd be quick to calculate if I understand you right - sum of vectors divided by number of vectors...

3. INTER/TOTAL blocks ratio: total_INTER_BLOCKS/total_BLOCKS

These are already reported via the ublk/kblk/mblk variables.

I'd like to code these things, but I'm leaving the country in 2 weeks and will not be able to work on (or look at) XviD for 3 or 4 months. So I'm trying to focus on things like the glaring bugs in vfw, sse2 code, field-based motion estimation (might help trbarry with his deinterlacer), and maybe image-based preinterpolation to speed up encoding.

I only have so much time :)

-h

[canadian_fbi]

Originally posted by -h
[i]I'd like to code these things, but I'm leaving the country in 2 weeks and will not be able to work on (or look at) XviD for 3 or 4 months. So I'm trying to focus on things like the glaring bugs in vfw, sse2 code, field-based motion estimation (might help trbarry with his deinterlacer), and maybe image-based preinterpolation to speed up encoding.

I only have so much time :)
ever seen the movie "multiplicity", with michael keaton? the basic premise is that he needs more time to do stuff, so he clones himself, and hilarity ensues. at least i think that's it - i saw the end of it on hbo today because i had nothing better to do. i think we need some -h clones... maybe name them -i, -j, etc.

where are you headed, if you don't mind me asking?

[-h]

I'm heading over to America to live with cousins for a while, though we'll probably travel all summer and end up in a beach house in South Carolina.

Hard work.. but I think I can do it.

-h

[soujir0u]

Hi, I read the readme file and it says to put the dbg.txt file into the same folder as the dbg.exe, but what is this dbg.txt file? Can anyone give a brief tutorial on how to start using debugview? Thanks.

[soujir0u]

Just checked the Q&A again, think I got an idea of how to use it already... First I need the DebugView program that captures the debug output, then I can analyse the output file with DebugView Analyzer?

[MoonWalker]

@soujir0u

1)Run debugview
2)make your 1-pass,2-pass (or only the 2-pass)
3)save the results from debugview to a file called dbg.txt
4)run dbg.exe and a file called analyze.txt will be created.

make sure you've got the latest version of the debugview..you can get it from my signature...

MoonWalker

[soujir0u]

Ok, thanks!

[ookzDVD]

@MoonWalker,

I'm using your dbg v0.4,
I got the error message : "Floating point error domain" :(

I'm using Win98SE.

Thanks.

[MoonWalker]

Originally posted by ookzDVD
@MoonWalker,

I'm using your dbg v0.4,
I got the error message : "Floating point error domain" :(

I'm using Win98SE.

Thanks.

Hmm...weird..I can't find any bugs in the code..anyway..i will release v0.5 soon (later today) with some more features...

MoonWalker

[Mefisto]

@MoonWalker,

I'm using your dbg v0.51,
I got the error message : "!!Error!! The file was not openned!! Floating point error: domain"

I'm using WinXP & AMD Athon 1 Ghz.

Thanks.

[FFiXx]

I got the same msg in win2k as well :confused:

[FFiXx]

Im using DebugView which came with the nandub ver I have and it works fine. :D

[MoonWalker]

Originally posted by FFiXx
I got the same msg in win2k as well :confused:

Did you save your file as dbg.txt?? Be careful if you choose Save As you have to enter the filename with quotes like this "dbg.txt"..

Version tested 4.13

MoonWalker

[FFiXx]

Nope, nothing happens when i double click the file. Ive alredy created dbg.txt and analyze.txt. If i use cmd dbg then this comes up:!!...Please Wait...I am Working...!!Floating point error: Domain.

[MoonWalker]

You said you have Win2000..I will check my prog to friend of mine that has WIn2000...You HD are NTFS or FAT32??


MoonWalker

[FFiXx]

:confused: There r ppls who uses FAT32 on win2k? heh, well its still wherd if the proggie isnt ntfs compatible. Besides Im allergic to FAT32 :scared:

[MoonWalker]

Originally posted by FFiXx
:confused: There r ppls who uses FAT32 on win2k? heh, well its still wherd if the proggie isnt ntfs compatible. Besides Im allergic to FAT32 :scared:

No, it's NTFS compatible..I have WinXP and have tested it with NFTS HD..

I look into it!

MoonWalker

[Bulletproof]

Hey I got some suggestions for your program:

1. Add up all the final scaled sizes for each frame, then add up total first pass sizes. Find out difference between both to find out compressability perecentage (You might have already done this, it's late at night and I'm tired so I may have forget, sorry)

2. Track how many "prevents" the codec makes from high quantizer changes, and list them and make a percentage of that as well. Debugview logs this.

3. Find out the percentage of H.263 and MPEG quantizer used.

4. This might be a bit advanced right now, but it would be nice if you could make it so that the program builds a graph of deviation. Getting the info would be from 1st pass vs final scaled size, and putting both lines into a single graph to visualize deviation. Maybe even add a third line into the graph to show quantizer distribution. Having all 3 shown at once would help lots with research.

5. I guess by reading the overflow (I still don't quite understand what that means), you can also tell the person whether the movie can have better or needs more compression. But I could be confused what the overflow does, nobody has answered the thread where I asked about it and there are a couple of other debugview outputs I don't understand.

6. I think Debugview even logs the time for each frame compression. I'm sure these are useful in determining compression speed performance. You write how fast the average frame encoded or how long it took to encode the movie.

I'm sure this program can be even more useful if you keep working on it ;)

[MoonWalker]

Sugestion 1 : Done already(except the compressibility result witch it's easy ;) )

Sugestion 2: I haven't experiance any "prevention". Anyone has a small log(about 20 lines) with this to send me?

Sugestion 3 : Easy. :)

Sugestion 4: It's a bit difficult as far as I know programming.But I cound make a txt file that can be open with excell and the do the graph(Or I could go to code all the prog to Visual C ;) )

Sugestion 5 : Well let's say some things. The 1-pass size is the size a frame had after aplying a quantizer of 2x at tha first pass. Scaled size is the size that the XviD codec calculates the frame need to be compressed with the setting.And finaly Actual size is the true size the frame got after the compression.So scaled size is the size calculated BEFORE the compression and actual the size AFTER the compreesion.So overflow=scaled-actual..In other words it's the bits that were got spared from the compression(if scaled>actual,possitive overflow) or the bits that were needed(if scaled<actual,negative overflow)..Also is means how well the codec has decided the quantizers to reach the projected size..Usually(from my experience) this is too small..And about deviation : It's the same as overflow(I think ;) )

Sugestion 6 : I will try to figure out how I can calculate the time and the frames/sec..

MoonWalker