small, n00bish xvidvfw patch (no longer available)

[virus]

Patch no longer available. Sorry.

Hi there!

For those of you who like stats, here's a small patch for XviD 1.0.x that enables full logging for all passes. Here's a screenshot of the lower half of the reworked Debug dialog (note that the disabled "Number of threads" box has been removed):


EDIT: screenshot removed


Outputs to file: infos on every frame (number/quant/type/length/psnr), plus the following overall infos -> status window stats (not the graph of course :)), quantizer distribution (overall/I/P/B), encoding time & speed in fps, compression ratio (bits/pixel), frame resolution, framerate, xvidcore & build versions, average PSNR (excluding the frames with infinite PSNR).

Choosing Full log will output detailed per-frame stats + the overall stats, while Summary only will save only the overall stats (much shorter).
Everything is simply appended at the end of the logfile so you can run several passes in a row without worrying (warning: the log can be several Mbytes for a pass on a 2h movie if you use "full log" -> 1 text line for each frame).

There's only the xvidvfw.dll library, so you'll need a full build (like Koepi's) installed. Simply replace the library... hitting Load Defaults should not be required (though some new stuff needs to be written in the registry).

Currently this build needs testing, so use it at your own risk. (I can easily produce way much more bugs than our "inventor" sysKin :D). Nevertheless, it (kinda) works for me.

Note: the code is actually based on XviD 1.0.2, but will work with every 1.0.x version.

If you want to take a look, grab it here. [link removed Nov 12, 2004]
Feedback would be appreciated.

enjoy :)
virus

EDIT:
23/5: revised for version 2
25/5: revised for version 3
12/6: patched the XviD 1.0.1 VfW frontend
18/9: code now in sync with XviD 1.0.2
12/11: patch no longer available, sorry. a big thx to the guys who helped me with that stuff.

[Asrial]

Haven't tested yet, but this is something I've wanted for a damn long time (my requests seemed to get overlooked, oh well).

Thanks!

[bond]

nice idea :)

[Zep]

Originally posted by Asrial
Haven't tested yet, but this is something I've wanted for a damn long time (my requests seemed to get overlooked, oh well).

Thanks!

you and me both. I posted a half dozen times i wanted this.
heck i did just again two days ago :D

[Soulhunter]

Yeah, and me as well... ;)

1000k thanks to virus !!!


Bye

[virus]

thanks for your support, guys :)

I can't believe nobody answered "hey dude, it's b0rked" :D
I will probably release an updated version very soon, which should include PSNR computation directly taken from xvidcore, so one can avoid writing a script just for retrieving the average PSNR.
New version codename: Serial tester's heaven ;)

more later
virus

[Soulhunter]

Originally posted by virus
I can't believe nobody answered "hey dude, it's b0rked"
Haven't found the time to test, coz Im still running a encode... :(

But when the next version is out, I will make heavy use of it !!!


Bye

[Asrial]

As far as I can tell, it's responding perfectly.

One suggestion I have is to have it do html output if possible. When the file gets appended to over and over (because I don't change the filename), it can make it difficult to compare information between sources (whereas with html you can use links to take you directly to the info you want).

[OUTPinged_]

does it display overflow values for each frame?

[virus]

@Asrial
Thanks for testing it. As for the "crowded" logfiles (I noticed that too), you can use the "Find" function in your text editor to look for the start of each report (searching "xvidcore"), or for a particular frame in many passes (entering "frame: 123" for example), and so on.

Adding HTML output is probably not so useful, and it brings some problems: formatting hyperlinks requires to _edit_ the whole log instead of simply appending the infos for a given pass --> makes things complicated. I'd like to keep the patch as simple and straightforward as possible because I wanna be able to patch easily the new versions, too (the upcoming stable 1.0.x releases for sure, let's see what happens for the 1.1.0 alphas/betas).

@OUTPinged
Overflow values aren't displayed. They belong to the ratecontrol system embedded in xvidcore. I don't even know if you can compute them using standard API calls, sorry.

BTW: new version due in a few hours

has someone measured the slowdown when enabling the logging? It should be really minimal, but I haven't managed to test it properly yet

[Zep]

Originally posted by virus


BTW: new version due in a few hours



i think you should try and get this added to CVS and have the Xvid
devs roll it into the next 1.0.X release

IMHO this type of option should have been added to Xvid option window
LONG AGO.




UPDATE: it works great. As for speed test i will do some after
you release the next version :D

[virus]

Version 2 ready. Hopefully, this is the "final" revision. You can download it from the link on my first post (which has been updated to reflect the changes).
New features:

- PSNR for each frame
- average PSNR for the whole video. Frames with infinite PSNR aren't counted so this stuff may be useful even if your source contains them (example: black frames).
- outputs quantizer distribution (with percentages) for I/P/B frames => delivers the same info as the status window graph.
- do not output detailed stats about B-frames if they aren't used.
- "may contain nuts"(TM) :D

Big thanks to GomGom who solved my problem with the SSE computation.

enjoy :)
virus

[virus]

Originally posted by Zep
i think you should try and get this added to CVS and have the Xvid devs roll it into the next 1.0.X release
I'm sorry, but despite being a "spare-time-programmer" since 1994, I know almost nothing about CVS, diff patches, commits and stuff like that. I compiled XviD for the 1st time 4 days ago.
Also, the GUI changes are probably incompatible with the 1.1 tree (which sports some new stuff in the VfW GUI). It's very simple & ugly code, mostly cut'n'pasted from the existing sources. Not worth a commit, IMHO.

But be sure that I will patch 1.0.x versions, too. Actually, if the new releases are going to bear an unchanged VfW frontend, that's not even needed - you can simply use the patch for 1.0.0. Otherwise, I will repatch and make that available. Maybe 3 or 4 people will find an use for that ;)

virus

[Asrial]

I'm not noticing any slowdown. I can't confirm it because I cleaned out the log, but pretty sure I'm still hitting ~15-18 minutes for the encode I'm working on.

[Soulhunter]

Fine, I gonna encode a movie with it this night... :)


Bye

[Soulhunter]

Ok, done my first encode with it...

And it works "nearly" like expected !!!

But, wouldnt it be possible to save the "real info" in a seperate file ???

Like...


C:/Blah.QuickInfo

Average bitrate: 5398.5 kbit/s

Compression ratio: 0.366 bit/pixel

Encoding speed: 5.5 fps

Average PSNR: 49.39 dB (Y 48.79 U 48.41 V 50.96)


Frametype distribution:

I-frames: 5534 (3.0%)

P-frames: 101333 (54.5%)

B-frames: 79076 (42.5%)


Overall quantizer distribution:

Quant 2: 106867 (57.5%)

Quant 3: 79076 (42.5%)


I-frames quantizer distribution:

Quant 2: 5534 (100.0%)


P-frames quantizer distribution:

Quant 2: 101333 (100.0%)


B-frames quantizer distribution:

Quant 3: 79076 (100.0%)


Frametype quantizers:

I-frames: min 2 max 2 avg 2.00

P-frames: min 2 max 2 avg 2.00

B-frames: min 3 max 3 avg 3.00

Overall: min 2 max 3 avg 2.43


Frametype size (bytes):

I-frames: min 6466 max 244138 avg 75391 tot 417216583 (8.3%)

P-frames: min 301 max 201023 avg 37970 tot 3847651580 (76.7%)

B-frames: min 8 max 116605 avg 9537 tot 754200109 (15.0%)

Overall: min 8 max 244138 avg 26992 tot 724100976



Bye

[virus]

Originally posted by Soulhunter
Frametype size (bytes):

I-frames: min 6466 max 244138 avg 75391 tot 417216583 (8.3%)
P-frames: min 301 max 201023 avg 37970 tot 3847651580 (76.7%)
B-frames: min 8 max 116605 avg 9537 tot 754200109 (15.0%)
Overall: min 8 max 244138 avg 26992 tot 724100976

Looks like you've spotted a bug (overflow! the total should have been more than 5 thousand millions bytes; only the last 32 bits are printed). Here we enter the dangerous realm of "how to print a 64-bit number using a syntax which all compilers understand". The fix would be simple for gcc, but I don't know if it's supported by other compilers as well.

As for the "separate file", it all comes down to the fact that, for large encodes, all the per-frame info may be redundant (the log can be huge - no one is going to read/use 300000 lines). I was thinking about adding a new option to disable detailed per-frame stats and print only the totals, but there's a dumb GUI design problem then: there's no room in the dialog for a new option... :)
Suggestions?

[zulu]

Suggestions?
resize the window/tab :D

[Soulhunter]

Originally posted by virus
Suggestions?
Make two separate files by default then... ;)

Blah.FullStats = Detailed info

Blah.QuickInfo = Short version


Or simply make the short the default one !!!

Think nobody needs the long version... :rolleyes:


Bye

[zulu]

Blah.QuickInfo = Short version

sounds reasonable..or..

Blah.Summary = Short version

[Asrial]

Originally posted by Soulhunter

Think nobody needs the long version...I would, in case I find a problem frame and I want to see if it's 'frame type' related.

Though I have no idea how to force that frame to be anything other than what it's encoded as :p

[virus]

The "long version" will stay (I want it too).

The best solution I've thought so far is to remove the disabled "number of threads" widget from the GUI and make room for a couple of mutually exclusive radio buttons saying "full log" and "summary only" or something like that. Creating a new file silently is a quick-and-dirty hack that I don't like that much. Let's see what I can do (I'm always open to suggestions :)). I'm a bit busy right now.

cheers :)
virus

[virus]

Version 3 out. Download from the link on the first post of this thread (which has been updated, too - see the new scrshot).
New features:

- now there are two options: "Full log" will save per-frame stats + the overall stats, whilst "Summary only" (guess what?) will only save the overall stats... :)
- removed the disabled "number of threads" box from the GUI
- should print correctly total framesizes > 4 GB
- in the Save File As dialog, the default file extension (*.log) is now appended if there isn't one
- like for B-frames, don't save detailed stats for P-frames if they aren't used (yeah, I forgot that motion search precision 0 => clip with only I-frames)
- as always, "may contain nuts"(TM) :D

enjoy :)
virus

[Soulhunter]

Originally posted by virus
Creating a new file silently is a quick-and-dirty hack that I don't like that much. Let's see what I can do (I'm always open to suggestions :)).
Hehe, quick-and-dirty is exactly the way I like it... :D


Bye

[maciek_m]

Is there a chance for version 1.0.1 of this great gimmick?

[celtic_druid]

Well you would already get everything except for:
- (VfW) video was not playable via ICM (mplay32.exe, files embedded in Office
applications etc)
- (VfW) calculator did not open many useful audio/subs extensions by default
- (VfW) calculator calculated audio size wrong when given bitrate and length (2.4% error)

[virus]

Originally posted by maciek_m
Is there a chance for version 1.0.1 of this great gimmick? Of course. Hope to find time this week... but meanwhile, you can still use this hacked VfW frontend with XviD 1.0.1. The only features missing are the ones marked with (VfW) in the changelog (really minimal fixes imho).

The trellis fix and the other "core" changes belong to xvidcore.dll so you won't "overwrite" them if you apply this patch over xvidvfw.dll ;)

EDIT: ooops, celtic_druid was faster :D

[maciek_m]

Which is kind of surpsising, and proves that Good can be faster than Evil :)

[Asrial]

<cracks his whip and readies McAfee Stinger> :P

[virus]

Patched the 1.0.1 code. As always, check the first post in this thread for the link.

On a side note, I see that the xvidcore version is still "1.0.0" and the build is still "xvid-1.0.0"... aren't they supposed to be both 1.0.1 now? (the same applies to Koepi's build)

[celtic_druid]

http://img19.imageshack.us/img19/272/XviD.gif

[virus]

http://img19.imageshack.us/img19/9854/xvid101_koepi.png

[Soulhunter]

http://img13.imageshack.us/img13/3931/snap000093.png

This is Jawors 1.0.1 build including virus' hack !!!

Cool, three different Grey tones... :D


Bye

[celtic_druid]

$Id: xvid.h,v 1.33.2.7 2004/06/05 23:08:01 edgomez Exp $
#define XVID_VERSION XVID_MAKE_VERSION(1,0,1)


From xvid.h, unchanged. Guess everyone else somehow ended up with a different xvid.h?

[virus]

Originally posted by celtic_druid
Guess everyone else somehow ended up with a different xvid.h? No, the macro is the same in the official sources. This is a compilation problem. I've just recompiled xvidcore myself and the versions are now correct.

BTW I'm not sure this applies to ICL too, but hacking the sources with gcc I had the following problem: when changing the header foo.h but not the corresponding foo.c file, the foo.obj object code didn't get updated (this may result in bad crashes if you add/modify members in a struct). Only doing a "make clean" before "make" (thus forcing the update) solved the issue. That is, there are missing dependences in the makefile for gcc. Maybe the same happened for Koepi with ICL.

[Dams]

virus : great work!!

Very usefull tool :cool:

[Soulhunter]

Small request...

Is there a possibility to add SSIM into this hack as well ???


Bye

[virus]

Originally posted by Soulhunter
Small request...not so small man :D

Actually xvidcore doesn't compute SSIM, so the xvidvfw frontend simply cannot request SSIM and print it out. Keep in mind that the VfW interface relies on the functions provided by the core library.

And personally I don't think the devs will ever add SSIM to xvidcore, since there's already PSNR which is "good enough" to evaluate objectively the improvements... of course, the devs crosscheck everything using their eyes, too. ;) SSIM computation is also quite complex (and slow) if I recall correctly the basics of this algorithm. Guess you had to live with the AVISynth plugin :D

virus

[Soulhunter]

Didn't even newed that XviD computes PSNR... :D

No matter, your hack still rocks !!!


THX n' Bye

[Dams]

Hi,
I have too a small request :)
Is it possible to add the filename that is processed by Xvid codec ??
(like "myprojet.avs") :

xvidcore.dll version 1.0.0 build: xvid-1.0.0
filename : myprojet.avs
start date/time: sept. 18, 2004 - 12:23:00
end date/time: sept. 18, 2004 - 12:27:02
frames: 8001
length: 29582595 bytes (28889.3 KB, 28.21 MB)
average bitrate: 739.5 kbit/s
frame resolution: 640x272
framerate: 25.000 fps

[virus]

@Dams

Unfortunately that's impossible AFAIK.
XviD only inputs rough frames through the VfW interface, it knows nothing about the source itself. Example: whether it's Vdub(Mod) or AVS2AVI, or whether it's AVS, AVI or MPG and so on. Only the calling application knows the input filename. Think about VDub: you select the input filename in a dialog which has nothing to do with the codec itself... ;)

EDIT
btw: new version (based on XviD 1.0.2) ready. Check the first post, as always. (damn, a new release for a one-liner fix... :rolleyes: )

[Dams]

Cool to see you're working always on it!

I'll put the filename with my little hands in xvid log, so ... :D