Common Benchmark

[DDogg]

The script below creates this logfile:
----
Script for t1 started 08/05/2003 at 11:05:08.828; finished 08/05/2003 at 11:12:11.203
After 10001 frames for timer "t1", average time = 0.75 ms, std dev. 0.00 ms, average speed = 1333.07 fps
----

This works for my personal use using "The Matrix" (NTSC) as source, but is sloppy and not transportable between NTSC and PAL. The average speed = 1333.07 fps (I wish!), while not fps accurate is very consistent and dead accurate as an empirical number of measurement and really shows the speed differences between IDCT, FSB speed, CPU clock, and resizers. I think the kronos.dll was not designed for this specific use, but it works great. Most probably I am just not using it correctly? Suggestions appreciated.

Anyway, to the issue of NTSC, PAL and source, I was hoping one of you might write a avisynth only script that generated / approximated a source somewhat similar to a real movie so that it could be portable between NTSC and PAL and not require a true video source.

I thought maybe just one of the examples that come with avisynth, but I don't think just colorbars and timecodes will really approximate a real encode. If somebody could address this maybe it could then be included in future releases of avisynth with a standard benchmark script.

Kronos.dll
http://kurosu.inforezo.org/avs/Kronos/index.html

#start
#Using "Kronos.dll"
mpeg2source("D:\matrix\DVD2AVI_PROJECT_FILE.d2v",idct=6)
StartTimer("t1")
trim(146700,156699)
crop(8,0,704,480)
#BicubicResize(480,368,0,0.6)
simpleresize(480,368)
#bilinearresize(480,368)
#LanczosResize(480,368)
AddBorders(0,56,0,56)
StopTimer("t1",mb=true,log="d:\Logfile.txt")
#end

[Kurosu]

Your result looks a lot like you're using a build with threads enabled, which give bad results in my opinion. I've updated Kronos (check website in sig) to 1.6 to get this is enabled (and give people a chance to notice the update). Concerning accuracy of results, it's better to use the measurement in comparisons. For instance, my system being under heavy load (but settign priority to idle for the cause of this load and VDub's priority to normal), I got those figures (with same script, except source which was a 640x480 avi):
- bicubic = 1.43ms
- bilinear = 0.98ms
- lanczos = 1.68ms
I couldn't find back simpleresize on my drive and was too lazy to search it further.

Under other circumstances (better english? = with other settings), ranking would be the same, but results a bit different (by 0.05ms I guess).

[DDogg]

Yes, I was using the earlier version. Thanks for replying and pointing that out.

Kurosu, actually the FPS is the most useful thing for me. While it may not be true FPS, it is a number that goes up and down depending on the speed of the encode. It seems a very accurate indicator and shows up small changes to FSB speed, ram timings, as well as the obvious script and encode related stuff like resizers and encoders.

Even though it works fine for me, I am curious why it would show as such a large number? As I said, I don't think I am using your tool as you intended so perhaps you could correct my usage? [Edit: for my use, I see it as a "Relative Speed Indicator" rather than FPS.]

[Kurosu]

[I may mix "time" and "measure of time" in the following, simpler, and that's how it works in French, so...]

Well, FPS is just the inverse of the average time you see: if the comp is able to get a frame through the filter chain in, say, 20ms, then the FPS value is just 1/(20*0.001) = 50fps. And, as you figured (I'm writing this for other readers in fact), it is by no means a true indicator of the real speed, as the source's processing time and other stuff (other parts of the script for instance) also consume processing time.

This time should be seen as the time needed by a comp to process a frame if it was the only thing the comp had to do.

As explained by Sh0dan, here is how it work (maybe not relevant to your questions, but still), very grossly:
- the avisynth script is processed backwards, ie, source doesn't push frames into filters, but software running the script requests frames from the last filter of the avisynth filter. Therefore, the request goes first through Stoptimer.
- stoptimer starts measuring how much time is needed untill a corresponding frame is requested at starttimer level. This time is the first part, that you can see as initialization time for each frame.
-then, starttime measures each period of time for each frame requested that are necessary to provide stoptimer a frame. This can be seen as the sum of processing times for all frames requested by the filter chain (for instance, decimate(cycle=5) will request 5 frames every 5 frames, and 0 the other times)
-stoptimer sums the previous sum and the initialization time. This can be seen as the time needed by the filter chain to produce a frame, and is the value displayed.

The measures you wrote are totally screwed and unrelevant, so I guess you don't ask a comment on them, but on my measures. Resize filters can really slow down a processing, and Sh0dan has done a _very_ good job at optimizing them. I still find the measures impressive, but they aren't that surprising. They are just put out of context: the actual fps display I got in Vdub was around 120fps (bicubic was 700fps). This Vdub fps accounts for the whole time elapsed between 2 frames are given to Vdub. It's directly affected by process priority and other stuff, while the Kronos measure should only change with hardware settings (those you point out, but also OS caching, ...).

Therefore, your usage is perfectly correct, as it really is a "Relative Speed Indicator". You can either choose the fps value (which proves to be more meaningfull for you - "what is the top speed of your car?") or the time measure (which I prefer - "How much time do you need to do 1km at top speed?").

[DDogg]

Kurosu, thanks for the explanation.

Q: Would you consider adding elapsed time to the logfile?

----
Script for t1 started 08/05/2003 at 11:05:08.828; finished 08/05/2003 at 11:12:11.203 Elapsed time: xx:xx:xx:xxx
After 10001 frames for timer "t1", average time = 0.75 ms, std dev. 0.00 ms, average speed = 1333.07 fps
----

[Kurosu]

(I hope you just copied the only line, and you're not having the same result with the new dll - but you may have some insanely fast system after all :)

In fact, you'd like I print in the log the difference of end of job minus start of job (ie 11:12:11.203-11:05:08.828 -> ~7 minutes, which also means the average time was totally unrealistic :)

[DDogg]

Well, I have a xp1700 jacked up to 2.2 ghz (183 clock x 12 multiplier) so yes it is pretty fast. Here are two runs I did. One with converttoyuy2() and the other using ffvfw to handle colorspace with CCE 2.67.00.11.

Note: IMO, we REALLY need a common benchmark for both PAL and NTSC users so we can accurately communicate between ourselves. This has been a real problem. I hope you and others might consider working with me toward that goal for the group.

----
using ffvfw
Script for t1 started 08/05/2003 at 23:23:05.15; finished 08/05/2003 at 23:26:34.453
After 10000 frames for timer "t1", average time = 1.93 ms, std dev. 0.45 ms, average speed = 519.03 fps

----
using convertoyuy2()
Script for t1 started 08/05/2003 at 23:27:50.718; finished 08/05/2003 at 23:34:19.187
After 10000 frames for timer "t1", average time = 2.01 ms, std dev. 0.14 ms, average speed = 498.72 fps

----
Using this script:

#loadplugin ("Kronos.dll")
mpeg2source("D:\matrix2\DVD2AVI_PROJECT_FILE.d2v",idct=6)
StartTimer("t1")
trim(146700,156699)
crop(8,0,704,480)
LanczosResize(480,368)
AddBorders(0,56,0,56)
converttoyuy2()# 1st run without and second with
StopTimer("t1",mb=true,log="d:\Logfileffvfw.txt")

[sh0dan]

I don't think "stop-start" times can be used for benchmarking, as there might pass some time from the script is opened until the encoding begins. CCE will also most likely request a frame when the script is opened, so there is no way of actually benchmarking inside AviSynth.

For an _external_ benchmark tool I'd suggest avs2avi, and using XviD "Null filter". I've used it a lot for profiling, before Kronos was available.

[Kurosu]

Originally posted by sh0dan
I don't think "stop-start" times can be used for benchmarking, as there might pass some time from the script is opened until the encoding begins. CCE will also most likely request a frame when the script is opened, so there is no way of actually benchmarking inside AviSynth.
[EDIT]
Start/Stop times as written in the log file (I first thought you were talkign of start/stoptimer, which didn't make sense :) ), and which maybe lead to the "elapsed time" request. Yes, that's a very bad benchmark method, as measuring how much time vdub has been processing the video.

[DDogg]

If I were a developer I would sure like to get feedback from users that was replicatable and dead accurate instead of "yeah, it kinda seemed faster to me, but Joe Smoe said on the xxx board that it was not.".

For that reason alone I think having a common benchmark included in avisynth of SOME kind that would be transparent to NTSC/PAL is something I hope you guys will consider. For your own sake, as well as it would be useful to many.

AS for Shodan's comments, I just can't understand what you mean when we are talking about a 10000 or 100000 frame test on an entire process. I am not speaking of the benchmarking of a particular filter in a chain.

[Kurosu]

Well, what should be this benchmark? Applying to some fixed sequence a resizer, do some math-intensive (more than memory -bandwidth intensive) stuff, and some other stuff? Well, that's good to benchmark synthetically (sp?) a system against another one. Kronos can help provide such benchmark. And that would bring so much benefit to some hardware sites that hardly have a clue about what can be involved in an encoding besides the codec (no bashing intended).

To continue on this almost off-topic disgression, avisynth "power users" are maybe 10% of the people using their comp to encode video, and 60% is using XMPEG and the like, so simpler tests as done now are maybe more relevant. But Doom9 posted some weeks ago a hardware review where some actual processing showed quite different comparisons between AMD's Athlon and Intel's P4 (ie Athlon wasn't biting the dust that much, thanks to more math-intensive processing).

But I don't see what kind of help I could give in that case, as the results made out of Kronos are of no relevance to Kronos - that's also why I rejected Sh0dan need to add some measures. There are simply too many possible uses, but all are external to Kronos.

And people would still have different results, as nothing can be replicatable if you don't have the same settings (memory, CPU, ...). Take for instance VagueDenoiser: the ICL7 compile said to be a lot faster on P4s was almost performing slower than a VC6 compile on my AMD Barton CPU.

[DDogg]

First, on a lighter note, you use of "disgression" is excellent. A combination of discussion and digression. It should be in the dictionary, or is it French? :-p

Ok, as background, this thread drove be batty and is what got me thinking (Shodan winces) about this stuff again.
http://forum.doom9.org/showthread.php?s=&threadid=55231

My thinking was based upon two things:

1> A avisynth script of internal commands only which somewhat mimicked a video source which could be used as a "Standard" for both NTSC and PAL users. It would be the equivalent of maybe 200,000 frames in length. It would load in any encoder/tool that will support avs scripts. Thus CCE users, DIVX or XVID or whatever users could all use the same standard "source".

2> A tool to measure the time to completion of a given set of frames. In CCE that might be 10,000, or 100,000, or through to completion of the fake source. Running a smaller frame set of say 1000 frames would allow easy comparisons of encoder, resizer, and hardware tweaks.

I see no reason why something like this could then not be used to compare machines and settings as well as give important feedback to developers about speed of processes.

In theory, I should be able to send you my working script like the one above, you test it on your Barton, and we should be able to compare our "logs". Double entendre? Of course :-)

[bilu]

Idea, not tested yet:

Using CHR.DLL

http://www.avisynth.org/~warpenterprises/files/chr_25_dll_20030404.zip


Codes for Output formatting
%a Abbreviated weekday name
%A Full weekday name
%b Abbreviated month name
%B Full month name
%c Date and time representation appropriate for locale
%d Day of month as decimal number (01 – 31)
%H Hour in 24-hour format (00 – 23)
%I Hour in 12-hour format (01 – 12)
%j Day of year as decimal number (001 – 366)
%m Month as decimal number (01 – 12)
%M Minute as decimal number (00 – 59)
%p Current locale‚s A.M./P.M. indicator for 12-hour clock
%S Second as decimal number (00 – 59)
%U Week of year as decimal number, with Sunday as first day of week (00 – 53)
%w Weekday as decimal number (0 – 6; Sunday is 0)
%W Week of year as decimal number, with Monday as first day of week (00 – 53)
%x Date representation for current locale
%X Time representation for current locale
%y Year without century, as decimal number (00 – 99)
%YYear with century, as decimal number
%z, %Z Time-zone name or abbreviation; no characters if time zone is unknown
%% Percent sign

As in the printf function, the # flag may prefix any formatting code. In that case, the meaning of the format code is changed as follows.

Format Code Meaning
%#a, %#A, %#b, %#B, %#p, %#X, %#z, %#Z, %#% # flag is ignored.
%#c Long date and time representation, appropriate for current locale. For example: “Tuesday, March 14, 1995, 12:41:29„.
%#x Long date representation, appropriate to current locale. For example: “Tuesday, March 14, 1995„.
%#d, %#H, %#I, %#j, %#m, %#M, %#S, %#U, %#w, %#W, %#y, %#Y Remove leading zeros (if any).


Concept:

1.AVS
=====
Loadplugin(chr.dll)
s_hoursecs=3600 * Value(Time(%H))
s_minsecs=60 * Value(Time(%M))
s_secs=Value(Time(%S))
s_totalsecs= s_hoursecs + s_minsecs + s_secs
mpeg2source(movie.d2v)
s_tsecs="\ns_tsecs="+string(s_totalsecs)
framecnt="\nframecnt="+string(last.framecount)
Call("nicecho.exe s_time.txt "+s_tsecs+framecnt, "-2")

Filters()

2.AVS
=====
Loadplugin(chr.dll)
e_hoursecs=3600 * Value(Time(%H))
e_minsecs=60 * Value(Time(%M))
e_secs=Value(Time(%S))
e_totalsecs= e_hoursecs + e_minsecs + e_secs
e_tsecs="\ne_tsecs="+string(e_totalsecs)
Call(BlankClip,"nicecho.exe e_time.txt "+e_tsecs, "-2")

3.AVS
=====

totalsecs= e_tsecs - s_tsecs
performance= float(framecnt / totalsecs)
ts="\nElapsed time: "+string(totalsecs)+" secs"
perf="\nAvg. framerate: "+string(performance)+" fps"
Call(BlankClip,"nicecho.exe perf.txt "+ts+perf, "-2")

DO.BAT
======
avs2avi 1.avs null.avi -c xvid
avs2avi 2.avs null.avi -c xvid
avs2avi 3.avs null.avi -c xvid


Again said, not tested or polished, but I think the concept works :)


Bilu

[Kurosu]

Originally posted by DDogg
First, on a lighter note, you use of "disgression" is excellent. A combination of discussion and digression. It should be in the dictionary, or is it French? :-p
The word is also "digression" in french, so just consider it as a typo.

Ok, as background, this thread drove be batty and is what got me thinking (Shodan winces) about this stuff again.
http://forum.doom9.org/showthread.php?s=&threadid=55231
Ok, I'll comment on it later in this post.


My thinking was based upon two things:

1> A avisynth script of internal commands only which somewhat mimicked a video source which could be used as a "Standard" for both NTSC and PAL users.
In my opinion, it's hardly possible:
- the benchmark you are expecting both need processing and encoding, and processing a dark slow motion video can have quite different speed differences with another one (MPEG's motion prediction, temporal filters...)
- the best you can make with a video entirely produced by avisynth is a synthetic benchmark; you need real-life videos of all kind.
For that reason, such synthetic videos as produced by ColorBars() (whatever the name of the actual function is) are not suited for benchmarking.

It would be the equivalent of maybe 200,000 frames in length. It would load in any encoder/tool that will support avs scripts. Thus CCE users, DIVX or XVID or whatever users could all use the same standard "source".
Well, in the image processing field, there is what I think is universally called corpus: various videos very typical of most situations that an encoder (from a codec) can find. I doubt it is meaningful to talk about comparisons based on foreman, weather and mobile.cif on doom9 forums; but they are the standard, and you can easily find them, and they have been used for years (not yet decades :) ).
On the other hand, not everybody can reproduce results if they don't have, say, the sources used by Doom9 in his codec comparison, so an intermediary solution should be found prior to discussing whether avisynth should offer extended (meaning: more than Kronos) benchmarking abilities.

2> A tool to measure the time to completion of a given set of frames. In CCE that might be 10,000, or 100,000, or through to completion of the fake source. Running a smaller frame set of say 1000 frames would allow easy comparisons of encoder, resizer, and hardware tweaks.
True, but then comes into mind the accuracy problem: CCE doesn't provide any way to accurately measure what's happening. Benchmarks often require you to run only one thing, the benchmark, at a time. Yet in the case of encoding, and although video processing and video coding are parts of the same thing (ripping a DVD for instance), those 2 parts should be put apart.

I see no reason why something like this could then not be used to compare machines and settings as well as give important feedback to developers about speed of processes.
Well, avisynth can provide something that could help benchmark filters: that's what Kronos offers. But it can't measure how much time is spent in the decoder, and benchmarking should be seen at a higher level than the one Kronos operates.
I do agree that the thread you refer to is really meaningless: I'm sure I could do encodes in CCE at speed of 0.2 (I've never used CCE) on Barton 2800, and someone running a P3@1GHz would maybe achieve 1.0

In theory, I should be able to send you my working script like the one above, you test it on your Barton, and we should be able to compare our "logs". Double entendre? Of course :-) [/B]
Yes, provided we use exact same plugin and software versions. This is more for system benchmarking, while Kronos was intended to accurately measure filter chains performance. 5s difference as reported by Vdub in a 3 min encode is meaningless to me, it could just be the system performing differed writing, backup and what not.

For that reason, I don't think bilu's methodology is usefull either. It might work, but it would be a bother to check everything performed exactly as expected (and is reproducable).

[DDogg]

Kurosu, thanks for your thoughtful and detailed reply. I think some of the "disgression" is caused by language barriers.
Yes, provided we use exact same plug-in and software versions. This is more for system benchmarking, while Kronos was intended to accurately measure filter chains performance.
This is what I meant when I said I was NOT using your plug-in as you intended when you wrote it. However, the truth is it does work very well for this use and I intend to keep using it for system benchmarking as this is my focus. A few additions like the elapsed time would have even made it more so.

I was really surprised when you resisted adding the elapsed time. You have the start time and the finish time. I was just asking for simple subtraction so an elapsed time could be shown for the length of test.

As for errors, a 10,000 or 100,000 frame test is a test large enough to reduce any start up delays for getting the first frame from an encoder and should be replicatable assuming extraneous background processes are kept in mind and are controlled.

Anyway, again thanks for taking the time to reply and whether you intended it or not, Kronos is excellent for complete system. benchmarking of a video encode and works very well. Thanks for developing it.

[Kurosu]

Originally posted by DDogg
Kurosu, thanks for your thoughtful and detailed reply. I think some of the "disgression" is caused by language barriers.
Thanks to use such a nice mean to say I write with a crappy english. :D
(just kidding)

However, the truth is it does work very well for this use and I intend to keep using it for system benchmarking as this is my focus. A few additions like the elapsed time would have even made it more so.
I do agree it can give good results for comparing different systems with a particular filter chain. (might need another wording, correct me on it if needed).

I was really surprised when you resisted adding the elapsed time. You have the start time and the finish time. I was just asking for simple subtraction so an elapsed time could be shown for the length of test.
But I didn't resist at all! That's something that doesn't hinder the work, and can have a meaningful purpose. I did resist Sh0dan proposal however, because it was too particular a use, and source code was available for any tweak. I did my best on that part to provide a mean to do that externally, but avisynth scripting makes it rather difficult to achieve.

I'll add it sooner or later, but everything in good time. If you need it, I'll even make it available during this weekend, but I generally don't like to release stuff with so little change (and inflate version numbers :) )

As for errors, a 10,000 or 100,000 frame test is a test large enough to reduce any start up delays for getting the first frame from an encoder and should be replicatable assuming extraneous background processes are kept in mind and are controlled.
Yes, though a 100,000 length may take a while to process (vdub joblist, I love you :D)
What I have noticed is that this is particularly needed: testing for instance Vaguedenoiser optimizations, I noticed that "instantaneous" speed can fluctuate a lot, but in a reproducible way (ie every time on the same frames) while nothing in the filter chain tested could be directly responsible for this.

Anyway, again thanks for taking the time to reply
Well, I'm not a busy person that is very tight on his (free) time either. So, you're welcome :)

and whether you intended it or not, Kronos is excellent for complete system. benchmarking of a video encode and works very well.
Don't worry, I'm not saying you shouldn't, I'm just saying that it might not be the most precise way to do that benchmarking. But I agree that, given a good workflow (natural video, particular filters and such), it is more meaningful than the "multimedia" benchmarks i was referring to and was almost bashing.

Thanks for developing it.
Again, you're welcome. I particularly enjoy feedback on my filters, and at the opposite - as many developpers I guess - quick to drop the public part of a project when I don't feel there's enough support/feedback. So, thanks for that disgression ;)

In the particular case of Kronos, I'm still not convinced it works the best way. But unfortunately, what is considered a better way - running threads and only measuring the time the system really spends actively on the processing - gives worse results in my opinions. And checking for accuracy of the measure is the least exciting part of its development :)