how about a benchmark file?

[russellm]

Alright, I'll admitt I am bored and just reading the message board and decided to post an idea.

I have read many threads on the forum about different speeds people get with different processors. One of the biggest problems with this was that it was never an apples to apples comparison.

How about someone making a small VOB file and maybe someone can make a script file to use with DVD2SVCD and we can all use it as a benchmark. Heck, maybe it could even be sent into some of the hardware sites who are always doing reviews on new motherboards and processors. They want to test something people are really using... Well, I can think of nothing better then DVD2SVCD for that :)
Problem is, we need to make sure it is a packaged deal so it's always an apples to apples comparison.

Russell

[oliv]

Hi russelm.

I'm 100% for this idea. You're fully right in saying that the VOB has to be the same for everybody to be able to compare the performances.
Not only the source but the quality settings and so on.
So, we should have in the package:
- bundle of the DVD2SVCD release which has to be used (an option maybe, as it makes a 5mb difference in download, but this is surely a requirement to be able to compare)
- VOB file(s)
- DVD2SVCD settings
- A way (program) to describe or retrieve system info (Is the Microsoft tool 'Sysinfo' aka 'msinfo32' (can be run via 'Win Key'+R+'msinfo32') good enough?). It seems that this tools is there since Win98-first edition. The command line (in Win2000 at least) can be:
C:\>"\Program Files\Common Files\Microsoft Shared\MSInfo\Msinfo32.exe" /report result.txt /categories +SystemSummary.
Can someone confirm that this also works in Win98, WinMe, WinNT 4.0 and WinXP?
- A way (program) to send the results in a standardized format. Maybe a submission in a dedicated thread of the Doom9 forum? Or a separate site, via a PHP or Perl script.

Hmmm... I'm volunteer to prepare the whole thing.

In the meantime, can someone bring comments + ideas?

[Hunter]

greta idea!

I've checked the command line in XP and it works flawlessly!

Hunter

[markrb]

Although not a bad idea there are a few things that I think will either hold it off or stop it completely.

Doom9 is already beyond capicity and adding another file to download will just about bust him. The file will have to be big, at least 400-500mb's. Hosting that size a file and paying for the bandwidth is very expensive. A smaller file will not give you enough time to get CCE running at speed for any length of time.

The second is getting the time to create such a file. The only ones capable to make a file like that the way it should be are already busy coding etc...


Mark

[oliv]

Hi markrb,

I agree with you, the file will have to be rather big. The hosting is a problem (I don't think the download is for all ADSL/Cable/Sat owners).
What about a movie trailer? Is this big enough for CCE?
I have yet no idea on how big this can be, but I can check this out.
For the creation of the file, as I said before, I can take care of it. I don't want to disturb our coders :)
Again, I'm looking at it and will keep you posted :D

[DDogg]

This is where this idea always gets slowed down everytime it is brought up. Trust me, use chapters from a common dvd like the matrix chapter 28-32. If anybody is serious about benchmarking they can go by the dvd or rent it and save these chapter vobs to disk for reuse.

[mrbass]

What's the point? Honestly everyone knows faster CPU = increased CCE RT speed..so go buy the fastest CPU your wallet can afford. Everyone is at different time and point in their pc buying choices. I jumped from a P450 to Althon XP1700.

-- 256MB makes a huge difference over 128MB. Anything over 256MB is minimal gain in speed if any.

-- motherboard chipsets probably can affect speed to a good degree but.

-- 5200rpm rather than 7200rpm makes a minimal difference (I've tested it)...very low difference. 5400rpm to a 15,000rpm scsi raid 0 striped might make a difference. DMA does need to be turned on otherwise PIO might suck 40% or more of CPU.

To me benchmarks (not in all cases though) are bragging rights (I have a screamer and you don't..blah blah).

So all one really needs to know for getting DVD2SVCD to process quickly a fast CPU (faster the better) and at least 256MB and the various parameters that affect encoding speed. I suppose with Tsunami dual cpu would help significantly unlike currently with CCE which results is such a small performance gain it'd make you cringe.

[russellm]

MrBass:

I think you are being a bit shortsighted with regard to the usefullness of benchmarks. I use benchmarks all the time to tweak my system and try to get the most out of it.

Also,
I think a consistent repeatable benchmark for a realworld application would be very helpful when people are making buying decisions. I for one, spend most of my CPU cycles on video editing/converting.
I went and bought what I believe is the best hardware to do the job.

I know more then the average Joe about benchmarks and system components (I manage the design of Servers for a living) but not everyone has that luxury. So lets assume someone comes in here looking for ideas on the best possible system to buy to get the best possible performance, what are you going to tell them? 512meg RAM of memory? 7200RPM hard drive? IDE RAID? SCSI? Intel P4? AMD Athlon XP? RDDRAM(Intel CPU only)? DDR Memory?

That is the nice thing about a realworld benchmark. No one needs to get into a discussion about which memory is best, which CPU is best, etc... You let the facts speak for themselves and buy based on those facts.

I do not believe you would need a file greater then 50meg to get be able to get an accurate benchmark reading.

Lastly,
You can brag about your system without needing to run a benchmark. People do it all the time :)

Thanks,
Russell

[markrb]

russellm I disagree that a 50mb file will give any kind of usefull results in CCE. It takes my system at least a minute to get a consistant speed. Maybe 400-500mb's is a bit high, but you would probably need at least 100mb's unless you have stripped the audio all ready and then maybe 90mb's of pure video.
Now this still does not address the issue of where it would be hosted. I can pretty much assure you it won't be at Doom9.

As for benchmarks. I believe they have their place. It is nice to know how my new system is running. Many people over use them and try to squeeze every point possible when 99.999% of the time it won't make a bit of difference in the real world.


I do have a possible solution that may satisfy the curiousity of those that would like to know how their system stacks up. Let's take DDogg's idea and expand on it a little. Let's get two groups together. 1 for NTSC and one for PAL, find a pretty common DVD and choose 2-3 chapters depending on their length and create a base level. All the while trying to keep everything the same no matter how trivial it may seem. If DVD2SVCD agrees lets have a couple of people, one or two for each type, take over, create a set of rules and common settings and then track the results. There is no way I have that kind of time, but maybe the ones most interested can become the leaders here.

Mark

[russellm]

Mark,

I understand that CCE takes a bit of time for its speed to get consistant. But, what may look like inconsistancy to the person viewing it, may not actually be. I would be willing to bet that if you ran a 50meg file a few times, you would find that the results were very repeatable. Just because you see the speed changing, does not mean that is not a repeatable test. It could just be how CCE manages the first part of the movie.

Secondly, I have spent a lot of time trying different combinations to squeeze out the most performance. for instance:
Which is better... a 133mhz FSB and a lower memory bus... or, a lower FSB and a higher memory bus...

I have been able to get my speed up to about 2.25 but I have seen anywhere from 2.1-2.25 dependant on which combinations I have tried. I would love to be able to get comparisons with other people's data so I could leverage the learnings.

With all that said, maybe a few of us, doing it would be a good start to see if we could make something repeatable. As far as finding a place to store the file, that is the easy part. I can find a few places to store it without a problem.

Thanks,
Russell

[oliv]

Actually, if the size of the video is a problem (and it can easily be), DDogg and markrb's idea is more than valid.
- Take a DVD which has been sold a lot (one in PAL and one in NTSC). To my mind, "The Matrix" is a good idea, along with Gladiator and SW-Episode I.
- Choose 1/2 chapters
- Provide the dvd2svcd settings (actually, I realised that in this case, we cannot give the "dvd2svcd_project_file.d2s" as it contains the directories to use). If we don't provide the video, we need to give a series of snapshot of the settings we use in the different tabs.
- Provide a standard program to analyse PC caracteristics and submit the results to a certain site (or, at least, a template of what needs to be filled out)

Important things to describe the configuration:
CPU
- Brand
- Frequency
- Model
- Nb of CPUs (might become useful with the support of TMPGENC :D )
Memory
- Amount
- Type (SDRAM, DDRAM, ...)
- Speed
Misc
- FSB
HardDisk
- Brand
- Model
- RPM
- Capacity (this only matters because the bigger the capacity, the bigger the density, the lowest the access time is and the bigger the transfer rate)

Any other idea?

[markrb]

Making a 50mb or larger benchmark file is ok for those of us lucky enough to have Cable, DSL or other high speed connections, but what about those many people that have no such option? It would take a person on a 56k modem days to download such a file and even with my cable modem it would take longer then I would want it to. Also I am pretty sure that a file of this size and the number of people intested in it would overwhelm most hosting sites. I just can't see this idea being very practical in use.

Now Oliv seems to have understood what I was thinking of. We could create a group out of the members here to create some sort of basic test using a template of settings and collecting the data and showing what a baseline system gets.

I don't see this as a very important idea, but for those interested it might be worth their time to create this.

Mark

[mrbass]

more useful than a benchmark showing total time it took to encoding a few chapters would be individuals trying there best to post percentages.
One such test I've been meaning to do is testing the % gain for CAS 2.0 rather than CAS 2.5.

For benchmarks people will have 10 or more variables and one if one had a significant speed over the other how is one supposed to know which component, setting, etc. will give them the most increase percentage wise. Obviously CPU, memory are up there.

This is good because the individual changes just one setting, component and reports the percentage increase gained using the same source (doesn't matter). Since we wouldn't be comparing PAL to NTSC, 4:3 or 16:9 all we would care is about is % gain.

A while back I did go thru that huge thread of CCE RT speeds and tried to summarize it but obviously it was flawed since so many variables existed it wasn't a good measure. But of all those posts I remember the CPU as easily the #1 factor and those who used the same CPU were generally within the same ballpark. 4:3 and 16:9 and PAL seem to be the biggest factors affecting speed at that time. Now there are quite a few more factors as listed in the newbie faq. Posted here for your convience. So why don't we continue expanding the percentages gained by adding this, tweaking this, etc.

Speed
-- NTSC 480x480 = 230,400 pixels, PAL 480x576 = 276,480 pixels. PAL encodes slower than NTSC since it has 17% more pixels to encode.
-- 4:3 source encodes quicker than 16:9 source
-- Verify that DMA (direct memory access) is enabled for your harddrive(s) to avoid wasting precious CPU cycles.
-- Don't downsample 48Khz --> 44.1Khz to save time (unneccessary)
-- CCE multipass VBR use 3 or 4 passes instead of 5
-- BicubicResize results in the slowest encoding speed
-- Bilinear increases encoding speed about 10%
-- SimpleResize increases encoding speed about 20-25%
-- TemporalSmoother slows encoding speed about 40%
-- Safe Mode (VFAPI Slow!) slows encoding speed about 30%

[Caliber]

It would be possible, IMO, to post a vob - of about 400-600MB to a newsgroup such as a.b.v I download from there quite often and it is very fast. In this way only one person needs to up it and we can all download it. Maybe I am lucky to be able to download quickly and freely from my news server and not everybody can do this from theirs, but it solves the problem of bandwidth coming down from doom9 or another site. The real question seems to be: how fast is it in CCE. Only problem with this is that who wants to watch CCE - it has no timer that I know of. If it is only an CCE test, we can all use one .ecl file and .avs (edit them for file locations of course). In the case of using DVD2SVCD, it would allow us to say "audio took XX minutes, Video took XX minutes, Ripping took XX minutes", while this is better IMO, we would need screenshots to show all settings - as oliv pointed out. If not the newsgroup idea, I will buy the Matrix this weekend to post my results next week - I'd be interested in seeing some results :)

I think benchmarking is great. I had a P3 450 and wasn't sure if I should upgrade for a long time. I finally just did - to an XP1800. Of course I knew it would be way faster, but being able to compare my speed to that of other computers would help me gauge when to upgrade - when it would give ENOUGH of an increase in speed to warrant spending the dough. It is because of this that I think benchmarking is important - not for "bragging rights" as some people think, but to help each other know how fast their computer SHOULD be (maybe need to optimize drivers etc...) and when it's time/worth it to buy a new processor or mobo.