x264 benchmark?

[turbojet]

Maybe a bit late to the party but is there a chance for an official x264 benchmark? Something that feeds a clip like parkrun directly into x264 x86/x64 that's updated with x264 and hosted by official mirrors?

The TechArp benchmarks sites are currently using is very inaccurate; source is a bunch fades, depends on dgdecode and avisynth and doesn't max quad\octa core cpus.

I'm willing to help with a batch file, if that's a preferred method.

[benwaggoner]

Sounds like an interesting idea. Although Parkrun is pretty short, and hence not very representative.

I think we want to have benchmark that represents a good baseline real-world scenario, so that optimizing for the benchmark would be generally useful.

Also, we'd want a benchmark that focuses on x264, no IO or other libraries. So an uncompressed format that x264 can read directly without any scaling, color space changes, etcetera. And maybe using something like --preset veryslow so that read speed of the source file would be less of a gating factor. I think --preset placebo wouldn't be appropriate since it uses a lot of modes that aren't typically used for mainstream x264.

Maybe standardize on a 1920x800 .y4m file of Tears of Steel or something like that?

[Groucho2004]

Sounds like an interesting idea. Although Parkrun is pretty short, and hence not very representative.

I think we want to have benchmark that represents a good baseline real-world scenario, so that optimizing for the benchmark would be generally useful.

Also, we'd want a benchmark that focuses on x264, no IO or other libraries. So an uncompressed format that x264 can read directly without any scaling, color space changes, etcetera. And maybe using something like --preset veryslow so that read speed of the source file would be less of a gating factor. I think --preset placebo wouldn't be appropriate since it uses a lot of modes that aren't typically used for mainstream x264.

Maybe standardize on a 1920x800 .y4m file of Tears of Steel or something like that?

I agree, a simple "slower" or "veryslow" preset without any bells and whistles would suffice.

Not sure about the source - Any useful raw file would be huge. I'm thinking a MPEG-2 file, 1080p with 20~30Mbps would be better. MPEG-2 decodes with little CPU overhead and the file would be much smaller.

Once a source is chosen, putting a benchmark together is trivial. A batch file is one option, a small launcher written in C/C++ that also provides accurate timing routines would be the other.

[turbojet]

Probably a good idea to use a small, short source and cat a few times to get a longer video to use. Getting an accurate measure in a short amount of time would in be good, preset slow or slower, the latter is rather slow with 1080p. It's not really about quality but loading the CPU for the duration. 2000 frames should be enough to be accurate shouldn't it?

Park run is 60fps which isn't very representative of real world use, might be better to use another small copyright-free clip. Good point about MPEG decoding but x264 implementation of libav would be decoding it so it should be a fair comparison no matter encoder is used. Is HD MPEG all that common outside of ntsc tv broadcast? Maybe AVC instead?

An exe comes with an advantage of most people not messing with it but leaving x264.exe and x264_x64.exe external might be a good idea for updating purposes. It just needs to get x264 version, the x264 commands, checks cpu with something like 'wmic cpu list full' which should be available since windows 2000. Maybe offer a choice to run 32 and/or 64 bit (after checking environment) and maybe resizing to 720p or 480p?

If x264 guys aren't in on it maybe it'd be better off suggesting things to techarp. However I think x264 devs would have better ideas than I do.

[Atak_Snajpera]

Convince them to use x264 FHD benchmark instead of their duct-taped benchmarking tool ;) Yeah I know x264 version included is rather old but still better that their shitty benchmark. Updating x264 would mean that I would have to erase whole results. I don't think I have time and patience to collect new results now.

[Asmodian]

Making a new version and starting to collect data with it doesn't mean you need to throw out the old one or stop collecting data with it. I would love to see comparisons of even just the few of the newer processors with a good benchmark running a new version of x264. :)

Sadly when a benchmark gets too out of date it loses it's relevance, the performance of the x264 I run might not be equivalent to the benchmark. Let me stress that I don't know this to be true and in fact I think your benchmark matches the performances differences I have noticed between the few computers I run x264 on. It is still nice to be sure with a regularly updated benchmark. ;)

[turbojet]

The FHD benchmark is a lot cleaner and shorter (2 minutes vs 40) and probably more accurate but there's a few problems with it:
1. doesn't handle high DPI correctly, can't see the buttons
2. fully loads the cpu about 90% of the time, which messes with turbo on modern cpus
3. 50fps source, does this represent a typical situation?
4. computer becomes unresponsive, too high priority given to x264
5. 180MB is a large download, concatating just one of those scenes would greatly reduce it and might resolve issue #2
6. sees AMD 8320 as 4C - 8T, cpuid could use an update, along with compare of modern cpu's

Other than that though it has some really good ideas, the x264 and settings could also use an update. The conflict with current info could be resolved with a counting version of the benchmark program.

[Atak_Snajpera]

1. true
2. blame x264 ;)
3. why not
4. real time priority is set in order to 'disable' all other proceses running in background. (stable results)
5. downloading 180mb these days is not a problem. have you heard people complaing about size of latest 3d mark?
6. i use windows api. before patch for win7 fx 8xxx was detected as 8C/8T. in order to improve performance m$ decided to change that to 4C/8T. afterall fx 8xxx has only 4 fpus. sorry but fx 8xxx is not true eight core.

[turbojet]

2. or cpu/mobo wherever the logic exists. Or the one thing that can be controlled, source. No problem sustaining full load on most sources I deal with here, TV caps. It's the complicated sources that have the problem, I'm guessing the lake scene and the tree scene are the biggest culprits but all are well above average complexity.
3. As long as it's consistent, it's not much of a difference speed wise from 24/25/30 sources is it?
4. High does the same thing without freezing the computer, crashing services, etc.
5. It's not really a problem but hosting it might be, could just as well be 30-40 MB and just as accurate of a benchmark
6. It still has 8 cores just doesn't have a fully capable pipeline. wmic for instance recognize it as 8 core and it's available to all NT OS's.

Anyhow I'll probably contact techarp next week and see what they have to say about it.

[Atak_Snajpera]

4. I have never experienced any crashing on my win7 with my Q6600@3Ghz and Q8200@2.8Ghz and I'm still able to terminate benchmark so even x264 is not able to freeze OS in real time priority.
5. I use mega.co.nz so it is not a problem for me. I do not care about the rest.
6. Exactly 8 INT / 4 FPU . For example Phenom X6 was true 6 core cpu. 6 INT / 6 FPU. So fx is still not a true 8 core cpu like amd's marketing says. BTW this is true eight-core cpu http://ark.intel.com/products/64596/