Multiple Encoder Processes: My experience

[Harrysmiith]

Does the Intel Conroe utillize 100 CPU when encoding?
Or does it use 70-80% like AMD?

Yes I have heard alot of god things about that new Intel CPU :D

Using an Intel duo2 6300 I have found that using CCE I get 80%cpu utilization. With HC and "Multiple Encoder Processes" checked I get 100% BUT still slower than CE.
Checking "multiple encoder processes" with CE makes no difference to speed for me

[blutach]

Latest results for me - dual core 3800, no o/c, 2hr 1mins encoded in 110 minutes, Procoder 2 Mastering. 100% utilization.

With CCE, I would expect 1/2 the time.

BTW: Anyone figured out how to stop the stupid Procoder 2 splash screen? It takes ages loading all its stuff for every segment, and stealing focus. It's supposed to run minimised, but somehow, only 1 process is. :(

Regards

[SiKKo]

Here is Firewall, R1, Movie only mode with QuEnc HQ, single pass CQ VBR (same as DDog and Nem). I stripped subtitles and all languages except the main 5 channel English track.

This was done with an Opteron 165 clocked just under 2.8Ghz and 2 GB RAM running at DDR460 3-4-4-8. Unfortunately I only have the one 320GB hard drive now on that computer, as I store movies, music etc on my file server which is connected via Gigabit network. So movie was ripped as .iso w/ DVD Decrypter to file server, mounted with Daemon tools on local computer, and so the work path and destination are on the sole hard drive of the computer, which I would assume explains the slower rebuild.

[06:16:12] One Click encoding activated...
-----------------
[06:16:13] Phase I, PREPARATION started.
- DVD-RB v1.10.5
- QuEnc 0.7.1.0 encoder selected.
- "One Pass VBR (w/analysis)" mode is enabled.
- "Movie Only" mode is enabled.
- VTS_01: 2,855,999 sectors.
-- Scanning and writing .D2V & .AVS files
-- Processed 155,610 frames.
-- Building .AVS and .ECL files
- Reduction Level for DVD-5: 91.0%
- Overall Bitrate : 6,466/5,173Kbs
- Space for Video : 4,098,236KB
- Analyzing VTS_01 for optimal CQ factor.
-- TargetSize (sectors):2,458,942
-- Sampling 3120 of 155610 frames.
-- Predicted size (sectors) at CQ=2: 1,201,726
-- Predicted size (sectors) at CQ=1: 2,601,279
- CQ Value selected: 2
- HIGH/LOW/TYPICAL Bitrates: 6,249/4,678/5,173 Kbs
[06:23:26] Phase I, PREPARATION completed in 7 minutes.
[06:23:26] Phase II ENCODING started
- Creating M2V for VTS_01 segment 0
- Creating M2V for VTS_01 segment 1
- Creating M2V for VTS_01 segment 2
- Creating M2V for VTS_01 segment 3
- Creating M2V for VTS_01 segment 4
- Creating M2V for VTS_01 segment 5
- Creating M2V for VTS_01 segment 6
- Creating M2V for VTS_01 segment 7
- Creating M2V for VTS_01 segment 8
- Creating M2V for VTS_01 segment 9
- Creating M2V for VTS_01 segment 10
- Creating M2V for VTS_01 segment 11
- Creating M2V for VTS_01 segment 12
- Creating M2V for VTS_01 segment 13
- Creating M2V for VTS_01 segment 14
- Creating M2V for VTS_01 segment 15
- Creating M2V for VTS_01 segment 16
- Creating M2V for VTS_01 segment 17
- Creating M2V for VTS_01 segment 18
- Creating M2V for VTS_01 segment 19
- Creating M2V for VTS_01 segment 20
- Creating M2V for VTS_01 segment 21
- Creating M2V for VTS_01 segment 22
- Creating M2V for VTS_01 segment 23
- Creating M2V for VTS_01 segment 24
- Creating M2V for VTS_01 segment 25
- Creating M2V for VTS_01 segment 26
- Creating M2V for VTS_01 segment 27
[06:43:41] Phase II ENCODING completed in 20 minutes.
[06:43:41] Phase III, REBUILD started.
- Processing VTS_01
- Reading/processing TMAP table...
- Rebuilding seg 0 VOBID 5 CELLID 1
- Rebuilding seg 1 VOBID 5 CELLID 2
- Rebuilding seg 2 VOBID 5 CELLID 3
- Rebuilding seg 3 VOBID 5 CELLID 4
- Rebuilding seg 4 VOBID 5 CELLID 5
- Rebuilding seg 5 VOBID 5 CELLID 6
- Rebuilding seg 6 VOBID 5 CELLID 7
- Rebuilding seg 7 VOBID 5 CELLID 8
- Rebuilding seg 8 VOBID 5 CELLID 9
- Rebuilding seg 9 VOBID 5 CELLID 10
- Rebuilding seg 10 VOBID 5 CELLID 11
- Rebuilding seg 11 VOBID 5 CELLID 12
- Rebuilding seg 12 VOBID 5 CELLID 13
- Rebuilding seg 13 VOBID 5 CELLID 14
- Updating NAVPACKS for VOBID_05
- Rebuilding seg 14 VOBID 6 CELLID 1
- Rebuilding seg 15 VOBID 6 CELLID 2
- Rebuilding seg 16 VOBID 6 CELLID 3
- Rebuilding seg 17 VOBID 6 CELLID 4
- Rebuilding seg 18 VOBID 6 CELLID 5
- Rebuilding seg 19 VOBID 6 CELLID 6
- Rebuilding seg 20 VOBID 6 CELLID 7
- Rebuilding seg 21 VOBID 6 CELLID 8
- Rebuilding seg 22 VOBID 6 CELLID 9
- Rebuilding seg 23 VOBID 6 CELLID 10
- Rebuilding seg 24 VOBID 6 CELLID 11
- Rebuilding seg 25 VOBID 6 CELLID 12
- Rebuilding seg 26 VOBID 6 CELLID 13
- Rebuilding seg 27 VOBID 6 CELLID 14
- Updating NAVPACKS for VOBID_06
- Updated VTS_C_ADT.
- Updated VTS_VOBU_ADMAP.
- Updated IFO: VTS_01_0.IFO
- Correcting VTS Sectors...
- Building ISO Image...
- ISO Image successfully created.
[06:55:04] Phase III, REBUILD completed in 12 minutes.

Done.
[06:55:04] PREPARE/ENCODE/REBUILD completed in 39 min.



Basically what I expected as far as results. Clock for clock, no AMD right now is going to beat the new Intel Core 2 Duos, however they will destroy any older Intel chips such as Pentium D8xx or D9xx. Again, I am talking clock speed vs clock speed. As you can see, Opteron @ 2780mhz encoded a minute faster than Intel D930 @ 4500mhz, while Core 2 Duo @ 2520mhz had the fastest encoding time by 2 minutes.

As far as real world differences, obviously a minute or two in encoding time is not very substantial, so it basically breaks down to cost. If you already have a socket 939 AMD, and want to upgrade from say a single core Venice 3000+, then I think your best bet right now is an x2 3800+ for about $150. You don't need a new motherboard, RAM, heatsink, etc - you could just plop in the new chip and you are good to go. Pretty much every x2 3800+ should be able to reach 2.6ghz, many will get to 2.8ghz or more. I have a single core San Diego 3700+ clocked to 2.8ghz, and definitely notice a substantial difference in encoding time between that and the Opty when using CCE or any multi-threaded application. Soon enough I will replace that San Diego with an X2 3800+, and am anxious to see if there is any noticable difference between that and the Opteron.

And if you lean towards Intel, there is no doubt that the Core 2 Duo's are the way to go right now. Pentium D's are way too inefficient, run hot, consume a lot more energy, etc etc. Even if the chip itself is $50-80 cheaper, you will end up spending that on extra cooling and on your energy bills.

[dragongodz]

i do have to pipe in and say people need to not worry so much about cpu usage not being 100%. its possible to saturate cpu usage with many threads but actually slow encoding down because of many stalls etc. so long as you are seeing both cpus are being used substantially and are getting good encoding times then dont stress over the cpu usage.

[DDogg]

And if you lean towards Intel, there is no doubt that the Core 2 Duo's are the way to go right now. Pentium D's are way too inefficient, run hot, consume a lot more energy, etc etc. Even if the chip itself is $50-80 cheaper, you will end up spending that on extra cooling and on your energy bills.

As a point of reference, my D930@4.5 GHz idles @ 24C and dual load peaks at 42C, but I do tend to agree with with your general thoughts. The D930 on the AW8D was a cost effective solution and should serve me well for quite a while.

As far as the Core 2 Duo's, I'm not yet comfortable with the available motherboards and think waiting a few months for the bios' and hardware to better stabilize would be prudent.

The cost reductions on the AMDx2's are a welcome byproduct of the additional competition caused by the Core Duo release, and may be the best alternative as that platform is much more mature at this stage.

Either way, all of us have benefited.

[SiKKo]

Dang DDog, what are you using to cool that baby to keep it at those temperatures? Although I don't know from personal experience, everything I have heard about Pentium D's is that they run hot. I know my old p4 3.2ghz Prescott was horribly hot, and got near 68C while under load, and indeed doubled as a heater for the office.

I also agree, the Core 2 Duo's release has benefitted everyone across the board with huge price reductions on Athlon 64s and Pentium D's! Being able to get dual core chips for $100-$150 is fantastic. I sure wish I would have waited about 4 months when I got my Opty, because I could have got two 3800+ x2's for the price I paid for the one Opteron earlier this year! Oh well, such is life!

[shoarthing]

. . . . BTW: Anyone figured out how to stop the stupid Procoder 2 splash screen? It takes ages loading all its stuff for every segment, and stealing focus. It's supposed to run minimised, but somehow, only 1 process is. :( . . . Splashkiller (http://www.digitallis.co.uk/) . . works fine for me; tho' for some reason the second Procoder instance on the first round of encodes shows for that one cell.

[Sophoclesdrf]

Go back through the threads here and it's amazing how many overclocking instabilities CCE has caught


That's true to a point. Any processor of a given core has the same potential because the lowest in a series and the highest in a series have the same core. What makes them different is how well they're manufactured and binned. Often this is determined by the batch in which each series was manufactured which is sometimes referred to as stepping. I have both an Opteron 175 and 170 system. Logic would tell one that the 175 should be the easiest to overclock because of its higher clock multiplier and the fact that it costs a $100 more than the 170 but that is not the case. My Opteron 175 system is stable and will run CCE in batch mode without a single error overclocked to 2.6 GHz and it's prime 95 stable for 36 hours at 2.55 GHz. My 170 However will run CCE at 2.75 GHz and it is Prime 95 stable at 2.7 GHz. They both have the Denmark (really just a Toledo) core but the lower clocked and lower priced Opteron 170 is the clear winner. Now jdobbs, you've known me online for more than two years and in that time you will not recall a single problem that I've had that could be traced to my overclocked rigs. The problem isn't overclocking, it's badly match equipment and unrealistic overclocing that causes problems.

[jdobbs]

It's not the overclocking itself that is the problem. It's that people have a tendency to always keep inching it up until they "think" it's stable. Then when CCE dies... rather than make the logical conclusion (that they should back their clock down) -- they post a problem with CCE or DVD-RB. Almost always when I see those same symptoms (like running the same ECL through twice and one of them works while the other fails) I know that it is a system instability. There is no room for doubt.

The problem is that everyone is certain that it isn't their computer or their overclocking... because program x or program y runs fine. CCE isn't program x or program y -- it is probably the most CPU intensive piece of software you can buy.

Since I can't sit down with them, look at their computer, and say "back the FSB down, your memory can't take that" -- or "Man, you're running hot, you need a better heat sink." -- all I can do is say is either "don't post your problem" or "don't overclock"... The latter is more palatable.

[blutach]

. . . Splashkiller (http://www.digitallis.co.uk/) . . works fine for me; tho' for some reason the second Procoder instance on the first round of encodes shows for that one cell.Thanks shoarthing.

@jdobbs - have you seen any other incidences of run encoders minimised not working 100% when multiple encoder processes are selected?

Regards

[jdobbs]

@blutach

No, I don't recall any reports. The same routine is called to run the app regardless of the instance -- so I can't imagine why they would run differently. Has it happened to you?


@All

For you folks that aren't running at 100% with multiple encoder instances... here's something you might want to try. Edit the REBUILDER.INI and change this line:

ENCODE_PROCESSES=2

to

ENCODE_PROCESSES=3

:eek:

Note: Consider it a beta test. I've never actually done any tests. I don't have a dual processor or a supercomputer, so I can't even seen an improvement with two instances.

[blutach]

@jdobbs - yes, every time. 1 process runs minimised, the other in the foreground. And each time Procoder 2 starts, the lousy splash screen steals focus, whether the process minimises or not (yet to try shoarthing's utility).

rahzel mentioned in post 11 (http://forum.doom9.org/showthread.php?p=857237#post857237) that it is easy to o/c the 3800+. I have done this reasonably conservatively with quite spectacular results.

The same source I mentioned in post 52 (http://forum.doom9.org/showthread.php?p=869669#post869669) is now encoded at 32 frames per second (95 minutes encoding time for the 122 minute movie) at 2250Mhz (no change to memory timings). This is now the equivalent of a 4200+ :). The system was a solid as a rock during that time - temp was only 1-2 degrees higher than normal (51-52C), no voltage problems at all. Dunno how much higher I will push the machine without "conservatising" the memory timings, but that is a great result, as far as I am concerned.

Thanks rahzel!

Regards

[jdobbs]

@blutach

Interesting. What MB are you using? Also, are you overclocking just by increasing the FSB or are have you somehow modified the processor to accept a higher clock multiplier? Is your memory anything special?

I'll do the testing and see what I can do about Procoder (if anything). One thing I noticed when I just looked at my code, when I shell to the process I give it focus even when it is set to run minimized -- I'll change that.

[blutach]

Appreciate the change re foreground stuff, jdobbs.

Memory is 2 x 512Mb Corsair XMS (CMX512-3200C2). Mobo is ASUS A8N-SLI Premium.

Only thing I have done is up the FSB by 12% or so. rahzel seems to think it can go up by 35% but I am too timid for that :D. The Nvidia benchmark tests I did quickly yesterday seemed to think the memory could go a bit faster too, although its SPD is 3-3-3-8 @ 200Mhz.

I may go to 2.4Ghz but at that speed, I expect the memory timings will need to go down a notch. Before that, I wanna do a few encodes at this current level to see how things are (BTW: The Phillips Verifier showed nothing really nasty on the output - just the normal things it moans about).

Here's the current setup.

http://img135.imageshack.us/img135/4479/09022006113804mx5.png

Regards

[Sophoclesdrf]

@jdobbs

I've assisted enough people over the last couple of years to know that what you're saying is true, some people assume that anything that boots up and surfs the net is stable. I just couldn't figure out why that would stop you from overclocking, you would know better. I have three overclocked systems running beautifully D945, Opty170, Opty, 175, and I just ordered an Intel Conroe E6600, and I intend to over clock it too.:rolleyes:

[Sophoclesdrf]

blutach


Your clock multiplier is way to low and you're underclocking your sytem. The X2 3800 has a native clock multiplier of 10X and you have yours set at 5X which is 5X your hyper transpot of 225 which gives you a core speed of only 1.125 GHz, the default speed of the X2 3800 is almost twice that 2.0 GHz.

Go into your bios. Leave your clock multiplier at 10X and set your Hyper Transport to to 4X and then up your memory speed until you find a stable setting. Where it is now will over clock it to 2.25 GHz which is nice little boost. The key is that your HT should not exceed 1000 MHz by much. At 4X your current speed it will be 900 MHz.

If you want more info contact me at www.dvdhounds.com as just Sophocles and I will be glad to help you boost it to a higher but stable setting. That chip is easily hitting 2.6 GHz depending on when and where your chip was made.:)

[jdobbs]

@blutach

I see what you mean about the second instance staying visible. I don't have any idea why it does that... but changing the shell parameter to "no focus" didn't seem to have any effect at all.

@Sophocles

Actually I know quite a bit about overclocking... as I said I'd been doing it for years. Hell, I was overclocking 8088's. But eventually I decided I'd had too many surprises at inopportune moments. The amount of time saved in overclocking was easily outweighed by in the increase in "redo"...

To each his own...

[SiKKo]

The reason Blutach's multiplier was at 5 and the voltage at 1.10 is because he has Cool N Quiet enabled, which you should not do when overclocking.

With a 3800+ x2 you should be able to hit 250 HTT x 10 Multiplier pretty easily, but you will likely have to use a memory divider unless you have PC4000 RAM.

I highly, highly recommend reading this guide for overclocking AMD chips before messing with anything.

AMD Overclocking Guide (http://www.dfi-street.com/forum/showthread.php?t=20823)

Its fairly lengthy, but I've learned a ton over the last year from this site. I think its pretty fun to turn a $150 chip into something that is faster than a $800-$900 chip. You just want to work your way up slowly, and not do a suicide mission right away trying to break 3Ghz or something dumb like that.

@JDobbs

I tried changing the command line in Rebuilder.ini to ENCODE_PROCESSES=3 and it still does not put both cores to 100% with CCE.

*Edit* Posted full results in post #73.

[blutach]

@Sophocles and Sikko - thanks for the links. I'll turn CNQ off and play around in the BIOS.

This is what the Nvidia ntune says, however.

http://img245.imageshack.us/img245/8450/09022006144005wv7.png

Regards

[rahzel]

np blu. overclocking any processor should be quite easy, and it's pretty similar for all socket types.

for your particular setup, make sure your temperatures are below 50-55c load, don't set the voltage any higher than 1.55-1.6, and make sure your system is stable. like i said before, all 3800+ X2's should reach 2.4ghz quite easily, so i'm sure youre ok. but to test stability, especailly for DC processors, i recommend using ORTHOS SP2004.
get it here:
http://sp2004.fre3.com/beta/beta2.htm

run that for at least 6-8 hours. it will stress both cores (probably more than anything youll ever run). this is also a good time to monitor your load temperatures.

overclocking the processor benefits the most out of any piece of computer hardware imo. and if you follow the safety precautions i mentioned above, it's completely safe (just make sure you only go up in small increments at a time, though). if you do, your processor should still last longer than you'll keep it, and it should be completely stable.

[blutach]

Updated results:

Well, I have turned Cool N Quiet off and there is little change except in the way CPU-Z reports things (in that lightly loaded mode, it reports a 10 mutiplier not 5 and a higher core voltage - 1.36v vs 1.12v)

I actually kicked the processor up to 2300Mhz and did a full encode on the same source - the encoding time was 94 minutes (1 minute better than before), however the rebuild time was 9 minutes (down from 11).

Trying to kick it up to 2500MhZ without other changes makes the rig unstable - the memory can't handle those timings, I don't think.

Might run that auto tuner that came with the PC to find the best memory and clocking parameters.

Regards

[rahzel]

all 939 motherboards should have memory divider options. you can always use them if your memory can't run 1:1 with your processor.

[SiKKo]

Apparently I had way too much free time this weekend, so I did a bunch of tests to see what the difference in encoding time would be in such things as single core vs dual core, and stock speed vs overclock.

I used the movie Firewall again, as in previous posts in this thread, with Main Movie only, subtitles stripped as well as stripping all audio except for 5.1 English track. I used two encoders, CCE 2-pass, and QuEnc with HQ Mode with Single Pass CQ VBR to test time differences. Computer 1 used an AMD San Diego 3700+ (single core), and Computer 2 was AMD Opteron 165 (dual core). To compare the differences in single core vs dual core, at least in overclocked mode, I ran the two rigs at approximately the same speed (2700MHz). I kept them at 2700MHz instead of 2800MHz on these tests because that is the highest I can go on stock voltage on both chips, and thus what I think the average AMD chip should be able to handle as far as an overclock. The difference between 2700Mhz and 2800Mhz was fairly insignficant anyhow as far as speed increases.

Here are the results:

1) San Diego 3700+, 2200 MHz (stock spd), CCE 2-pass
Phase II ENCODING completed in 70 minutes.

2) San Diego 3700+, 2706MHz (246x11), CCE 2-pass
Phase II ENCODING completed in 56 minutes.

3) San Diego 3700+, 2200MHz (stock spd), QuEnc HQ
Phase II ENCODING completed in 49 minutes.

4) San Diego 3700+, 2706MHz (246x11), QuEnc HQ
Phase II ENCODING completed in 41 minutes.

5) Opteron 165, 1800MHz (stock spd), CCE 2-pass, DISABLED Multiple Encoder Processes
Phase II ENCODING completed in 53 minutes.

6) Opteron 165, 1800MHz (stock spd), CCE 2-pass, ENABLED Multiple Encoder Processes
Phase II ENCODING completed in 53 minutes.

7) Opteron 165, 2700MHz (300x9), CCE 2-pass, DISABLED Multiple Encoder Processes
Phase II ENCODING completed in 38 minutes.

8) Opteron 165, 2700MHz (300x9), CCE 2-pass, ENABLED Multiple Encoder Processes
Phase II ENCODING completed in 38 minutes.

9) Opteron 165, 1800MHz (stock spd), QuEnc HQ, DISABLED Multiple Encoder Processes
Phase II ENCODING completed in 59 minutes.

10) Opteron 165, 1800MHz (stock spd), QuEnc HQ, ENABLED Multiple Encoder Processes
Phase II ENCODING completed in 30 minutes.

11) Opteron 165, 2700MHz (300x9), QuEnc HQ, DISABLED Multiple Encoder Processes
Phase II ENCODING completed in 40 minutes.

12) Opteron 165, 2700MHz (300x9), QuEnc HQ, ENABLED Multiple Encoder Processes
Phase II ENCODING completed in 21 minutes.



Maybe this should have been obvious to me beforehand, but whether you have Multiple Encoder Processes disabled or enabled makes no difference for CCE, whereas it makes a huge difference for QuEnc.

I didn't post the Phase I and Phase III times because there was no significant difference between the runs - about 3 minutes was the largest gap.

[jdobbs]

CCE doesn't allow multiple instances -- so setting multiple encoder processes has no effect.

[rahzel]

yup, CCE does not run 2 instances. however, having a dual core alone makes the encoding faster. with my old opteron 144 s939 at 2.68ghz, my average speed was about 3.9. now with my 3800+ X2 at 2.75ghz, my average speed is about 5.7. a movie that would take around 2 hours, now takes just over an hour.

[SiKKo]

Yeah, I know it won't run two instances, but I thought maybe enabling Multiple Encoder processes would make CCE use 100% (or close to it) of both cores, and in effect reduce the encoding time by 50% over its single core counterpart - but apparently that does not happen. But using a dual core does significantly reduce the time to encode (in my case from 56 minutes down to 38 minutes using CCE 2 pass).

[jdobbs]

When you set multiple encoders on, all it does it tell DVD-RB to increase then number of concurrent instances of the program while encoding.