Dual Xeon Hyperthreaded - incredible speed boost with 2.59!!!

[ninja.rogue]

Hi folks,
I just upgraded from P4 1.9 512 Rambus to Dual Xeon 2.4 1024 Rambus.
I also used latest 2.59 Plus release of Tmpgenc.

The results in term of speed bust are simply amazing.
Source file is DV AVI PAL 720x576, with a lot of movement.
Encoding time with highest quality, 2-pass vbr, min 2000 max 9000 avg 8000 is just 5:1!
With my previous configuration it was 14-15:1
And I still haven't tried motion estimate...

I have obviously enabled all options in environmental settings.
What are your experiences in values for memory caching - 16 mb are not enough, I have used 384.

Cheers!
Paolo

[tiki4]

Hell, I don't belong in the SVCD section of doom9, but maybe your incredible speed increase is due to using so much caching memory? I mean, HT is nice, but 1 Gig of RAM is even nicer...

Just some thoughts,

tiki4

[ninja.rogue]

Hi Tiki4,
presumably you are partially right.
There could possibly be a speed increase in having much higher memory cache.
But the point is: which is the optimum value for this item?
I am using 12GB cache for 2-pass vbr temporary file.
How much memory should I cache for dual processing?

[tiki4]

Sorry ninja.rogue,

I'm not so experienced with TMPGEnc. I don't know what the ideal setting for all those values in TMPGEnc is. I think there are some guys around here who know that much better.
Anyway your combo of P4 and RD-RAM should do pretty well when it comes to speed.

Regards,

tiki4

[ninja.rogue]

It is really a devil of speed...
And I can have four of five sessions of TMPGENC running contemporarily.
Or Procoder...
When using just one session of Tmpegnc I have just 25-50% of processor being used.
Is this normal? Do I have raised such a monster that it needs to be fed with multiple encoding sessions?
This is a wholly new world.
Can anybody help me to get oriented?
Thanks in advance
Paolo

[drafty]

This is probably due to :
1) 2 very fast processors interms of core clock speed vs command execution
2) A 533 mhz front side bus the higher the better because the processor can access memory faster.
3) 512 K of on board cache memory * 2 , you got two of them.

Mpeg encoding needs lots of cache and processing power and fast memory access.
The encoding algorythm simply won't fit into my Athlons 256K of cache. Thus the system relies on fast memory access which is not that fast... 15-18Hrs for me :-(
The HT technology reduces system latency also making your system more usuable when your not using 100% CPU. Ah well I can only dream of such a system...


Drafty.


:rolleyes:

[TyMac]

Can TMPG account for 4 processors? What OS are you using?

[ninja.rogue]

@ Drafty,
the answer is:
No, Tmpgenc can not see four processors. It just sees two of them, in spite of its hyperthreading awareness...
My OS is XP-PRO, which sees 4 processors.

[TyMac]

Originally posted by ninja.rogue
@ Drafty,
the answer is:
No, Tmpgenc can not see four processors. It just sees two of them, in spite of its hyperthreading awareness...
My OS is XP-PRO, which sees 4 processors.

Hold up... XP pro (not server) sees 4 processors? Are you sure?

[ninja.rogue]

Yes, TyMac,
I can see without lenses :D
BTW, it is no news: search Intel or Microsoft about Hyperthreading.
The software which can use 4 processors is not very common, though.
One possible way-through could be to use more than one instance for each 2proc-smp program - i.e. 2 Tmpgenc encodes (using file split).

[Arky]

BTW, does anybody knwo what is happening about the "server" version of TMGPEnc Plus? According to the original TMPGEnc Plus Press Release the server version supports/will support "3 or more CPU"! :D

( http://www.pegasys-inc.com/e_release02.html )


Arky ;o)

[TyMac]

Originally posted by ninja.rogue
Yes, TyMac,
I can see without lenses :D
BTW, it is no news: search Intel or Microsoft about Hyperthreading.
The software which can use 4 processors is not very common, though.
One possible way-through could be to use more than one instance for each 2proc-smp program - i.e. 2 Tmpgenc encodes (using file split).

How do you run multiple instances of a program (I'm off subject, I know... but it sounds like you know how, so why not ask?)

[ninja.rogue]

Tymac,
You can split your source file into multiple source segments of smaller size. Then you can encode them in parallel, running Tmpgenc several times with different files. Finally, you will merge the files using Tmpgenc merge & cut function.

[TyMac]

Originally posted by ninja.rogue
Tymac,
You can split your source file into multiple source segments of smaller size. Then you can encode them in parallel, running Tmpgenc several times with different files. Finally, you will merge the files using Tmpgenc merge & cut function.


Yes but what do I have to do to make XP run multiple instances of a single program?

[ninja.rogue]

It is very simple. Just start the program twice - i.e. double click on Tmpgenc's icon onto your desktop - the tmpgenc.exe file.
You should have two windows of the program and use each of them as an individual instance of the program.
You aren't kidding me, are you?

[TyMac]

Originally posted by ninja.rogue
It is very simple. Just start the program twice - i.e. double click on Tmpgenc's icon onto your desktop - the tmpgenc.exe file.
You should have two windows of the program and use each of them as an individual instance of the program.
You aren't kidding me, are you?

And this works for any application? Never really had to... but I have applications that have *features* that run mulitple instances as an option. I did not know if this was a feature of the app or just simply double clicking.

[ninja.rogue]

Just an update:
I could get encoding time as low as 4.5 times the length of the movie through the use of highest priority for Tmpgenc.exe process using crtl+alt+del - processes.
And, as a comparison, I can have 0.5x (i.e. half the length of the movie) using CCE 2.66 and 1.5x using Main Concept Encoder 1.3.1.
This dual rig is getting a hell of a missile...

[Stereodude]

Originally posted by ninja.rogue
Just an update:
I could get encoding time as low as 4.5 times the length of the movie through the use of highest priority for Tmpgenc.exe process using crtl+alt+del - processes.
And, as a comparison, I can have 0.5x (i.e. half the length of the movie) using CCE 2.66 and 1.5x using Main Concept Encoder 1.3.1.
This dual rig is getting a hell of a missile... If it only takes 1/2 the time of a movie then it's encoding at 2x, not .5x. Sounds like you should be using CCE, not TMPGEnc. It's faster and has better quality to boot (well at DVD bitrates anyhow).

Stereodude

[ninja.rogue]

Stereodude,
I encode using both Procoder (for very low Q values) or Tmpgenc with MB1 interlaced DV matrix.
To my eyes, Tmpgenc's picture quality seems still better than CCE.
And with these encoding times, it really is not a pain in the *** to wait for the outcome.
I don't have my pc running for a whole day anymore, and often it is quicker than expected.
Moreover, I still am suffering a field disorder with CCE - presumably it is curable, but haven't found the medicine.
CCE should be superior for DVD ripping but with interlaced DV PAL source material TMPGENC rules - at least IMHO.
Cheers
Ninja

[Stereodude]

Originally posted by ninja.rogue
Stereodude,
I encode using both Procoder (for very low Q values) or Tmpgenc with MB1 interlaced DV matrix.
To my eyes, Tmpgenc's picture quality seems still better than CCE.
And with these encoding times, it really is not a pain in the *** to wait for the outcome.
I don't have my pc running for a whole day anymore, and often it is quicker than expected.
Moreover, I still am suffering a field disorder with CCE - presumably it is curable, but haven't found the medicine.
CCE should be superior for DVD ripping but with interlaced DV PAL source material TMPGENC rules - at least IMHO.
Cheers
Ninja I have found that for progressive video (24fps or 29.97fps) CCE has less artifacts in flat areas than TMPGEnc. CCE also seems to be sharper. Recent versions of CCE allow manipulation of Quantization Dithering, and Quantization tweaks specific to flat areas. The results have been nothing short of outstanding.

In my previous test TMPGEnc was superior to CCE when dealing with SVCDs (less artifacts). I have yet to try the latest CCE on SVCD because I don't generally make SVCDs.

Stereodude

[Crucio]

I just ordered a dual xeon system (well the parts to one)

and i got dual ddr memory, its 266. Do u guys think this will be as fast as rambus?


Also, what are the programs that can use 4 cpus?

and ninja, have u tried any games on that system yet? Do they work ok?

[ninja.rogue]

Mainc Concept Encoder 1.3.1. uses all four processors
Haven't tried any game
Let us know whether games use more than one
Cheers
Ninja

[lancer]

that is interesting.

on a 1.9 you were getting a factor of 15:1 for the conversion to mpeg.

I thought mpeg compression was horsepower based. ie faster machine, faster encode.

on my PIII 733 512mb SDRAM I get a 15:1 factor on the encode. ie 10 minutes of footage takes about 2.5 hours.

but according to your post you 1.9 took the same factor. hmmmm either my machine is bloody fast or something else.anyone wanna cast a thought?

[ninja.rogue]

Lancer, remember that encoding times are a function of the source fine and the encoding parameters.
I am encoding DV AVI 720x567 files, both audio and video multiplexed, with the following parameters:
Highest quality
2-pass vbr min 2000 avg 6000 max 9000
Interlaced, bottom field
Audio 384kbps
No filters enabled

If you are re-encoding mpeg2 files, things change a lot, or even if you are doing VCD or SVCD.
what are you source and setting specs?

[lancer]

I'm frameserving from virtualdub an interim Huufyuv avi file.

this is a capture interimed with the filters applied.

size is 44GB. files of 2GB in segments.

using script to splice them all together to tmpgenc I use following

720x480 @ 16:9 non interlaced, 23.976 internally (externally 29.97), 2pass vbr 4550 avg, max 8000, min 2000. motion search normal.

takes 30 hours to encode the 2 hour film

[ninja.rogue]

Ok, the riddle is solved.
You are using normal motion estimate search, while I am using highest quality.
Your speed should lower considerably when using my settings.
I believe you should put from two to three times more than what you are getting now.
But trying is the best solution.