Better Multicore / Multi CPU support anyone ?

[Darksoul71]

It´s really amazing that we see dual core CPUs below 200.- € these days. In earlier times
dual CPU system were something only a few of us could buy / were willing to buy. Now
nearly each new system in the >= 800.- € category sports a DualCore system. And this
trend will continue. I would not be surprised if we see quad core systems in the medium
price segment in two or three years.

I´ve written a simple AutoIt script called QuEnc^n which splits up an AVISynth script
into "n" segments and starts "n" instances of QuEnc for encoding. Thus you are able
to run QuEnc "multi-instanced" rathern than multi-threaded. Revgen was so kind to do
some tests on his beefy AMD 4600+ X2 Dual-Core. Those were the results for full D1
29.97i interlaced:

Quenc 0.70 - single instance: 1006 Secs 8.925 Total FPS
Quenc^n using QuEnc 0.70 - dual instances: 608 Secs 14.768 Total FPS

This is a performance boost of ~ 65%.
Here is the original posting:
http://forum.doom9.org/showpost.php?p=825589&postcount=12

His tests with progressive material also revelead that one can expect over 80 % speedup
depending on your system specs, filtering used, etc.
See here: http://forum.doom9.org/showpost.php?p=826001&postcount=16

I´m aware that this "multi-instance" approach is not new and was implemented a while
ago by Rumbah (DualDVDRB) and is now supported by jdobbs with DVD-RB pro.

While the "segmented encoding" approach has it´s downsides (lower quality compared
to one instance encoding, more I-frames) to me it still clearly shows how superior this
approach can be speedwise compared to multithreaded encoders. From what I´ve seen
even Canopus Procuder only shows up with something like 30% speed increase on
a dual core machine. That is not bad at all but much lower compared to the increase
you see with something like QuEnc^n and those who are that concerned with the drop
in quality can still run the encoder of their choice without segmented encoding.

Multithreading does not neccessarily mean speed increase if you have a dual core
machine. I´ve got numbers of BR7 from his new dual core P4 running AutoMatQuEnc
multithreaded and the results were more or less disillusioning for me. The speed "increase"
measured between running AQE single threaded and with 6 threads even showed speed decrease
of around 30% and a speed increase of 24% maximum. The average speed increase when
running AQE MT was only 0.8% ! I discussed this with SAPSTAR and he indicated that
the MT code also adds some overhead which can easily eat up the speed increase or
even turn into the opposite. While the MT implementation shows speed increase on
SAPSTARs machine (a single core P4 with HT), users with dual-core machines without
HT would be better off to run two separate instances of AQE in singlethread mode.
I wouldn´t be surprised if they see similiar speed increases as reported above (> 60%).

Although I´ve never touched a line of code for an MPEG2 encoder I think the main
reason for relatively low speed increase you get from MT implementations is simply
the fact that you / the developer is not able to parallelize any aspect of the encoding
process. So if he is able to enable multithreading for 20% of his encoding process
and sees 100% speed increase, this would result in 10% overall speed increase
for the complete encoding process only.

Segmented encoding is rather the KISS approach (Keep It Stupid Simple) for
parallelizing encoding. Since you simply start multiple encoding processes you
could expect up to 100% increase in theory. Of course there are internal limits
since both cores share the same infrastructure (cache, HDD controller, RAM)
but in general 80% and more are realistic.

Since I do not own a dual core CPU currently I rely on other people owning
such machines to do the testing but to me "#´s don´t lie !" :)

My script implementation is rather crappy but I think of QuEnc^n more as
of a "proof of concept". While the multi instance encoder feature of DVD-RB Pro
is nice, it´s limited to DVD-RB. Thus not the optimal solution.

Wouldn´t it be nice to have "^n"-support integrated within the encoder itself ?
One could even remove the drop in quality which comes with my external approach.
The encoder could split the movie in "n" segments for writting the log file. Afterwards
the encoder knows how much "bitrate" needs to be assigned to each segment and
can (relatively easy) fork out multiple instances of the encoding function. As the encoder
also detects all scene changes the final segments could be based on GOPs.

I can see no real reason why a segmented encoding process results in a quality drop
when the implementation is integrated encoder.

Since we have now many freeware encoders which can rival qualitywise with CCE,
Procoder and TMPEGEnc, I could imagine that good support for multicore CPUs
will be an extremely valuable feature. Why should someone buy CCE when HCEnc(^n)
or QuEnc(^n) reaches the same encoding time at identical quality on a dual core machine ?

Any comments / thoughts ?

BTW: Those of you who want to do the math themselves can grab the AQE logfiles provided
by BR7 themself here: http://www.vmesquita.com/forum/index.php?topic=6737.msg45264#msg45264

[foxyshadis]

Avisynth is actually one of the primary limiters of multithreading encoders as well, which is a strong argument in favor of multiprocesses, until multithreaded avisynth is officially released, since Avisynth is used as the source to pretty much everything now.

[Revgen]

@foxyshadis

TSP's SetMTmode filter works pretty well with most of the filters I use for Avisynth. I've hadn't had a problem with it unless I'm using some convoluted script like MVBob.

Even if SetMTmode doesn't work the old MT.dll can usually works well in it's place.

@DarkSoul71

I haven't been able to test AQE yet. I've been very busy with other projects. I might have some time at the end of the week.

[foxyshadis]

It works great for you, but I use 2.5.7 cvs builds, so I don't have tsp's alterations available. :p (And the majority of users wouldn't have it unless you specifically recommended using it with the encoder download.) And I don't have his source to try patching it in.

[Mug Funky]

wasn't there talk a while ago about the possibility of an encoder splitting p and b frames into 2 cores...

so with a GOP structure of: IBBPBBPBBPBB, you'll get:
- core 1 doing IPPP
- core 2 doing BBBBBBBB

that's not the most efficient, but it's also possible to throw a couple of B frames to core 1 i'd imagine, like:
I-P--P--PBB
-BB-BB-BB

which gives 6 frames to each core, and wouldn't have any dependency issues (if core 2 is a few frames behind core 1 then the p and i frames will be available for the b-frames to predict off).

maybe hank, or nic/dragongodz could look at it... but i doubt this stuff will happen much until more developers have dual cores. though they're getting cheaper, they're not necessarily in everyone's reach.

[Revgen]

I'd say Dual-Core's are definitely within reach. AMD 3800+ are selling for $300. This could definitely fit into a Sub-$1000 prebuilt PC.

EDIT

err..I forgot. They are probably higher in Austrailia right?

[dragongodz]

that's not the most efficient
umm can i say major understatement ? ;)
splitting to a GOP per core on the second pass would be better but even that would have certain issues making it less than perfect and very hard to get right. so spliting a video in 2 sections on second pass would be better still really.

while this is all possible it would also be a LOT of work. so do not expect anything like this from me atleast.

err..I forgot. They are probably higher in Austrailia right?
a quick estimate from 1 place for you.
cpu = $435(socket 939)
motherboard = $130
512k memory = $60(cheapo brand)
pcie nvidia 6200 = $90(which would be a step down from my 6600)

sorry but i dont happen to have a spare $715 to spend. i have a family with a mortgage and other bills to pay.

[ronnylov]

Sounds like a good idea to implement multithreading functionality in freeware encoders. I have an AMD Athlon 64 X2 3800+ (overclocked to 4600+) and it is much faster on commersial multithreaded encoders like TMPGEnc compared to my other PC with a single core 3400+.

Another (probably much more difficult) enhancement would be if it is possible to use GPU support for encoding like the ATI avivo is supposed to do. There are some avisynth filters (not many though) which are available in GPU accelerated versions.

I have done some encoding with HC encoder 0.18 and my CPU is only loaded 50 % which indicates that only one thread is used. I did not use any multithreading functionality for avisynth nor any GPU filters in avisynth.

[foxyshadis]

Ahaha, Avivo, the joke of encoding. It makes Xing look like a good mpeg encoder, ten years later. The only claim to fame it has is speed, and x264 or xvid (and probably any mpeg2 encoder) can easily be made faster and better at the same time.

dragongodz, would you be willing to spend any time on supporting splitting the video into discrete parts to run both passes? Even if it was only as complex as compositing stats files and making quenc^n do the hard work.

[Inc]

Ahaha, Avivo, the joke of encoding. It makes Xing look like a good mpeg encoder, ten years later. The only claim to fame it has is speed, and x264 or xvid (and probably any mpeg2 encoder) can easily be made faster and better at the same time.Thank you for that info as I was about to buy a x1300 card just to try out that Avivo support when upgrading to a PCIe system.
So if I gonna buy a PCIe Board Ill get a cheaper GraCard as also now my Radeon 9250 is fast enough for my purposes.

Maybe such an Avivo supporting Card makes at least sense on a HTPC for fast HD playback?!

BTW. in here in germany the price of the x2 3800+ is about 180€
http://www.geizhals.at/deutschland/a166487.html

[Adub]

Remember, AMD announced a huge price drop a few weeks ago. Rumor has it that the new prices will go into effect around the end of july.

For example the AMD X2 3800+ will be about $160.

[dragongodz]

use GPU support for encoding like the ATI avivo is supposed to do.
for anyone who didnt get the remarks by people after this, the last time avivo was tested it was found to be very fast but absolutly shocking(as in really bad) quality output. also it was found while there was a card check, so it would run only on ATI cards unless that check was hacked out, it was also found it didnt actually use the gpu.

would you be willing to spend any time on supporting splitting the video into discrete parts to run both passes?
not currently. maybe in the far future when i feel QuEnc is more complete. i would envision normal dual-cpu support coming first.

For example the AMD X2 3800+ will be about $160.
and then add on all the other hardware i listed needed for someone like me to upgrade. it does get quite annoying when people just quote the cpu price and ignore that for some of us there is more to buy than that.

[Inc]

it does get quite annoying when people just quote the cpu price and ignore that for some of us there is more to buy than that.
OT/
Thats why for example there do exist nice Mobos like the Asrock 939DualSATA which does still support DDR1 and AGP! And its proofed to be very stable. So you only would have to buy the CPU and the Mobo = $160 + $50 = $210 in fact.
As you can still use your rams and your graphics card. Ok you would have to invest also into a new CPUHeatsink which costs about 20$ if your heatsink wasnt already delivered including adapters for diverse sockets.

Anyhow ... $230 in fact is much money which could better be invested in nice senseful things like kids/family ;)

/OT off

[Darksoul71]

Hi Dg,

agreed, for some of us upgrading to a dualcore machine requires mucho $$$ / €€€.
Speaking for myself I have S939 mobo which would allow me to upgrade to a X2 3800+
as soon as the price drops below a certain value. Others may need to upgrade everything.

My main points simply are:
- upgrading to a dualcore system can be suprisingly cheap very often.
I´ve seen an upgrade bundle for 300€ like this:
INTEL Pentium D 805 Dual Core 2.66 GHz
ASUS MoBo P5ND2-SLI
1024 MB DDR2 RAM PC-667 from MDT

Of course this would require you have a PCI-X graphic adapter, which I do :)

- almost any system I see shipping in the 1k€ price range (something I consider
the medium price range for PC systems) features dual core.

- it´s expectable that we´ll see dual-core cpu´s even in the entry CPU level (Sempron /
Celeron / whatever they will be called then) over the next year(s).

- I´ve seen a lot of postings asking about dualcore support for encoder A or application B.

Currently I neither own a dualcore, nor do I plan to buy one in the next half year. Nevertheless
I´m very interested in the area of multicore / multinode encoding applications. Technically and
implementationwise there is no big difference between coding something like QuEnc^n or a
network distributed encoding solution. At least this was my impression from my latest coding
experiments.

I for one can live with "single-threaded" implementation of any freeware encoder out there
since I have the possibilities to add a multi-node encoding approach writing and using external
tools.

For me the main reason for doing my inital posting is simply the fact that the only two real
problems I see with my external approach should be easily solvable if you integrate "my"
implementation into any encoder out there.

Problem 1:
I don´t pay attention to scene changes. Thus using QuEnc^n will produce at least one
additional I-frame per segment start compared to doing a normal "one segment"
encoding. Since each encoder has it´s own scene change detection algorithm adjusting
the segment borders to scene changes is only possible using the same algorithm with the
same settings.

Solution 1 if implemented in encoder:
Using the internal SCD algorithm of the encoder for adjusting the segment borders to
a scene change will ensure that no additional I-frames are produced by segmented
encoding.

Problem 2:
The segmented, independent encoding causes a drop in quality (I can and will not discuss if
this is really visible on medium sized TV screen at reasonable bitrates) based on the fact that the encoders do not take possible different complexities for each segments into account. I try to compensate this by doing a CQ encoding for each segment and adjusting the average bitrate accordingly.

Not perfect but at least a countermeasure :cool:

Solution 2 if implemented in encoder:
The encoder can adjust the bitrate for each segment according to the internal rate control
algorithm by taking the informations of the complete movie in account (versus ony taking
the information of each segment into account).

Implementationwise my thinking might seem quite naive to any of you encoder developers but
while truely adding multithreading to any encoder out there clearly requires a lot of work
I simply thought that integrating the functionality of a few lines of autoit code into an encoder shouldn´t be so hard to do. Obviously I was wrong.

One thing that should not be forgotten:
Both multithreaded and multi instance encoding support can be developed without really owning
a dual-cpu / dual-core machine. You simply do not see the real-life speed increase, but hey...
there are enough dual-core users out there willing to test and provide feedback ;)

For me coding QuEnc^n (and all the other stuff that is currently in the work) is
really only for fun. I´m not in a hurry anymore and if my encoding solution is able
to encode two movies captured as MJPEG from TV to DVD-compliant MPEG2
material on my A64 / 3200+ I´m happy.

If I would require faster encoding I could simply finish my network encoding script,
switch to CQ encoding with size prediction using AQE or fall back on CCE (which I dislike since there are free alternatives available which provide at least equivalent if not superior quality).

Stuff like QuEnc^n is more an "investment in the future" *ggg*

Finally I would like to add that this posting shouldn´t sound like a feature request for you, hank
or any other freeware encoder out there. It´s more "thinking aloud !"

Keep up your good work !

Kind regards,
D$

[dragongodz]

Thats why for example there do exist nice Mobos like the Asrock 939DualSATA
the problem there is future upgradability. that is it would be good for now but in a year or more i could very well have to upgrade the motherbaord again plus all the extras. so in the end would actually cost more.

this looks like it also could be worth waiting for, for people wanting to get in to dualcore at a more budget level.
http://www.theinquirer.net/default.aspx?article=32717

I simply thought that integrating the functionality of a few lines of autoit code into an encoder shouldn´t be so hard to do. Obviously I was wrong.
to do it right then yes i think there it would be quite complex. for example lets quickly look at splitting on a scene change. say we are splitting the source in 2, for simplisity, so you would need to check something like 100 frames around the estimated split point and find a good candidate, such as the frame with the highest amplitude of change. doesnt sound much right ? well i can tell you that would be more than just a few lines to add all by itself. :)

Finally I would like to add that this posting shouldn´t sound like a feature request for you, hank
or any other freeware encoder out there. It´s more "thinking aloud !"
ye no problems. i am just trying to point out that while it may seem straight forward and not so complex actual good implamentation would be more work than people may realise.

[foxyshadis]

Of course this would require you have a PCI-X graphic adapter, which I do :)
Whoa, a server graphics board? =o Oh, I see, it's just PCIe. PCI-X is Extended PCI, a partly compatible faster PCI. (Whereas PCIe is entirely different and supercedes both.)

My biggest issue with network-based parallelism is that the network is usually the bottleneck when I've tried (and it adds quite a bit of cpu overhead as well). So you have to at least have the sources on each machine, or have them split and sent ahead of time. I don't have a gigabit hub yet, maybe that would help (the systems all have it). Protip: Don't even bother trying over wireless, all I got for my trouble was a crashed router.

A neat tip to speed up avisynth on dualcore that I just thought of (if you're like me and need to use current cvs): TCPdeliver and tcpsource in two scripts, but run on the same machine! No network latency! =D Thanks for jogging my mind on that. (Don't turn lzw/zip compression on!)

I think #1 is only a problem if you split into hundreds of files, it's neglegible for 2-8 pieces, and problem #2 is solved with a stats parser to stitch and split the files as necesary, so you can run the first pass on the same farm and have all the information you need to split the job. ELDER did it this way, in perl. The trouble is just writing it (I half-finished one for x264 before drifting away from the project).

btw, gave ELDER a try on the new 12-core Opteron farm at work... scary fast.

[Darksoul71]

ye no problems. i am just trying to point out that while it may seem straight forward and not so complex actual good implamentation would be more work than people may realise.
I second that ! It´s clear to me that my "QuEnc^n" scripting stuff is more "quick and dirty" and a more clean implementation requires some more work. Since I never touched any encoder source code I have no good understanding how complex such a change may be. Thus I thank you even more for posting !

Whoa, a server graphics board? =o Oh, I see, it's just PCIe. PCI-X is Extended PCI, a partly compatible faster PCI. (Whereas PCIe is entirely different and supercedes both.)
Mate, I feel major stupid ! Of course I was talking about PCIe :cool:
My mistake....

PCI-X was this long 64 Bit PCI bus, right ?

My biggest issue with network-based parallelism is that the network is usually the bottleneck when I've tried (and it adds quite a bit of cpu overhead as well). So you have to at least have the sources on each machine, or have them split and sent ahead of time. I don't have a gigabit hub yet, maybe that would help (the systems all have it).
Hm, that´s strange. In my experiments with an early version of LAN_AQE-RB (a tool similiar to RB Farm which supports AutoMatQEnc) I experienced only 10% network load when encoding a DVD with two nodes. My estimation was that a standard 100 MBit connection (incl. hub / switch) should be capable to support up to 8-9 nodes in a network prior maxing out. And you also know that it´s very depending on your network adapter, the driver, etc. wether network load adds a lot CPU load to your system. Example my NForce 2 network adapter eats up a lot more CPU power compared to the 3Con network card I have in the same system.

In regard to #2: Unfortunately I have no clue how to read, analyse and adjust the log files of QuEnc. I don´t even know where QuEnc stores its log file when doing 2 pass.

btw, gave ELDER a try on the new 12-core Opteron farm at work... scary fast.
Yummie, can I borrow such one ? Of course only for...errr....experiments and LAN_AQE-RB development. LOL....