MT() script not capable of using 2 cores when used in HCEnc

[Malcolm]

i just got my new c2d and quickly started to play with MT() and HCEnc.

I have a script starting with SetMTmode(2) followed by some heavy processing. When i run this script inside Virtualdub, nearly 100% CPU time is used (=both cores). when i run the same script in HCEnc, encoding it to mpeg2, only 50% of the CPU time is used (=one core).

Why do Virtualdub and HCEnc behave differently? Is it possible to request frames from avisynth in different manners?
Is it because HCEnc requests one frame, encodes it, requests the next frame, encodes it, and so on? All single threaded?
Is Virtualdub simply multithreaded and that's it???

Maybe Hank315 only has to change 2 lines of code and HCEnc is able to use both cores like Virtualdub? :rolleyes:
Am i the only one who sees this behaviour with HCEnc? :confused:

[Mug Funky]

probably the best way to get multithreading in HC is to help finance a new motherboard for hank :)

[Malcolm]

seems to me too... ;)

But seriously: shouldn't the multithreading in avisynth via mt() work completely separate from HCEnc beeing able to encode multithreaded or not?

[tsp]

seems to me too... ;)

But seriously: shouldn't the multithreading in avisynth via mt() work completely separate from HCEnc beeing able to encode multithreaded or not?

it should yes. Maybe using getmtmode to see if multithreading is actually enabled in the script in HCEnc might be an idea

[Malcolm]

it should yes. Maybe using getmtmode to see if multithreading is actually enabled in the script in HCEnc might be an idea

Thanks for the tipp! actually i haven't thought of that yet. checked it, but it's 2 threads + mtmode 2 just like in Virtualdub. still it uses only one core...
But here are 2 screenshots from Process Explorer showing VDub and HCEnc while playing / encoding the MT() avisynth script.
http://img164.imageshack.us/img164/719/vdubmtyh1.th.png (http://img164.imageshack.us/my.php?image=vdubmtyh1.png) http://img103.imageshack.us/img103/6121/hcencmtzo0.th.png (http://img103.imageshack.us/my.php?image=hcencmtzo0.png)

As you can see, there are 2 threads spawned in VDub with each slightly under 50% CPU time. Obiously avisynth processing the video in parallel.
In HCEnc, there is NO thread spawned at all. just 4 threads inside HCEnc and one of them uses 50% CPU time.
So, HCEnc seems to deal with avisynth in a totally different manner.

[IanB]

Probably all the CPU goes into HCEnc code and a trivial amount in Avisynth code. So multithreading a 1% target saves you 0.5%

[Malcolm]

@IanB
i'm not sure if i understand you correctly, but to give you some numbers:
Using the .d2v from the mentioned video in HCEnc directly (no avisynth involved), i get average fps of 82 while encoding! (single threaded).
Using the avisynth script in HCEnc, i get average fps of 5 while encoding! (single threaded).
So nearly all encoding time goes into the avisynth filtering (did i mention 'heavy filtering' in my first post? ;) ) and only a small amount goes into mpeg2 coding.
If i could use 2 cores instead of only one, the average fps would go to something like 9 or 10!
So since my 'target' really is >90%, my saving would be >45% of the whole encoding time ;)

[techreactor]

Multi threading is not supported in HCenc yet, probably hank will think about including it in the future.

[Leak]

So, HCEnc seems to deal with avisynth in a totally different manner.
What's the CPU affinity of the HCEnc process? If it's tied to one CPU it won't get more than 50% anyway, so MT probably won't bother spawning a second thread...

[Malcolm]

Multi threading is not supported in HCenc yet, probably hank will think about including it in the future.

This really isn't about multithreading in HCEnc, which would mean: parallelizing the internal algorithms for mpeg2 encoding. It's about multithreading in avisynth, which is accomplished by MT() and should be totally separate from HCEnc even if the avisynth script is used inside HCEnc. I guess the reason is that HCEnc uses avisynth.dll directly and not like Virtualdub, which i guess treats avs files just like avi files and let Windows do the stuff. (which seems to work)

I hope Hank315 will comment on this....

[techreactor]

when i run the same script in HCEnc, encoding it to mpeg2, only 50% of the CPU time is used (=one core).


Well it is, even if you are able to use the MT() script so as avisynth throughputs are on the fly, HCenc will not be able to use both cores to their 100%, since HCenc is not multithreaded. for eg people use multiple encoder option in DVD-RB to use both their cores to 100%, with avisynth.

Its like avisynth is trying to feed you two burgers(many frames) at a time but you(HCenc) can only eat one.

[Boulder]

When you use MT, the number of threads in the encoder process does increase so something is happening. It's just that it's not really useful most of the time.

[foxyshadis]

Well it is, even if you are able to use the MT() script so as avisynth throughputs are on the fly, HCenc will not be able to use both cores to their 100%, since HCenc is not multithreaded. for eg people use multiple encoder option in DVD-RB to use both their cores to 100%, with avisynth.

Its like avisynth is trying to feed you two burgers(many frames) at a time but you(HCenc) can only eat one.

If that was the case, multithreading avisynth would never be useful in any situation and thus wouldn't exist. In fact, in this case HC spends most of its time waiting on avisynth to deliver a frame, so something is clearly blocking it. Avisynth should be creating secondary threads entirely on its own, without any interference from HC, even if an affinity is set, or should just error out if it's an unsupported version.

[Malcolm]

When you use MT, the number of threads in the encoder process does increase so something is happening. It's just that it's not really useful most of the time.

foxyshadis is right!

@Boulder
As you can seen in the screenshots from Process Explorer, there are no threads created when i use the mt() script from within HCEnc, whereas there are 2 threads created when i use the mt() script from within Virtualdub.

[Leak]

@Boulder
As you can seen in the screenshots from Process Explorer, there are no threads created when i use the mt() script from within HCEnc, whereas there are 2 threads created when i use the mt() script from within Virtualdub.
Like I asked earlier - what is HCEnc's CPU affinity when it's running?

(I guess Process Explorer should be able to show it since Task Manager does so too, but since I currently don't have a multicore CPU in front of me I can't check...)

[Boulder]

foxyshadis is right!

@Boulder
As you can seen in the screenshots from Process Explorer, there are no threads created when i use the mt() script from within HCEnc, whereas there are 2 threads created when i use the mt() script from within Virtualdub.Do you use Vista? I'm not familiar with that screen. When I use MT with HC in WinXP, Task Manager's process list shows that the encoder process has more than 2 threads. It has 2 threads if MT is not used.

[Malcolm]

@Leak
Sorry! Did it, but forgot to post (too busy at work atm.) Process Explorer shows affinity to both cores for HCEnc, (both checkboxes are checked).

@Boulder
I have WinXP. See http://www.sysinternals.com for Process Explorer. The Windowdecoration is just a theme for WinXP.
Process Explorer shows 4 threads regardless if i use mt() or not. :confused:

@all
i just want to say thank you all for your contributions in this thread! even if there's no solution yet. :thanks:

[Boulder]

Try using MT with DegrainMedian. That is one filter which should be faster when you encode with HC using MT.

[techreactor]

If that was the case, multithreading avisynth would never be useful in any situation and thus wouldn't exist. In fact, in this case HC spends most of its time waiting on avisynth to deliver a frame, so something is clearly blocking it. Avisynth should be creating secondary threads entirely on its own, without any interference from HC, even if an affinity is set, or should just error out if it's an unsupported version.

Let me take an example to make things clear as there are two aspects here that we are talking about.

I have a dual core CPU and HC is not the only encoder, you can use many others which use multiple threads and 100% CPU during the encodes.

For eg. with same scripts in avisynth and same source/encode

Scenario 1 (avisynth + autoqmatenc)

I get 100% cpu usage due to the fact that AQMenc uses as many thread as your CPU can support.

Scenario 2 (avisynth + HCenc)

I get some 60-70% CPU usage since HCenc cannot use multiple threads and hence CPU usage is less

Now to segregate the problem you need to use the same software with and without avisynth and see if avisynth is the bottleneck or the encoder.

Depending on the filters used, the fps might go down considerably or might show a marginal decrease, therefore I would suggest using a light filter like removegrain() to test.

Here malcom is comparing oranges with apples (Vdub with HCenc) and assuming that the problem lies with avisynth wherein it could be otherwise.

[foxyshadis]

I understand that completely, but we're not trying to intuit what might be happening based on cpu usage and guessing about the script - we already know the script is by far the slowest element, and we know what threads have been spawned by HC.

I tested HC v21 and QuEnc and can confirm the results, didn't test v20:
HC, MT()
http://foxyshadis.slightlydark.com/random/hc-mt.png
HC, SetMTMode(2)
http://foxyshadis.slightlydark.com/random/hc-setmtmode.png
QuEnc, SetMTMode(2)
http://foxyshadis.slightlydark.com/random/quenc-setmtmode.png

Those extra avisynth.dll threads should be showing up in HC, as they show up in everything else you can load avisynth into. (Note that the nature of MT() is that the first thread is part of the primary execution thread, so MT()+HC show up as one thread and the other MT threads as their own threads. SetMTMode doesn't have this behavior, it operates entirely out of the primary thread, anything listed as HC is HC alone. Or rather QuEnc in this case.)

[tsp]

does using setmtmode(2,2) makes any difference? (it forces the use of two threads instead of autodetecting the number of cores available)
I will check this out some more when I get visual studio reinstalled