Tons of threads created unnecessarily

[MysteryX]

I got this plugin here: Shader
https://github.com/mysteryx93/AviSynthShader/blob/master/Src/Shader.cpp

If I disable MT and use this script

AviSource("Preview.avi", audio=false, pixel_type="YV12")
ConvertToFloat()
#Shader(PluginPath+"GammaToLinear.cso")
#Shader(PluginPath+"LinearToGamma.cso")
ConvertFromFloat()

This creates 12 threads. Then, for each call to Shader, it creates 8 additional threads. Which means that if I run MT=8 and have 10 Shader calls, it creates 640 threads!!

Why is it doing this? I don't have any code within the plugin to create threads, so I really wonder where that could be coming from.

Any idea?


... or I could leave it that way; burning calories

[Groucho2004]

Why is it doing this? I don't have any code within the plugin to create threads, so I really wonder where that could be coming from.
How about using Process Explorer to look at the thread details of the process?

[MysteryX]

There is the main thread followed by 3 ntdll.dll, then 7x d3d9.dll each followed by 2 ntdll.dll

So these threads are created by DirectX 9 somehow

[MysteryX]

When creating the device, removing behavior flag D3DCREATE_MULTITHREADED and adding D3DCREATE_DISABLE_PSGP_THREADING slightly increases performance, but I still get all those threads created. However, when debugging, I'm not seeing all those extra d3d9.dll threads anymore; only when running it live.

[MysteryX]

It is becoming apparent that the question is related to DirectX, not AviSynth.

[foxyshadis]

At the point when you just don't know wtf is going on, it's time to liberally sprinkle OutputDebugString() or fprintf(stderr,...) around your codebase to look for magical fun thread-spawning statements. Throw in __FILE__ & __LINE__ so you can see from where they come. Here's a function to return the threadcount at any point. (http://stackoverflow.com/a/3750019/321935)

[StainlessS]

Further to Foxy's suggestion


...

#include <stdio.h> // for OutputDebugString()
#include <tlhelp32.h> // for GetCurrentThreadCount()

....


// Somewhere after includes

#define BUG // Comment out to disable all debug output

#ifdef BUG
int dprintf(char* fmt, ...) {
char printString[1024]=""; // MUST NUL TERM THIS // EDIT: at some point before va_list argp call
char *p=printString;
const char *nam="MysteryX: "; // EDIT: static name optional, comment out 3 lines
while(*p++=*nam++);
--p; // @ nul term
va_list argp;
va_start(argp, fmt);
vsprintf(p, fmt, argp);
va_end(argp);
for(;*p++;);
--p; // @ null term
if(printString == p || p[-1] != '\n') {
p[0]='\n'; // append n/l if not there already
p[1]='\0';
}
OutputDebugString(printString);
return p-printString; // strlen printString
}

// OLD C89 Compiler: (eg VS6)
// Call as eg DPRINTF(("Forty Two = %d",42)) // Enclosed in double parentethis, No Semi colon
//#define DPRINTF(x) dprintf x;

// Call as eg DPRINTF("Forty Two = %d",42) // No Semi colon
// C99 Compiler (eg VS2008)
#define DPRINTF(fmt, ...) dprintf(fmt, ##__VA_ARGS__); // No Semi colon needed
#else
// OLD C89 Compiler: (eg VS6)
// #define DPRINTF(x) /* x */

// C99 Compiler (eg VS2008)
#define DPRINTF(fmt,...) /* fmt */
#endif

...

// Foxy's link

int GetCurrentThreadCount()
{
// first determine the id of the current process
DWORD const id = GetCurrentProcessId();

// then get a process list snapshot.
HANDLE const snapshot = CreateToolhelp32Snapshot( TH32CS_SNAPALL, 0 );

// initialize the process entry structure.
PROCESSENTRY32 entry = { 0 };
entry.dwSize = sizeof( entry );

// get the first process info.
BOOL ret = true;
ret = Process32First( snapshot, &entry );
while( ret && entry.th32ProcessID != id ) {
ret = Process32Next( snapshot, &entry );
}
CloseHandle( snapshot );
return ret
? entry.cntThreads
: -1;
}




And use as in eg

DPRINTF("Threads=%d Line=%d File=%s",GetCurrentThreadCount(),__LINE__,__FILE__)


Just comment out the #define BUG thing to disable it all.
DPRINTF() configured for VS 2008 +, must use double parentesis for VS 6 config (EDIT: Need comment/uncomment lines in #ifdef BUG and #else).

EDIT: Need DebugView: https://technet.microsoft.com/en-us/library/bb896647.aspx

EDIT: With supplied snippet can filter out only your debug messages in debugview via Filter on "MysteryX:".

[MysteryX]

Great, thanks. I haven't tried your code yet, but looked further into it.

Creating the DX9 device with flag D3DCREATE_DISABLE_PSGP_THREADING removes the extra threads... at least when running in VirtualDub with Release build and pausing in Visual Studio.

When I run this script
AviSource("F:\AVSMeter\Preview.avi", audio=false, pixel_type="YV12")
SuperRes(2, 1, 0, true, """nnedi3_rpow2(2, cshift="Spline16Resize", Threads=1)""")

In VirtualDub, I get 17 threads when I Pause in Visual Studio.

Running the exact same script with AvsMeter gives me 37 threads!?

What the heck

[MysteryX]

I found out that the difference is whether the application is configured as "High Performance" (Radeon 7670M) or "Power Saving" (Intel HD 4000) in ATI's dual-graphics control panel.

If using the Radeon 7670M, my full script creates 471 threads, processes ~4.3fps @ 67% CPU usage, and has memory usage of 1170MB phys / 1723MB virt

If using the Intel HD 4000, my full script creates 150 threads, processes ~4.5fps @ 67% CPU usage, and has memory usage of 1604MB phys / 2189MB virt


I find it weird that the memory usage is higher when the amount of threads is lower.

[Groucho2004]

Etienne,

I suggest you remove the link to the blog in your signature. The rules forbid spamming and your blog clearly falls into that category.
It may look innocent at first sight but almost every link leads to a page with a big fat "add to cart" button where you try to sell your "spiritual services".

[MysteryX]

It definitely isn't spamming but I agree this isn't the best place for it. I removed it.