Debugging plugin DLLs?

[wonkey_monkey]

What's the best/easiest way to debug a plugin DLL while you're developing it?

I've got a few unreleased plugins that are in various stages of development and exhibiting those annoying "20% of the time" crashes. When I'm compiling a stand-alone program, I can just hit F5 in Visual Studio and it'll run it in its own environment and can report back to me if it crashes, taking me to the offending line.

I recently resorted to writing a C++ program which #includes my plugin's code, creates an Avisynth environment, calls AddFunction to add the plugin at runtime, rather than as an auto-loading DLL, then either imports a suspect script of uses invoke to call the offending functions until I get a crash.

Is there a better way of doing all this? I just want to know which line or even just function in a plugin's code is causing a crash and possibly to inspect a few variables.

[videoh]

Open and compile the plugin in Debug mode in VS.
Make a script that loads the Debug version plugin from [project]/Debug/plugin.dll. (Make sure
there is no autoloading version.)
In Debugging properties, specify VirtualDub2 as the command to run.
Ensure exception handling is enabled.
Do Start Debugging and VirtualDub2 opens.
Open the script in VirtualDub2 with File/Open video file...
Play.
A crash due to an exception will stop you in the debugger at the location.
Set appropriate breakpoints as needed for further information on the cause.

[wonkey_monkey]

Thanks, I didn't think it would be that easy! Computers eh...

Edit: Great, it's one of those bugs that disappears when you add more code to find it. In fact the code that the debugger says is to blame doesn't even seem to be getting called... ah well.

[videoh]

In fact the code that the debugger says is to blame doesn't even seem to be getting called... ah well. Doesn't sound right. Set a breakpoint and then start stepping until it crashes. The last line you step over is the culprit. If it is a function call then step into it and continue.

[wonkey_monkey]

Turns out I hadn't updated my Debug build steps to copy the DLL into the Avisynth plugin folder. I guess Visual Studio was just doing something undefined and taking me to some unrelated line of code.

Of course now that I've fixed that, the debug version isn't crashing at all...

[videoh]

It crashes only in release mode?

https://stackoverflow.com/questions/186237/program-only-crashes-as-release-build-how-to-debug

I typically use OutputDebugString() as if it is printf. If you run in the debugger they will appear in the output window. Otherwise run it externally and use DebugView to watch the output.

[kedautinh12]

Hello @videoh, can you mod blindPP to HBD and more colors space support?? I think it's need for modern days. Thanks
https://forum.doom9.org/showthread.php?p=1943548#post1943548

[GMJCZP]

Hello @videoh, can you mod blindPP to HBD and more colors space support?? I think it's need for modern days. Thanks
https://forum.doom9.org/showthread.php?p=1943548#post1943548

I join the initiative, +1!

EDIT: Although the initiative is good, sorry if I wrote this here, I did not have a good night and I distracted.

[wonkey_monkey]

I've labelled my clips with a debugging filter that outputs debug strings whenever GetFrame is called. Apparently there is heap corruption which almost always occurs once it gets to the source filter, which is ffmpegsource2, but occasionally crashes elsewhere or (once) got stuck in an infinite loop where none should happen. I'm guessing I'm messing up the heap and it's not causing a problem locally enough for me to figure out where right away. Time to trawl my code...

Edit: turns out VS was right the first time. Even though Avisynth was using the Release DLL, it was still able to work out which line of code was causing the heap corruption (it couldn't show me the value of the variable that was to blame, though). No idea how it managed to do it.

[vcmohan]

I am not adept at going through all this. I always use the release version. If the code crashes, I will introduce the handy
env->ThrowError(" here x = %d y =%5.3f....", x,y,...);
I first check completion of create plugin section and see whether variables are as expected. If OK I delete this and place at end of the constructor. If crashes place this above at a place by judging.
If constructor is ok, I place this with suitable variables in progressively deeper in GetFrame section. This way I can check at every stage what are the values of variables and where its going wrong. If within a loop I use if (h == a && y == b) env->ThrowError("...",); So I come to know values of variables at various stages. Only problem is each time I nchange, need to compile agai. Hardly takes a minute. I open in vitualdub the script file to see the error being reported.

[StainlessS]

For an easy compile/nocompile DPRINTF(), [switched on/off by presence of BUG define].

in some header [I use Compiler.h, below for Visual Studio 2008+]

#ifdef BUG
// Call as eg DPRINTF("Forty Two = %d",42) // No ending Semi colon
// C99 Compiler (eg VS2008)
#define DPRINTF(fmt, ...) dprintf(fmt, ##__VA_ARGS__); // No Semi colon needed, will be auto appended
#else
// C99 Compiler (eg VS2008)
#define DPRINTF(fmt,...) /* fmt */
#endif

extern int __cdecl dprintf(char* fmt, ...);


Switch it on/off by line in CPP, Before including above header.

#define BUG // Allow DPRINTF() style stuff, comment out removes DPRINTF()'s from binary. (requires DebugView)


Also later in same CPP,

// Use as in:- dprintf("Hello %s",__DATE__); // Produces eg "Hello Oct 23 2017".
int __cdecl dprintf(char* fmt, ...) {
char printString[2048]="Example: "; // MUST NUL TERM THIS, minimum "", or eg "Test: "
char *p=printString;
for(;*p++;);
--p; // @ null term
va_list argp;
va_start(argp, fmt);
vsprintf(p, fmt, argp);
va_end(argp);
for(;*p++;);
--p; // @ null term
if(printString == p || p[-1] != '\n') {
p[0]='\n'; // append n/l if not there already
p[1]='\0';
}
OutputDebugString(printString);
return int(p-printString); // strlen printString
}

Can then use

dprintf("Hello %s","fred"); // MUST have trailing semicolon. always in binary.

or

DPRINTF("Hello %s","fred") // NO trailing semicolon, and output compiled into binary via switch, #define BUG.



Also, here some meanderings originally posted here:- https://forum.doom9.org/showthread.php?p=1877934#post1877934
I have re-formatted as looked a bit wide and hard to read on 4:3 monitor.
Related to heap corruption due to bad string handling, probably the hardest of all bug types to locate and squash.

malloc/new

Library writers long ago found that it is inefficient to allocate small blocks of memory, as that
produces memory fragmentation, and so slower finding of suitable sized mem block.
So, was usual in C library, when user requested block size of eg 1 bytes, to carve off 8 bytes, and
hand pointer back to user. Only 1 byte of this block belonged to the user,
but there would be no error encountered if user used up to 8 bytes as carved out from free memory lists.
Memory size returned was rounded up to a multiple of 8 bytes, so requesting 1 to 8, would carve out 8
bytes, 9 to 16, 16 bytes etc.
An additional bonus of this round up to 8 bytes, was that the memory block when free'ed, there was
always enough room in that block to create a link in a linked list, for use in the free memory list
arrangement. The free memory block kept track of 'itself', without any additional overhead, and mem
fragmentation was much less and also faster to find block of sufficient size. [link list struct would
hold 32 bit int size of mem block, and 32 bit pointer to next link in list, both 4 byte values on 32
bit system].
Above round up to 8 bytes also enabled the ANSI spec that malloc() returns a pointer to mem bock
that is castable to a memory block structure of any type [or words something like that],
ie, some structures need be located on a memory boundary of maybe 2 bytes, 4 bytes or 8 bytes.
If the initial heap base pointer is rounded to multiple of 8 bytes, and all mem blocks are a multiple
of 8 bytes, so all malloc blocks will return a mem bufer on 8 byte boundary.

Unfortunately, above used to cause novice C proggers to cock up when allocating string buffers, for
small strings (less than 8 chars), as there were no problems produced when forgetting to add 1 bytes
for the nul term sentinel (zero byte) at end of string. This due to habit of writing C examples
using "fred" and "ted" sized strings.
Novice C proggers would get the impression that it was unnecessary to add that 1 extra byte for
the nul term because they never experienced any errors, but of course this is bad, and perhaps the
hardest of all bugs to track down.

Imagine you created a DBase for some project, and that it held Name field, and Title field, and
Address etc, but that in the title field, you forgot to add the 1 byte for nul term.
Your DBase might go on seemingly OK for 5 or more years without any trouble at all, then one day
you get new accound created for title "Princess", exactly 8 bytes in length, but requiring 9 bytes
in total for the buffer. you ask for 8 bytes, and this time you actually need 9, kaboom!!!, you
blast a byte that does not belong to you, belongs to some other structure or maybe even a link in
the free memory list. As you repeatedly save and load (at day end/start) the DBase, each and every
time, the problem is likely to move about, causing
corruption in differeing parts of your DBase, and eventually it will be noticed and become a problem.
Where do you start to look for the problem ???, you got 5 years spare to keep tabs on your next newly
created DBase?

Because of the C libs malloc/calloc/new etc optimisation stuff, you have to be paranoid about that 1
extra byte, and suggest always allocate string buffers via a single function which auto adds the
required extra nul term byte. [look up the non ANSI function 'strdup()' and knock it up yourself if
not supplied with your compiler [STRICT_ANSI (or whatever the define is) will exclude it even if it
does exist in your compiler].

For x64, it is most likely [ie definite] that the 8 byte fragment size mentioned above will be 16 bytes,
so if you request 1 bytes from malloc, you will actually get 16, making the above mentioned failure to
add 1 byte for nul term on string buffer, an even bigger problem for string buffer bug hunting.

Also, a little aside,
Some C docs for calloc() seem to have ommitted the fact that calloc will auto zero memory [where malloc
does not], think maybe MicroSoft (or someone, forgot about this more than insignificant fact and just
missed it out), and everybody else copied the erroneous docs. Perhaps your compiler mentions this clearing
of memory by calloc, perhaps not [somewhere about year 1992->2000, they all/many [cross platform] seemed
to have forgotten about it].


EDIT: I've googled "calloc", and checked 6 results from the links, and it seems that
everybody now documents calloc as zeroing the contents of the returned memory block,
so at least they fixed that docs omission from years ago. [concerning last paragraph of above code block]