Memory leak in Compressibilty test?

[leadman584]

I'm posting on behalf of a few folks that have had a problem with compressibility test in newest release of GKnot. Apparently several people have had a problem with high memory usage during comp tests. The most notable is a gent who claims mem usage during comp test approaches 1 Gig. Has this issue been experienced by any other users?

[colordog]

It might help if you provide some more details - this is pretty information poor. What codec, and what version of codec, for instance, blah blah blah. Also, which 'memory' are you refering to?

I just did a 5% comp test with GKnot 0.28.6.2 with DivX 5.1.1. The VOB was 3.2 GB, amounting to 100 minutes of video. This was performed on WinXP Pro, with 1GB of RAM, and 1.5GB of swap space.

During the compression, GKnot occupied ~22MB of RAM, and VDubMod held constant at ~121MB. After the encode was over, VDubMod released its supply to the Available Physical Memory. GKnot continued to hold at 22MB.

The test compression file it generated (named Movie_5_perc_Divx5_sDeIntrlc_Lnoise_LanczosResize_432x256.log in this instance) was 407 KB, well under a meg.

Soooo..... don't know where you're getting that. Is this topic posted elsewhere in the forum?

[leadman584]

I'm still using an older version of GKnot which works flawlessly for me. A gent over at Divx.com was experiencing this prob and got a tad frustrated with the cooling off period here at Doom9.

The thread describing his situation is here:

http://forums.divx.com/viewtopic.php?topic=55880&forum=5

Maybe you can be of more help to him in this matter than I. You can probly get a better decription from him than I, concerning this situation. Thanks for you're time in this matter.

[Doom9]

the first order of business would be to establish a common procedure to reproduce the problem, then note which program is using the memory.. it might not be GKnot after all (and it's extremely easy to establish, just press alt-ctrl-delete, select the task manager, switch to processes, and there you go).

[colordog]

@leadman584

I checked out the DivX forum thread you provided - I'm almost wondering if you meant this as a joke, but I'm sure you meant it in good spirits. The thread you gave doesn't have any additional information other than "there might be a memory leak", but instead is filled with "boy, aren't those doom9 people an arrogant bunch?"

Not to detract from the real topic of this thread, but I have to disagree with your statement that it's arrogent to make people wait 5 days until posting, stating that 24 hours is enough time to make people wait to use the search button instead.

Here's some links to the forum in the within the last week, where someone did indeed wait the 5 days to post a question that was easily answered by searching - you'll note that many are related to the new speed definition of "standard" relating to the DivX 5.1(.1) codec.

http://forum.doom9.org/showthread.php?s=&threadid=65965
http://forum.doom9.org/showthread.php?s=&threadid=65752
http://forum.doom9.org/showthread.php?s=&threadid=65454
http://forum.doom9.org/showthread.php?s=&threadid=65450

Etc., Etc. In general, I really appreciate how this forum really encourages learning, and is supportive of people who provide feedback about problems (i.e., details), rather than treating this board as a place to dump their problems and frustrations with free software (i.e., a customer support department). Your 'notable gent' gives much more time discussing his unhappiness with the Doom9 forums than does describing his problem with Gordian Knot. It generates a 'reverse' frustration from his own perspective, when people here are flooded with "this won't work fix it now" messages, often because people won't / can't use a search feature (see last link above), and won't / can't provide detailed information on their problem (like a majority of computer users). I happily support this forum's 'arrogent' features in order to remove these types of posts, so that we're not constantly saturated "Why isn't this working?", "Well, what program version are you using?", "I Don't Know...", etc.

However, back to the original question at hand, like Doom9 said, there needs to be a level of reproducibility / detail level. You mentioned in your DivX forums thread that you thought the memory leak was "back" in GKnot? Was this an issue before? Off the cuff, I'd be surprised if it was GKnot, and not one of the bundled components, but who knows? I've used GKnot for about a year and a half now, and can't recall ever having a memory leak - I mean, I reboot my computer about once a month just for those damn Windows updates.

Perhaps you can see if you can tease anymore information from your 'reliable source'? I'm sure people here would be more than glad to look into it provided with a starting point.

[jggimi]

Searching the GKnot forum here with the word "leak" I found the following posts:

http://forum.doom9.org/showthread.php?s=&threadid=54334

This cites sluggish behavior of the workstation after compressibility tests with Gknot 0.28.2 beta, and poses the question if "resource leaks" can happen but not show up in the task manager. Manono responded with a search link for VdubMod leaks. That link is no longer valid, so if you search the VirtualDub & VirtualDubMod forum for "leak" you will find 11 posts in 9 different threads on memory leaks in the Vdub or VdubMod encoder, which, if you think about it, is the tool being executed by Gknot when you run compression tests or do encoding. That's assuming, of course, that your codec is Xvid or DivX 4 or 5.

http://forum.doom9.org/showthread.php?s=&threadid=55039

This cites a Vdub program crash at 0.28.3 beta. Both colordog and leadman584 posted in this thread, though leadman describes his "leak" in more detail there, not Vdub crashing.

-----------------

This is from memory, so forgive any slight inaccuracies:

As I recall, we instituted a waiting period for new members to reduce the replication of the same FAQ questions by just-registered users over and over. Again, and again. Ad infinitum. Repeatly. Non-stop.
Continuously. Without end. Intermidably. (You get my drift?)

More FAQs were written and published. Warnings were announced. Nothing seemed to stop it. Many members requested something be done, as it was a distraction, a nuisance, unwelcome, and preventable.

Doom9 and the mod team determined that since the majority of those were from instant registrants, that a waiting period be established. When Doom9 first instituted the waiting period, it was for three days. It reduced these duplicate already-answered-a-billion-times questions, but only by about 50%.

The wait was increased to five days as an experiment. Low and behold, these posts were drastically reduced from about 50% of the previous level to about 5% of the previous level.

Nothing will eliminate them entirely. (Against stupidity, the gods themselves contend in vain. --William Shakespeare--)

-----------------

And as to the kind of information needed to actually have someone take action (such as attempting to duplicate your problem) I recommend this old post of markrb's (http://forum.doom9.org/showthread.php?s=&threadid=19566) on the kinds of information needed by an application developer that you have not yet provided.

[ZZZERO]

Going back to the original topic, I also experience extreme memory usage during compressibility tests. Next time I run one I'll post all the details that I can. It happens it both Gordion Knot and Auto G-Knot.

[Fat80y]

I also get a lot of disk swap occurring on my system, yet task manager reports similar numbers to what colordog reported. This has occurred through many revisions of divx and gknot.

Win 2000, all service packs, K7D Master, tbred b's @ 13.5 x 150 fsb, 512 mb ram.

The problem is w2k disk cache - because the compression test is essentially random access to a very large file, w2k increase the cache size as much as possible to increase performance. If you look at task manager while doing a comp check, (in my case) free memory will drop to 5mb, while cache size will be ~350mb. As soon as the test is over, windows releases the memory from the cache, and everything returns to normal.

I found this pretty annoying, so upon research at 2cpu.com Jed (http://forums.2cpu.com/showthread.php?s=&threadid=37326) (their resident expert on w2k) said there isn't anything u can do except buy more ram!!!

Anyhoo, hope this helps.

[colordog]

I certainly haven't seen that effect, but I'm using WinXP and a static swap file. Regardless, what you are describing really isn't a memory leak though. That's just a windows thing. Windows increasing the swap file size while processing isn't a bug, especially if it shrinks back. Do you only see this during the compression test?

I think the original issue of the memory leak was referring to the compression test causing either physical memory or disk memory to be used, but not released. Unfortunately, we never got any additional information about whether that's really the case or not. Soooo.....

Fat80y, what you might want to try doing is setting your swap file to a large, static permanent size. This is a Windows tweak that a lot of people use. Since drive space is so plentiful these days, you can set your swap file to a permanent 1.5GB or so, which is more than it'll ever use, and you won't have to wait around while Windows trys to grow or shrink the swap file while you're performing tasks. It's one of the first things I do after a new installation. Additionally, you might want to use a good defragmentation program like Executive's DiskKeeeper which can do boot-time defraging of the swap file.

[Fat80y]

Colordog,

It is not the swapfile growing or shrinking, as I have a permanent size of 2 gig. What happens is Windows swaps everything possible out of ram to the page file to make the disk cache as big as possible to cope with the random access to the vobs. My peak memory usage hardly ever goes above 600mb which equals ~350mb cache + 120mb vdub (ram used) + unused windows dlls + other progs swapped to disk.

If for example, I wish to web browse etc while doing comp test, the computer decreases the cache by a small amount, whatever is necessary to load iexplorer. Because the cache is so large (max amount of memory available) anything that requires memory involves swapping to disk

The system is perfectly usable, just very slow beacuse of the disk swapping. As soon as the comp test finishes, the cache ram is reduced, allowing more free memory for other apps, and performance returns to normal.

I realise this has nothing to do with a application memory leak, but it may explain high ram usuage during comp check...

[colordog]

fat -

Sorry, I mistook your cache size for your memory commit.

That is actually about what I remember before I went with my latest upgrade. Previously, I had a normal P4 and half-gig of RAM, and I recall having stuff similiar to what you were describing. After launching VDubMod, with other Windows programs running, I had no free physical RAM left, and the caching did suck.

More recently, I've upgrade to a P4 with HT, and increased memory to 1 GB, and don't have that slowdown anymore. (I really like to have lots of things open at once). With the gig of memory, I rarely ever have to cache, and the HT lets me be able to do other things in other programs while VDub is running without lag.

The tough thing here is so many people have such different computer setups, that it gets really tough to track things like this. Although at this point, it seems like the original people that suspected a memory leak don't care enough (?) to pursue the matter, since no details have been posting describing this possible bug yet. Regardless, what you're describing sounds pretty normal to me. Try to save up for more memory? I was really amazed at what a difference the half-gig to full gig did for me. But I know other people just set up encodes and walk away from their computer, so they don't care about having extra memory for other programs. (Not me though!)

[Fat80y]

I'm with u colordog, smp is where it's at tho, none of this pseudo ht stuff :) - yeah I am currently saving up for another half gig, good to hear it makes a real difference...

[colordog]

I'll do SMP as soon as there's a dual P4 board (none of this Xeon crap!)