Pulling my hair out: dropped frames

[dilly]

I'm at a loss here. I formatted my capping box about 6 weeks ago. For the first five, and for months before that, everything worked fine. Now, all of a sudden I'm getting unusable levels of framedrops (300+ /min) whereas in the past I was seeing 0. I can't think of what triggered this, no software installations, no hardware upgrades.. nothing changed.

So, I decided that it was a hard drive speed issue. This came to me because I tried uncompressed avi (huffy usually) and frame drops increased exponentially, and decreasing the capture resolution decreased the amount of frame drops*. So, I tried defragging, didn't help. I then deleted the partition (which is a WinXP dynamic disk, spans over two physical disks) and created it again, quick-formatted, and then I was back to 0 FDs.

This only lasted about two or three days. Today, I'm back to 300+/min.

I don't really know why I'm posting this... it's such a specific problem that I doubt anybody can help me, but here's hoping.

I'm going to try deleting the partition again, and this time, leave the disks independent and see if I'm able to keep it up. Wish me luck :(

* edit:

"Less frame drops than at full resolution huffyUV, but still >0, which is what I've come to expect - I am usually able to capture at 0FD with VDub running and encoding and other things going without worry."

[dilly]

Well I may as well keep track of what I've done as I go, if for nobody else, myself.

Solutions that have produced false positives:

- Turned off overlay preview (so as to save system resources)
- Turned off many Windows XP services, used EndItAll before capturing, reducing minimum RAM usage (without VirtualVCR running) from 130 to 70
- Formatting, repartitioning capture drives

Solutions that seemingly haven't done anything:

- Defrag

I still haven't tried breaking up the software RAID, as there's data to be moved off still. After doing the above(false positives) , I've noticed the pattern has become more sporatic(sp?) -- instead of constant framedrop levels over 300/min, one capture started dropping frames immediately, and another took 10 minutes before it began heavy frame dropping.

Now my theories:

- that one disk is faulty, not capable of maintaining the 9MB/sec it takes; and the other is fine, fragmentation explaining why it works 'sometimes'
- that some mysterious process / network request is eating CPU cycles and system resources, constantly for a duration but said duration is random

[2ZOD.COM]

what are your system specs?

Software RAID and Huffy UV capturing at the same time doesn't sound like a task for any kind of slow cpu. Even if you have a fast processor most RAID that isn't totally integrated will produce some latency no matter what, and that could defineatly cause drops.

What are your sector sizes on your hard drives? If you bump up the size of those you should see a slight speed increase. I don't know how you can change it without using another program. I used Partition Magic to do it. Works great.

[ppera2]

Windows XP has habit to turn off DMA for every little problem.
Solution to back DMA is to turn of IDE driver in device manager and then let Win to recognise disks. So, it could be answer why you had 0 drops after repartitioning.

I suggest to you that change IDE cable of that disk - cables have often bad contact(s). Also some replacement may help - it's not goog if cables go to close each to other.

[dilly]

OK, thanks a lot for the input.

System specs:

P4-2.4Ghz (FSB/etc I can't be bothered)
512MB no-name
"80" (74.5) GB Western Digital WD800JB-00CRA1
80GB Maxtor 6L080J4
Leadtek Winfast 2000XP Deluxe
Creative Sound Blaster Live! 5.1


Software RAID and Huffy UV capturing at the same time doesn't sound like a task for any kind of slow cpu.


I was doubtful too, but when I thought about it, I knew it was a great idea, and it worked flawlessly for months. Anyway my CPU is fine ;)

I have pinpointed the problem now. I broke up the software raid. The Western Digital disk is the one causing the trouble, when I capture to it, hundreds of FD's. Maxtor works flawlessly, 0 FD's.

So I ran a SiSoft Sandra test and found out that the Maxtor performs at 30000KB/s whereas the WD performs at 18000KB/s. This is more than sufficient for capturing, but I can't see any reason for the huge speed difference. (Note that I had never done one of these speed tests before the problem, so this could be wholly unrelated).

I suggest to you that change IDE cable of that disk - cables have often bad contact(s)

Fair enough, I'm going to try some software approaches before opening it up. I just changed it about 6 months ago for an unrelated reason, so I don't see why it should just stop working on me.


Windows XP has habit to turn off DMA for every little problem.
Solution to back DMA is to turn of IDE driver in device manager and then let Win to recognise disks.


That sounds like the most logical explanation yet... when something spontaneously breaks, it must be Windows :)

I will try that as soon as I can. However if it makes a difference, in the controller settings, they all claim to be running in UDMA5. Sounds good, but who knows.


So, it could be answer why you had 0 drops after repartitioning.


How so? I didn't actually reformat/reinstall Windows, there was no disk detection AFAIK. Just Disk Management MMC snap-in, delete, create, quick format.

I can't believe that my disk just died, it doesn't make sense that a disk slows down for no apparent reason. So, we'll see.

Thanks again, hope to hear from more people.

[dilly]

Originally posted by 2ZOD.COM
What are your sector sizes on your hard drives? If you bump up the size of those you should see a slight speed increase. I don't know how you can change it without using another program. I used Partition Magic to do it. Works great.

I'm not sure, I'll have to look for PM, I have it somewhere, never used it once though on consideration I don't have a floppy drive anywhere in the house.

Anyway, isn't that set when you format the drive?

[dilly]

Update:

I'm still having this problem. Nothing has helped so far, including the DMA solution (I also tried using my drivers, from SiS, which I never needed before).

I'm about to test the drive in another machine to confirm that it's a bad motor or cable or something along those lines.

[dilly]

Update:

Well, I've pretty much confirmed that the WDC drive is the culprit, I don't know whether to keep it or not.

Anyway, I had a good idea: I'd capture to the working drive, and send it off to the bad one for processing, figuring that processing never reaches anywhere near 9MB/sec, especially with heavy filters that drop you to 2.0FPS. But, what I've figured out: if there is ANY activity at ALL on the bad drive, the good drive drops frames too.

This is getting frustrating.

[ppera2]

Yes, that's the behavior of IDE bus - if one device reacts slow, it stops whole bus, and other device must wait until problematic finishes or time-outs.

[dilly]

That's logical. I tried putting one on IDE0 the other on IDE1 and it still had the same problem... so who knows.

OK, now the situation has entirely changed.

I formatted today and put Windows on the Maxtor drive, the one that let me capture before.

Now, guess what? The WD drive caps fine and the Maxtor doesn't work for capping at all.

Why would this be a problem? I can't comprehend why it would work for months, then stop working because of Windows.

So I'm going to try what I said above again (switching up the IDE channels, removing IDE drivers, etc) and then I'm gonna shoot myself in the head.

Frustrating ^ infinity.

[jggimi]

I've moved this to the PC H/W forum, in the hopes that you may get some hardware specific help for your problem. Don't do anything drastic just yet.

[dilly]

Thanks jggimi.

I'll summarize then, thread is long if you haven't been following it:

Specs:
"80" (74.5) GB Western Digital WD800JB-00CRA1
80GB Maxtor 6L080J4
(more above)

The good ol' days: Both disks working together in WinXP dynamic disks (160GB). Windows on WD disk.
Then all of a sudden: WD disk can't sustain huge transfer rates anymore (write 9mb/sec).
Format, install WinXP on Maxtor: Now WD disk is fine, Maxtor disk can't sustain huge transfer rates anymore.

everything I tried is above.

[ppera2]

In such case, I would try with another motherboard.

[dilly]

That's not going to be easy.

I've done some tests, and indeed the system drive is slower. I discovered auxsetup.exe in Virtualdub, very useful. This has to be able to be solved through software.

[colordog]

It's hard to say, but I agree with ppera2 that based on the available information, it's mostly likely due to the MB. (Especially if you've checked Window's setting for DMA, and verified that the drives are UDMAx and not PIO, which you stated above).

I'd like to ask some clarifying questions though.

1) When you read from the 'bad' drive that maxs out at 9MB/sec, does your CPU usage have a sustained increase more than 2%?

2) Could you go over the exact locations of the drives on the IDE bus and channel? For instance, what worked where? When you switched them around, were they are separate buses, or the same bus different channel?

Like:
WD on IDE 0 master
MAX on IDE 1 master

or something like

WD on IDE 0 master
MAX on IDE 0 slave

etc?

2a) When you say that if the 'bad' drive is installed and causes the good drive to slow down as well, is then when it's installed on the same or different bus?

3) What other devices do you have attached to the IDE bus?

[dilly]

1) When you read from the 'bad' drive that maxs out at 9MB/sec, does your CPU usage have a sustained increase more than 2%?


I can't think of any way to test simply the read and see the CPU, besides copying file from 'bad' to 'good' -> avg 20% CPU usage. (But that's in my current config, below)


2) Could you go over the exact locations of the drives on the IDE bus and channel? For instance, what worked where? When you switched them around, were they are separate buses, or the same bus different channel?
3) What other devices do you have attached to the IDE bus?



Orginally, worked fine:

Primary (good IDE cable)
--------
Master: 80GB Western Digital
Slave: 80GB Maxtor

Secondary (older IDE cable)
----------
Master: PIONEER DVD-RW 106A
Slave: LITEON 48/24/48

That configuration stopped working without any modification on my part.

I am now running like this:

Primary (good)
--------
Master: 80GB Western Digital
Slave: -

Secondary (old)
----------
Master: -
Slave: 80GB Maxtor

The problem is still happening in this configuration, and in addition the Maxtor is now in UDMA2 mode, because of the cable, which can't be good at all.

(I don't have my CD drives hooked up because I'm too lazy to re-arrange them, cables don't reach.)


2a) When you say that if the 'bad' drive is installed and causes the good drive to slow down as well, is then when it's installed on the same or different bus?


That only happened when they were together. Now they're apart, for this exact reason.

I've done more SiSoft tests, and it's telling me that the write index is substantially lower than read. (although it won't let me access 'real' r/w numbers, just a single rating) I'm not so sure, but I can believe that read is OK off of the 'bad' drive. It tells me to try enabling write caching in Windows, which i've tried off and on, doesn't work either way.

Another thing: SiSoft tells me my Maxtor can go to UDMA6, I only see it in UDMA5 in Windows. I went into my BIOS and it's set to 6, I tried putting that value to 5 manually, still no good.

[dilly]

BAH

OK, I guess you guys can stop helping now. Things have gone from bad to worse to disatrous. Now, I'm getting Delayed write errors to C:\$Mft, unable to run CHKDSK because it doesn't exist (!) and no control panel, etc etc. Computer is dead.

I guess I'm getting a new motherboard.

[dilly]

and the plot thickens...

The above errors were most likely due to me disabling paging, in a desparate attempt to reduce disk activity. bad idea :)

I got the machine working for regular tasks, but the write error was still bugging me. So, I was messing around and I noticed off to the corner there was a third IDE controller. I forgot that my motherboard has a Promise RAID controller onboard. Problem with it is, it needs a SATA and a ATA133 to run RAID (no 2xATA133). So I made a one-drive array with the non-Windows and tested it out in capture. Works, as it should. I changed the plugs around and made the Windows drive on the Promise controller, the non-Windows drive on the SiS IDE.

Lo and behold, the pattern continues: the drive Windows is installed to remains slow.

Therefore I'm going to do the only thing left I can do before resorting to violence: take a third drive and install Windows to it, and hope the other two can get along.

To the people who said it may be my motherboard: of course, but did you mean my motherboard, or the IDE controller on it? I suppose some kind of motherboard malfunction could cause a broken transfer between one onboard controller and another, but heres the breakdown: I have two drives, and two controllers. I have made each combination both WORK, and NOT WORK. Therefore, I can only deduce that the controllers are fine, the drives are fine.

There may yet be hope...

[ppera2]

Originally posted by dilly

...
To the people who said it may be my motherboard: of course, but did you mean my motherboard, or the IDE controller on it? I suppose some kind of motherboard malfunction could cause a broken transfer between one onboard controller and another, but heres the breakdown: I have two drives, and two controllers. I have made each combination both WORK, and NOT WORK. Therefore, I can only deduce that the controllers are fine, the drives are fine.
There may yet be hope...

Motherboard and controllers on it is same - could you replace controllers separately?

I meant earlier that you try another board, not to immediately buy another.

[dilly]

Of course, I didn't think you meant to buy another :)

It is kind of hard to get my hands on one, and is a hefty job.

Right now I'm in the middle of formatting / installing Windows on a third hard drive. We'll see how it goes.

[dilly]

This has made me want to take nuclear physics and organic chemistry. Anything is easier than fixing this problem ;)

Well I installed a third drive, Windows on it, and the pattern is broken. It works fine for capture. And now, the Maxtor drive is giving me trouble (it doesn't matter how I arrange the drives, however if I put the Maxtor drive on the RAID controller it has *better* but not sufficient performance) and I'm beginning to think it was that drive all along, causing a slowdown on the IDE bus as ppera2 pointed out. I'm franticly copying all my data over to it to see whether pulling it will fix everything.

I doubt it though, as my initial findings showed there was the exact same issue on the WDC drive.

Anybody still pulling for me? :/

[dilly]

BINGO

80GB Maxtor 6L080J4

Yes, that's the behavior of IDE bus - if one device reacts slow, it stops whole bus, and other device must wait until problematic finishes or time-outs.

I took out that drive, and now everything works. Obviously there's something wrong with it. I would have figured it out earlier, if I hadn't had troubles writing to my Western Digital drive -- which were probably caused by the Maxtor tying up the IDE bus.

In my P3 machine, I tested the Maxtor and the write benchmark is too low for capture. Looks like the one drive was the culprit all along.

:) :) :) :)

[foznikjj]

You guys are talking over my head, but I think I'm having some of the same problems. Around 1st of DEC.03 I captured 47.xxGB of Dv video from my camcorder, using Pinnacle Studio 8. It rarely dropped a frame. Around Dec 26 03 I noticed the video files on my 2nd HD were jerky video and distorted audio. I deleted some of these captured files thing they had been corrupted. Finally found my problem (with forum help)was my 2nd HD had got switched to running in PIO mode. I reinstalled primary IDE driver and both HD's are running in ultra DMA mode 5. BUT,when I try to recapture these files I deleted, Pinnacle clock will run smooth and then hangs, when it starts running again I drop maybe 200 frames. It does this over and over every few seconds and at each pause It will create a new scene or separate clip in libiary. Hope you can help me. Foz

Also I have used Pinnacle to test drive speed.
E-drive-Read,1391462Kb/sec--write,33707Kb/sec. (said drive capable of capturing Dv data.
F-drive-Read,1143458Kb/sec--write,17531Kb/sec. (same drive capable)
E-2nd test-Read,1340759Kb/sec.--Write,44161Kb/sec(same drive capable)

[ppera2]

I have similar situation now with one Maxtor 40 GB drive, 2 years old.
It worked fine couple days, but now starts slow to write, and is not able to capture well. DMA is OK - I checked it in XP and 98 too.

Buying new drive... perhaps not Maxtor. What are experieces with new WD drives?

[foznikjj]

I'm pretty new at this puter stuff. I got my machine in June of 03. my wife has had her's since Jan. 03. Mine came with a Seagate 120GB 2MB buffer.I got the Western Digital in Nov.03. It is 200Gb 8MB buffer and seems to be OK. The problem I had with it going into PIO mode was caused by Windows XP,(see link below). Now that I have both drives running in ultra DMA mode 5, I don't have a clue why I'm dropping frames. Did not have this problem when I began my 1st captures in Nov. 03. Foz


http://wdc.custhelp.com/cgi-bin/wdc.cfg/php/enduser/std_adp.php?p_sid=kk5h_qXg&p_lva=&p_faqid=525&p_created=1031702376&p_sp=cF9zcmNoPTEmcF9ncmlkc29ydD0mcF9yb3dfY250PTE0NiZwX3NlYXJjaF90ZXh0PVBJTyZwX3NlYXJjaF90eXBlPTQmcF9wcm9kX2x2bDE9fmFueX4mcF9wcm9kX2x2bDI9fmFueX4mcF9jYXRfbHZsMT1_YW55fiZwX3NvcnRfYnk9ZGZsdCZwX3BhZ2U9MQ**&p_li

[foznikjj]

Finally got my frame dropping cured. I had 359 files in my IE
favorites, I copied this favorite folder to 2nd HDD, and deleted the
favorites in IE down to 63 files. I captured two 30 minute DV tapes without
dropping a frame. Thanks to all. Foz

[GrofLuigi]

Originally posted by foznikjj
Finally got my frame dropping cured. I had 359 files in my IE
favorites, I copied this favorite folder to 2nd HDD, and deleted the
favorites in IE down to 63 files. I captured two 30 minute DV tapes without
dropping a frame. Thanks to all. Foz


You capture with IE?!

:)

[foznikjj]

You capture with IE?!
I do not capture with IE, but a friend suggested that all the favorites (359) I had was loading my 768MB of memory. This could cause Pinnacle Studio 8 to have to access the page file. I'm not smartt enough to figure it all out, but I can say that since I deleted these files to a smaller amout (63) I no longer drop any frames.I have tried for months,using the enditall program,check disk,defragmenting, and many other things, and this is the first thing that has worked. Thanks Fozj