Fix or extract streams from large Divx+AC3 AVIs with header problems

[alfonzotan]

I have a number of HD .avi files encoded with Dr. Divx. For any of these files larger than 4GB, virtually every player or tool that I've tried (and trust me, I've tried a lot of them), playback stops at the 4GB point. Because somebody is sure to ask, no, I'm not using any FAT32 drives.

Examining the headers with AVIMaster, it appears that Dr. Divx incorrectly writes in a truncated total frame count corresponding with the 4GB point in the AVI Main Header (for this example, 161626 frames, when the actual total of encoded frames is 204328). While I can open the full file using the AVIFile compatibility option in VirtualDub, VDMod, or NanDub, no tool that I've tried will re-write the full-length file in a new .avi. VirtualDub stops with an error at frame 83129, which corresponds to the 2GB point (this makes no sense to me--VDub is ODML-friendly and shouldn't dork up at the archaic AVI limits). The log simply says, "Video frame 83129 could not be read from the source. The file may be corrupt." Trying to skip the "problem" frame or save as multiple smaller .avis doesn't work, either.

I'd be perfectly happy to just extract the elementary A/V streams and remux in a new container, but I'm not even able to do that. I was able to pull out the full video stream (painfully) with VisualHub on a Mac, but I've had no luck at all in getting out the audio stream. Not one of VDub, VDubMod or a Graphedit script will fully extract the AC3 audio track (they all stop at the 4GB point).

I've spent a ton of time trying to find a fix for this, and I'm about at the end of my rope. Any suggestions (other than don't use Dr. Divx--I figured that one out)?

[Guest]

Why don't you re-encode from your original streams?

[alfonzotan]

Why don't you re-encode from your original streams?

That's exactly what I'm going to do in the cases where I still have the original streams. I don't have all of them, though.

[setarip_old]

Have you tried loading it into Virtual Dub - and (with both "Video" and "Audio" set to "Direct Stream Copy") setting the "Set selection start" point to, for example, 162,000 frames and save that portion?

[Guest]

I don't have all of them, though. What happened to them?

[alfonzotan]

What happened to them?

Tossed after being reencoded. They were encoded to Divx quite a while back, and I didn't realized there was a problem with full playback until more recently. My own dumb fault, of course.

[alfonzotan]

Have you tried loading it into Virtual Dub - and (with both "Video" and "Audio" set to "Direct Stream Copy") setting the "Set selection start" point to, for example, 162,000 frames and save that portion?

I think so (I've tried so many things that they're all running together), but I'll give that one another shot.

[Guest]

Tossed after being reencoded. You tossed your original disks?

[alfonzotan]

You tossed your original disks?

No, these originated on a DVR, not optical media.

[alfonzotan]

Have you tried loading it into Virtual Dub - and (with both "Video" and "Audio" set to "Direct Stream Copy") setting the "Set selection start" point to, for example, 162,000 frames and save that portion?

No dice. It just grabs that many frames from the start of the file (not the selection point), then writes an un-playable result. Only audio out, no readable video.

[setarip_old]

tool that I've tried (and trust me, I've tried a lot of them)Please state the specific tools you've tried...

[alfonzotan]

Please state the specific tools you've tried...

Well, let's see.

VirtualDub, VDubMod, Avisynth (a very simple script in that case, I'm not a whiz with that one), NanDub, AVIDemux, Graphedit, AVIMaster, DviFix, MKVToolNix, MPEG4Modifier, StaxRip, TMPGEng, AVI-MuxGUI and Yamb. On the OS X side, VisualHub, VLC, MPEG Streamclip, QuickTime Pro, ffMPEGX, and I'm sure one or two more that I've forgotten about by now.

[setarip_old]

You might want to try "Avifix" and "AVI Fixed"...

[alfonzotan]

You might want to try "Avifix" and "AVI Fixed"...

Thanks, I'll give both a shot.

[joseph5]

And DivFix.

[alfonzotan]

You might want to try "Avifix" and "AVI Fixed"...

No dice. Truncates the file at the 2GB point for opening in VDub.

[setarip_old]

Have you tried using VirtualDub as follows?:

Open the file in VirtualDub (or VirtualDubMod), but check the "popup extended open options" in the open dialog. In the extended open options check "re-derive keyframe flags". Wait for re-parsing and rebuild index block.
Video-direct stream copy, Audio-direct stream copy, File- Save as avi (with a new filename) ...

[alfonzotan]

And DivFix.


Okay, I think I'm getting somewhere now.

Choosing the "delete index" option in Divfix and then opening the file with VDub or VDubMod (which are forced to write a new index as a result; fixing with Divfix itself doesn't work) allows me to write a new AVI from there that contains and will open the full file length. That file isn't terribly portable/playable, but at least I can get to the full streams, and from there I ought to be able to get a working-again final product.

Appreciate all the help, I'm on travel for a couple of weeks, so it'll be a while before I can continue with this. Thanks again, all.

[alfonzotan]

Have you tried using VirtualDub as follows?:

Open the file in VirtualDub (or VirtualDubMod), but check the "popup extended open options" in the open dialog. In the extended open options check "re-derive keyframe flags". Wait for re-parsing and rebuild index block.
Video-direct stream copy, Audio-direct stream copy, File- Save as avi (with a new filename) ...

Thanks, I did try that right off the bat (no joy), but I haven't tried it yet with the index-stripped version. Next on the list when I get home.

[alfonzotan]

Whew.

Well, it really took a long time to figure this out, but I think I have a fix. Standard YMMV disclaimers apply, but if anybody else runs into the 4GB limit thanks to Dr. Divx, here's how I fixed it:

Run the affected file through AVIFixed (I used 1.1b1). That'll strip the old, faulty header off the .avi. Then open the result with VirtualDub, which will rebuild the header when it loads (this takes a while). Save another copy with VirtualDub using Direct Stream Copy for both streams.

In Graphedit, use the 3ivx 5.0 (4.5 will not work) codecs to convert the AC3 audio (if that's what you have) to AAC and remux to .mp4 or .mov (I prefer .mov, YMMV). The working graph for the video side looks like this:

[source.avi] > [AVI Splitter] > (video) > [3ivx Media Muxer] > [filewriter]

... and for the audio track coming out of AVI Splitter:

[AVI Splitter > (audio) > [AC3Filter] > [3ivx Audio Encoder] > [3ivx Media Muxer] > [filewriter] (same muxer and filewriter as above, natch)

The result plays back fine on either a Windows PC or a Mac, assuming the 3ivx codecs are installed.

Thanks to everybody here for the suggestions and advice. I should note in closing that Dr. Divx still works great for smaller files (i.e. hour-long TV episodes), but I wouldn't trust it for anything longer than 1:45 or so.

[setarip_old]

Run the affected file through AVIFixed (I used 1.1b1). That'll strip the old, faulty header off the .avi. Then open the result with VirtualDub, which will rebuild the header when it loads (this takes a while). Save another copy with VirtualDub using Direct Stream Copy for both streams.Glad to hear that the combination of two of my suggestions helped resolve your dilemma ;>}

[alfonzotan]

A bit of an update here... since 3ivx 5.0 won't connect to a 5.1 audio source without buying the ($99) Pro version, and because the 3ivx 5.0 codecs interfere negatively with so many other codecs on the Mac side, I've had to find another method. Fortunately for me, Perian 1.0 came to the rescue. Here's my current conversion process:

1. Run the long .avi file through AVIFixed to strip the header.
2. Open resulting file with VirtualDubMod, then save a new copy.
3. Demux the .ac3 stream from the new copy with VDM.
4. Open the new .avi and .ac3 files with an MKV muxer and build an MKV file of the full movie. If you're happy with using .mkv-friendly playback tools, you can stop here. Since I use FrontRow at my HTPC (er, Mac), and as Quicktime doesn't do MKV without some help, I needed another step.
5. On a Mac, load Perian 1.0, then open the .mkv file in Quicktime Player (you'll have to choose "Open With" and then select "All Applications" to get to QT Player as an option).
6. Allow the full movie to load (watch the status bar at the bottom of the player window). Once the movie is loaded, Save As to a reference movie. Do NOT delete the .mkv file!

Quicktime will now open and play the full movie from the small reference .mov. This is quite nice, since this way you don't have to recode the .ac3 audio.

[Stux]

A bit of an update here... since 3ivx 5.0 won't connect to a 5.1 audio source without buying the ($99) Pro version

With the release of 3ivx MPEG-4 5.0.1 we've added a personal Multichannel version with 5.1 aac encoding/decoding support.

http://store.3ivx.com/3ivxStore/product_details.php?id=18

So no need to purchase the 99$ pro version just for Multichannel support :)

[Blue_MiSfit]

Just out of curiosity, did you try remuxing this b0rken AVI to an MKV using MKVToolNix, or maybe just remuxing it to AVI using ffmpeg or mencoder?

~MiSfit

[alfonzotan]

Just out of curiosity, did you try remuxing this b0rken AVI to an MKV using MKVToolNix, or maybe just remuxing it to AVI using ffmpeg or mencoder?

~MiSfit

Yeah, tried that in ffmpeg and MKVToolNix (and I think mencoder; don't hold me to that, it's been a while), but wound up with a 4GB file. Had to strip the header and resave before I could get a useful MKV remux.

[alfonzotan]

With the release of 3ivx MPEG-4 5.0.1 we will be adding personal Multichannel versions with 5.1 aac encoding/decoding support.

So no need to purchase the 99$ pro version just for Multichannel support :)

Thanks for the info, but using the free MKVToolNix and not having to reencode anything suits me better.

[alfonzotan]

A bit of an update (and hopefully the last one): if you have the patience for still one more demux-and-remux step, once you have a working .mkv file from the procedure above, you can demux THAT file using MKVE Wizard, then remux the resulting "laundered" components with VirtualDubMod into a regular old .avi that actually works. Gives you the advantage of portability if that's important to you (it is to me), since the Perian MKV fix for Quicktime/FrontRow is not just ungainly but also not entirely dependable.