issue avs2avi with big AVI ouput file (127 GB limit)

[yup]

Hi All!

I have problem with avs2avi v1.39.
I am deal with big clip 2.5 hours, 960x720 resolution, 50 fps, YV12 colorspace.
When ouput file reach size 127 Gb (136*365*211*648 bytes) avs2avi give error that programm could closed.
First time I think that problem is complex script (using multithreded Avisynth, QTGMC with upscaling), but I am see the same issue on 3 PC (1 Windows 7 and 2 Windows 10).
File avs2avi.exe set LAA.
Please advice how can solve problem or other way for conversion Avisynth scripts to AVI.

yup.

[mariush]

my guess from a very brief look at the source code...

avs2avi uses a bunch of source code from an older version of Virtualdub, that code handles the writing to avi file and creating the index for the video and audio frames (index is read by applications to know at which offsets to jump in the file if they want to go to a particular moment)

AVI has a "superindex" which is made out of several "index"-es, each with a number of entries. In the avs2avi source code, the number of entries in each index is hardcoded to 2048, and I suspect there's a maximum of 256 indexes , which works out to about 127 GB of content ... so that's why I suspect it fails, you reach the maximum of index entries defined in the source code.

code in avs2avi\VirtualDub\AVIOutput.h


class AVIOutputFile : public AVIOutput {
private:
enum { MAX_AVIBLOCKS = 64, MAX_SUPERINDEX_ENTRIES=256, MAX_INDEX_ENTRIES=3072 };



code in avs2avi\VirtualDub\AVIIndex.cpp


class AVIIndexChainNode {
public:
enum { ENTS=2048 };

AVIIndexChainNode *next;

AVIIndexEntry2 ient[ENTS];


Newer virtualdub versions use 256 indexes but each index holds 8192 entries, which works out for 2 GB per index as the text says in the options box:

http://savedonthe.net/image/2492/vdub-superindex.png

So with 2048 entries, it would be only logical to have one index per 0.5 GB ( 1/4th of 2GB), so the avi output would crash at 0.5x256 = 128 GB.

I guess the fix would be to patch the source code and recompile the application.

[yup]

mariush!
:thanks:
Do not see Your attached picture two days. waiting approval.
May be exist other way (program)? VirtualDub for me unstable for scripts with multithreaded Avisynth.
yup.

[StainlessS]

Yup, I can do a special compile for you of TWriteAVI v2.0 (which also uses Vdub v1.411 source) and you can write via eg


TWriteAVI("out.avi", Overwrite=True, FourCC="ULY0") # Write at same time as playing clip (UT_Video)
ForceProcessAVI() # Force write AVI, dont play clip until fully written
Avisource("out.avi") # Load new Lossless clip
return Last # Plays lossless clip after writing


I can do this mod (AVIIndex.cpp)

class AVIIndexChainNode {
public:
enum { ENTS=8192 };

AVIIndexChainNode *next;

AVIIndexEntry2 ient[ENTS];


Are you willing to give it a try (I will not make it public) ?

TWriteAVI v2.0:- http://forum.doom9.org/showthread.php?t=172837&highlight=TwriteAvi

EDIT: Sent PM link to TWriteAVI mod, totally untested (let me know if it works without problem, and I'll make public).

[StainlessS]

Here Yup, mod of avs2avi:- http://www.mediafire.com/download/3zx6e0dbnzojlkt/avs2avi_1.39Mod_20160403.zip

[kolak]

Newer virtualdub versions use 256 indexes but each index holds 8192 entries, which works out for 2 GB per index as the text says in the options box:

http://savedonthe.net/image/2492/vdub-superindex.png

So with 2048 entries, it would be only logical to have one index per 0.5 GB ( 1/4th of 2GB), so the avi output would crash at 0.5x256 = 128 GB.

I guess the fix would be to patch the source code and recompile the application.

Had similar issue, but for me it was 2TB boundary. After changing Vdub default values it worked.

[yup]

StainlessS!

Moded version avs2avi do not solve problem.
I get error when file size reached 127 GB.
yup.

[poisondeathray]

I quite frequently use AVI's larger than that. FFMpeg can do it from avs input

Some other options you can look at are dividing up script into parts (perhaps with some "overlap" for filtering purposes), or using stronger lossless compression to reduce filesize

If the problem is with the script (MT can often cause problems, usually due to memory), consider dividing up the script into sections and lossless intermediates, or using non MT

[yup]

VirtualDub make job for output file size 170GB with default setting (see post 2 and 6 this thread).
poisondeathray thanks for advice I am try ffmpeg instead of avs2avi.
yup.

[poisondeathray]

yup, some MT scripts in ffmpeg work better with distributor() , some do not (I mean speed-wise). Do a quick test with avsmeter before proceeding to the full encode

[yup]

Hi poisondeathray!
I am try simple command
ffmpeg -i test.avs -codec:v utvideo test.avi
fps dropped to three times compare to avs2avi.
Interesting I am try encode my script using avs4x264 and avs4x265 without problem.

yup.

[kolak]

Try also adding -threads. Not sure if utvideo does it by the default. Older versions of ffmpeg did not support threading in utvideo.

[yup]

kolak!
threads=8 do not increase fps.
poisondeathray!
Yes when added distributor() as last line in script speed like using avs2avi.
Now testing stability.
yup.

[poisondeathray]

ut video ? Just an FYI, the ffmpeg ut video variant is slower (about 2/3) than the VFW variant for decoding speed . Something about differences optimization, instruction sets. For example, if I use it in another application like a NLE, the ffmpeg ut variant is noticably slower/sluggish

[yup]

When I try convert my working script (QTGMC + nnedi3 enlargement) using ffmpeg, I do not have problem for clip 1.5 hour, but during conversion 2.5 hours clip ffmpeg crashed.
I am only have success with this big clip using encoding x264 and x265, both use interface from 32 bit avisynth to 64 bit encoder.
No one 32 bit programm do not work properly (avs2avi, VirtualDub and ffmpeg).
Any recommendations please.
May be can organize pipe from 32 bit avisynth to 64 bit AVI writer?
yup.

[Emulgator]

137,4GB... hm, 28-bit LBA limit comes into mind, but only if you have an old ATA-5 controller sitting somewhere.
External HDD possibly ? But then the system would see the first 137GB of that HDD anyway...
Some older Chipsets have that limitation: InSystem ISD300A1 (USB2.0), ATA-5 brings it.

137GB (FAT-32) is a Microsoft problem. Even though they claim to use up to 28 bits & 32K clusters to give 8TB, there are other limitations. Some older system tools like ScanDisk and Format are 16 bit and have a memory block maximum allocation size of 16 MB less 64 KB. Using one memory block for the FAT with 4 bytes for each FAT entry and 64KB overhead, there can be only 4,177,920 clusters allocated. Also some older programs can not handle clusters larger than 32KB. . . . Therefor . . . 4,178,920 clusters x 32,768 Bytes per cluster = 137GB

Note;
According to Microsoft's FAT32 spec, the cluster numbers are really only 28 bits and the upper 4 bits of a cluster number are "reserved". If that were the case then FAT32 could handle up to 1 Terabyte partition size with 4K clusters. Microsoft doesn't even adhere to their own spec.

137GB is also a BIOS limit where the system BIOS can only handle 28 bit LBA but not “large" 48 bit LBA.
28 bit LBA limit = 2^28 x 512 bytes per sector = 268,435,456 x 512 = 137,438,953,472 bytes
- - - This is also a limit on Windows XP-SP0 and Windows 2000-SP2 which failed to support 48 bit LBA and corrected on later versions of Windows.

[Groucho2004]

137,4GB... hm, 28-bit LBA limit comes into mind, but only if you have an old ATA-5 controller sitting somewhere.
External HDD possibly ? But then the system would see the first 137GB of that HDD anyway...
Some older Chipsets have that limitation: InSystem ISD300A1 (USB2.0), ATA-5 brings it.
Firstly, the limit that was mentioned is 127 G, not 137 G. Secondly, it was determined that avs2avi is to blame and that the OP can create files larger than 127 G with VirtualDub.
Read the thread title and the replies before posting solutions to unrelated problems that were an issue 16 years ago.

[Emulgator]

Yes, me too tired..

[yup]

Hi All!

Now I am try switch from Set MT build to Avisynth+(little change script, replace MT call to SetFilterMTMode and add Prefetch), speed drop from 17 fps to 11 fps, but using Virtualdub I can save whole file, 166 GB.

yup.