How do you tell how big a file is ?

[StainlessS]

Concerning Large File Support, How do you tell how big a BIG file is in Visual CPP 6.0 (1998) ?

I know that fpos_t is 8 bytes in size in VS6 without any kind of configuration (at least with 2003 Platform SDK
1st in include paths), and I know that I can use fsetpos and fgetpos both using fpos_t.

But, as I cannot use fseek (4 byte offset) with it's SEEK_SET, SEEK_CUR, SEEK_END, but I do need to seek to the end of file,
how can I find out how big a BIG file is, so I can seek to the end of file (equivalent to SEEK_END) when I dont know
how big the file is.

Can anybody assist here ?

I want to be able to write RT_Stats DBase and Array files bigger than 2GB.

Perhaps fseek SEEK_END actually does work on big files, can anybody clarify.

[Groucho2004]

This also works for files > 4GB:

WIN32_FIND_DATA fd;
unsigned __int64 iSize = 0;
char szFind[MAX_PATH];
...
//copy file name to szFind
...
HANDLE hFind = FindFirstFile(szFind, &fd);

if (hFind != INVALID_HANDLE_VALUE)
iSize = (((unsigned __int64)fd.nFileSizeHigh) << 32) + (unsigned __int64)fd.nFileSizeLow;

FindClose(hFind);


As for seeking, writing and reading large files I suggest you search google. I don't think that the good old fseek can handle it.

[StainlessS]

Thanx Groucho, I'm sure that will come in handy, but,
Is there any way using FILE * (pointer to FILE), is needed to be used in conjunction with read/write using file pointer,
I am under the impression that fwrite() and fread() work OK with big files (just limited to 2GB chunk max at a time).

[Groucho2004]

Have a look at "_lseeki64", "_write" and "_read".

[mariush]

Open the file using CreateFile (windows api function) : http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx (note, there's also CreateFile2 but it's not required for your particular needs)

CreateFile can create files or open existing files, it all depends on the parameters you pass to it. Read documentation at link above.

Use GetFileSizeEx to get the 64 bit file size : http://msdn.microsoft.com/en-us/library/windows/desktop/aa364957%28v=vs.85%29.aspx

Use SetFilePointerEx to position yourself in the file... http://msdn.microsoft.com/en-us/library/windows/desktop/aa365542%28v=vs.85%29.aspx
As the documentation says, you can also use this function to determine where you are in the file [ex setfilepointerex( filehandle, 0, variable_for_current_position, file_current) ] or to determine the file size by saying setfilepointerex(filehandle,0,variable,file_end) but then you'd have to reset the point to read from beginning.


There's also GetFileInformationByHandle function that could be useful... you need to pass a structure to it and it populates that with info based on a file handle you give it. See http://msdn.microsoft.com/en-us/library/windows/desktop/aa364952%28v=vs.85%29.aspx

[Groucho2004]

Open the file using CreateFile (windows api function) : http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx (note, there's also CreateFile2 but it's not required for your particular needs)

CreateFile can create files or open existing files, it all depends on the parameters you pass to it. Read documentation at link above.

Use GetFileSizeEx to get the 64 bit file size : http://msdn.microsoft.com/en-us/library/windows/desktop/aa364957%28v=vs.85%29.aspx

Use SetFilePointerEx to position yourself in the file... http://msdn.microsoft.com/en-us/library/windows/desktop/aa365542%28v=vs.85%29.aspx
As the documentation says, you can also use this function to determine where you are in the file [ex setfilepointerex( filehandle, 0, variable_for_current_position, file_current) ] or to determine the file size by saying setfilepointerex(filehandle,0,variable,file_end) but then you'd have to reset the point to read from beginning.


There's also GetFileInformationByHandle function that could be useful... you need to pass a structure to it and it populates that with info based on a file handle you give it. See http://msdn.microsoft.com/en-us/library/windows/desktop/aa364952%28v=vs.85%29.aspx
Most of the APIs you mention like GetFileSizeEx() are not available in VC6.

[mariush]

They're Windows API functions ... just #include <windows.h> and if needed download the windows sdk from microsoft, it's free.

As a suggestion to see how others do it, download Virtualdub's source code and browse the source code. It's not perfect (or modern in some instances) but it's readable and interesting.

[TheFluff]

Visual CPP 6.0 (1998) ?

jesus christ upgrade to a compiler from this millennium already, it won't even cost you anything

Is there any way using FILE * (pointer to FILE), is needed to be used in conjunction with read/write using file pointer

_fseeki64() and _ftelli64() are what you want but who knows if they're available in your ancient runtime
(looks like they're not, VS2005 or newer I think)

[StainlessS]

I'm not altogether happy about mixing eg file handles and FILE*, but will investigate all things suggested.
Thanx guys, I guess I'll also try dredging VirtualDubMod source (From TwriteAVI plug as I cant find the full source).

Sleep well my friends.

EDIT: Does anybody know, does fseek SEEK_END actually work for big files ?

[Groucho2004]

if needed download the windows sdk
Good point, the latest platform SDK that is compatible with VS6 does indeed support these functions.

[foxyshadis]

Are you trying to take the mantle from IanB for supporting the most out-of-date development environment? I can't think of a single good reason to use VC6, when newer ones are free; even IanB finally upgraded after a couple of 2.6 alphas proved how limited it was. VC6 has crash bugs you can drive a truck through.

[StainlessS]

NO, so far as I am concerned, IanB has not (As Yet) been declared dead (I see him sometimes lurking about, like something dead),
but seems to be quite alive. Twould be nice if the big man could speak from the grave (Or wherever) and declare to the world
whether he has or has not abandoned the world that is avisynth.
Foxy, you clearly like to take the easy path in life, some of us like the struggle against adversity, we do it just because it's there.
(EDIT: Some people like to climb moutains, WHY)

EDIT: Final Plugs are compiled with VS TK 2003. (Because generally faster, sometimes faster than VS2005/2008 but sometimes VS6 faster)

@TheFluff, Some things that I used to use more than 20+ years ago on Atari and Amiga machines are not implemented well into the future
(present being 1998) on MS machines, I really cannot understand how the MS PC prevailed when the system is/was so lacking.

[Groucho2004]

jesus christ upgrade to a compiler from this millennium already, it won't even cost you anything
Since he uses VC6 mainly for Avisynth plugin development and the latest official Avisynth Alpha is built with VC6, I guess he's following your advice from this post (http://forum.doom9.org/showthread.php?p=1653737#post1653737).

[StainlessS]

HeHe, hoisted by his own petard.

I have in the past tried both VS 2005 and VS 2008 and found that I got a performance hit of something like 20% when compared to
VS TK2003, and I think even VS6 was faster (No alterations to standard optimization config in 2008 or VS6, TK3).

Doing time test comparisons between Avisynth Standard AverageLuma and RT_AverageLuma, I was getting something like 4200FPS (STD) and 5000FPS (RT) in
VS6, some of that would be down to the fact that it would locate RT (external plug) quicker in the name lookup table.
On TK3, was something like 7083FPS(RT) on my lowly dual core Core Duo 2.4GHz.
Actual timing for RT_AverageLuma TK3 7083FPS and 4200FPS for STD in VS6.
TK3 seems to quite efficiently compile these type of loop implementations (more efficient in TK3 than VS6, some processors can handle the --x>=0 and
loop in a single instruction eg Motorola M68K family although x would need to be 16 bit int)

for(x=width; --x>=0;) {
....
}



rather than this


for(x=0; x < width; x++) {
....
}


or this (better than prev on some compilers/processors EDIT: No real difference in either VS6 or TK3)

for(x=0; x < width; ++x) {
....
}


On tight loop, can have a significant performance boost simply by using the --x>=0 thing.

I also did time test in Avisynth+, not much difference in speed between A+ and RT for eg YPlaneMin equivalent and other non AverageLuma functions, but,
A+ blew my socks off with 30,000FPS for AverageLuma using 2008 (I think) intrinsics.

[TurboPascal7]

People like you are the primary reason why we can't have nice things.

[StainlessS]

TheFluff, I think turbo must be addressing you.

[TheFluff]

Since he uses VC6 mainly for Avisynth plugin development and the latest official Avisynth Alpha is built with VC6, I guess he's following your advice from this post (http://forum.doom9.org/showthread.php?p=1653737#post1653737).

Only someone reading my posts like the devil reads the bible could come to that conclusion. Obviously nobody sane would get the idea to use VC6 in 2014. The point of the post was to a) point out that C++ API's are dumb, Jeroi is delusional and that you probably shouldn't be using Avisynth, and b) remind people about the things you have to keep in mind when developing Avisynth plugins (using a modern compiler, natch).

TK3 seems to quite efficiently compile these type of loop implementations (more efficient in TK3 than VS6, some processors can handle the --x>=0 and
loop in a single instruction)
While it is true that if you translated this verbatim into asm, testing for sign inversion or overflow would save a comparison, that's almost certainly not what happens. Fix your benchmarks, check what asm is actually generated and stop writing code like it's 1985. I also wonder what kind of tight loops you're writing that are significantly impacted by the loop counter.

[StainlessS]

Obviously nobody sane would get the idea to use VC6 in 2014.

Your other post was only 1 Year + 4 days ago, how things have changed.

Any kind of tight loops are faster, especially with TK3 7083FPS(RT)" for pretty much the same code in standard
avisynth (4200FPS VS6) and the --x>=0 in RT_AverageLuma.
I dont speak i86 so examining it would be pointless. EDIT: 2 of my compilers on M68K compiled the --x>=0 thing to a single instruction.
stop writing code like it's 1985Nah, I like to write efficient code I'll leave the alternative to you.

[StainlessS]

Here output of timer, I stopped it early as I have to go out

00000006 0.47205281 RT_DebugF: RT_Stats Timing Test
00000007 0.47207168 RT_DebugF: Timing 13x5(65) runs on 100000 Frames (Total Frames=6500000)
00000008 26.10106659 RT_DebugF: RT_YankChain 5.122Secs :19524.66FPS :: Basic GetFrame() [ADJUSTMENT]
00000009 496.45959473 RT_DebugF: YPlaneMin 94.071Secs : 1063.02FPS :: ADJUSTED88.950Secs 1124.23FPS
00000010 868.93853760 RT_DebugF: RT_YPlaneMin 74.495Secs : 1342.37FPS :: ADJUSTED69.373Secs 1441.48FPS
00000011 1339.47485352 RT_DebugF: YPlaneMax 94.110Secs : 1062.59FPS :: ADJUSTED88.988Secs 1123.75FPS
00000012 1711.73962402 RT_DebugF: RT_YPlaneMax 74.452Secs : 1343.15FPS :: ADJUSTED69.330Secs 1442.38FPS
00000013 2197.82421875 RT_DebugF: YPlaneMinMaxDif 97.216Secs : 1028.64FPS :: ADJUSTED92.094Secs 1085.85FPS
00000014 2571.29956055 RT_DebugF: RT_YPlaneMinMaxDif74.694Secs : 1338.79FPS :: ADJUSTED69.573Secs 1437.34FPS
00000015 3036.29052734 RT_DebugF: YPlaneMedian 93.009Secs : 1075.16FPS :: ADJUSTED87.888Secs 1137.82FPS
00000016 3407.98608398 RT_DebugF: RT_YPlaneMedian 74.338Secs : 1345.20FPS :: ADJUSTED69.217Secs 1444.74FPS
00000017 3551.94287109 RT_DebugF: AverageLuma 28.930Secs : 3456.65FPS :: ADJUSTED23.808Secs 4200.26FPS
00000018 3648.14013672 RT_DebugF: RT_AverageLuma 19.239Secs : 5197.90FPS :: ADJUSTED14.117Secs 7083.76FPS


The figures are adjusted by subtracting time taken for basic GetFrame ie removing timing for sourcefilter, script etc and so are timing only
the functions themselves.
Does five timings on each routine and throws away fastest and slowst times then averages remaining 3.

The non AverageLuma functions are slower due to them using a count array rather than single accumulator.


Here the script:

TITLE="RT_Stats Timing Test"
RUNS = 5 # Number of runs of each function, slowest and fastest discarded, timed on average of remainder
NFRAMES=100000 # Number of Frames to use from Source Clip

#AVISource("D:\avs\TEST.avi")
Colorbars(pixel_type="YV12")

Trim(0,NFRAMES-1).KillAudio()


# Function names for Display only
Names="
RT_YankChain
YPlaneMin
RT_YPlaneMin
YPlaneMax
RT_YPlaneMax
YPlaneMinMaxDif
RT_YPlaneMinMaxDif
YPlaneMedian
RT_YPlaneMedian
AverageLuma
RT_AverageLuma
Avisynth_ALL
RT_Avisynth_ALL
RT_ALL
"

TESTS=13 # Total number of test incl RT_Yankchain # EDIT: Should read 'not including RT_Yankchain'

GSCript("""
Function Test_00(clip c){c for(i=0,framecount-1){current_frame=i RT_YankChain(Last)} return c}
Function Test_01(clip c){c for(i=0,framecount-1){current_frame=i YPlaneMin(Last)} return c}
Function Test_02(clip c){c for(i=0,framecount-1){current_frame=i RT_YPlaneMin(Last)} return c}
Function Test_03(clip c){c for(i=0,framecount-1){current_frame=i YPlaneMax(Last)} return c}
Function Test_04(clip c){c for(i=0,framecount-1){current_frame=i RT_YPlaneMax(Last)} return c}
Function Test_05(clip c){c for(i=0,framecount-1){current_frame=i YPlaneMinMaxDifference(Last)} return c}
Function Test_06(clip c){c for(i=0,framecount-1){current_frame=i RT_YPlaneMinMaxDifference(Last)} return c}
Function Test_07(clip c){c for(i=0,framecount-1){current_frame=i YPlaneMedian(Last)} return c}
Function Test_08(clip c){c for(i=0,framecount-1){current_frame=i RT_YPlaneMedian(Last)} return c}
Function Test_09(clip c){c for(i=0,framecount-1){current_frame=i AverageLuma(Last)} return c}
Function Test_10(clip c){c for(i=0,framecount-1){current_frame=i RT_AverageLuma(Last)} return c}
Function Test_11(clip c){c for(i=0,framecount-1){current_frame=i \
YPlaneMin(Last) YPlaneMax(Last) YPlaneMinMaxDifference(Last) YPlaneMedian(Last) AverageLuma(Last)} return c}
Function Test_12(clip c){c for(i=0,framecount-1){current_frame=i RT_YStats(Last,flgs=$1F)} return c} # As Test_11
Function Test_13(clip c){c for(i=0,framecount-1){current_frame=i RT_YStats(Last,flgs=$7F,lo=0,hi=255)} return c}
# As Test_12 + RT_Stdev+RT_YInRange
Assert(RUNS>=3,"RUNS MUST be at least 3")
Frames=FrameCount()
YankAve=0.0 # Adjustment, will be subtracted from timings of non YankChain fns
PStr=RT_String("%s\nTiming %dx%d(%d) runs on %d Frames (Total Frames=%d)",TITLE,TESTS,RUNS,TESTS*RUNS,NFRAMES,TESTS*RUNS*NFRAMES)
RT_debugF("%s",Pstr)
TStart=RT_TimerHP() # Total Runtime Start
For(Test=0,TESTS) {
EStr="Test_"+RT_NumberString(Test,10,2)+"()" # Func to EVALuate
Times=""
for(Run=1,RUNS) {
s=RT_TimerHP() Eval(EStr) e=RT_TimerHP() t = e-s # Time Function
Times=RT_TxtAddStr(Times,String(t))
# RT_DebugF("%s %d] S=%6.2f E=%6.2f Time=%6.2f",EStr,Run,s,e,t)
}
Times=RT_TxtSort(Times,9) # Sort Float strings ascending
Ave = 0.0 #
for(i=1,RUNS-2) {Ave=Ave+Value(RT_TxtGetLine(Times,i))} # excluding slowest and fastest run
Ave=Ave / Float(RUNS-2) # Get Average Time, discarding slowest and fastest run
FPS=Frames / Ave # Ave FPS
Title = RT_StrReplaceMulti(RT_TxtGetLine(Names,Line=Test+1)," "+Chr(10)+Chr(9),Chr(10)+Chr(10)) # Remove any SPACE + TAB
Title = RT_StrPad(Title,18) # Align
OStr=RT_String("%s%6.3fSecs :%8.2fFPS",Title,ave,FPS)
if(Test==0) {
YankAve=Ave # Average of basic GetFrame() ONLY time
OStr=OStr+" :: Basic GetFrame() [ADJUSTMENT]"
} else {
AdjustedTime = Ave-YankAve
AdjustedFPS = Frames / AdjustedTime
OStr=RT_String("%s :: ADJUSTED%6.3fSecs %7.2fFPS",OStr,AdjustedTime,AdjustedFPS)
}
RT_DebugF("%s",OStr)
PStr=RT_TxtAddStr(PStr,OStr)
}
TEnd=RT_TimerHP() # Total Runtime End
TTime=RT_String("Total Runtime = %6.2fMins",(TEnd-TStart)/60.0)
RT_DebugF("%s",TTime)
PStr=RT_TxtAddStr(PStr,TTime)
# RT_DebugF("%s",PStr)
SStr=RT_StrReplace(PStr,Chr(10),"\n") # Convert Chr(10) to '\n' for correct Subtitle display
SubTitle(Sstr,font="Courier New",size=16,lsp=0)
""")


EDIT: The 'Tight Loop' in question from RT_AverageLuma on Planar Y

for(y=0 ; y < hh; y += ystep) {
for (x=ww ; --x>=0 ; ) {
sum += srcp[x];
}
if(sum & 0x80000000) {acc += sum;sum=0;} // avoid overflow (big frame support, not available in AverageLuma)
srcp += ystride;
}

[foxyshadis]

Loop unrolling would be dramatically more beneficial than bashing one byte at a time here, so it's no wonder that isn't particularly well optimized for anymore. Hell, accessing the pointers as ulongs (or ulong longs in 64-bit) would be a great improvement, without having to get into intrinsic/ASM/OpenMP territory.

[LoRd_MuldeR]

_fseeki64() and _ftelli64() are what you want but who knows if they're available in your ancient runtime
(looks like they're not, VS2005 or newer I think)

Either that, or just use the _fstati64() function, which avoids having to move the file pointer around. Also you don't need to go down to the Win32 API this way.

It appears to be available in VS 6.0:
http://msdn.microsoft.com/en-us/library/aa246905%28v=vs.60%29.aspx
#include <sys/stat.h>

__int64 getFileSize64(FILE *const fp)
{
struct _stati64 stat;
if(_fstati64(_fileno(fp), &stat) != 0)
{
return -1; /*some error occurred*/
}
return stat.st_size;
}


I'm not altogether happy about mixing eg file handles and FILE*, but will investigate all things suggested.

On Windows, each FILE* has an associated "native" operating system HANDLE, which you can get via _get_osfhandle() (http://msdn.microsoft.com/de-de/library/ks2530z6%28v=vs.80%29.aspx), if you really need to.

It is returned as an intptr_t, but you can typecast to HANDLE safely. I would still prefer the fstat()-based solution, since it's cross-platform.

[StainlessS]

Thanx everybody for input, I shall investigate all (bit busy doing something else at the moment).

Loop unrolling would be dramatically more beneficial than bashing one byte at a time here

True, I used something like that in ShowChannels(), perhaps a bit of copy and paste is needed.

Hell, accessing the pointers as ulongs (or ulong longs in 64-bit) would be a great improvement, without having to get into intrinsic/ASM/OpenMP territory.

I'm not altogether sure I know what you mean there, are you meaning ULONG *, accessing 4 bytes at a time with shift, and AND extraction ? I have no knowledge of the inner workings or peculiarities of i86 assembler or what instructions execute faster.

EDIT: Perhaps you meant something like,


sum += ((BYTE*)srcp)[x+0];
sum += ((BYTE*)srcp)[x+1];
sum += ((BYTE*)srcp)[x+2];
sum += ((BYTE*)srcp)[x+3];

where srcp is ULONG*, and ULONG aligned.


EDIT: I have SDK 2003 installed but can find no docs on _fseek64 and so thought it was not available, also searched include directory.
It does exist in the src\crt\ directory (header and source) , dont know if that means that it is included in libs or not (I assume it is), I shall have to investigate.

[LoRd_MuldeR]

EDIT: I have SDK 2003 installed but can find no docs on _fseek64 and so thought it was not available, also searched include directory.
It does exist in the src\crt\ directory (header an source) , dont know if that means that it is included in libs or not (I assume it is), I shall have to investigate.

Functions like _fseeki64(), _ftelli64() and _fstati64() are from the C-Runtime (CRT), not the Win32 API, and thus will not be found in the Win32 SDK.

And you don't need to link a special library for those CRT functions, because the VC linker will link against the suitable MSVCRTxxx.DLL automatically, unless you explicitly prevent linking of that library.

The CRT headers should be in "VS_INSTALL_PATH\VC\include" (at least that's how it is with recent VisualStudio). Simply include <sys/stat.h> to get _fstati64(), for example.

[StainlessS]

Windows search on SDK Include directory failed to find _fseek64, but looking into
"C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2\Include\crt\sys\stat.h"
It is there. (EDIT: All files and folders, search system files, search hidden files, search subfolders, case insensitive)

Thank you mi'lud.

EDIT: Just did search again, not found again ???

EDIT: And I really should read the VS documentation one day.

[LoRd_MuldeR]

I think what you are looking for should be _fseeki64(), not _fseek64(). As a little extra challenge for us, Linux/glibc calls this fseeko64() ;)

Also, any reason you prefer determining the file size using the _fseeki64() plus _ftelli64() "workaround" rather than calling _fstati64() right away?

EDIT: I just noticed that the MSVC CRT even has a _filelengthi64() (http://msdn.microsoft.com/en-us/library/aa246860%28v=vs.60%29.aspx) function, available in VC 6.0, which makes it even simpler :D

#include <io.h>

__int64 getFileSize64(FILE *const fp)
{
return _filelengthi64(_fileno(fp));
}

[StainlessS]

Oops yes, I meant _fseeki64, just re-did search again, still not found.

I was just looking for that one to get file location, wanted _ftelli64() too but did not search on that (EDIT: searched on that too, still not found).
Also, thanx for the getFileSize64 thing, I had already located that one on net.

[StainlessS]

OK, think I'll give VS2008 another try,
perhaps that will placate the Fluffy one (why is he so angry, all of the time).

Been having a few crashes lately due to trying a hack up of adb over USB for three Android devices,
think I've screwed my registry and need to re-ghost or do fresh install, might as well get rid of all trace of VS6
and install 2008.