Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
21st November 2014, 00:20 | #1 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
How do you tell how big a file is ?
Concerning Large File Support, How do you tell how big a BIG file is in Visual CPP 6.0 (1998) ?
I know that fpos_t is 8 bytes in size in VS6 without any kind of configuration (at least with 2003 Platform SDK 1st in include paths), and I know that I can use fsetpos and fgetpos both using fpos_t. But, as I cannot use fseek (4 byte offset) with it's SEEK_SET, SEEK_CUR, SEEK_END, but I do need to seek to the end of file, how can I find out how big a BIG file is, so I can seek to the end of file (equivalent to SEEK_END) when I dont know how big the file is. Can anybody assist here ? I want to be able to write RT_Stats DBase and Array files bigger than 2GB. Perhaps fseek SEEK_END actually does work on big files, can anybody clarify.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? |
21st November 2014, 00:28 | #2 | Link |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
This also works for files > 4GB:
Code:
WIN32_FIND_DATA fd; unsigned __int64 iSize = 0; char szFind[MAX_PATH]; ... //copy file name to szFind ... HANDLE hFind = FindFirstFile(szFind, &fd); if (hFind != INVALID_HANDLE_VALUE) iSize = (((unsigned __int64)fd.nFileSizeHigh) << 32) + (unsigned __int64)fd.nFileSizeLow; FindClose(hFind); Last edited by Groucho2004; 21st November 2014 at 00:38. |
21st November 2014, 00:38 | #3 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Thanx Groucho, I'm sure that will come in handy, but,
Is there any way using FILE * (pointer to FILE), is needed to be used in conjunction with read/write using file pointer, I am under the impression that fwrite() and fread() work OK with big files (just limited to 2GB chunk max at a time).
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? |
21st November 2014, 01:10 | #5 | Link |
Registered User
Join Date: Dec 2008
Posts: 589
|
Open the file using CreateFile (windows api function) : http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx (note, there's also CreateFile2 but it's not required for your particular needs)
CreateFile can create files or open existing files, it all depends on the parameters you pass to it. Read documentation at link above. Use GetFileSizeEx to get the 64 bit file size : http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx Use SetFilePointerEx to position yourself in the file... http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx As the documentation says, you can also use this function to determine where you are in the file [ex setfilepointerex( filehandle, 0, variable_for_current_position, file_current) ] or to determine the file size by saying setfilepointerex(filehandle,0,variable,file_end) but then you'd have to reset the point to read from beginning. There's also GetFileInformationByHandle function that could be useful... you need to pass a structure to it and it populates that with info based on a file handle you give it. See http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx |
21st November 2014, 01:14 | #6 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
|
|
21st November 2014, 01:24 | #7 | Link |
Registered User
Join Date: Dec 2008
Posts: 589
|
They're Windows API functions ... just #include <windows.h> and if needed download the windows sdk from microsoft, it's free.
As a suggestion to see how others do it, download Virtualdub's source code and browse the source code. It's not perfect (or modern in some instances) but it's readable and interesting. |
21st November 2014, 01:43 | #8 | Link | |
Excessively jovial fellow
Join Date: Jun 2004
Location: rude
Posts: 1,100
|
jesus christ upgrade to a compiler from this millennium already, it won't even cost you anything
Quote:
(looks like they're not, VS2005 or newer I think) Last edited by TheFluff; 21st November 2014 at 01:55. |
|
21st November 2014, 02:26 | #9 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
I'm not altogether happy about mixing eg file handles and FILE*, but will investigate all things suggested.
Thanx guys, I guess I'll also try dredging VirtualDubMod source (From TwriteAVI plug as I cant find the full source). Sleep well my friends. EDIT: Does anybody know, does fseek SEEK_END actually work for big files ?
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 21st November 2014 at 02:31. |
21st November 2014, 03:06 | #11 | Link |
Angel of Night
Join Date: Nov 2004
Location: Tangled in the silks
Posts: 9,559
|
Are you trying to take the mantle from IanB for supporting the most out-of-date development environment? I can't think of a single good reason to use VC6, when newer ones are free; even IanB finally upgraded after a couple of 2.6 alphas proved how limited it was. VC6 has crash bugs you can drive a truck through.
|
21st November 2014, 03:28 | #12 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
NO, so far as I am concerned, IanB has not (As Yet) been declared dead (I see him sometimes lurking about, like something dead),
but seems to be quite alive. Twould be nice if the big man could speak from the grave (Or wherever) and declare to the world whether he has or has not abandoned the world that is avisynth. Foxy, you clearly like to take the easy path in life, some of us like the struggle against adversity, we do it just because it's there. (EDIT: Some people like to climb moutains, WHY) EDIT: Final Plugs are compiled with VS TK 2003. (Because generally faster, sometimes faster than VS2005/2008 but sometimes VS6 faster) @TheFluff, Some things that I used to use more than 20+ years ago on Atari and Amiga machines are not implemented well into the future (present being 1998) on MS machines, I really cannot understand how the MS PC prevailed when the system is/was so lacking.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 21st November 2014 at 04:50. |
21st November 2014, 10:46 | #13 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
|
|
21st November 2014, 14:08 | #14 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
HeHe, hoisted by his own petard.
I have in the past tried both VS 2005 and VS 2008 and found that I got a performance hit of something like 20% when compared to VS TK2003, and I think even VS6 was faster (No alterations to standard optimization config in 2008 or VS6, TK3). Doing time test comparisons between Avisynth Standard AverageLuma and RT_AverageLuma, I was getting something like 4200FPS (STD) and 5000FPS (RT) in VS6, some of that would be down to the fact that it would locate RT (external plug) quicker in the name lookup table. On TK3, was something like 7083FPS(RT) on my lowly dual core Core Duo 2.4GHz. Actual timing for RT_AverageLuma TK3 7083FPS and 4200FPS for STD in VS6. TK3 seems to quite efficiently compile these type of loop implementations (more efficient in TK3 than VS6, some processors can handle the --x>=0 and loop in a single instruction eg Motorola M68K family although x would need to be 16 bit int) Code:
for(x=width; --x>=0;) { .... } rather than this Code:
for(x=0; x < width; x++) { .... } Code:
for(x=0; x < width; ++x) { .... } I also did time test in Avisynth+, not much difference in speed between A+ and RT for eg YPlaneMin equivalent and other non AverageLuma functions, but, A+ blew my socks off with 30,000FPS for AverageLuma using 2008 (I think) intrinsics.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 21st November 2014 at 20:41. |
21st November 2014, 16:25 | #16 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
TheFluff, I think turbo must be addressing you.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? |
21st November 2014, 16:39 | #17 | Link | |
Excessively jovial fellow
Join Date: Jun 2004
Location: rude
Posts: 1,100
|
Quote:
While it is true that if you translated this verbatim into asm, testing for sign inversion or overflow would save a comparison, that's almost certainly not what happens. Fix your benchmarks, check what asm is actually generated and stop writing code like it's 1985. I also wonder what kind of tight loops you're writing that are significantly impacted by the loop counter. Last edited by TheFluff; 21st November 2014 at 17:04. |
|
21st November 2014, 17:18 | #18 | Link | ||
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Quote:
Any kind of tight loops are faster, especially with TK3 7083FPS(RT)" for pretty much the same code in standard avisynth (4200FPS VS6) and the --x>=0 in RT_AverageLuma. I dont speak i86 so examining it would be pointless. EDIT: 2 of my compilers on M68K compiled the --x>=0 thing to a single instruction. Quote:
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 21st November 2014 at 20:47. |
||
21st November 2014, 19:22 | #19 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Here output of timer, I stopped it early as I have to go out
Code:
00000006 0.47205281 RT_DebugF: RT_Stats Timing Test 00000007 0.47207168 RT_DebugF: Timing 13x5(65) runs on 100000 Frames (Total Frames=6500000) 00000008 26.10106659 RT_DebugF: RT_YankChain 5.122Secs :19524.66FPS :: Basic GetFrame() [ADJUSTMENT] 00000009 496.45959473 RT_DebugF: YPlaneMin 94.071Secs : 1063.02FPS :: ADJUSTED88.950Secs 1124.23FPS 00000010 868.93853760 RT_DebugF: RT_YPlaneMin 74.495Secs : 1342.37FPS :: ADJUSTED69.373Secs 1441.48FPS 00000011 1339.47485352 RT_DebugF: YPlaneMax 94.110Secs : 1062.59FPS :: ADJUSTED88.988Secs 1123.75FPS 00000012 1711.73962402 RT_DebugF: RT_YPlaneMax 74.452Secs : 1343.15FPS :: ADJUSTED69.330Secs 1442.38FPS 00000013 2197.82421875 RT_DebugF: YPlaneMinMaxDif 97.216Secs : 1028.64FPS :: ADJUSTED92.094Secs 1085.85FPS 00000014 2571.29956055 RT_DebugF: RT_YPlaneMinMaxDif74.694Secs : 1338.79FPS :: ADJUSTED69.573Secs 1437.34FPS 00000015 3036.29052734 RT_DebugF: YPlaneMedian 93.009Secs : 1075.16FPS :: ADJUSTED87.888Secs 1137.82FPS 00000016 3407.98608398 RT_DebugF: RT_YPlaneMedian 74.338Secs : 1345.20FPS :: ADJUSTED69.217Secs 1444.74FPS 00000017 3551.94287109 RT_DebugF: AverageLuma 28.930Secs : 3456.65FPS :: ADJUSTED23.808Secs 4200.26FPS 00000018 3648.14013672 RT_DebugF: RT_AverageLuma 19.239Secs : 5197.90FPS :: ADJUSTED14.117Secs 7083.76FPS the functions themselves. Does five timings on each routine and throws away fastest and slowst times then averages remaining 3. The non AverageLuma functions are slower due to them using a count array rather than single accumulator. Here the script: Code:
TITLE="RT_Stats Timing Test" RUNS = 5 # Number of runs of each function, slowest and fastest discarded, timed on average of remainder NFRAMES=100000 # Number of Frames to use from Source Clip #AVISource("D:\avs\TEST.avi") Colorbars(pixel_type="YV12") Trim(0,NFRAMES-1).KillAudio() # Function names for Display only Names=" RT_YankChain YPlaneMin RT_YPlaneMin YPlaneMax RT_YPlaneMax YPlaneMinMaxDif RT_YPlaneMinMaxDif YPlaneMedian RT_YPlaneMedian AverageLuma RT_AverageLuma Avisynth_ALL RT_Avisynth_ALL RT_ALL " TESTS=13 # Total number of test incl RT_Yankchain # EDIT: Should read 'not including RT_Yankchain' GSCript(""" Function Test_00(clip c){c for(i=0,framecount-1){current_frame=i RT_YankChain(Last)} return c} Function Test_01(clip c){c for(i=0,framecount-1){current_frame=i YPlaneMin(Last)} return c} Function Test_02(clip c){c for(i=0,framecount-1){current_frame=i RT_YPlaneMin(Last)} return c} Function Test_03(clip c){c for(i=0,framecount-1){current_frame=i YPlaneMax(Last)} return c} Function Test_04(clip c){c for(i=0,framecount-1){current_frame=i RT_YPlaneMax(Last)} return c} Function Test_05(clip c){c for(i=0,framecount-1){current_frame=i YPlaneMinMaxDifference(Last)} return c} Function Test_06(clip c){c for(i=0,framecount-1){current_frame=i RT_YPlaneMinMaxDifference(Last)} return c} Function Test_07(clip c){c for(i=0,framecount-1){current_frame=i YPlaneMedian(Last)} return c} Function Test_08(clip c){c for(i=0,framecount-1){current_frame=i RT_YPlaneMedian(Last)} return c} Function Test_09(clip c){c for(i=0,framecount-1){current_frame=i AverageLuma(Last)} return c} Function Test_10(clip c){c for(i=0,framecount-1){current_frame=i RT_AverageLuma(Last)} return c} Function Test_11(clip c){c for(i=0,framecount-1){current_frame=i \ YPlaneMin(Last) YPlaneMax(Last) YPlaneMinMaxDifference(Last) YPlaneMedian(Last) AverageLuma(Last)} return c} Function Test_12(clip c){c for(i=0,framecount-1){current_frame=i RT_YStats(Last,flgs=$1F)} return c} # As Test_11 Function Test_13(clip c){c for(i=0,framecount-1){current_frame=i RT_YStats(Last,flgs=$7F,lo=0,hi=255)} return c} # As Test_12 + RT_Stdev+RT_YInRange Assert(RUNS>=3,"RUNS MUST be at least 3") Frames=FrameCount() YankAve=0.0 # Adjustment, will be subtracted from timings of non YankChain fns PStr=RT_String("%s\nTiming %dx%d(%d) runs on %d Frames (Total Frames=%d)",TITLE,TESTS,RUNS,TESTS*RUNS,NFRAMES,TESTS*RUNS*NFRAMES) RT_debugF("%s",Pstr) TStart=RT_TimerHP() # Total Runtime Start For(Test=0,TESTS) { EStr="Test_"+RT_NumberString(Test,10,2)+"()" # Func to EVALuate Times="" for(Run=1,RUNS) { s=RT_TimerHP() Eval(EStr) e=RT_TimerHP() t = e-s # Time Function Times=RT_TxtAddStr(Times,String(t)) # RT_DebugF("%s %d] S=%6.2f E=%6.2f Time=%6.2f",EStr,Run,s,e,t) } Times=RT_TxtSort(Times,9) # Sort Float strings ascending Ave = 0.0 # for(i=1,RUNS-2) {Ave=Ave+Value(RT_TxtGetLine(Times,i))} # excluding slowest and fastest run Ave=Ave / Float(RUNS-2) # Get Average Time, discarding slowest and fastest run FPS=Frames / Ave # Ave FPS Title = RT_StrReplaceMulti(RT_TxtGetLine(Names,Line=Test+1)," "+Chr(10)+Chr(9),Chr(10)+Chr(10)) # Remove any SPACE + TAB Title = RT_StrPad(Title,18) # Align OStr=RT_String("%s%6.3fSecs :%8.2fFPS",Title,ave,FPS) if(Test==0) { YankAve=Ave # Average of basic GetFrame() ONLY time OStr=OStr+" :: Basic GetFrame() [ADJUSTMENT]" } else { AdjustedTime = Ave-YankAve AdjustedFPS = Frames / AdjustedTime OStr=RT_String("%s :: ADJUSTED%6.3fSecs %7.2fFPS",OStr,AdjustedTime,AdjustedFPS) } RT_DebugF("%s",OStr) PStr=RT_TxtAddStr(PStr,OStr) } TEnd=RT_TimerHP() # Total Runtime End TTime=RT_String("Total Runtime = %6.2fMins",(TEnd-TStart)/60.0) RT_DebugF("%s",TTime) PStr=RT_TxtAddStr(PStr,TTime) # RT_DebugF("%s",PStr) SStr=RT_StrReplace(PStr,Chr(10),"\n") # Convert Chr(10) to '\n' for correct Subtitle display SubTitle(Sstr,font="Courier New",size=16,lsp=0) """) Code:
for(y=0 ; y < hh; y += ystep) { for (x=ww ; --x>=0 ; ) { sum += srcp[x]; } if(sum & 0x80000000) {acc += sum;sum=0;} // avoid overflow (big frame support, not available in AverageLuma) srcp += ystride; }
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 22nd November 2014 at 17:52. |
23rd November 2014, 13:20 | #20 | Link |
Angel of Night
Join Date: Nov 2004
Location: Tangled in the silks
Posts: 9,559
|
Loop unrolling would be dramatically more beneficial than bashing one byte at a time here, so it's no wonder that isn't particularly well optimized for anymore. Hell, accessing the pointers as ulongs (or ulong longs in 64-bit) would be a great improvement, without having to get into intrinsic/ASM/OpenMP territory.
|
Tags |
file system, large file support |
|
|