Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 25th November 2008, 16:38   #81  |  Link
hanfrunz
Registered User
 
hanfrunz's Avatar
 
Join Date: Feb 2002
Location: Germany
Posts: 532
I have another wish. Is it possible to mount multiple .avs to one virtual folder or drive? So lets say i have clip1.avs, clip2.avs and i like to have \clips\clip1.avi and \clips\clip2.avi? So i could import two files via drag and drop to my encoder.

that would be realy helpful.

regards,
hanfrunz
hanfrunz is offline   Reply With Quote
Old 25th November 2008, 17:29   #82  |  Link
joe48
AVFS Developer
 
joe48's Avatar
 
Join Date: Nov 2007
Location: Portland OR US
Posts: 48
Quote:
I have another wish. Is it possible to mount multiple .avs to one virtual folder or drive? So lets say i have clip1.avs, clip2.avs and i like to have \clips\clip1.avi and \clips\clip2.avi?
On vista you might be able to fake this using a file search.

If the \clips\ folder is a real folder then this would be difficult. On vista it could maybe be done by having avfs create and delete NTFS symbolic links.

If the \clips\ folder is a virtual folder, then it would require more UI, probably an .avs file list mount file. It also would result in multiple avisynth instances running in a single process, which might be problematic.
joe48 is offline   Reply With Quote
Old 26th November 2008, 12:11   #83  |  Link
hanfrunz
Registered User
 
hanfrunz's Avatar
 
Join Date: Feb 2002
Location: Germany
Posts: 532
Quote:
Originally Posted by joe48 View Post
On vista you might be able to fake this using a file search.
filesearch is a very good idea and should work on xp as well, thanks!
hanfrunz is offline   Reply With Quote
Old 24th December 2008, 09:43   #84  |  Link
mikeytown2
Resize Abuser
 
mikeytown2's Avatar
 
Join Date: Apr 2005
Location: Seattle, WA
Posts: 623
Any chance of adding YUV output as well as RGB?
http://forum.doom9.org/showthread.ph...41#post1218241
mikeytown2 is offline   Reply With Quote
Old 24th December 2008, 13:20   #85  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
@mikeytown2,

It already does.

AVFS outputs the exit format of your Avisynth script, RGB24, RGB32, YUY2 or YV12
IanB is offline   Reply With Quote
Old 27th December 2008, 08:17   #86  |  Link
paulvdb
Registered User
 
Join Date: Jan 2005
Posts: 33
I can't get this to work. I download and installed Pismo and then registered the avfs dll. When I mount an avs file it creates a new folder with that same avs file in it and a file called error.log which just context the text "(null)". Should I do anything else to get it to work? I've tried this with several avs files including the sample.avs that came with AVFS. I'm using Windows XP SP2 with all the latest updates.

Last edited by paulvdb; 27th December 2008 at 08:22.
paulvdb is offline   Reply With Quote
Old 29th December 2008, 14:32   #87  |  Link
paulvdb
Registered User
 
Join Date: Jan 2005
Posts: 33
I managed to solve my problem by upgrading avisynth. Apparently AVFS doesn't work with avisynth 2.56.
paulvdb is offline   Reply With Quote
Old 23rd March 2009, 19:36   #88  |  Link
joe48
AVFS Developer
 
joe48's Avatar
 
Join Date: Nov 2007
Location: Portland OR US
Posts: 48
Buffering optimizations in AVFS/PFMnew PFM AP build

I released a new version of Pismo File Mount Audit Package earlier this month, build 050.

This new version has some substantial changes to buffering and caching to eliminate unnecessary memory copying. Depending on data sources and AVS scripts, there might be some performance increase using AVFS on build 050 and later. No changes to AVFS itself were necessary, so the AVFS build has not been updated.

Build 050 also uses 128kb sized cache buffers in the driver, instead of 256k in build 047-048, and 48k in build 046 and earlier. This only affects client applications that do not do unbuffered I/O.

Joe L.
joe48 is offline   Reply With Quote
Old 17th August 2009, 14:32   #89  |  Link
Quant
Registered User
 
Join Date: Jun 2009
Posts: 1
And how about BoxedApp SDK. I heared, that this SDK is for emulating file system and registry too? Does anybody use this?
Quant is offline   Reply With Quote
Old 11th December 2009, 23:59   #90  |  Link
a451guy451
Xbox Live: o 4lif o
 
a451guy451's Avatar
 
Join Date: Jun 2009
Location: Monrovia, CA
Posts: 64
I love this thing, but it only lets me host virtual files (that my Mac can see) on shared folders physically on the host windoz machine. Is there a way to get files on independent network storage to work, because while SD sourced files work great HD stuff chokes on my ethernet card? Right now, I technically can make virtual files on network storage, but they are only accessible as such via my host machine which defeats the purpose. Or perhaps this is impossible and I need to go buy a 10gbps ethernet card... Thanks for any input!
a451guy451 is offline   Reply With Quote
Old 12th December 2009, 00:31   #91  |  Link
a451guy451
Xbox Live: o 4lif o
 
a451guy451's Avatar
 
Join Date: Jun 2009
Location: Monrovia, CA
Posts: 64
Oops. I think I'm having a FSB bottleneck, not ethernet. Just swapped my penrin host machine with an i7 and it dramatically improved.

Oddly enough, if I connect to my virtual folder from any PC on my network, everything works great. It seems my Mac is having a personal problem (or perhaps quicktime is specifically). Does anybody else make virtualfiles using HD sources and use them on their Mac? I'm low on apple expertise, and am not really sure how to troubleshoot the system for the bottleneck there.

Last edited by a451guy451; 12th December 2009 at 01:28.
a451guy451 is offline   Reply With Quote
Old 21st December 2009, 22:06   #92  |  Link
Dreadkillz
Registered User
 
Join Date: Aug 2009
Posts: 19
Hi, I would really love to use this, but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. Loading the virtual file is A LOT slower than loading the actual script file for me. Also, the heavier that my filters are, the slower it is to load. BTW, I have an XP with AMD 3600 x2
Dreadkillz is offline   Reply With Quote
Old 31st March 2010, 20:30   #93  |  Link
joe48
AVFS Developer
 
joe48's Avatar
 
Join Date: Nov 2007
Location: Portland OR US
Posts: 48
Quote:
Originally Posted by Dreadkillz View Post
Hi, I would really love to use this, but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. Loading the virtual file is A LOT slower than loading the actual script file for me. Also, the heavier that my filters are, the slower it is to load. BTW, I have an XP with AMD 3600 x2
Performance is often slower with AVFS than a frame served or VFW accessed AVS, but it depends on the script and on the program accessing the virtual AVI file. The problem is not that AVFS adds any significant overhead to rendering, because it doesn't (there are file systems built on PFM doing hundreds of MBs per second sustained throughput). The problem is that the programs accessing the virtual AVI files often issue reads that require frames to be rendered that the application does not use.

With the limited types of scripts I use, the performance issue I see is a delay when opening the virtual AVI. This is caused by sloppy reads of the AVI file headers and indexes inadvertently causing unneeded frames to be rendered.

There may be some things that can be done with the tunable AVFS parameters in the script to improve performance. Also, someone with some C/C++ coding experience, who is seeing performance issues, could experiment with caching and/or padding in a custom avfs.dll.
joe48 is offline   Reply With Quote
Old 31st March 2010, 22:26   #94  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
Quote:
Originally Posted by Dreadkillz View Post
... but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. ...
Various tools and explorer plugins like to 'investigate' .AVI files. And then there are virus scanners and security products. It is crucial these never be allowed to read through the contents of the pseudo .AVI files. If they do then the pseudo contents has to be rendered. If it takes 2 hours to render the full Avisynth script, then it will take 2 hours to read through the pseudo .AVI file.

Joe, does PFM have knowledge of the "Application name" when a pseudo file access happens? Could various applications be black listed and the returned contents just be 1 big zero fill. Or maybe a white list so only say VirtualDub.exe, X264.exe and Vegas*.exe get to see the rendered pseudo contents.
Quote:
Originally Posted by joe48 View Post
... The problem is that the programs accessing the virtual AVI files often issue reads that require frames to be rendered that the application does not use.

... This is caused by sloppy reads of the AVI file headers and indexes inadvertently causing unneeded frames to be rendered. ...
In the original implementation, I went for up tight maximum packing in the pseudo AVI file. On reflection perhaps a sparse highly padded approach would have been better.

In an AVI structure 'JUNK' segments can be freely used to align/pad other data segments, this is normally done when capturing uncompressed video to match the clustering alignment of physical disks and maximise performance.

It would be trivial to have the frame data aligned on some nice round offset (say 64K or more) and add a 'JUNK' padding/guard block between each frame chunk. The data contents in the 'JUNK' blocks doesn't even matter, so no actual memory transfer/zero fill needs to occur, further boosting performance.

I assume the marshalling code is happy if no actual data is written to the application's buffer.
_____________________________________

Case study :- On mapped Linux systems the /dev/zero driver can apparently achieve throughputs of 100's of GB/s, way in excess of the actual memory bandwidth of the host CPU. This is achieved by zero filling 1 physical memory page and mapping it repeatedly onto the users buffer. E.g. App does a 1Mb _read of /dev/zero, on a modern CPU one might expect this to take 40 microseconds (25GB/s), profile the _read call and it only takes 5 microseconds (200Gb/s). Woooohoooo!
IanB is offline   Reply With Quote
Old 1st April 2010, 00:17   #95  |  Link
joe48
AVFS Developer
 
joe48's Avatar
 
Join Date: Nov 2007
Location: Portland OR US
Posts: 48
Quote:
Originally Posted by IanB View Post
Various tools and explorer plugins like to 'investigate' .AVI files. And then there are virus scanners and security products. It is crucial these never be allowed to read through the contents of the pseudo .AVI files.

.....

Joe, does PFM have knowledge of the "Application name" when a pseudo file access happens? Could various applications be black listed and the returned contents just be 1 big zero fill. Or maybe a white list so only say VirtualDub.exe, X264.exe and Vegas*.exe get to see the rendered pseudo contents.
No current mechanisms exist in PFM to allow the formatter to change behavior based on the process name. If it were supported, it would only be available during open, not at read.

But, I really do not like this sort of a solution. It looks like a quick fix, but it has a lot of drawbacks.

Examples:
1) Explorer related I/O, such as various shell extensions, is going to happen in the open dialog in applications in the white list.
2) Some of the anti-virus products and system extensions open files and do I/O in the context of the triggering process (this is true of prefetch as well).
3) Some apps/plugins may do work out-of-process, so the user has to track down the name of processes they did not run and add them to the white list.
4) Dllhost.exe, rundll32.exe, and the like.

One thing that should help is to set the FILE_ATTRIBUTE_OFFLINE flag on the virtual media files. This is a trivial change to avfs.dll. I would have done this originally, but it was not supported two years ago when we first put AVFS together. This flag will convince explorer to leave the file alone, only show default icon, etc.. Many 3rd party shell extensions and applications also honor this flag, otherwise they are unusable in enterprise environments that use any sort of HSM. The shell extensions that do not check this flag should be bugged back to the developer.

The anti-virus issues can be handled (usually) with exclusions. Well, if the anti-virus product does not support user exclusions then the user might want to look for another vendor.

Quote:
In the original implementation, I went for up tight maximum packing in the pseudo AVI file. On reflection perhaps a sparse highly padded approach would have been better.

...

It would be trivial to have the frame data aligned on some nice round offset (say 64K or more) and add a 'JUNK' padding/guard block between each frame chunk. The data contents in the 'JUNK' blocks doesn't even matter, so no actual memory transfer/zero fill needs to occur, further boosting performance.
I can not recall if there were any compat issues with the padding between frame and audio chunks. There is the existing AVFS_AVI_NoInterleave option to disable interleaving the audio and video chunks. Some more padding options would probably be useful.

The below thread indicates the no-interleave option solves some performance issues. I am suspicious about the comment that it introduces audio sync issues, but this is the sort of compat trouble that playing with the index and with padding is likely to cause.
http://www.dvinfo.net/forum/canon-eo...tml#post990122
joe48 is offline   Reply With Quote
Old 1st April 2010, 02:15   #96  |  Link
Robert Martens
Registered User
 
Join Date: Feb 2010
Location: New York
Posts: 113
Talk about a small world; after years of lurking, I registered an account here a few weeks ago to post about an Avisynth helper script I'd developed. I ended up revising it practically every other day, though, and thought it best to wait until I'd settled on something more stable before bringing it up here, so I have yet to get around to that (maybe later tonight, actually), but I was just browsing the boards and stumbled across this.

As evidenced by my username, I'm the same guy from the aforementioned DV Info Net thread. It's been over a year since I last experimented with the AVFS as it applies to 5D Mark II Quicktime files, so I'm a little fuzzy on the details, but yes, I do remember that while there was a performance improvement brought about by AVFS_AVI_NoInterleave, we subsequently ran into an issue with A/V sync. Barely being able to "Hello, World!" my way out of a wet paper bag, I was and still am in no position to diagnose the cause, and the simplest solution was just avoiding that option. Ultimately I believe most people started using a Cineform transcoding solution for their 5D (and now 7D and T2i) footage for use in their NLEs.

I still have the sample clip I was using for tests in that DVi thread (on the same system, no less, with the same processor, RAM and hard drives, though now I'm using Avisynth 2.5.8 MT), and I'm happy to help if there's anything I can do as far as testing. Or, at least, shedding some light on that particular discussion. It's been awhile, but I'm sure I could remember the details if you have any questions.
Robert Martens is offline   Reply With Quote
Old 1st April 2010, 16:31   #97  |  Link
joe48
AVFS Developer
 
joe48's Avatar
 
Join Date: Nov 2007
Location: Portland OR US
Posts: 48
Quote:
Originally Posted by Robert Martens View Post
I do remember that while there was a performance improvement brought about by AVFS_AVI_NoInterleave, we subsequently ran into an issue with A/V sync.
....
and I'm happy to help if there's anything I can do as far as testing. Or, at least, shedding some light on that particular discussion. It's been awhile, but I'm sure I could remember the details if you have any questions.
If you get a chance, could you rerun the test with and without the no-interleave option and verify the audio sync issue comes and goes?

Last edited by joe48; 1st April 2010 at 16:34. Reason: spelling
joe48 is offline   Reply With Quote
Old 1st April 2010, 20:20   #98  |  Link
Robert Martens
Registered User
 
Join Date: Feb 2010
Location: New York
Posts: 113
Turns out more had changed than I remembered, in that I had been using the CoreAVC decoder and loading the fake AVI files into a trial copy of Vegas Pro. I've downloaded Vegas too many times now (I've needed it more than once to help someone diagnose a problem; it now tells me the trial period has expired, so I can't register the download), and pathetic as it may seem I can't throw any money at CoreAVC for the moment.

I have nonetheless confirmed the behavior in question by other means. I have ffdshow-tryouts version 3154 installed, and I have the H.264/AVC "decoder" dropdown set to ffmpeg-mt for both DShow and VfW, since I find that performs best on my system.

I created a simple script, AVFSsynctest.avs, containing these lines:

AVFS_AVI_NoInterleave=false
DirectShowSource("C:\Downloads\mvi_1419.mov")
ConvertToRGB24(matrix="PC.709")

The source clip doesn't appear to be in the same place as when I grabbed it, but that Vimeo user referenced in the DVi thread has made the clip available here: http://drop.io/larktav

I swapped NoInterleave between true and false, and commented out the ConvertToRGB line for a couple of tests. The script was mounted with the latest PFM/AVFS, the files were opened in Liquid 7.2, combustion 3.0.4, and MediaInfo, and the script was explicitly Unmounted between each change. Results were as follows:

1.) NoInterleave=false->DirectShowSource->ConvertToRGB24->Liquid 7.2

On a 1080/30p timeline (exactly 30p, not 29.97), everything displays as if it's fine; the audio in the AVI file shows up as being the same duration as the video, as do the two WAV files generated (AVFSsynctest.wav and AVFSsynctest.00.wav). I'm inclined to believe this is a quirk of the NLE, however, thanks to tests 3 through 8.

2.) NoInterleave=true->DirectShowSource->ConvertToRGB24->Liquid 7.2

Same as 1; everything looks fine, durations all appear to match.

3.) NoInterleave=false->DirectShowSource->combustion 3.0.4

Here's where the first sign of trouble begins. The video track is reported as 491 frames, but audio is reported as 490 whether I load it from the AVI container or either of the WAVs.

4.) NoInterleave=true->DirectShowSource->combustion 3.0.4

Even stranger; the durations behave as in test 3, but trying to load the AVI as an audio source causes only the last ten frames or so to display a waveform in the combustion interface; further, the audio track is silent even during those ten frames. The WAVs as audio sources had the 490 duration, but played back without issue.

5.) NoInterleave=false->DirectShowSource->ConvertToRGB24->combustion 3.0.4

For tests 3 and 4 I had initially removed the ConvertToRGB line, as combustion doesn't require it, but thinking it might have some impact on my results I restored the line and tried again. Results were identical to test 3.

6.) NoInterleave=true->DirectShowSource->ConvertToRGB24->combustion 3.0.4

Same as number 4.

7.) NoInterleave=false->DirectShowSource->ConvertToRGB24->MediaInfo

Finally, I loaded the phantom files in MediaInfo 0.7.19 to see what it could tell me, and found that in this case the AVI video track showed a duration of 16s 366ms, and the audio a duration of 16s 333ms. The external files, AVFSsynctest.wav and AVFSsynctest.00.wav, were also both 16s 333ms.

8.) NoInterleave=true->DirectShowSource->ConvertToRGB24->MediaInfo

Changing only the NoInterleave option, this time I found the AVI video track was its expected 16s 366ms, but the audio packed into the AVI had shrunk to 15s 867ms. AVFSsynctest.wav and AVFSsynctest.00.wav, though, were both 16s 333ms, as I'd seen in the other examinations.

All of this is in line with what we experienced in that thread; I was unclear at first, but as soon as I saw those numbers in MediaInfo I remembered running into the same audio-shrink last time.

Last edited by Robert Martens; 1st April 2010 at 20:25.
Robert Martens is offline   Reply With Quote
Old 1st April 2010, 22:22   #99  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
Joe, Seems you have a bug, you forget to put back the 0.5 secs of preload when not in interleave mode.
IanB is offline   Reply With Quote
Old 2nd April 2010, 03:58   #100  |  Link
joe48
AVFS Developer
 
joe48's Avatar
 
Join Date: Nov 2007
Location: Portland OR US
Posts: 48
@Robert, I sent you a PM with a link to a test build of AVFS. It would be great if you could give it a look.
joe48 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 03:02.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.