no luck playing mp4 from an xcd

[avih]

ok, test7f was sent. hopefully, this version WILL be faster.

changelog (test7f): removed debug messages during playback.

under normal conditions, the filter sends a debug message to the OS appx. once every 3 seconds, however, in the case of the 3ivx splitter, it sends about 1,000,000 (1 million) messages before the 1st frame is displayed. this could be the reason for the slowdown, 100% cpu, etc.

your feedback is greatly appreciated.
thx again,
avih

[P0l1m0rph1c]

Yay! Way faster, no doubt about it :)

Takes less than a sec to load here now, just as much time as a regular cd.

Nice work! :thanks:

[avih]

Originally posted by P0l1m0rph1c
Yay! Way faster, no doubt about it :)

Takes less than a sec to load here now, just as much time as a regular cd.

Nice work! :thanks:
:) at last. good.
guys, i'd like some more confirmations pls, and if all is well, i'll release this version publicly later. pls give this version as much test cases as u can, i want the public version to be as bug-free as i can.

thx again,
avih

[calinb]

Thumbs up, way up!!! :) 7f is working very well indeed, avih. xcd2file is working well too. I've run quite a few codecs with it and will continue to watch a variety of movies through the weekend. Many thanks to your efforts and thanks to shitowax for his interest and help too.

[avih]

Originally posted by calinb
Thumbs up, way up!!! :) 7f is working very well indeed, avih. xcd2file is working well too. I've run quite a few codecs with it and will continue to watch a variety of movies through the weekend. Many thanks to your efforts and thanks to shitowax for his interest and help too.
:) thx. enjoy your movies ;) (and pls report back later)

the trigger for the (right) guess about the debug messages as the cause for the slowdown was calinb's report of 'proportional load times' and shitowax's report of 100% cpu. the description of the file access pattern from shitowax helped much as well. the debug messages do not affect the performance under 'normal' conditions, but they definately do when requesting half a million requests from the filter before the 1st frame is displayed. my bad for not figuring it out earlier. nevertheless, due to the great testers that u were, and the quality of your reports, we've managed to solved the issue. many thx again to all of u.

of course, the new cache system improve the performance considerably as well, but the final acceleration came from the removal of the debug messages. how dumb of me ;) cheers guys.

:)
avih

ps.
shitowax, i still think that u'll gain much in other scenarios as well with your new cache system. congrats for that one ;) (although to test your new system you'll have to use the 'old' test6b release of the filter, as the new one does the hard work for u anyway ;)

[shitowax]

My personnal opinion is that it's not up to the splitter to perform caching. The source knows in theory much better the physical layer, sector size, bottleneck factors, ... As the 3ivx splitter totally abstracts the physical layer, it requires bytes only when it needs them. Of course this causes problems with sources that assume things about the file request patterns like avih's XCD did and like other sources do (for example we have problem reports when serving files from MacOSX to windows via samba)... For the moment the latest 3ivx alpha splitter contains a cache of previous bytes read, it's fairly interesting with poorly muxed streams. The 4.5.2 or maybe 4.6 one should contain something a bit smarter ;)

[avih]

shitowax, i generally agree with you. EVERY layer should do it's best. in our case, the filter should indeed have provided some sort of caching. i didn't implement caching since i, as yourself, left the caching (and possibly prefetch) to underlaying layer. in my case it's the windows OS. in your case, it's my filter (and the filter has NO overhead when assumming the OS cache). and we now know that the OS cache is bad in our case. the filter 'cache' was bad as well due to this fact. in order for our code to work well together, I had to OVERRIDE the OS cache mechanism, a thing i didn't want to do. it's not really overriding, but providing another layer of caching.

as i said, test6b had no cache, and it performed pretty well because the HIGHER layers usually did well, and asked for the amount of data that they KNEW they'd use. in our case (3ivx and xcd) it performed bad since NEITHER layer did what it should have done.

generally speaking, caching/prefetching without prior knowledge about the access pattern would always have a performance hit (should it be memory usage, cpu load, etc). my cache system indeed introduces such performance hit (although neglectable). it uses several hundreds of KB for auxiliary storage, and it has a tiny cpu overhead for searching the cache for every request. a performance hit that wouldn't have been there should your splitter behaved 'properly'. that's why i made it cache only 64 sectors. to make the cpu overhead of my cache system as minimal as possible. i do NOT know how will my 'clients' ask the data. so the cache is provided for clients that don't do THEIR job well. kind of protecting bad 'users' from hurting themselves. this introduces an overhead for 'users' that KNOW to behave well.

and this brings us to the bottom line. the only component in this pipe that actually know the access pattern is the object that request the data. therefore, the requesting object MUST request the data in an optimized manner. an optimized manner in our case is an optimized FILE-SYSTEM access. and file systems are slow. that's a known fact. you should NOT assume that the filter abstracts the file into a huge MEMORY space, and ask the data in chunks of 4 bytes.

imagine you would have implemented a function that copies a file to another file. let's say the file's size is 300M. would u read/write the file byte by byte and assume that the OS will cache/prefetch it in such a way that the performance are maximized? or would u read the file in least in 1K chunks??? it's a file!! the cache is for 'users' of the file system functions that don't take into account the fact that they're fetching data from a FILE.

nevertheless, i think that the my cache system is worth it. since there could be more clients that misbehave, and it will give the final user an improved perfocmance in such cases. the downside is the performance hit on the rest of the 'good' cases, but i've written the cache in such a way that it would be neglectable.

do u think that the performance hit you're experiencing with samba is due to the bad implementation on the samba's side? maybe on OS/X's side? if that's what you're thinking then you're mistaking. it's due to the fact that every access has a higher overhead than access to a local file system. it's up to the requesting object to optimize its access pattern. what u should assume is that file access HAS overhead. the amount of it depends on the media and medium.

u should assume that the time it takes for a request (for any kind of external data) to complete is:

singleRequestTime=REQUEST_SIZE*throughputFactor+fixedLatency

both throughputFactor and fixedLatency are out of your control, but u should assume they're there (and they're always there). not taking this factor into account will result in worse total performance. since u can't control them, u should optimize your access pattern such that the total time is minimized. the total time calculation is NUMBER_OF_REQUESTS*singleRequestTime. or:

totalAccessTime=TOTAL_REQUESTED_BYTES*throughputFactor+NUMBER_OF_REQUESTS*fixedLatency

it's easy to see that the only factor that u can control here is the number of requests. the lower it is, the lower the total access time will be. if you use many requests, you're in the mercy of the fixedLatency constant and/or some kind of caching system (again, which u can't control). you MAY be lucky, but you may also not be so lucky, as is the case of the xcd filter and the samba issue.

the only GOOD solution is to optimize NUMBER_OF_REQUESTS according to your prior knowledge as for the nature of your requests. if you know you're going to search 2M, that you should ask for 2M, and then parse it from memory, where u know that both of these factors are considerably lower than these of a file system abstraction.

that's it ;) as i said, i think that your decision to implement the cache will have much gain in other scenarios as well. u reassured my assumption with at least one case in which it currently performs badly.

[Phobos]

sorry about the delay

something must be VERY bad about my system. Ive seen improvement in the lastest version indeed. In the first one it took 4 min to show the error in wmp, in the second it took a couple of minutes, now it takes 20 seconds to show the error. Note that this is from Daemon-tools so my supposed drive crappyness is out of the question since polimorphic had success. Anyway ill try reinstalling 3ivX, adding the registry hack and rebootin and lets see if that helps.

EDIT
Alright, i tried the method mentioned above and i still get this message:

"ClassFactory no puede suministrar a la clase requerida (Error=80040111)"

its something like "ClassFactory cant retrieve the required class"

anybody knows anything about this errror?

[P0l1m0rph1c]

@Phobos, just a quick question. What extension do you use for the m2f2 file? Are you using .mp4 or .dat? I just can't get things to work if i use other extension rather than .dat. That could be a problem.

[Phobos]

i use dat of course

i also try graphedit, it shows the very same error and even faster than wmp.. wtf?

[shitowax]

Are you sure you don't use gabest's xcd source instead of avih one ?

[calinb]

Originally posted by avih
the trigger for the (right) guess about the debug messages as the cause for the slowdown was calinb's report of 'proportional load times' and shitowax's report of 100% cpu. Thanks for this information, avih. Sometimes when users try to help developers, our reports just fall into a "black hole" and we never find out if our reports are helpful. I appreciate your efforts to keep us informed, though I realize such communication takes valuable time away from development and your "day jobs," studies, etc.

I'll continue to feedback my experiences using your excellent filter.

[avih]

Originally posted by calinb
Thanks for this information, avih. Sometimes when users try to help developers, our reports just fall into a "black hole" and we never find out if our reports are helpful. I appreciate your efforts to keep us informed, though I realize such communication takes valuable time away from development and your "day jobs," studies, etc.

I'll continue to feedback my experiences using your excellent filter.
;) thx

[snakeman]

Please send me the 7f filters to panther182@yousef.com

I would love to test it on my server 2003 install for compat testing.


plus I am still on the fence nero digital or xvid, but thats another topic.



snakeman

[avih]

Originally posted by snakeman
Please send me the 7f filters to panther182@yousef.com
I would love to test it on my server 2003 install for compat testing.
snakeman
sent. pls report back to this thread.

[snakeman]

Stream 0

Media Type 0:
--------------------------
AM_MEDIA_TYPE:
majortype: MEDIATYPE_Stream {E436EB83-524F-11CE-9F53-0020AF0BA770}
subtype: MEDIASUBTYPE_QTMovie {E436EB89-524F-11CE-9F53-0020AF0BA770}
formattype: TIME_FORMAT_NONE {00000000-0000-0000-0000-000000000000}
bFixedSizeSamples: 1
bTemporalCompression: 0
lSampleSize: 1
cbFormat: 0

results from media player classic,


Windows Media Player encountered an unknown error.



results from WM9


Details are as follows

previous version was 6B everything worked fine except nero (xcd) with daemon tools

OS Server 2003
cpu dual processor MP2000 AMD
Mem 512 megs of ram
400 gigs of raid 0+1 for video and audio recording
ldw-811s for writing 99min XCDs :)

using DVX latest version for XVID/RV10/DiVX/OGG creation

XVID rc2 DiVX 5.11 pro RV10 beta ..............

Did I miss anything?
Everything works fine except nero digital .mp4 files in XCD format.

snakeman

[calinb]

Originally posted by snakeman
Did I miss anything?
Everything works fine except nero digital .mp4 files in XCD format.
snakeman With nero mp4/XCD, you must use the 3ivx splitter, which means you must install 3ivx after nero or any nero updates. Otherwise, Directshow will use the nero filesource/splitter filter which doesn't work here. Surprisingly, I've found that the nero Showtime player also works with mp4/XCD, but I prefer MPC.

If using MPC, make sure that the internal MPC VCD/SVCD/XCD Reader is unchecked in the filters options page because it doesn't work with mp4/XCD.

If you still have problems, try to render the XCD with graphedit and see what happens.

[snakeman]

Trying it now, hmmm I installed 3ivx not just the splitter, and it worked. It took about 2 sec to start on my P4 2.4gig 512 1066 rambus
box with all scsi drives. Oops almost forgot using Daemon tools I haven't burned a cd yet.


snakeman

[avih]

snakeman, could u pls answer explicitly:
have u or have u not found a scenario in which the filters test6b and test7f behave differently?

if you have found such scenario, what's the difference?
thx
avih

[snakeman]

6b (with or without 3ivx installed,it just hangs and then it gives the error I posted above)does not work for me on any of my boxes, Server2003 or XPsp1. 7f works with 3ivx installed on XPsp1, I have not tested my other two machines yet I have an appointment in a few mins.



snakeman

[avih]

thx. sounds good so far. pls let us know when u test on your other machines.

thx again,
avih

[snakeman]

Ok here we go, it works fine on my server2003 box. But, I am getting some strange error messages and nero showtime will not work at all. The error messages only occur when closing the xcd. The messages occur with both WMP9 and WMPclassic 6.4.xxxxxx I will post the error message later on tonite.




snakeman

[snakeman]

http://cardude.dyndns.org/~byron/error.jpg

[avih]

this is crutial. you MUST compare the results with test6b. otherwise i can't tell whether it is a new bug that i introduces in test7f, or it's the same problem as with test6b.

[snakeman]

Oops sorry for the incomplete post. Basically it was the same as before. It does not work at all with 6b installed with or without 3ivx. But, works as stated above with 7F installed andbv 3ivx installed with the error messages. Strange, nero showtime doesn't work at all. On the Server2003 installations(XCD). Again using Daemon tools to play the XCDs. I have not burned a disk yet.

Machines:

Dually MP 2000 AMD with 512 megs of ram etc
other 2003 Server box
P4 2.6 gig 512 megs of ram and the usual suspects of 200 gigs of raid space, etc

I hope this helps


snakeman

[avih]

ok, so test7f behaves differently on different machines. it's working on one of your 2003 servers, and not working on others. it didn't work with test6b on any machine. correct?

i'd like you to supply me with some more info pls. go here (http://www.sysinternals.com/ntw2k/freeware/debugview.shtml) and download debugview (small freeware utility that logs system messages). run it before you open the video file. then open the file, and wait till it crashes. export the log from debugview, zip it, and email it to me pls. i'll try to further analyze it.

thx in advance.
avih

[snakeman]

Incorrect, it does not work on any of my Server2003 machines. I will send you the debug results.



snakeman

[bond]

as the error seems to have to do with timemanager.ax i would propose to unregister it
afaik this timemanager filter was developed to make native matroska possible, but isnt needed anymore for anything

[snakeman]

Send details on how to unregister it.



snakeman

[P0l1m0rph1c]

Go to start menu -> Run. Then type:

regsvr32 /u timemanager.ax

Then windows shall give you a message saying if it was unregistered successfully or not.

[Phobos]

Originally posted by shitowax
Are you sure you don't use gabest's xcd source instead of avih one ?

yes im using avihs 7f

i tried the debugview thingy and i got nothing, any n00b proof guide of being sure debugview worxs? mp4 xcd still doesnt work for me btw... and for some strange reason i stopped getting emails from this thread, sorry for the delay

[avih]

phobos, when you opened debugview, you saw NOTHING? was it left completely blank? or were there some sort of messages? how about playing a different kind of xcd (ogm/avi/etc). debugview should give plenty of messages when starting to playback. (the xcd filter should give some messages, other filters may display some messages, and even the player itself. i think zoomplayer shows some messages.)

[Phobos]

well, the first time i did it showed no messages, now i have a logfile, it worked :)

tell me your e mail or msg me so i can send results to you avih

[avih]

the one you got my filter from. zip it good before sending. this inbox is quite limited. thx. or, even better, if it's not more than 2-3 pages, just post here such that others can analyze it as well.
thx.

[phrentec]

Avih, can you send me 7f(or whatever it is the version that seems to be working at least). email at p h r e n t e c @ y a h o o . c o m I'de really like to try it.
Also if someone can please restate the proper registry additions that would be nice.

[avih]

@phrentec: sent.
don't distribute, report back, and compare with test6b (must).
enjoy.
avih

[avih]

reminder:
calinb, snakeman, shitowax:
still waiting for your (detailed) findings with test7f. that's the reason i haven't publicly released the filter so far.

thx.
avih

[calinb]

Originally posted by avih
still waiting for your (detailed) findings with test7f. avih--sorry, man! I don't really have anything new to report on 7f--except the increased speed. I've had some crashes with ogm/Div5/dts XCD, but 6b crashes on those files too. I haven't had time to play the failing XCDs with graphedit yet--the crashes were with MPC. I'll investigate further, if you're interested, but as I said, 6b behaves the same way with these XCDs (internal MPC filter works on them though). 6b and 7f yield a jumbled screen (mostly green) and I get a MPC crash w/ "mpclassic has encoutered an error" >>> CLOSE ...or something like that.

[avih]

thx calinb. no worries ;) nothing bad happens if it's delayed. your report is good enough. thx again.

shitowax, snakeman: still waiting.
thx

[phrentec]

I know we are trying to get mp4 xcds to work in directshow but aside from that Is there any chance that the quicktime player(which is supposed to support mp4 natively) can be made to open mp4 files from an xcd? Right now it obviosly can't. I was hoping for support for mp4 xcds on macs.

[avih]

well, let's focus on releasing this filter 1st, and then see what else can be done.

have u played with it?

[calinb]

Originally posted by phrentec
I was hoping for support for mp4 xcds on macs. Yeah, my wife's got a mac, but I haven't found a way to do it yet. It can't read XCDs at all (can't even copy the dat file). I have to convert them to files on a networked windows machine. BTW, I've found QT playback to be very marginal, performance-wise, with most of the media that I've encoded. That is, when it can play it at all (no HE-AAC, ogm, mkv, etc.!) on her 800GHz/640MB G3 iBook. mplayer is best and VLC has pretty good performance too. VLC plays the most formats. I still haven't tried 3ivx on it (only have the windows version). She's more into viewing than encoding :)

[snakeman]

Sorry guys I have been real busy with some projects I let slip. I will be with you guys some time in the next 24 hours.


Sorry :(


snakeman

[avih]

no need to be sad snakeman ;) we're ok...
if you can't test it just let me know.

cheers
avih

[Phobos]

man, sorry to see this thread didnt have a real ending. avih, would you resend me the 7f filter??? im starting to believe its my 16x lg combo drive messing up. maybe i have the b0rked one shown at the xcd faq... anyway i have plenty of computers to test on in my dads business (where i spend most of the time now on "vacation" :/ ). so pls send the filter to phobmx@yahoo.com.mx if you can

and i have an update for you guys, the new phillips player supports MP4 in xcds, as tested in its respective thread.

snakeman and shitowax, i hope you can still review the filter so avih releases it public, i want to do my part too

[Phobos]

alright bro i have great news for ya!!! i dunno what changed, but i reinstalled windows a week ago and now with your bat everything works like a charm!!!

i used an 800mb mp4 with xvid 1.01 and he-aac, even subtitles worked flawless. Seeking through the file is as fast as it gets and the file loads as fast as if it was an avi (dunno if this is because of my fast puter but i doubt so)

so the guys left must give it a shot, this one is worth a public release if you ask me

so no more problems playing mp4 from an xcd :D

[avih]

hmmm, quite an old thread ;)
i'm glad to hear it's finally working for u.

the reason i didn't eventually release the filter is because the 3ivx filter was fixed, and now doesn't exhibit the problem it used to have when playing xcd.

the filter was a workaround for otherparts of the system that missbehave, and now they behave properly. so no need for it afterall.

although i must admitt it was an interesting 'mini-project' ;)
if you still think that it's worth releasing, plase name the reasons, and i'll reconsider it.

thx.
avih

[bond]

Originally posted by avih
the reason i didn't eventually release the filter is because the 3ivx filter was fixed, and now doesn't exhibit the problem it used to have when playing xcd.3ivx! i knew it :D

if you still think that it's worth releasing, plase name the reasons, and i'll reconsider it.hm if i remember it right the version had something to do with enhancing the caching?

was this only a workaround, or is it really not useful for other situations? does it have downsides compared to the current filter?

[avih]

@bond:
the only difference is that it has cache, while the original filter doesn't. so it has bigger memory footprint (about 2M compared to probably less than 100K), and it has higher cpu load (the cache needs to be searched for every request). the extra cpu load is probably neglectible on modern computers, and so is the higher mem footprint, but they're there.

BUT all this cache mechanism needs not be there unless the parts that use the filter to read XCDs don't do it properly. so while the overheads are quite low, it will allow writing bad software that still behaves pretty good with the filter. as i usually don't tend to encourage bad programmings habbits, and all the currently known filters read properly from the xcd filter, i thought that the current filter is very good with it's minimalistic approach.

so, again, since you too have a copy of the 7f filter, you may compare it to the original 6b release, and if you find other filters that read the cd slow with the xcd, you can try the 7f one for speed improvements. if it works better than i suggest that u contact the (other) filter author, and point him to this thrad ;)

of course, if enough ppl think that it's worth releasing the 7f filter, i'll do that. but again, i don't like encouraging bad programming habbits....

avih

[calinb]

Originally posted by avih
the reason i didn't eventually release the filter is because the 3ivx filter was fixed, and now doesn't exhibit the problem it used to have when playing xcd. Really? Was the 3ivx splitter fixed?

I have the "plus" version (paid-version)of 3ivx installed on my computer, but I downloaded the latest free 3ivx demo. Comparing the size and dates on the splitters (3ivxDSMediaSplitter.ax), they are both 284KB and 1/27/2004, which predates our problems.

Does somebody have a newer version of 3ivx D4? I'm still using avih's unreleased test 7f version of the XCD filter because of this problem.

Also, didn't we see a performance improvement playing files across a network with test 7f? I think I remember an advantage.

Phobos, glad to hear it's working, regardless. :)