no luck playing mp4 from an xcd

[Phobos]

aww man, well, i followed the guide to create my mp4 xcds, everything goes fine but the dat file wont play, i get a "classfactory" error thingy... pls help

i muxed the mp4 using 3ivx, i use ffdshow for video and 3ivx for audio,,,

and YES i installed the xcd ds filter...

[Phobos]

update:

i tried all kinds of things with mode2cdmaker:

tried to put the form2 ext to mp4: didnt work

tried putting subtitles also as form2: didnt work

nothing on the form2 ext: didnt work

should i try using single track?? i cant think about anything else... i hope its possible to play this babies through xcd, id h8 to reencode...

[bond]

did you try both xcd dshow filter: from gabest and avih?

[Phobos]

i first tried avih, my above post shows what happened

now i tried gabest and i get a different error, like illegal path or something like that...

btw, none of the filters could render the file in graphedit...

[calinb]

How about trying Media Player Classic? It supports XCD/dat files natively. If it plays it, it's probably some kind of filtergraph-DSF mgmt'or merit level problem, though I'll admit that that kind of thing doesn't usually result in the error msg's you're getting.

[Phobos]

not even playable with media player classic :( it says "failed to render the file"

pls help me guys...

[bond]

ok i tried now playing both a mode2 nero produced mp4 file and a 3ivx produced xvid mp4 and it worked fine

does 3ivx have inbuilt m2f2 support?

[Stux]

Originally posted by bond
does 3ivx have inbuilt m2f2 support?

Umm... no

[bond]

hm i did the following:
transform the mp4 with mode2 creator to a mode2 .bin
extract the .dat file out of the .bin with isobuster
played the resulting file (which is little bigger than the source mp4) in graphedit

the wierd thing is it also plays if i dont a xcd filter installed?

is the .dat file i have not a m2f2 file?

[Stux]

Could be because the 3ivx Splitter will split anything which smells like a mov or mp4 file ;)

[Phobos]

Originally posted by bond
hm i did the following:
transform the mp4 with mode2 creator to a mode2 .bin
extract the .dat file out of the .bin with isobuster
played the resulting file (which is little bigger than the source mp4) in graphedit

the wierd thing is it also plays if i dont a xcd filter installed?

is the .dat file i have not a m2f2 file?

thnx bond but whats the point on extracting the file each time?? so there is no way i can play it directly from the cd???

[bond]

well i did extract it from the image cause i didnt want to burn a cd only for testing

if you want to archieve a xcd you shouldnt extract but burn the image

[Phobos]

right now i havent burned any cd, im testing with daemon tools, is that a problem??? i still have no success playing the file directly from daemon tools. Anyhow bond, i tried your method and it works, i still believe the use of xcd this way is pointless, i want media reproduction, not something i must copy to the hd...

So in conclusion, is this the container's fault? the muxer's fault? or the splitter's fault? The way i see it, it is maybe 3ivx's splitter fault... so any chance this will change ever?? if not i will just dump the whole xcd idea...

thnx for your help guys, just answer this last one

[bond]

hm if the extracted file works, the file is ok (of course)

so i guess the problem is maybe the burning prog!? try another one if possible

[Phobos]

do you know about daemon tools??? is a program that reads directly isos and acts as a virtual dvd rom drive, ive never had bad reads with it which turn out bull burned, its near perfect. i dont want a coaster, thats why i wont burn until im sure it will play. well not a coaster really, but as i said its pointless making xcds if i always have to copy them to the hd....

[bond]

you dont have to copy the files to hd from xcd to play them! it of course will work from cd!

i guess the problem is daemon tools having problems with the cd image produced by mode2 creator
isobuster doesnt seem to have this problem, but with it you have to extract the file to play it

if you want to risk it i would burn the image to cd :)

[Phobos]

errr, i dont agree, 3ivx wont support m2f2 right now, so i dont see why it would change if i burn it, i can still give it a try, but if i dont succeed, maybe mp4 just doesnt work on xcds atm :(

[bond]

3ivx has nothing to do with xcd!

you need to install a xcd/cdxa dshow filter to make the files play (independantely of mp4 (3ivx), avi...)! checkout the xcd faq!

[Phobos]

already done bro, i tell you what, ill do a fresh reinstall and see what happens

[Phobos]

well, i did a fresh install, i have all the ax files working and i get the same g@y errors... conclusion, MP4 DOESNT SUPPORT XCD ATM (lets hope 3ivx supports m2f2 later), unless somebody proves me wrong... believe me ive tried anything

[bond]

well i proved you wrong ;)

once again i guess the problem is daemon tools
try burning the .bin outputted by mode2 maker on cd and report if you can play the .dat from the burned cd

[calinb]

Phobos is right--mp4 and XCD don't get along! Bummer. I've been encoding 795MB files with Nero thinking I might burn them to XCD someday. :(

I made a QT profile Nero Digital Encoding (so it would play on my wife's Mac and also work with 3ivx decoder). I burned it to an XCD. MPC couldn't play it with 3ivx or Nero filters. I installed ff-cdxa. Still broken!

Next I tried graphedit 3ivx splitter >> matroska muxer. The HE-AAC broke! It CoreAAC reports SBR but, instead of 24000/48000, it says it's 12000/24000 and no sound to be heard. Time for a new filter graph.

Finally, 3ivx splitter >> ogg muxer worked, but now my file won't fit on an XCD! VdubMod 1.5.10.1 supports the new mkv lacing so I tried to remux ogm >> mkv. No dice. It just sat there after F7. So I went back to VDubMod 1.5.4.1 and it worked (793MB). Small enough--even w/o the lacing so onward to burn a 2nd coaster...err XCD.

And....it plays! MPC, Nero Showtime, WMP, ff-cdxa, 3ivx splitter, Nero splitter--totally compatible. I thought Nero Recode was supposed to be convenient :) Hey Bond, do you think if we let menno, shitowax, ChristianHJW, and Dext know about the above, we could get some of these problems fixed :) Maybe you can can figure it out. I'm looking for a mp4 >> mkv utility--preferably with the new efficient lacing!

[bond]

Originally posted by calinb
I made a QT profile Nero Digital Encoding (so it would play on my wife's Mac and also work with 3ivx decoder). I burned it to an XCD. MPC couldn't play it with 3ivx or Nero filters. I installed ff-cdxa. Still broken!what error message exactly did you get in mpc? or only crash/freeze?

also to avoid any player related problems its very important to test these things in graphedit!
if you open the mp4xcd file in graphedit how does the graph look like, what filters are used? does it play?

[calinb]

Bond, I did use Graphedit--that's how I tried so many filters. Sorry I didn't provide details.

There's no way that Nero filter will connect to XCD filters; they only connect to their own file source/splitter filter. So we must use the 3ivx splitter. As Shitowax explained to me in the 3ivx forums, you have to choose your solution.

I just tried the 3ivx splitter again. I thought it hung had at first (yesterday) but it just takes a very long time to connect to the riff/CDXA filter. Once it connects, it's fine.

MPC worked with the internal VCD/SVCD/XCD reader too, but it also takes F-O-R-E-V-E-R!

Phobos, try it again and be very, very patient--at least until Shitowax can tells us how to get the filters to connect quicker :).

[bond]

hm interesting, maybe its a problem with 3ivx, not liking the xcd/cdxa filters or so (i hope stux or shitowax will read this)

did you try both filters in graphedit, from avih and gabest?

[shitowax]

You seem to have found a small problem in the 3ivx splitter, in OQT to be precise: the parsing of the header of the .mp4 file is performed by reading a lot of times a few bytes. This way to read files is in general not a problem because of reading cache of modern file systems. Problems start occuring with file systems that don't use any reading cache as it seems to be the case with XCD and some remote file system as well... It's not a new problem and a solution is already in OQT (OQT's internal caching system), but it just need to be used in the 3ivx splitter. It should be in the next 3ivx version (if I can manage to create properly an XCD myself ;))

[calinb]

Gabest's filters (cdxareader and MPC internal) don't work at all. You get a "can't render" with cdxareader and "cannot load any source filter" with the MPC internal filter. mkv XCDs play fine.

[calinb]

Originally posted by shitowax
(if I can manage to create properly an XCD myself ;)) Thanks Shitowax! If you don't have time to do this, I can send you an XCD image. Just tell me what format (nrg, iso, or a cue+bin.) Also what streams you want and how large of a sample. (I assume you'd like 3ivx, but if you feel lucky or want a variety of others....) If it's < 30MB, I'll just put it on my comcast webpage. If you want something > 30MB, I'll have to open up my ftp server to you. PM me, if I can help.

[Phobos]

hmm, after readin shitowax's post it seems im not wrong after all, it was 3ivX fault after all (p0wn3d bond, lol jk) well ill burn these xcds and wait patiently for a new 3ivX version... btw, i hope this new one also supports chapters in mp4 ;)

[bond]

Originally posted by Phobos
hmm, after readin shitowax's post it seems im not wrong after all, it was 3ivX fault after all (p0wn3d bond, lol jk):p :D

but i still wonder why my with iso buster from the mode2 image extracted .dat played without any xcd filter installed with 3ivx...

[shitowax]

Hehehe .... my post described a possible problem, it's not sure. Indeed I've just looked at Gabest's XCD source filter, and it does cache an entire sector (2352 bytes) BUT it doesn't scan for the QT/MP4 mediatypes ... so it may not be 3ivx's fault at all ... ;)
I need to do some real tests now.

Originally posted by Phobos
hmm, after readin shitowax's post it seems im not wrong after all, it was 3ivX fault after all (p0wn3d bond, lol jk) well ill burn these xcds and wait patiently for a new 3ivX version... btw, i hope this new one also supports chapters in mp4 ;)

[bond]

also have a look at avih's filter, its maybe more used than the one from gabest:
http://webs.ono.com/usr016/de_xt/

[calinb]

Originally posted by bond
also have a look at avih's filter, its maybe more used than the one from gabest:
http://webs.ono.com/usr016/de_xt/ Yes! I can't get the gabest filter to work at all with XCD/mp4. avih's works, but it's slow to launch.

[bond]

yes, gabests filter seems to totally lack support for mp4, me and also shitowax got a "filter failed to load file" message

[Phobos]

Originally posted by bond
:p :D

but i still wonder why my with iso buster from the mode2 image extracted .dat played without any xcd filter installed with 3ivx...

i think its because isobuster automaticaly removes m2f2 data and you get a file just the same as the source mp4, thus you dont need any xcd filters

Hehehe .... my post described a possible problem, it's not sure. Indeed I've just looked at Gabest's XCD source filter, and it does cache an entire sector (2352 bytes) BUT it doesn't scan for the QT/MP4 mediatypes ... so it may not be 3ivx's fault at all ...

so what does this mean? is it gabest's fault?? i dont think so, i tried avih's filter and it doesnt work either, any chance these guys update their filters to support mp4? or is 3ivx able to find some workaround??

[bond]

Originally posted by Phobos
i think its because isobuster automaticaly removes m2f2 data and you get a file just the same as the source mp4, thus you dont need any xcd filtersyep that was it, but it seems isobuster isnt able to extract the m2f2 correctly :(

so what does this mean? is it gabest's fault?? i dont think so, i tried avih's filter and it doesnt work either, any chance these guys update their filters to support mp4? or is 3ivx able to find some workaround??well as calinb and shitowax reported the filter from avih works, but it needs very long to load!
gabests filter doesnt work at all

[bond]

ok it seems 3ivx together with avihs filter doesnt handle xcd atm without flaws, did anyone try the mpegable filters from dicas (http://www.mpegable.com/showPage.php?SITE=mpegable&PAGE=download)?

[calinb]

Originally posted by bond
did anyone try the mpegable filters Not for several months. Last I tried them, I think I had trouble decoding some files I made with mp4creator and gave up. I'll try them again.

Okay--I tried it. Dicas uses a file reader/splitter like Nero so I uninstalled 3ivx and....

"Filter Failed to load File."

So 3ivx + avih's riff/CDXA filter seems to be the only (painfully slow loading) solution ATM.

[Phobos]

so in the end whats more likely to happen? 3ivX adapts? xcd filter updates? or should i just dump the xcd idea due to lack of support( id wait some time because 100mb are pretty tempting)?

btw, gabest xcd filter works better for me, it says a route error, while avih's riff/CDXA say theres a "Classfactory" error no matter how long i wait or how many times i try...

calinb, would you pm me with some straightforward instructions to make this work? your first post is a bit confusing, thnx in advance

[bond]

ok here is gabests answer on the problems with his cdxa reader:

Originally posted by gabest
This filter knows how to create media types for a few formats like avi, mpeg1/2 or matroska, but the rest is generated using the registry where certain byte patterns are listed with the corresponding media types. These entries are usually put there when you run regsvr32 on a splitter or source filter. Try to reregister the 3ivx splitter and see if the cdxa filter will work then.so the thing would be to first register cdxareader and only after that the 3ivx filter

[Phobos]

didnt work for me, tried to do exactly what you said bond and i still get error 80004005 :( , any other posibility? note that now this is a BURNED mp4 (dat extension). So the questions prevails... what updates to seek for??

[calinb]

Originally posted by bond
ok here is gabests answer on the problems with his cdxa reader:
so the thing would be to first register cdxareader and only after that the 3ivx filter I tried unregistering and reregistering the 3ivx filter and it didn't help. Also keep in mind that gabest's internal MPC filter, which usually works fine all alone, doesn't work here.

The only filter that works at all is xcdsrc.ax. It also doesn't matter whether the XCD is burned, mounted image, or .dat file, IME.

[Phobos]

well, i deregistered the cdxareader filter and registered the avih's one... i tried to play it in wmp and stood "connecting" like for 5 min reading the xcd until it poped the same "error found" or stuff, secondly i downloaded bsplayer and it also stood there making hundreds of painful-fast head movements on the dvd-rom, so i dumped it...

how exactly did you do it calinb?

update: i tried to render the dat file into graphedit and i got the same lame error asking if i have installed proper filters: 0x80040111

help :(

[calinb]

Originally posted by Phobos
how exactly did you do it calinb? Sounds like it should be working (slowly) for you, Phobos. I use regdrop to register/and unregister filters, but that shouldn't matter. I've tried it on XP sp1 and Win2K sp4--both with DirectX 9b. I don't know what else you could try. If you PM me, I could send you a small .dat file that works for me and you could try it. I think you already said your files render okay, if you extract them with dat2file, right?

[avih]

the xcdsrc.ax filter doesn't do caching at all.
i don't have time to modify the files, but it shouldn't be too hard. the sources are open and are available from dext's page.

@shitowax:
can u pls describe your file access pattern?
i would like to know in the 1st place if a 'normal' caching system will do.

thx,
avih

[Phobos]

Originally posted by calinb
Sounds like it should be working (slowly) for you, Phobos. I use regdrop to register/and unregister filters, but that shouldn't matter. I've tried it on XP sp1 and Win2K sp4--both with DirectX 9b. I don't know what else you could try. If you PM me, I could send you a small .dat file that works for me and you could try it. I think you already said your files render okay, if you extract them with dat2file, right?

yea, i tried isobuster and it worked well, but i cant play the dat no matter what...

feel free to send that mp4 file to my email, i have 20mb..

thnx in advance

[shitowax]

The 3ivx splitter looks first for the "header" atom ("moov"), atom by atom. An atom always start by 8 bytes containing the atom size (4 bytes) and the atom type (4 bytes). Of course, the "header" can be at the beginning or at the end of the movie ;)
Then when it finds the "moov" atom, it reads it entirely sequentially with a lot of small readings. Then it's ready to play.
As far as I've debug, there are problems in the mediatype negociation (I've got something ready for this problem), but there are also problem when reading sequentially the "moov" atom ...

Originally posted by avih
@shitowax:
can u pls describe your file access pattern?
i would like to know in the 1st place if a 'normal' caching system will do.

thx,
avih

[avih]

shitowax, in your oppinion, would a 1 sector (~2k) cache improve the performance considerably? or is the file beeing seeked across a wider area? (assumming for a moment that the seeks don't cross sector boundries. is this assumption good enough?)

[shitowax]

I think yeah. I already saw that Gabest's source filter contains a cache of 1 sector. Basically, there are a lot of atoms read like this:

read size on 4 bytes
for(i = 0; i < size; i++)
{
read a record of a few bytes.
}

So, the splitter performs a lot of small readings of a few bytes. If the source physically reads the data on the disk each time, it would slow things down a lot.

[avih]

thx. i'll see if i can do anything about this.

[avih]

ok, i need volunteers to test a new version.
improved features in the new version (test7) of the xcdsrc.ax filter:

1. has cache (currently 64 distinct sectors, each sector is cached on it's own {=~128K}. number of sectors is totally configurable).
i expect the cache to be usefull during the negotiation phases (start of playback, etc) because in these phases, the same sectors are read over and over again (as is the case with 3ivx, but not only).

2. reads more efficiently from the cd (the whole request in one go instead of the whole request sector-by-sector. usually request is for about 32K during actual playback, and 1byte-1K during negotiations phases).
i expect the improved reading mechanism to give a more fluent playback, (if it wasn't so far for u).

i don't know what other effects the new version might have (especially if there are bad sectors).
i didn't test it extensively, so new bugs may appear. adding the cache system was a bit more complicated than i expected, and much new code was written. if you wanna experiment (absolute descretion is a MUST), post on this thread.

if the tests by the volunteers prove that the new version is usable, i'll release it to the public asap. otherwise, i don't want ppl that are not specifically interested in testing using the buggy version (and it WILL spread as the 'latest' version if u let it).

cheers,
avih

[Phobos]

well, you have here a glad volunteer with plenty of material to test on, feel free to send it to my email or contact me as desired

[shitowax]

I can help as well :)

[ChristianHJW]

Link pls :D .... although MKV from XCD works, there is a lot room for improvement, especially for accessing tags and the like. Looking forward to test the new version of the filter avih ! You rock ! :)

[avih]

shitowax, chris: sent.
phobos, gimme your email, i can't find it.


chris, had a delivery failure with your email:
"Remote host said: 550 Error: Message content rejected"
gimme another address, or contact me by icq, etc.

[P0l1m0rph1c]

@avih: if you need anything for testing, please let me know. I am willing to test and give feedback :) I'm looking forward to that new filter.

Oh, btw, my mail is dcoder@alexandria.cc

[avih]

it crashed constantly for for shitowax. i wanna have another debugging session before releasing another private test version.
i'll let u know when it's ready, thx.

avih

[calinb]

Originally posted by avih
i wanna have another debugging session before releasing another private test version. i'll let u know when it's ready, thx.avih Okay, I'd like to test it, when you're ready. I have bunch of XCDs that use a variety of codecs and splitters.

[avih]

ok, version test7b was sent to shitowax, phobos and P0l1m0rph1c.
calinb, PM me your email if u're still interested.

reports here pls.
thx.
avih.

[P0l1m0rph1c]

Hi. I've tested this new version and it seems not to be working properly. :confused:

With AVI and MKV files, it crashes all players, caused by an access violation.

With MP4 files, it seems to work, and it even reads the XCD, but keeps telling "opening..." and doesn't do anything more (I'm using 3ivx's splitter).

I hope this helps.

Edit: btw, i've tried xcd2file and it looks nice, and works just fine. Nice proggie! ;)

[avih]

P0l1m0rph1c:
if xcd2file works well (meanning: the extracted file plays back ok), then the new system is probably working well.

both the filter and xcd2file use exactly the same core reading library (although in a slightly different manner), so i think that on files which xcd2file works correctly, the filter should at least perform as well as the test6b filter.

i didn't change any of the dshow stuff, i just made it read faster from the cd and use cache.

how did these files play with test6b? comparison with test6b is essenital (as i noted on my email) for me. pls also post your OS, and drive used. thx.

thx.

[P0l1m0rph1c]

On my machine (P4 2.0 GHz, 768 MB, Win2000 SP4, ASUS CRW-2410A):

With AVI and MKV files, it crashes all players (didn't try WMP9 here), caused by an access violation.

With MP4 files, it seems to work, and it even reads the XCD, but keeps telling "opening..." and doesn't do anything more (I'm using 3ivx's splitter). And the XCD2FILE extraction makes a perfectly playable mp4 file.

On another machine(P4 2.4 GHz, 256 MB, WinXP SP1, LG GMA-4020B):

With AVI and MKV files, it plays fine with WMP9, but with MPC or WMP6.4 it says "no combination of filters....".

With MP4, same beahavior as above, in all players.

Where i could play the XCD's, the video was just as smooth as always (i don't have much problems with that), but the seeking was way faster.

With Gabest's filter i was able to play all AVI and MKV XCD's, but not any of the MP4 ones, in both PC's. This also applies to the test6b filter.

[Phobos]

well, ill just post my impressions on the format given by avih:

functionality: ds filter still doesnt work, it acts just the same it did on b6, it just keeps "connecting" in wmp9 and does the same in bs player

size of file: 799MB

speed: a bit faster but still slow, it just took like 3 minutes opening until i got an error message (opposed to 10 min in b6), never played

strain: wmp9 and bsplayer both have normal cd use, in b6 wmp9 was slower and bsplayer did hundreds of head seekings in the drive

OS: Windows XP SP1 up to date

drive: LG DVD-ROM\CD-RW 16x\52X GCC 4520G

On what xcd2file matters, i succesfuly converted the same dat file to mp4 reading directly from cd, maybe its just me, but isobuster feels a little faster, but heck its good enough and saves the hassle of installing a program, not to mention its freeware... thnx avih :) The drive had no strain, it read smootly like any other app.

The output file played fine in any player, i just found that filesize was different:

original mp4
Size: 836,133,315 bytes
Size in Disc: 835,100,672 bytes

output from xcd
Size: 836,133,368 bytes
Size in Disc: 835,100,672 bytes

im not sure if the translation of my spanish explorer is okay, but you get the idea :rolleyes: i dont find it anoying, maybe its that i have ntfs compression enabled by default.

I hope we find the cure for xcd mp4s on directshow soon

:D

P.S.: of course i didn't share the file i got ;)

[shitowax]

To make the XCD source filter connect automatically to the 3ivx splitter, you need to add this to your registry:

[HKEY_CLASSES_ROOT\Media Type\{e436eb83-524f-11ce-9f53-0020af0ba770}\{E436EB89-524F-11CE-9F53-0020AF0BA770}]
"2"="4, 4, 66747970"

This helps DirectShow to recognize mov/mp4 files. It will of course be done automatically in the new 3ivx version. On my two tests systems, it works nicely with that. Feel free to report problems and congrat to avih ;):

- I tested successfully a medium file (180MB) and a small file (10MB) using both daemon tools and real CD on two different CD readers.
- with the registry addition, it works with both b6 and b7. With b6 it takes more than 1 minute to connect, with b7 about 7 seconds.
- the playback is perfect and maybe even better in b7 ...

hope that helps.

[avih]

thx shitowax, and congrats.

@P0l1m0rph1c, Phobos:
thank u very much for your tests and reports. just to summary the new riff/cdxa filter issues, can u pls confirm or show otherwise for each point (for all of your tests so far):

1. the new (test7b) is working in every scenario that test6b was working for.

2. test7b is sometimes working faster than test6b, but never slower that test6b (start/playback/seek).

3. in cases in which both test6b and test7b fail, they behave differently (7b causes a crash, 6b causes a... ? )

4. xcd2file is working properly always (i guess that's confirmed already).


i need these answers since i can't release test7b unless it's at least as good as test6b. if there's even one scenario in which 7b performs worse than 6b, then i must fix it before releasing.

remember that the thread is about xcd with mp4. it was suggested that the long time interval during start of playback may be the cause. i addressed this issue ONLY!, there are NO dshow improvements in 7b, only better cd usage.

it also seems that 3ivx splitter is working better now with shitowax's registery fix, as now (if i understand correctly) mp4/mov files can also have other file extensions and still be recognized by a dshow player. the improvements with xcd is a derived result (but still a good one).

thx.
avih

[Phobos]

Originally posted by shitowax
To make the XCD source filter connect automatically to the 3ivx splitter, you need to add this to your registry:

[HKEY_CLASSES_ROOT\Media Type\{e436eb83-524f-11ce-9f53-0020af0ba770}\{E436EB89-524F-11CE-9F53-0020AF0BA770}]
"2"="4, 4, 66747970"

how exactly do i add that to the registry?

what type? i tried alphanumeric and it didnt work :(

[shitowax]

You can either do it by hand using regedit or try to create an xcd.reg file, put those following lines in it and double-click it in explorer:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Media Type\{e436eb83-524f-11ce-9f53-0020af0ba770}\{E436EB89-524F-11CE-9F53-0020AF0BA770}]
"2"="4, 4, 66747970"

[Phobos]

well, i copied the info you posted including the first "windows registry editor 5" line to a txt, renamed to reg, applied the changes and rebooted... still wmp just sits there.... :mad:

[P0l1m0rph1c]

Well, i added those values to registry too, but the result is the same.:confused: It doesn't do anything, no matter how much time i wait.

[calinb]

Originally posted by shitowax
[HKEY_CLASSES_ROOT\Media Type\{e436eb83-524f-11ce-9f53-0020af0ba770}\{E436EB89-524F-11CE-9F53-0020AF0BA770}]
"2"="4, 4, 66747970" Shitowax, are you sure that's correct? All the other entries have two commas, like this:

"2"="4, 4, , 66747970"

In fact, would you mind checking the entire key name and value again?

Thanks!

[calinb]

So far, I've only found one thing that's different with test7b -- Nero Showtime will now play a Nero Digital mp4/XCD whereas it just hung with 6b.

Otherwise, it still takes about a minute to load -- just like graphedit or the directshow players with the 3ivx splitter.

I played a Nero Digital mp4/XCD made with QT profile + bframes. HE-AAC 5.1 and CoreAAC 1.0b9 filter. I used both DivX 5.11 and 3ivX D4 4.5.1 decoders. I haven't tried 3ivx source yet but plan to try it soon.

I used a IBM Thinkpad T23 1.2GHz PIII Mobile, 512MB RAM, XP-Pro sp1a. I read directly from the CD and I read .dat files from the HD.

I have bunch of larger, faster machines and many XCDs to try, if avih has any further test cases or suggestions. Right now, not much difference from 6b :( and I'm not sure what else I should try.

Phobos, can you get anything to play by right clicking on the dat file >> open with >> browse >> graphedit? You can google for graphedit and download it. Take the player out of the test and go straight to a graphedit filter graph.

[shitowax]

You're right ... I wonder what happened cause I export the key with regedit :( ?!? Anyway you can try with that:


Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Media Type\{e436eb83-524f-11ce-9f53-0020af0ba770}\{E436EB89-524F-11CE-9F53-0020AF0BA770}]
"2"="4, 4, , 66747970"

Originally posted by calinb
Shitowax, are you sure that's correct? All the other entries have two commas, like this:

"2"="4, 4, , 66747970"

In fact, would you mind checking the entire key name and value again?

Thanks!

[avih]

@calinb:
thx, that's a very good report. i guess your computer/cd are fast enough and there's no difference with 6b. i wonder, if shotowax's registry fix will make it work with the 3ivx splitter for you. if it will, i'd appreciate a speed comparision with test6b.

the nero player issue puzzles me, since as i said, it should be 100% the same as test6b, only with faster access to the cd. so the fact that it didn't work with 6b and does work with 7b may imply that it handles dshow connectivity differently. and since i didn't touch that part of the filter, i don't understand how is it possible. one possible solution to this 'mystery' is that with the old filter it's slow enough for the nero player to quit on a timeout, while the new filter reads faster, and the player manages to start the playback before reaching that timeout.

perhaps someone of the nero team could comment on this? (are there any nero ppl around here anyway?)

regarding other test cases, well, just throw at it the mode2form2 cds that u've got ;). i don't urge u to do that, but it will make me feel more comfortable to publicly release the new version.

one implication of the caching system is that it may cache bad sectors as well (it knows they're bad, but i decided to cache them anyway). so i think it may behave differently if there are bad sectors on the cd. comparisions of this kind are very welcome.

thx for a good report.

@phobos:
i'm still waiting for you to comment on my previous post. if you, too, report that it's not worse than 6b, then i'll release 7b to the public.

thx again.
avih

[P0l1m0rph1c]

Originally posted by avih

1. the new (test7b) is working in every scenario that test6b was working for.

2. test7b is sometimes working faster than test6b, but never slower that test6b (start/playback/seek).

3. in cases in which both test6b and test7b fail, they behave differently (7b causes a crash, 6b causes a... ? )

4. xcd2file is working properly always (i guess that's confirmed already).


As for the #1, here's confirmed. The problem with non-mp4 xcd's was a flaw in my system :(
I confirm the #2 too. #3 also.
#4 - xcd2file works in all cases.

However, even with the registry tweaks, mp4 doesn't work in any case here :eek:

anyway, thx for all your work :)

[avih]

@P0l1m0rph1c:
thx (and sorry for forgetting you, i remembered i sent it to 3 ppl, but i forgot i also sent to calinb afterwards.).

ok, that's good. one to go ;)

[calinb]

shitowax, Did you cut and paste the entire key? If so, you don't have a "0," "Source Filter," or a "1" value.

Here's what I have:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Media Type\{e436eb83-524f-11ce-9f53-0020af0ba770}\{E436EB89-524F-11CE-9F53-0020AF0BA770}]
"0"="4, 4, , 6d646174"
"Source Filter"="{E436EBB5-524F-11CE-9F53-0020AF0BA770}"
"1"="4, 4, , 6d6f6f76"
"2"="4, 4, , 66747970"

I don't know what the other values do but I'd sure like to speed things up.

avih, I'll throw everything I've got at 7b tonight.

[shitowax]

This is correct. It should work. Btw, I wonder how you managed to play mp4+XCD with Showtime as the NDParser seems to use its own source filter and doesn't seem to be able to open .dat files at all ... Almost sure it uses the 3ivx splitter in this case =).

Originally posted by calinb
shitowax, Did you cut and paste the entire key? If so, you don't have a "0," "Source Filter," or a "1" value.

Here's what I have:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Media Type\{e436eb83-524f-11ce-9f53-0020af0ba770}\{E436EB89-524F-11CE-9F53-0020AF0BA770}]
"0"="4, 4, , 6d646174"
"Source Filter"="{E436EBB5-524F-11CE-9F53-0020AF0BA770}"
"1"="4, 4, , 6d6f6f76"
"2"="4, 4, , 66747970"

I don't know what the other values do but I'd sure like to speed things up.

avih, I'll throw everything I've got at 7b tonight.

[Phobos]

Originally posted by calinb
Phobos, can you get anything to play by right clicking on the dat file >> open with >> browse >> graphedit? You can google for graphedit and download it. Take the player out of the test and go straight to a graphedit filter graph.

Calinb: ok i have graphedit already, thats what i use to mux mp4s lol. Well i tried your suggestiong and after 5 minutes graphedit just pops up with "have you installed necessary filters?" error. Note that i have the registry values added as shitowax said. Would you post a noob proof way to enter the registry settings you have?

Avih: I'd be glad to confirm your statements but im still unable to open any kind of mp4/xcd... :( Here are my hardware specs:

P4 800MHz Bus @ 3GHz
1GB PC3700 DDR RAM
10000 RPM 36GB WD Raptor HD

I guess that my system speed aint a factor here...

any other suggestions?

[calinb]

Originally posted by shitowax
Btw, I wonder how you managed to play mp4+XCD with Showtime as the NDParser seems to use its own source filter and doesn't seem to be able to open .dat files at all ... Almost sure it uses the 3ivx splitter in this case =). Yeah, I was wondering the same thing. I tried Showtime with 7b on a lark. I could hardly believe it when it opened it from the CD drive and played. It must be using the 3ivx splitter but I don't know how to check. Just to be sure that I remembered it NOT working with 6b I uninstalled 7b and went back. I waited forever and nothing. I think I finally killed it with task mgr. because it had hung. 7b loaded the XCD file in about a minute, as with the other players. Weird, huh!

Phobos, sure I'll help with regedit tutoring, but it's like shitowax said--you can make a text file with notepad or whatever. Put the lines in it as he described above, but put the two commas in it, like I discovered and name it xcd.reg. Double click it and that should install it into the registry. You could then go to "start" button>> "run" and type "regedt32. Then navigate down to the key. The hierarchy follows the from the key path and name. Careful 'cause many of the keys are have very similar names and you've got to find the correct one. You shouldn't have to change anything, but you can find the entry for the "2" value, etc. and then just close regedt32.

I'll give you some more details later, but I've got to test these XCDs right now (had a long day a work so not much time). All the values except the "2" entry were in the registry when I opened the key with regedt32 to do the entry manually. I guess 3ivx install did the other ones. If your keys are different, maybe it's because I have the licensed version vs. demo 3ivx???...I dunno. I would think they'd be the same.

[calinb]

Originally posted by Phobos
P4 800MHz Bus @ 3GHz
1GB PC3700 DDR RAM
10000 RPM 36GB WD Raptor HD Ahaa! That's it! Your ultra fast system can't keep up with my ol' IBM T23 and 1.3GHz PIII :) It must be hyperthreading its little GHz's out.

avih, I've played mkv, mp4, and ogm containers from XCDs and they all work fine. The streams were combinations of DIV5, Nero Digital w/ HE-AAC, mp3, and DTS and they were played with the 3ivx splitter, DivX decoder, 3ivx audio and video decoders, CoreAAC.9b, MPC built-in filters, and gabest/intervideo DTS. I also tried a K7S5A/1.2GHz Duron machine with Win2K spk6a. The mp4 XCDs loaded even faster than on the T23 (the Duron box CD is faster.) It loaded a full 795MB mp4 from an XCD in less than 30 sec. Smaller files seem to load proportionitely faster than larger files.

I'll try to run some XCDs with 3ivx, AC3, and vorbis later this weekend and I might have some XVID too, but these 3 container formats all seem to be working. There's also the surprising new functionality of Showtime working with a Nero mp4 XCD too, which I reported above.

Looks like you've got a "keeper" here. Thanks so much!

[avih]

Originally posted by calinb
... It loaded a full 795MB mp4 from an XCD in less than 30 sec. Smaller files seem to load proportionitely faster than larger files....

... Thanks so much! [/B]

my pleasure :) glad it works better.

now, the proportional load time is an interesting issue indeed, and imho, can be addressed.

shitowax, i need some more info pls. regarging the 30 seconds it takes for a ~800M file, it probably means that it has cache misses. a lot of them. now the filter drops only the last used sectors from the cache when there's a need to use a new sector. this means that before starting the actual playback, it reads from sectors that sum up to more than 128K (!!) in a manner that's causing cache misses. could this be possible? coud you try to describe again your file access pattern? where does it read from during the negotiation phase? how much does it read? does it 'scan' the file repeatedly over a large area?

calinb, i'll make u a version with a larger cache (say, 256 sectors =~ 0.5M) and send it to u. i want to release it with the minimal cache size that still working well.

thx so much for the tests. invaluable diagnostic data ;)

avih

[shitowax]

The 3ivx splitter does a lot of small sequential readings. That means in theory, the only cache miss that it does is when it fills the source cache and keep on reading sequentially ... Basically, I wonder if the cache system wouldn't be better with pre-readings i.e when the cache is full and data after the cache is required, you could imagine flushing the oldest portion of the cache and prereading not 1 sector but 10 or 100 ... What do you think ?

[Phobos]

Originally posted by calinb
Phobos, sure I'll help with regedit tutoring, but it's like shitowax said--you can make a text file with notepad or whatever. Put the lines in it as he described above, but put the two commas in it, like I discovered and name it xcd.reg. Double click it and that should install it into the registry. You could then go to "start" button>> "run" and type "regedt32. Then navigate down to the key. The hierarchy follows the from the key path and name. Careful 'cause many of the keys are have very similar names and you've got to find the correct one. You shouldn't have to change anything, but you can find the entry for the "2" value, etc. and then just close regedt32.

well, i now tried the registry settings you posted above and still no luck... just a question, does your drive starts spinning madly at full speed or just a calm like 8x read speed? mine doesnt spin fast at all, i cant hear it but i can see the led working... maybe my supposedly 'b0rked' LG combo drive (as said in the xcd faq) is the responsible.

[avih]

Originally posted by shitowax
The 3ivx splitter does a lot of small sequential readings. That means in theory, the only cache miss that it does is when it fills the source cache and keep on reading sequentially ... Basically, I wonder if the cache system wouldn't be better with pre-readings i.e when the cache is full and data after the cache is required, you could imagine flushing the oldest portion of the cache and prereading not 1 sector but 10 or 100 ... What do you think ?

i intentionally didn't introduce a prefetch mechanism, since the amount of prefech is unknown to the filter. guessing the amount wrongly will result in decreased performance if the reader doesn't need the prefetched data. the only place that knows the amount needed is the function that requests the file (i.e. the splitter). and i don't want the filter to perform better with your splitter, if it's worse with others.

how about requesting on your 1st request (just once) the whole portion of the data that you're going to search? this will cause all of it to get into the cache in one go, and on subsequent requests it will be served much faster. you could also request chunks of 32K if you're not sure how large the cache is (atm it's about 128K, but u shouldn't count on it).

but the best solution would be, imho, that you improve YOUR reading mechanism. i.e. if u know you're going to scan for atoms over 8M, then read these 8M 1st, and then scan them from your memory.

every filesystem is much more efficient when reading/writing large chunks of data, instead of 4 bytes at a time. it just doesn't make sense, and the result is decreased performance.

i.e. try to read/write a 10M file on the HD in chunks of 4bytes. then try to do that in chunks of 1K, then in chunks of 1M. the 4bytes one will be MUCH slower than the others. filesystems are not built to access such small chunks of data.


regarding your assumption that the slowdown is only because of the missing prefetch.. it still sounds strange to me. let's say that you scan a section of 128K. it will access the CD only 64 times, once for each sector read. the rest of the requests will be served from the cache. that means that 64 sectors read are 30s on a fast CD, 1min on slower CD. sounds too much to me.
possible reasons:
- something's wrong with the cache system.
- you read much more than 128K. can u pls comment on this one?


let me know what u think.
cheers
avih

[shitowax]

hehehe ... yeah, of course, we could improve the 3ivx splitter reading scheme and indeed we already did (the current CVS has it's own internal caching system ...). But I still think there is a problem somewhere else:

XCD beta 6 : 30 seconds to open a 180 MB mp4 file.
XCD beta 7 : 7 seconds.
normal CD : less than 1 second.

During the duration of the building of the graph the CPU is at 100% and the CD doesn't do a lot of readings it seems. I also tried to connect manually the source to the splitter filter and the time is the same so it's not another problem of directshow registration ...

[shitowax]

On a .mp4 file containing a 2 hours movie, the splitter has to read about 2 MB before being able to play the first frame. Almost all those bytes are currently read sequentially with fairly small readings.

Originally posted by avih
regarding your assumption that the slowdown is only because of the missing prefetch.. it still sounds strange to me. let's say that you scan a section of 128K. it will access the CD only 64 times, once for each sector read. the rest of the requests will be served from the cache. that means that 64 sectors read are 30s on a fast CD, 1min on slower CD. sounds too much to me.
possible reasons:
- something's wrong with the cache system.
- you read much more than 128K. can u pls comment on this one?

[calinb]

Originally posted by shitowax

XCD beta 6 : 30 seconds to open a 180 MB mp4 file.
XCD beta 7 : 7 seconds.
normal CD : less than 1 second. ogm XCD beta 7 less than 1 second to open a 800MB file.
mkv XCD beta 7 less than 1 second to open a 800MB file.

Well, maybe I'm exaggerating slightly :)

shitowax, No criticism of 3ivx or mp4 intended! I think you guys are doing great work and this issue is new simply because we haven't had good directshow mp4 tools before. You guys are changing that and I very much appreciate that you're interested in making them universally compatible across many platforms and formats. Thanks to you and avih, I think were making good progesss and mp4/XCD is usable now :).

avih, I'll keep testing anything you send my way.

[avih]

Originally posted by shitowax

...During the duration of the building of the graph the CPU is at 100% and the CD doesn't do a lot of readings it seems. I also tried to connect manually the source to the splitter filter and the time is the same so it's not another problem of directshow registration ...

On a .mp4 file containing a 2 hours movie, the splitter has to read about 2 MB before being able to play the first frame. Almost all those bytes are currently read sequentially with fairly small readings.


ok. now we're getting somewhere.

what u describe (100% cpu) makes sense in the light of the new info (2M before the 1st frame). imagine requesting 2M in chunks of 4bytes. that's 500,000 (0.5M!!) requests from the cache!!!! for each of those, the cache searches for the matching entry, copies the memory.. yada yada yada. u get the point. i'm actually surprised it performed that well ;) only 7 seconds :) and that includes accessing the cd 500(!!) times.

what adds to it, is that since it's the 1st version of the cache, it searches the entries in a linear form and not with a hash table.. i somehow imagined that scanning 64 entries won't make such a cpu load for each sector read... but 500,000 requests.. too much :) (although it DOES NOT scan the entries if it's requested with the same entry over and over [sort of a mini cache of the last request within the cache itself].. which is the case most of the time with your splitter. it still copies 2K chunks 500,000 times. which is a total of 1GB copied memory).

obviously, a prefetch mechanism won't help here much (and it already implicitly prefetches 2K while it reads the sector. that's 2K prefetch for 4bytes request! you can't ask much more from a prefetch scheme ;) )

what needs to be done here is improving the reading scheme of your splitter. once that's done, i think that the whole issue will be a none-issue.

and regarding the 1 second from a normal cd issue. i think that the OS treats mode2 form2 files differently than normal files. that could be the reason. also, i intentionally wanted to leave cache and prefetch of files to the OS. it now seems like it doesn't give much of these on a mode2form2 filesystems. so a cache is what i'm currently supplying.. ;)

let me know if YOUR new cache indeed improves the performance.

avih

[calinb]

I tested several CDs today, including vorbis and 3ivx streams, and everything is working better than ever. I'm still using the 7b version you sent to me, avih. In fact, I can't figure out why both Nero Digital and 3ivx 800MB cds are now loading in 3-4 secs today compared to 30 secs yesterday on the same K7S5A/1200 GHz Duron Win2K machine. I haven't changed anything. I rebooted previsouly after making the registry entries so that's not it. I really don't know what happened. Device Mgr. said DMA's been enabled on my CD drive the whole time and I can't figure it out. Weird! It sure works great now.

Hey Phobos, come to think of it, have your checked that DMA is working on your CD drive? Windows has a nasty habit of rolling back to PIO mode after a certain number of read errors (like after reading a scratched CD). It will forever stay in PIO mode--even through reboots and even after attempting to select DMA in the Device Mgr. pulldown box. You either have to edit the registry to disable this "feature" (this approach is covered in the MS knowledge base somewhere and you know how to use regedit now :) ) or you have to use Device Mgr. to delete the IDE/ATAPI controller driver for the primary or secondary controller, as appropirate for your CD drive, and reboot. Plug and play will do its thing and you'll get your DMA mode back after you select it again. About your drive spin-up question, the drive in the Duron box spins like a top. The drive in the laptop is very quiet and I can't even tell if it's running. Of course it's a slower drive too. The Duron box has some kinda 52x read/48x write unit made by "China, Inc." According to the Nero tools, it's really that fast too, but no mfg. is identifiable.

[Phobos]

i confirm not even this works, i tried exactly what you sad and STILL i cant play the friggin mp4 from the xcd... while the original file is playable and an extracted mp4 from xcd2file works well...

now my question is, what in this world is different from calinb's putter to polimorphic's one and mine???

[avih]

phobos, it's known that xcd doesn't work with all drives. perhaps u've got one of those?

calinb, that's great news. thx again for the time you're investing in the new version (i hope you're at least enjoying the movies ;) ).

as i noted on my prev post, the filter can still be improved on 'edgy' cases as with the 3ixv splitter. that's what i'm currently working on. regardless, if the 3ivx splitter is modified to behave better, i'm sure it'll gain on other scenarios as well (other than xcd that is). i hope to release the new version shortly.

thx
avih

ps.
phobos, maybe u can find another cd drive which u can try? i feel for u ;)

[avih]

test7c was sent.
pls report back to this thread.

improvements: should cope better now with tons of tiny requests (4 bytes anyone? ;) ).

avih

[Phobos]

well, sorry to say no use here :( ... ill try with another drive ASAP

[P0l1m0rph1c]

Hmm...now with the new filter (7c), it does connect to 3ivx's splitter, after a couple of seconds. :)
However, i'm still unable to play it after that (I tried ffdshow and xvid's dsf for the task, and CoreAAC for audio).

Now i gotta get some sleep, i'll do more tests tomorrow.

[avih]

thx P0l1m0rph1c. reports on success stories (regardless of mp4) are welcome as well, since the new filter is NOT mp4 specific, and it will be released as an improved GENERAL riff-cdxa filter. so, again, general filter test results are welcome as well.

calinb, shitowax, i'd appreciate some comments. phobos, i just wish you'll be able to use it for any playback ;)

thx guys.
avih

[P0l1m0rph1c]

This is weird. Now, all filters connect correctly (using either MP4, AVI or MKV), but when trying to play, there's simply no action. Either in graphedit or any media player, it just sits there, doing nothing. Maybe this issue is from the dshow part of the filter (?) Dunno. But i see we're making progress here :)

[P0l1m0rph1c]

Oops, sorry, but i guess the problem is from my drive. I made a virtual image of the XCD's i have, and it played just fine with either MKV, AVI or MP4 (MP4 took a few seconds to connect).

So, i guess the problem here is either of the drive or its mode (DMA, PIO).

Keep up the excelent work ;)

[shitowax]

With the same 3ivx splitter and the same file:
XCD beta7b: 9 seconds to open
XCD beta7c: 6 seconds to open
CD : less than 1 second.

With XCD, the CPU is still at 100% and only a few physical access during the opening phase, so physical access don't seem to be the bottleneck here.

[avih]

Originally posted by shitowax
With the same 3ivx splitter and the same file:
XCD beta7b: 9 seconds to open
XCD beta7c: 6 seconds to open
CD : less than 1 second.

With XCD, the CPU is still at 100% and only a few physical access during the opening phase, so physical access don't seem to be the bottleneck here.

hmmm, that was not the improvement i was hoping for... :(
will u pls open debugview and make me a log of the messages during file open? (there should be a L-O-T of them, since it prints a message for every access to the filter, and according to your estimation (and my calculation), there should be about 500,000 accesses to the filter. maybe u could make a log for a very small file? say 10-20M? it should note that it retrieves from the buffer (if indeed it does) for about 500 streight requests before bufferring another sector. reading from the buffer should have NO overhead. exactly copying the number of requested bytes, with no intermediate bufferring. unless i have more bugs of course ;)

i'd greatly appriciate such log report (do zip it, or prefferably, 7zip it).

thx
avih

[calinb]

I'm not seeing much difference with 7c either. I haven't had time to do as much testing as before and I haven't quantified the improvement, but it's not much. Maybe have more time in a couple of days!

[avih]

ok, test7f was sent. hopefully, this version WILL be faster.

changelog (test7f): removed debug messages during playback.

under normal conditions, the filter sends a debug message to the OS appx. once every 3 seconds, however, in the case of the 3ivx splitter, it sends about 1,000,000 (1 million) messages before the 1st frame is displayed. this could be the reason for the slowdown, 100% cpu, etc.

your feedback is greatly appreciated.
thx again,
avih

[P0l1m0rph1c]

Yay! Way faster, no doubt about it :)

Takes less than a sec to load here now, just as much time as a regular cd.

Nice work! :thanks:

[avih]

Originally posted by P0l1m0rph1c
Yay! Way faster, no doubt about it :)

Takes less than a sec to load here now, just as much time as a regular cd.

Nice work! :thanks:
:) at last. good.
guys, i'd like some more confirmations pls, and if all is well, i'll release this version publicly later. pls give this version as much test cases as u can, i want the public version to be as bug-free as i can.

thx again,
avih

[calinb]

Thumbs up, way up!!! :) 7f is working very well indeed, avih. xcd2file is working well too. I've run quite a few codecs with it and will continue to watch a variety of movies through the weekend. Many thanks to your efforts and thanks to shitowax for his interest and help too.

[avih]

Originally posted by calinb
Thumbs up, way up!!! :) 7f is working very well indeed, avih. xcd2file is working well too. I've run quite a few codecs with it and will continue to watch a variety of movies through the weekend. Many thanks to your efforts and thanks to shitowax for his interest and help too.
:) thx. enjoy your movies ;) (and pls report back later)

the trigger for the (right) guess about the debug messages as the cause for the slowdown was calinb's report of 'proportional load times' and shitowax's report of 100% cpu. the description of the file access pattern from shitowax helped much as well. the debug messages do not affect the performance under 'normal' conditions, but they definately do when requesting half a million requests from the filter before the 1st frame is displayed. my bad for not figuring it out earlier. nevertheless, due to the great testers that u were, and the quality of your reports, we've managed to solved the issue. many thx again to all of u.

of course, the new cache system improve the performance considerably as well, but the final acceleration came from the removal of the debug messages. how dumb of me ;) cheers guys.

:)
avih

ps.
shitowax, i still think that u'll gain much in other scenarios as well with your new cache system. congrats for that one ;) (although to test your new system you'll have to use the 'old' test6b release of the filter, as the new one does the hard work for u anyway ;)

[shitowax]

My personnal opinion is that it's not up to the splitter to perform caching. The source knows in theory much better the physical layer, sector size, bottleneck factors, ... As the 3ivx splitter totally abstracts the physical layer, it requires bytes only when it needs them. Of course this causes problems with sources that assume things about the file request patterns like avih's XCD did and like other sources do (for example we have problem reports when serving files from MacOSX to windows via samba)... For the moment the latest 3ivx alpha splitter contains a cache of previous bytes read, it's fairly interesting with poorly muxed streams. The 4.5.2 or maybe 4.6 one should contain something a bit smarter ;)

[avih]

shitowax, i generally agree with you. EVERY layer should do it's best. in our case, the filter should indeed have provided some sort of caching. i didn't implement caching since i, as yourself, left the caching (and possibly prefetch) to underlaying layer. in my case it's the windows OS. in your case, it's my filter (and the filter has NO overhead when assumming the OS cache). and we now know that the OS cache is bad in our case. the filter 'cache' was bad as well due to this fact. in order for our code to work well together, I had to OVERRIDE the OS cache mechanism, a thing i didn't want to do. it's not really overriding, but providing another layer of caching.

as i said, test6b had no cache, and it performed pretty well because the HIGHER layers usually did well, and asked for the amount of data that they KNEW they'd use. in our case (3ivx and xcd) it performed bad since NEITHER layer did what it should have done.

generally speaking, caching/prefetching without prior knowledge about the access pattern would always have a performance hit (should it be memory usage, cpu load, etc). my cache system indeed introduces such performance hit (although neglectable). it uses several hundreds of KB for auxiliary storage, and it has a tiny cpu overhead for searching the cache for every request. a performance hit that wouldn't have been there should your splitter behaved 'properly'. that's why i made it cache only 64 sectors. to make the cpu overhead of my cache system as minimal as possible. i do NOT know how will my 'clients' ask the data. so the cache is provided for clients that don't do THEIR job well. kind of protecting bad 'users' from hurting themselves. this introduces an overhead for 'users' that KNOW to behave well.

and this brings us to the bottom line. the only component in this pipe that actually know the access pattern is the object that request the data. therefore, the requesting object MUST request the data in an optimized manner. an optimized manner in our case is an optimized FILE-SYSTEM access. and file systems are slow. that's a known fact. you should NOT assume that the filter abstracts the file into a huge MEMORY space, and ask the data in chunks of 4 bytes.

imagine you would have implemented a function that copies a file to another file. let's say the file's size is 300M. would u read/write the file byte by byte and assume that the OS will cache/prefetch it in such a way that the performance are maximized? or would u read the file in least in 1K chunks??? it's a file!! the cache is for 'users' of the file system functions that don't take into account the fact that they're fetching data from a FILE.

nevertheless, i think that the my cache system is worth it. since there could be more clients that misbehave, and it will give the final user an improved perfocmance in such cases. the downside is the performance hit on the rest of the 'good' cases, but i've written the cache in such a way that it would be neglectable.

do u think that the performance hit you're experiencing with samba is due to the bad implementation on the samba's side? maybe on OS/X's side? if that's what you're thinking then you're mistaking. it's due to the fact that every access has a higher overhead than access to a local file system. it's up to the requesting object to optimize its access pattern. what u should assume is that file access HAS overhead. the amount of it depends on the media and medium.

u should assume that the time it takes for a request (for any kind of external data) to complete is:

singleRequestTime=REQUEST_SIZE*throughputFactor+fixedLatency

both throughputFactor and fixedLatency are out of your control, but u should assume they're there (and they're always there). not taking this factor into account will result in worse total performance. since u can't control them, u should optimize your access pattern such that the total time is minimized. the total time calculation is NUMBER_OF_REQUESTS*singleRequestTime. or:

totalAccessTime=TOTAL_REQUESTED_BYTES*throughputFactor+NUMBER_OF_REQUESTS*fixedLatency

it's easy to see that the only factor that u can control here is the number of requests. the lower it is, the lower the total access time will be. if you use many requests, you're in the mercy of the fixedLatency constant and/or some kind of caching system (again, which u can't control). you MAY be lucky, but you may also not be so lucky, as is the case of the xcd filter and the samba issue.

the only GOOD solution is to optimize NUMBER_OF_REQUESTS according to your prior knowledge as for the nature of your requests. if you know you're going to search 2M, that you should ask for 2M, and then parse it from memory, where u know that both of these factors are considerably lower than these of a file system abstraction.

that's it ;) as i said, i think that your decision to implement the cache will have much gain in other scenarios as well. u reassured my assumption with at least one case in which it currently performs badly.

[Phobos]

sorry about the delay

something must be VERY bad about my system. Ive seen improvement in the lastest version indeed. In the first one it took 4 min to show the error in wmp, in the second it took a couple of minutes, now it takes 20 seconds to show the error. Note that this is from Daemon-tools so my supposed drive crappyness is out of the question since polimorphic had success. Anyway ill try reinstalling 3ivX, adding the registry hack and rebootin and lets see if that helps.

EDIT
Alright, i tried the method mentioned above and i still get this message:

"ClassFactory no puede suministrar a la clase requerida (Error=80040111)"

its something like "ClassFactory cant retrieve the required class"

anybody knows anything about this errror?

[P0l1m0rph1c]

@Phobos, just a quick question. What extension do you use for the m2f2 file? Are you using .mp4 or .dat? I just can't get things to work if i use other extension rather than .dat. That could be a problem.

[Phobos]

i use dat of course

i also try graphedit, it shows the very same error and even faster than wmp.. wtf?

[shitowax]

Are you sure you don't use gabest's xcd source instead of avih one ?

[calinb]

Originally posted by avih
the trigger for the (right) guess about the debug messages as the cause for the slowdown was calinb's report of 'proportional load times' and shitowax's report of 100% cpu. Thanks for this information, avih. Sometimes when users try to help developers, our reports just fall into a "black hole" and we never find out if our reports are helpful. I appreciate your efforts to keep us informed, though I realize such communication takes valuable time away from development and your "day jobs," studies, etc.

I'll continue to feedback my experiences using your excellent filter.

[avih]

Originally posted by calinb
Thanks for this information, avih. Sometimes when users try to help developers, our reports just fall into a "black hole" and we never find out if our reports are helpful. I appreciate your efforts to keep us informed, though I realize such communication takes valuable time away from development and your "day jobs," studies, etc.

I'll continue to feedback my experiences using your excellent filter.
;) thx

[snakeman]

Please send me the 7f filters to panther182@yousef.com

I would love to test it on my server 2003 install for compat testing.


plus I am still on the fence nero digital or xvid, but thats another topic.



snakeman

[avih]

Originally posted by snakeman
Please send me the 7f filters to panther182@yousef.com
I would love to test it on my server 2003 install for compat testing.
snakeman
sent. pls report back to this thread.

[snakeman]

Stream 0

Media Type 0:
--------------------------
AM_MEDIA_TYPE:
majortype: MEDIATYPE_Stream {E436EB83-524F-11CE-9F53-0020AF0BA770}
subtype: MEDIASUBTYPE_QTMovie {E436EB89-524F-11CE-9F53-0020AF0BA770}
formattype: TIME_FORMAT_NONE {00000000-0000-0000-0000-000000000000}
bFixedSizeSamples: 1
bTemporalCompression: 0
lSampleSize: 1
cbFormat: 0

results from media player classic,


Windows Media Player encountered an unknown error.



results from WM9


Details are as follows

previous version was 6B everything worked fine except nero (xcd) with daemon tools

OS Server 2003
cpu dual processor MP2000 AMD
Mem 512 megs of ram
400 gigs of raid 0+1 for video and audio recording
ldw-811s for writing 99min XCDs :)

using DVX latest version for XVID/RV10/DiVX/OGG creation

XVID rc2 DiVX 5.11 pro RV10 beta ..............

Did I miss anything?
Everything works fine except nero digital .mp4 files in XCD format.

snakeman

[calinb]

Originally posted by snakeman
Did I miss anything?
Everything works fine except nero digital .mp4 files in XCD format.
snakeman With nero mp4/XCD, you must use the 3ivx splitter, which means you must install 3ivx after nero or any nero updates. Otherwise, Directshow will use the nero filesource/splitter filter which doesn't work here. Surprisingly, I've found that the nero Showtime player also works with mp4/XCD, but I prefer MPC.

If using MPC, make sure that the internal MPC VCD/SVCD/XCD Reader is unchecked in the filters options page because it doesn't work with mp4/XCD.

If you still have problems, try to render the XCD with graphedit and see what happens.

[snakeman]

Trying it now, hmmm I installed 3ivx not just the splitter, and it worked. It took about 2 sec to start on my P4 2.4gig 512 1066 rambus
box with all scsi drives. Oops almost forgot using Daemon tools I haven't burned a cd yet.


snakeman

[avih]

snakeman, could u pls answer explicitly:
have u or have u not found a scenario in which the filters test6b and test7f behave differently?

if you have found such scenario, what's the difference?
thx
avih

[snakeman]

6b (with or without 3ivx installed,it just hangs and then it gives the error I posted above)does not work for me on any of my boxes, Server2003 or XPsp1. 7f works with 3ivx installed on XPsp1, I have not tested my other two machines yet I have an appointment in a few mins.



snakeman

[avih]

thx. sounds good so far. pls let us know when u test on your other machines.

thx again,
avih

[snakeman]

Ok here we go, it works fine on my server2003 box. But, I am getting some strange error messages and nero showtime will not work at all. The error messages only occur when closing the xcd. The messages occur with both WMP9 and WMPclassic 6.4.xxxxxx I will post the error message later on tonite.




snakeman

[snakeman]

http://cardude.dyndns.org/~byron/error.jpg

[avih]

this is crutial. you MUST compare the results with test6b. otherwise i can't tell whether it is a new bug that i introduces in test7f, or it's the same problem as with test6b.

[snakeman]

Oops sorry for the incomplete post. Basically it was the same as before. It does not work at all with 6b installed with or without 3ivx. But, works as stated above with 7F installed andbv 3ivx installed with the error messages. Strange, nero showtime doesn't work at all. On the Server2003 installations(XCD). Again using Daemon tools to play the XCDs. I have not burned a disk yet.

Machines:

Dually MP 2000 AMD with 512 megs of ram etc
other 2003 Server box
P4 2.6 gig 512 megs of ram and the usual suspects of 200 gigs of raid space, etc

I hope this helps


snakeman

[avih]

ok, so test7f behaves differently on different machines. it's working on one of your 2003 servers, and not working on others. it didn't work with test6b on any machine. correct?

i'd like you to supply me with some more info pls. go here (http://www.sysinternals.com/ntw2k/freeware/debugview.shtml) and download debugview (small freeware utility that logs system messages). run it before you open the video file. then open the file, and wait till it crashes. export the log from debugview, zip it, and email it to me pls. i'll try to further analyze it.

thx in advance.
avih

[snakeman]

Incorrect, it does not work on any of my Server2003 machines. I will send you the debug results.



snakeman

[bond]

as the error seems to have to do with timemanager.ax i would propose to unregister it
afaik this timemanager filter was developed to make native matroska possible, but isnt needed anymore for anything

[snakeman]

Send details on how to unregister it.



snakeman

[P0l1m0rph1c]

Go to start menu -> Run. Then type:

regsvr32 /u timemanager.ax

Then windows shall give you a message saying if it was unregistered successfully or not.

[Phobos]

Originally posted by shitowax
Are you sure you don't use gabest's xcd source instead of avih one ?

yes im using avihs 7f

i tried the debugview thingy and i got nothing, any n00b proof guide of being sure debugview worxs? mp4 xcd still doesnt work for me btw... and for some strange reason i stopped getting emails from this thread, sorry for the delay

[avih]

phobos, when you opened debugview, you saw NOTHING? was it left completely blank? or were there some sort of messages? how about playing a different kind of xcd (ogm/avi/etc). debugview should give plenty of messages when starting to playback. (the xcd filter should give some messages, other filters may display some messages, and even the player itself. i think zoomplayer shows some messages.)

[Phobos]

well, the first time i did it showed no messages, now i have a logfile, it worked :)

tell me your e mail or msg me so i can send results to you avih

[avih]

the one you got my filter from. zip it good before sending. this inbox is quite limited. thx. or, even better, if it's not more than 2-3 pages, just post here such that others can analyze it as well.
thx.

[phrentec]

Avih, can you send me 7f(or whatever it is the version that seems to be working at least). email at p h r e n t e c @ y a h o o . c o m I'de really like to try it.
Also if someone can please restate the proper registry additions that would be nice.

[avih]

@phrentec: sent.
don't distribute, report back, and compare with test6b (must).
enjoy.
avih

[avih]

reminder:
calinb, snakeman, shitowax:
still waiting for your (detailed) findings with test7f. that's the reason i haven't publicly released the filter so far.

thx.
avih

[calinb]

Originally posted by avih
still waiting for your (detailed) findings with test7f. avih--sorry, man! I don't really have anything new to report on 7f--except the increased speed. I've had some crashes with ogm/Div5/dts XCD, but 6b crashes on those files too. I haven't had time to play the failing XCDs with graphedit yet--the crashes were with MPC. I'll investigate further, if you're interested, but as I said, 6b behaves the same way with these XCDs (internal MPC filter works on them though). 6b and 7f yield a jumbled screen (mostly green) and I get a MPC crash w/ "mpclassic has encoutered an error" >>> CLOSE ...or something like that.

[avih]

thx calinb. no worries ;) nothing bad happens if it's delayed. your report is good enough. thx again.

shitowax, snakeman: still waiting.
thx

[phrentec]

I know we are trying to get mp4 xcds to work in directshow but aside from that Is there any chance that the quicktime player(which is supposed to support mp4 natively) can be made to open mp4 files from an xcd? Right now it obviosly can't. I was hoping for support for mp4 xcds on macs.

[avih]

well, let's focus on releasing this filter 1st, and then see what else can be done.

have u played with it?

[calinb]

Originally posted by phrentec
I was hoping for support for mp4 xcds on macs. Yeah, my wife's got a mac, but I haven't found a way to do it yet. It can't read XCDs at all (can't even copy the dat file). I have to convert them to files on a networked windows machine. BTW, I've found QT playback to be very marginal, performance-wise, with most of the media that I've encoded. That is, when it can play it at all (no HE-AAC, ogm, mkv, etc.!) on her 800GHz/640MB G3 iBook. mplayer is best and VLC has pretty good performance too. VLC plays the most formats. I still haven't tried 3ivx on it (only have the windows version). She's more into viewing than encoding :)

[snakeman]

Sorry guys I have been real busy with some projects I let slip. I will be with you guys some time in the next 24 hours.


Sorry :(


snakeman

[avih]

no need to be sad snakeman ;) we're ok...
if you can't test it just let me know.

cheers
avih

[Phobos]

man, sorry to see this thread didnt have a real ending. avih, would you resend me the 7f filter??? im starting to believe its my 16x lg combo drive messing up. maybe i have the b0rked one shown at the xcd faq... anyway i have plenty of computers to test on in my dads business (where i spend most of the time now on "vacation" :/ ). so pls send the filter to phobmx@yahoo.com.mx if you can

and i have an update for you guys, the new phillips player supports MP4 in xcds, as tested in its respective thread.

snakeman and shitowax, i hope you can still review the filter so avih releases it public, i want to do my part too

[Phobos]

alright bro i have great news for ya!!! i dunno what changed, but i reinstalled windows a week ago and now with your bat everything works like a charm!!!

i used an 800mb mp4 with xvid 1.01 and he-aac, even subtitles worked flawless. Seeking through the file is as fast as it gets and the file loads as fast as if it was an avi (dunno if this is because of my fast puter but i doubt so)

so the guys left must give it a shot, this one is worth a public release if you ask me

so no more problems playing mp4 from an xcd :D

[avih]

hmmm, quite an old thread ;)
i'm glad to hear it's finally working for u.

the reason i didn't eventually release the filter is because the 3ivx filter was fixed, and now doesn't exhibit the problem it used to have when playing xcd.

the filter was a workaround for otherparts of the system that missbehave, and now they behave properly. so no need for it afterall.

although i must admitt it was an interesting 'mini-project' ;)
if you still think that it's worth releasing, plase name the reasons, and i'll reconsider it.

thx.
avih

[bond]

Originally posted by avih
the reason i didn't eventually release the filter is because the 3ivx filter was fixed, and now doesn't exhibit the problem it used to have when playing xcd.3ivx! i knew it :D

if you still think that it's worth releasing, plase name the reasons, and i'll reconsider it.hm if i remember it right the version had something to do with enhancing the caching?

was this only a workaround, or is it really not useful for other situations? does it have downsides compared to the current filter?

[avih]

@bond:
the only difference is that it has cache, while the original filter doesn't. so it has bigger memory footprint (about 2M compared to probably less than 100K), and it has higher cpu load (the cache needs to be searched for every request). the extra cpu load is probably neglectible on modern computers, and so is the higher mem footprint, but they're there.

BUT all this cache mechanism needs not be there unless the parts that use the filter to read XCDs don't do it properly. so while the overheads are quite low, it will allow writing bad software that still behaves pretty good with the filter. as i usually don't tend to encourage bad programmings habbits, and all the currently known filters read properly from the xcd filter, i thought that the current filter is very good with it's minimalistic approach.

so, again, since you too have a copy of the 7f filter, you may compare it to the original 6b release, and if you find other filters that read the cd slow with the xcd, you can try the 7f one for speed improvements. if it works better than i suggest that u contact the (other) filter author, and point him to this thrad ;)

of course, if enough ppl think that it's worth releasing the 7f filter, i'll do that. but again, i don't like encouraging bad programming habbits....

avih

[calinb]

Originally posted by avih
the reason i didn't eventually release the filter is because the 3ivx filter was fixed, and now doesn't exhibit the problem it used to have when playing xcd. Really? Was the 3ivx splitter fixed?

I have the "plus" version (paid-version)of 3ivx installed on my computer, but I downloaded the latest free 3ivx demo. Comparing the size and dates on the splitters (3ivxDSMediaSplitter.ax), they are both 284KB and 1/27/2004, which predates our problems.

Does somebody have a newer version of 3ivx D4? I'm still using avih's unreleased test 7f version of the XCD filter because of this problem.

Also, didn't we see a performance improvement playing files across a network with test 7f? I think I remember an advantage.

Phobos, glad to hear it's working, regardless. :)

[Phobos]

Originally posted by calinb
Really? Was the 3ivx splitter fixed?

I have the "plus" version (paid-version)of 3ivx installed on my computer, but I downloaded the latest free 3ivx demo. Comparing the size and dates on the splitters (3ivxDSMediaSplitter.ax), they are both 284KB and 1/27/2004, which predates our problems.

you are right, lastest 3ivx version was released before our problems, or was there an un-announced change??

[bond]

3ivx is working behind the scenes on new versions of their tools of course
i assume they sent avih a fixed unofficial one for testing

[SeeMoreDigital]

bond,

Can you remind us how well the Philips DVP642 stand-alone player coped with .mp4 files burned as XCD?


Cheers

[bond]

i think it has been reported hat the video was "delayed", dunno what this means tough

[SeeMoreDigital]

Please forgive me for asking an stupid/newbie question!

Does XCD use similar standards to the old Philips CDi system?

On this thread (http://forum.doom9.org/showthread.php?s=&threadid=74009) Joe Fenton describe CDi as being: -
discs are one track in mode 2 form 2 with a variant of ISO-9660 specific to Philips. I don't do much burning so I don't understand all the different modes etc. But I am interested why it is some players (like the Philips) can read/see discs that other players can't!


Cheers

[avih]

Originally posted by bond
3ivx is working behind the scenes on new versions of their tools of course
i assume they sent avih a fixed unofficial one for testing

no, thay actually haven't. Shitowax told me they changed the implementation (which also caused the same symptoms while playing a clip from a network drive, due to the same incorrect assumptions: "the underlaying layer should handle cache, and therefore, we can treat the stram as fully random access, and don't worry about delays in reading it from the medium").

i wasn't following this issue closely since, as i don't personally use the 3ivx splitter.

perhaps shitowax could provide some more info here.

sorry for the misinformation guys, i hope we can sattle this issue soon.

avih.

[bond]

okidoki, i now had the time to test avih's 7f filter with the latest internal 3ivx version available atm (should include all the goodies shitowax promised in this thread :D )

here are my findings:
1) with 7f loading needs about 6-7 secs (with graphedit, winme) on my 866mhz pentium3
2) with the latest official 6b it takes forever (surely more than 30secs)
3) only registering the 7f is not enough (takes very long like 6b), adding the mentioned registry entries is still necessary (shitowax wrote that 3ivx will do this automatically)
4) there doesnt seem to be much seeking, as the cd drive is very quiet
5) the whole playback process seems to need pretty much cpu power, as at the beginning and on some highmo scenes i get choppy playback

all in all 7f is working much better than 6b, still for me the current situation is far from being perfect :(

/me blames shitowax!

[calinb]

Originally posted by bond
okidoki, i now had the time to test avih's 7f filter with the latest internal 3ivx version available atm (should include all the goodies shitowax promised in this thread :D )
Bond, Thanks!

Your latest result seems to be in line with what I found with 7f and the released 3ivx splitter. I made the registry edit too. Doesn't sound like the internal version provides any improvement. Are you sure shitowax gave you a new splitter (did you diff the files :))

[bond]

Originally posted by calinb
Doesn't sound like the internal version provides any improvement. Are you sure shitowax gave you a new splitter (did you diff the filesi am absolutely sure, as i have access to internal alpha releases which are done for testing previews

what i find annoying is the high cpu usage it needs during decoding :(

[shitowax]

It's a VERY BAD idea to put a .mp4 file on an XCD... and this whatever chaching system we add in our splitter.
Anyway, did you check that the same bitstream plays correctly from a .avi ? Maybe it's just your CDROM drive that cannot handle the high bitrate of those scenes ...

Originally posted by bond
okidoki, i now had the time to test avih's 7f filter with the latest internal 3ivx version available atm (should include all the goodies shitowax promised in this thread :D )

here are my findings:
1) with 7f loading needs about 6-7 secs (with graphedit, winme) on my 866mhz pentium3
2) with the latest official 6b it takes forever (surely more than 30secs)
3) only registering the 7f is not enough (takes very long like 6b), adding the mentioned registry entries is still necessary (shitowax wrote that 3ivx will do this automatically)
4) there doesnt seem to be much seeking, as the cd drive is very quiet
5) the whole playback process seems to need pretty much cpu power, as at the beginning and on some highmo scenes i get choppy playback

all in all 7f is working much better than 6b, still for me the current situation is far from being perfect :(

/me blames shitowax!

[bond]

Originally posted by shitowax
Maybe it's just your CDROM drive that cannot handle the high bitrate of those scenes ...hm i noticed that whenever there are highmo scenes, which show choppyness, it seems that there is some heavy reading from the cd, as the lights of my drives flash strongly (in quiet scenes the lights flashes very rarely).

i tested it with my two drives: philips cdd4801 burner (reading speed 32x) and the samsung sd-612s dvd drive (40x)

does anyone know whats causing this heavy reading in highmo scenes and what i can do about it?
does the caching discussed in this thread influence this somehow? shouldnt be 40x drives fast enough to handle 1cd highmo bitrates?


btw i also noticed that the samsung drive doesnt seem to like reading the beginning of mp4s on xcd as it takes very long, makes lots of noice, spins around a lot...
i assume this has something to do with the way the 3ivx splitter handles the mp4 playback starting?

[avih]

@bond:
the buferring/prefetching affects mainly 2 scenarios:

1. most important: the startup time. during the 'negotiations' phase, some splitters (i.e. the 3ivx one) reads the a LOT of data in very small chunks (think 2M in 2-4 bytes chunk) to identify the properties of the clip, and synchronize with the stream. in this case, the buferring (and prefetching) helps performance a lot (again, only if the splitter ignores the fact that it's reading from a relatively slow media).

2. during playback, where relatively larger chunks are read (think 2K-32K), the buferring has small to non-existent improvement to very-slightly deteriorating the performance, because the cache is searched on every read, but it never finds anything it needs in the cache (it always reads forward and nearly-never the same sector twice). the preformance hit should be neglectible imho though.

[bond]

thanks avih! :)

any idea why (too) the heavy reading from the cd in highmo scenes happens and what i can do about it? :(

[avih]

@bond.
no, i don't know. iirc, i did not see any obvious anomalities with the 3ivx splitter during normal playback. the only thing i can think of, is that for extremely high bitrate bursts, the cd just doesn't read fast enough.

my only suggestion (for the implementation of the splitter), is that the splitter is programmed to prefetch the data from the cd in a separate thread, in large enough chunks (i'd say few hundreds Kb to few Mb), and always 'stay ahead' enough such that there's no acute need to use the cd for reading very large chunks of data during very short periods of time. but this is only based on my logic, and not on any aquantance with the internal implementation of the splitter.

sorry i can't help better.

[bond]

:(