XCD: automatic media detection and licensing

[Koepi]

Avig,

now i know what you mean, and yes, you're right.

But the number of sectors that fit into a small buffer (we could make it a ring buffer, would work even smoother and would be a software representation of the physical world ;) ) isn't really random. We should make the buffer size as a multiple of our sector size, e.g. 2352 or 2336 bytes, this is the kind of alignment we need.

I'll search about tracks/sectors on CDs later on, but maybe we really don't need such a huge buffer as ~500kb - depends how much data is stored in 1 track of a cd (800 tracks per inch if i remember correctly, would be something like a few kb per track).

Regards,
Koepi

[DeXT]

Hi there again. I was not sure about releasing this since I'm leaving tomorrow early and I won't be able to read the reports until I'm back in May 31. But I've done many tests with this and I think it works fine. Anyways I'm not deleting the previous release so if you think there is something wrong just use the 1.2 release.

Mode2 CD Maker 1.3 with the following new features:

- multiple subdirectories support (only for Form1 files)
- ASCII filenames (w/lowercase and whitespaces)
- removed old VCD-like filenames

To add subdirectories to the current ISO structure just add a -d option followed by a directory name. All subsequent Form1 files
after this option will be saved inside this directory instead of root. You can do this as many times as you want, but all subdirectories will be added at root (i.e. no nested subdirs support). An example:

mode2cdmaker -m movie01.avi -f readme.txt -d utils -f player.exe player.ini -d SECURE -f movie01.xch

This will generate the following directory tree:

MOVIE01.DAT (Form2)
README.TXT
UTILS\PLAYER.EXE
UTILS\PLAYER.INI
SECURE\MOVIE01.XCH

I know this is not the best method but I'll improve it later, probably through the use of a file list.

To avoid these ugly ISO-9660 filenames use the -a (ASCII) option. This will tell the tool to use ASCII character set which supports whitespaces and lowercase letters in the filenames. To use long filenames along with this you'll have to specify both options (-l -a).

And now by default the Form2 files will be saved with the original name instead of the ugly AVSEQ01.DAT (but trimmed to 8+3 unless you add -l of course).

Oh and good news, tonight in an IRC chat Avih, Koepi & me finished the first XCD specs so Koepi will be able to release a 100% compliant tool very soon. Definitely XCD (File Level at least) is here sooner than we expected.

Hope you enjoy it. I will read reports when I'm back. See you!

http://webs.ono.com/de_xt/

(wait until ISP cache is updated)

[Koepi]

Great work De_XT! :)

I'm glad that you could release a new version this fast!

So it's my turn to finish off the backup creator to support the final spec avih sent around today. It may take some hours though since we're once again having a small party here :)

Anyways, enjoy your time off,

CU then :)

Regards,
Koepi

[TheXung]

I must comment that this is the biggest thing that has happened to ripping since Nandub.

[Oletros]

I have used mode2cdmaker 1.2 with riff test5 without any problem.
Last film I burned was one done in xvid with 2 ogg stream and subtitles. It sized 803 MB and plays ok.
Today I have used mode2cd 1.3 with riff test6 to make an image with this parameters:

mode2cdmaker.exe -m "Rocky Horror.ogm" -o trhps -v TRHPS -e DAT -l -a

I load the image with daemon tools and load the file with bsplayer. I can only hear the sound but I don't have any video.

If I load the .DAT file in graphedit it plays perfectly.

Any clue for that behavior?

Thanks
(Excuses for my bad English)

[rui]

Just wanted to say that I had no problems in creating a mode2 cd with an avi movie, burning at the same time the subtitle files, and using with no bugs at all direct vobsub.
Nothing big, but I thought it was worth this little post, to someone out there who is wondering (like I was) if direct vobsub would work ok with this new burning method.

I used mode2cdmaker 1.3, and riff-cdxa filter 6, with this command line:
Mode2cdmaker –m movie.avi –o newmoviename –e avi –f newmoviename.idx newmoviename.rar

I only used the command –e avi, so the new file extension wouldn’t be DAT but AVI, and that way one can load the movie even from windows explorer, and not only with media player. But it isn’t necessary for direct vobsub to work. I tried burning with the default DAT extension, and direct vobsub worked very well

Hope the Mod finds this post useful enough so he doesn’t delete it;)

EDIT: Just saw that Oletros also burned a movie with subs. I also have problems when trying to load the DAT file in bsplayer (crashes)

[Koepi]

Might be an issue of BSplayer. Can you test with WMP 6.4 again please?

Btw., if you change the extension to AVI with an *.avi source file you don't have any troubles since avi files already contain RIFF headers. :)

Just don't try that with ogg, mcf or mp3 ;)

best regards,
Koepi

[Oletros]

I have played it with WMP 8 and Zoomplayer and it plays like a charm. Could it be a bsplayer issue.

By the way, is there any option to hide the subtitles muxed in the ogm? I haven't found it. I have movies with 2 subtitle streams and movies with 1 and I can't hide them.

For information, I have burned the image with a AOpen 24x and I can play the cds with it and with my Pioneer 16x DVD.

Thanks Koepi, avih and the rest of people for you great job.

[rui]

Koepi, like Oletros said, with media player 8, and with media player 6.4, plays just fine both the avi and dat extension.
But, when having avi extension, and trying to load on bsplayer, it gives a message of unknow format. When trying with dat extension, it crashes.
I will experiment more, because i really like the posibilities in bsplayer of loading winamp DSP filters.

[Oletros]

I have only tried to create the image with form2 extension equal to DAT and with this extension bsplayer loads the movie, it doesn't crash (I haven't used AVI extension rui).
When you watch ds filters it shows all the necessary codecs loaded (xvid decompressor, ogg decompressor, vorbis splitter) and you can hear the movie but you can watch it.

It's a really strange issue because you can move around the movie, you can use the chapter list muxed in the ogm, etc.

[avih]

bsplayer doesn't use a 'standard' dshow chain. it's a known facts, and it has advantages and disadavtages. one of the advatages is that it uses it's resize module, and can overcome crappy hardwarew resizers. but that's the situation atm.

if bsplayer can't play an mode2form2 clip for some reason (due to unique file handling), there's nothing we can do about it, except writing a plug-in (is that possible??) or let the author modifie the player to use the xcd library (when it's finished).

currently we're working on a dshow filter only as it allows most players (wmpx.x, zoom player, powerdivx and more) to play the file. when we're satisfied with the dshow dshow filter, we'll hopefully start making other plugins (like for mplayer, bsplayer, winamp, etc).

sorry for now
cheers
avi

[Oletros]

Thanks for the answer avih.

[nah]

Hi,

I've just updated my Mode2CDMaker_GUI to use with Mode2CDMaker 1.3.
Not enough time to test all items I've added.
http://membres.lycos.fr/zorgzorg

[MaTTeR]

Thx for the update nah.

Anyway you can start naming the archives with version numbers? It would make a little less confusion for me;)

[mustaneekeri]

Concerning BSPlayer i had no problems what so ever playing a .dat file from the image mounted with daemon tools.
I used latest Test6 filters, latest BSPlayer, Latest GUI from Nah. Mounted the image with Daemon tools (970MB .dat file (Ogg container with DivX 5.02 video and Vorbis audio) and a form1 .sub file).
The movie played perfectly with BSPlayer from the virtual DVD-drive and with external subtitles and all.

Im now goin to add BSPlayer straight to cd try to burn the image to 99min cd-r.
Reporting the result later...

Burned the image file 981Mb/97:14 containin 20 form1 files and 3 folders and one form2 media file.

Burner: LG GCE-8160b
Reader: Liteon LTD-163

It works like a charm.

[Koepi]

We're close!

I just sent around rc1 of XCDBackupCreator to avih and DeXT for reviews of the correct implementation of our spec.

:)

Regards,
Koepi

P.S.: some input is still necessary to find out which are the vital parts of avi, mcf and mp3... ;) (the first 128bytes, the last 256bytes, ...)

[ChristianHJW]

Originally posted by Koepi
I just sent around rc1 of XCDBackupCreator to avih and DeXT for reviews of the correct implementation of our spec.
P.S.: some input is still necessary to find out which are the vital parts of avi, mcf and mp3... ;) (the first 128bytes, the last 256bytes, ...) ... asking for help on MCF mailing list now ;)

[jurij]

@koepi: "..I just sent around rc1 of XCDBackupCreator to avih and DeXT..."
i tought DeXT was not gonna be back online before the 31st.... :) (haven't seen any new post from hims since he left...)

What do you mean we are close if it still miss the whole parts about avi, mp3 and mcf? you only seem to us to have completed the ogg part.
Let us know a little bit more :)

P.S. you all are great!!!!!

[Koepi]

Since I only have real information about how ogm has to be "secured" in the sense of which information of the media stream is vital, i first coded this.
MP3 will be added soon as well, I just have to find out about id3/id3v2 (how much bytes are reserved for them in the beginning/end of the file?).

I hope this is the information you wanted to have.

EDIT:
PS:
I wrote CLOSE, not FINISHED.
Guess why.

You shouldn't sound that "bad tempered" when you're posting in a development thread. All those issues WILL be addressed.

I'll stop posting about progress when _everything_ I post is bashed at.

[jurij]

well if ogm is working i believe an OGM release would be fine since i guess most of us has moved to ogm (being around this tread all day, eh eh...)

[Belgabor]

Originally posted by Koepi
MP3 will be added soon as well, I just have to find out about id3/id3v2 (how much bytes are reserved for them in the beginning/end of the file?).


Afaik, id3v2 has no fixed size in the manner of always being say 256 bytes. its variable and ususally at the end of the mp3. id3v1 has a fixed size (which I dont know) and is at the beginning of the file.

Regards
Belgabor

[Koepi]

About mp3 I found following information:

The spec for ID31 is simply 128 bytes at the end of the file divided into the following fields[...]

So I have to store the last 128bytes of an mp3.

The ID3v2(.x) tag is at the start of an mp3 and can be _any_ size *grrr*
Still figuring out how to find out the size information, www.id3.org is the address I'm searching now.

First results:
(from http://www.updatestage.com/previous/000501.html )
1. Get the entire ID3 info length from the ID3 header (byte 7-10)
2. Read byte 6 (flag byte). Value > 32 means there is an extended header
3. If there is an extended header, get its length from bytes 11-14 and read to the end of the extended header, otherwise the read head is already positioned before the first frame

Problem is, there might be a copy of this header at the end of the mp3 again (sic!) or it could _only_ exist there. I think I'll support this version:

Storing the id3v2 header (if present, detection through the chars "id3" at byte 1-3), fetch the length from byte 7-10 (int32 hopefully). Second segment stored is _generally_ the last 128bytes of the mp3.

Hu

[Tronic]

MCF protected areas:
http://mcf.sourceforge.net/cd/mcdcd.zip

The code is in mcfcdwr.c

[Koepi]

Hi Tronic,

the only actual value I found in those sources was 0xA8 - do you need these first 168 bytes _only_ as backup to keep the stream playable? Do you have a "rule of thumb" how much data you need to playback such a stream?
We don't want to in-depth analyse and parse the different media types as therefore much knowledge about a format must be implemented into the programs (we have still somewhat the standalone manufacturers in mind ;) ).

Our attempt is to store the backup as m2f1 instead inserting multiple (vulnerable) header copies into the m2f2 part of the CD...

Any further hint would be appreciated (like: the first 64kb of an *ogm video+audio+subs+... stream are vital, with mp3 the last 128bytes and the first [read byte7-10 here] bytes )

Thanks a million,

regards,
Koepi

[raistlin2k]

@Koepi

You're sure to keep always last 128bytes of each mp3? What for those mp3s containing no id3v1, but only id3v2.

I'm used to tag all my files only with id3v2 and delete all id3v1 if I've downloaded the file and not ripped it for myself.

WHY? (you will surely ask)

Well, just to be sure that there is no conflict (difference) between v1 & v2! Because some players prefer v1, some v2, so it may happen, that they show different info on the same file.

I know that 128 bytes is not so much, but it's no good to backup last 128 bytes, if they don't contain id3-tag, but only music.

Raist

[Koepi]

EDIT:

finally found a way to calculate the id3v2 tag length - stupid specs they made(c)Mastaa Yoda.

So I'm confident i get this baby rolling :)

Regards,
Koepi

[Koepi]

Well, well, first version running with mp3 support (still have to do something about non-existent id3 tags....). Btw., I _have_ to store something out of a file, else it would indicate it's stored as m2f1.

so gimme my 128bytes ;)

Now I just need final info on mcf and on avi.

Regards,
Koepi

[Koepi]

Ok, maybe someone can give me a helping hand with avi:

I found this link

http://www.rasnaimaging.com/people/lapus/avi.html

But here it is said that the index, fourcc etc. stuff is in the beginning of the file.

http://www.jmcgowan.com/avitech.html#Format

Still not too useful.

Any other good resources?

Regards,
Koepi

[Tronic]

For MCF, you must protect Main Header (in the beginning of file, size is at 0xA8). In ADDITION to that, you must protect all Small Elements which have protection-value below 0x0E (should actually be user-definable, but can be hardcoded).

The long piece of code, marked "MCF-dependant part" in the source, sorts and combines important stuff into few (typically 2-3) protected areas (protpos[] and protsize[], both on byte-precision).

There is no simple rule for n bytes from the beginning and m bytes from the end. Protected areas could be anywhere, the number of those may vary, etc.

Anyway, the MCF-CD code has handled all that correctly even before you started XCD ..

[Koepi]

Originally posted by Tronic
Anyway, the MCF-CD code has handled all that correctly even before you started XCD ..

I'm sad that you think of it this way. We try to support many formats with a nice CD structure and further enhancements to give some freedom on formats used. MCFCD sounds still very specific and misleading. Well, it's not too easy to implement MCF support into our scheme but it will be there in time (when MCF is ready).

Thanks for your help,

best regards,
Koepi

[ChristianHJW]

Originally posted by Koepi
I'm sad that you think of it this way. We try to support many formats with a nice CD structure and further enhancements to give some freedom on formats used. MCFCD sounds still very specific and misleading. ... well, there has been a nice conversation between avih and Tronic today on #mcf. Seems as if Tronic had some valid points about XCD, but in most other points he had a wrong impression about it. avih maybe wants to tell you a bit more about the outcome tomorrow ...

[Koepi]

This explains why he wrote me to "backup 2k from the start of every media file".

Well, kind of a summary, hm? ;)

[avih]

koepi: no :)

to make a public summary.
the 1st 2K was an initial idea i had long time ago, i think it's even in the spec from 30-apr-2002 ((in the spec from 30-apr, last item of section 3.1)). i just forgot to discuss that when u, dext and myself talked few days ago, that's it. i just remembered in that since you were 'looking for 128B' with mp3 ;). that also reminded me the potential problems of the 'implicit mode 2 form 1' method (#segs=0).

what chris is talking about, is that late last night i talked to him and tronic, and we thought again about merging efforts with mcf-cd, to make a single strong standard. we didn't reach any conclusion, just points were raised about advantages and disadvantages of xcd/mcf-cd. surely, both are not 'perfect', and both can be made better. that's it.

no hidden meaning in anything ;)

in the meantime, the show goes on as planned ;)

cheers
avi.

[ChristianHJW]

Originally posted by avih what chris is talking about, is that late last night i talked to him and tronic, and we thought again about merging efforts with mcf-cd, to make a single strong standard. ... and in fact the more i think about it the more i am convinced there is absolutely no alternative to doing that. It doesnt make sense at all to have 2 different formats aiming for the same goal.
we didn't reach any conclusion, just points were raised about advantages and disadvantages of xcd/mcf-cd. surely, both are not 'perfect', and both can be made better. that's it. Well, what can you do with these crazy coders ;) . Instead of merging efforts they insist on their solution being the best and go for it, no matter if it makes sense or not. And if they talk to each other about it its getting more and more clear that the real differences are only marginal, so i wouldnt even be a big problem to make one solution. But, as always, nobody listening to the marketing people ... developers simply do what they want, and to make things even worse, you cant blame them for that because here they are not paid for their work :D !!
no hidden meaning in anything ;) Well, yes and no ! I know you all are doing this for fun and pleasure, but my general perception of Life is that with improved communication a lot of double work and failures could be avoided. So, i am indeed a little bit disappointed that there is no chance to find a common sense. But this is probably better discussed on IRC tonight, not here.
in the meantime, the show goes on as planned ;) To reach hardware manufacturers attention ( if this is ever going to happen ) both formats dont give themselves a benefit by going separate ways. There is the need for a MCF-CD to complete the MCF specs and make it attractive to both users and manufacturers, but i cant see any reason why there should be another technical solution for XCD and MCF-CD , or however we call the baby. Lets talk again tonight on #mcf .... looking forward to de_xt being back online, he has a very good way of bringing things together.

[avih]

@chris ;)
amigo, no one is fighting, no one says we're NOT gonna merge, no one thinks only his solution is the best (that's true for me, and i think that's true for tronic as well).

all i said is that we didn't reach a conclusion LAST NIGHT! ;) that's it. as u can see, we still talk about that now and then. but i also don't think that we should STOP everything we do till we reach a conclusion. everyone is making progress, and if we merge, surely the time and experience spent/gained by each of us will be valuable and usable.

why do these creazy marketting ppl see everything in black and white?? ;)

best regards chris :)
avi

[Chibi Jasmin]

Using RIFF-CDXA test filter 6 for SVCD playback directly from the disc, I couldn't seek/jump in the file...is this a known issue?

BTW: Maybe make new threads for filter/mode2cdmaker and licensing/internal issues?

[avih]

well jasmin, the non-seekable svcd is not a known issue. i use it and it seeks fine.

what mpeg2 decoders (=filters) do u have installed? i use it with freedvd, and it's working fine.

care to give us more details?

about a different thread... well, don't really know about that, it's a 'fluid' stuff, everyone replies on the same thread a question was asked. there are not too many xcd related threads atm... i think about 4 to the most, so it shouldn't be too hard following the happenings with this subject.

cheers
avi.

[Chibi Jasmin]

Well, with DVD2AVI it always works, with DirectShow based players I only got one short clip to work with seeking (some other short clips and long movies failed), but it could as well be badly muxed/encoded SVCDs, so don't give to much on this, I didn't make these SVCDs...

I have WinDVD4 DS-Filters for SVCD playback. Anything else you need to know? But well, don't spend too much time with this, maybe it's just screwed SVCDs..

Can anyone confirm SVCD playback with DShow and WinDVD4 Filters to work?

[Koepi]

Works like charme here with my test-SVCD (80 minutes ;) ) and the latest test6-filters.

Regards,
Koepi

[Chibi Jasmin]

You mean with WinDVD? Okay, then it's probably really screwed SVCDs...or does anyone have any other ideas or maybe the same problem?

I will try to make an SVCD myself and test, if I have time...

[ChristianHJW]

I do have so many MPEG2 decoder filters on my machine, i cant really tell if its the test6 filter doing the S-VCD trick for me ... but i recently installed it on my bros PC and he could play all my S-VCDs fine with avih/de_xt test6 filters, normal WMP 6.4 and the installed PowerDVD MPEG2 decoder filters ....

[kagoru]

Sorry if my question is stupid, but isn't it kind of senseless to burn an avi to a xcd since avi doesn't have ANY kind of error correction; that would mean that with every little scratch you automatically would lose a bit of video info. ain't that right? Does the error correction of ogg actually work for such a case? Anybody experienced?


TIA.

[int 21h]

Originally posted by kagoru
Sorry if my question is stupid, but isn't it kind of senseless to burn an avi to a xcd since avi doesn't have ANY kind of error correction; that would mean that with every little scratch you automatically would lose a bit of video info. ain't that right? Does the error correction of ogg actually work for such a case? Anybody experienced?


TIA.

Naughty Naughty, someone didn't read the whole thread before posting.

Yes, you must use a format that has builtin error correction (i.e. floating index, etc) in the file itself (i.e. ASF, OGM, MP4, etc). Currently the tools that are available do not backup the crucial parts of those files (because you still need a backup copy of the headers of the files), but that is soon to change.

[kagoru]

Thank you and sorry again. I just got confused because doom9 had a picture where he selected an .avi-file. In theory you could just burn any kind of data to a xcd, I suppose, even though it doesn't make sense right now.

[Chibi Jasmin]

Originally posted by Chibi Jasmin
Well, with DVD2AVI it always works, with DirectShow based players I only got one short clip to work with seeking (some other short clips and long movies failed), but it could as well be badly muxed/encoded SVCDs, so don't give to much on this, I didn't make these SVCDs...

I have WinDVD4 DS-Filters for SVCD playback. Anything else you need to know? But well, don't spend too much time with this, maybe it's just screwed SVCDs..

Can anyone confirm SVCD playback with DShow and WinDVD4 Filters to work?

I have created my own test-svcd and seeking and everything is fine now.

UPDATE: Remuxing the before non-working ones with BBMpeg made them work, too...

[kxy]

Koepi,

Are you still planning to release XCDBackupCreator since now it seems that there is a argument about this ogm extension.

[avih]

r u gonna keep encoding? now that there's an argument about the ogg extension?? ;)

it will be released soon, don't worry. let us take care of few issues before.

cheers
avi

[Koepi]

I have to agree on avih here.

We're still discussing weather or not we're doing the backups "the right way" now.
Until now we agreed on releasing first tools soon and make them "version 0.1 beta" to indicate that we still are concerned.
But if we release them, we have to code backwards-compatible.

So we have to find a consense first about our minimum.

Sorry, can't give deeper insights yet.

Best regards,
Koepi

[arty_sin]

Originally posted by avih
Hi again.

attached is the latest RIFF/XDCA filter (test6).

Changelog:
----------
21-MAY-2002: test6:
- VCD and SVCD playback support added
- filter rewrite (for portability and future support for error protection)
- improved media detection
- file name changed from asyncflt.ax to xcdsrc.ax
- registry system modified to override ms VIDEO CD Source
(which doesn't read s/vcd correctly, and may introduce conflicts.
the uninstall script put back the original registry values)

SVCD playback may be a bit skippy since high bitrate is involved, and we didin't add buffering yet.

the filter DOESN'T support (=use) the backup files now. but the next version prob will.

this is a binary only release. the sources will be included again with the next release.

any feedback is welcome (especially regarding vcd/svcd).

enjoy
avi

Finally got this mode 2 stuff working with the 'test6' filters. Still had a bit of problem. Even after putting the full path in the .bat file for regsvr32 graphedit didn't show the new filter. So I used regdrop, at http://www.addisonsw.com, which got it in there. Also had to run the .bat file to get it all working. Some folks have had problems with CDRWIN. I found that unless the 'raw mode' check box is ticked you do indeed get problems. Goldenhawk says that the CD writer firmware attempts to put in error correction unless you do this which might explain things. It also says not all writers will use 'raw mode', which might explain a few more. I use LG GCE 8160B which also overburns well (recommended out of Tom's Hardware pages). Anyway I now have a 804,619KB avi on an autostart CDROM with codecs for sound and DivX included with their installers. All this on an 80 min CD (700 MB). Great stuff.:p

[DeXT]

I'm back... wow, I guess I'll have to read a lot of posts to be up to date :)

@arty_sin: well seems there are some problems with the new install script in Win9x systems, probably due to REGSVR32.EXE not being in the default PATH (weird?). I guess I'll have to make separate install scripts for 9x ad NT class OSes :(

Please confirm me if changing the following lines from the install.bat script solves the problem (unregister it first just to be sure):

copy /y xcdsrc.ax %WINDIR%\SYSTEM32\xcdsrc.ax
regsvr32 %WINDIR%\SYSTEM32\xcdsrc.ax

for

copy /y xcdsrc.ax %WINDIR%\SYSTEM\xcdsrc.ax
%WINDIR%\SYSTEM\regsvr32.exe %WINDIR%\SYSTEM\xcdsrc.ax

Thanks for the note about the "RAW" switch in CDR-WIN, I don't need it with my burner, but I'll recommend to turn it on when burning in the readme.

@kagoru & int 21h: just to remember that AFAIK neither OGG, MP4 nor ASF have got error correction but error detection and/or error concealment. It's the same situation as with RIFF/CDXA (detection only), and the error concealment must be done by the codec/player itself. The problem with AVI is that this format is based in a chunk scheme, where each chunk has a "size" field, and getting a corrupt value here may prevent the rest of the file from being properly readed. The other formats are designed to deal with corrupt chunks without breaking the playing.

@Chibi: thanks for your nice tests with SVCD. ;)

@Nah: thanks a lot for your continued support making such a nice GUI supporting every new feature in mode2cdmaker. I may add this "-n" option in the next release so you don't have to make custom builds with every release. Seems this form2 extension is so important for you ;)

BTW if you have any suggestion regarding how subdirs could be implemented either from the command line or a list file, or any other thing, just tell me.