800 MB AVIs to 1 CD ?

[ChristianHJW]

Sorry for being late guys ... this happens if you dont check the 'General Section' in regular periods ... shame on me :( !

You may read these threads here http://www.powerdivx.com/phpbb2/viewtopic.php?t=367 and here http://forum.doom9.org/showthread.php?s=&threadid=18969 also, the subject was raised before a few times already.

About 1300 MB CDs or even DVDs : using mode 2 form 2 will always give you more storage capacity, even if your CD is bigger !! A 1300 MB CD should well be able to give you up to 1500 MB then, so all your nice 2 CD rips will fit on 1 CD then ;) ...

So, here is some more input :

1. AVI : This subject was discussed really more than one time and as a matter of fact it seems that good old AVI format is not at all prepared for this ! I was told not only the header will be problematic in case of reading errors, the whole file can be messed with a few bits missing or being wrong on some place ... so it seems you had to do a complete hack for this, and with respect to Ogg and MCF standing outside the front door this is really not worth it IMHO.

2. Ogg : We havent got any clear response on this from Ogg dev team, but it seems even Ogg has some neat CRC in the format that would allow using it for mode 2 form 2 discs. It should be, always bare in mind it was designed for stremaing also.

3. MCF : MCF was defined with Adler32 error correction and is extremely forgiving if any bytes or even blocks are missing. It will be a wise thing though to write the header itself ( although there is a safety copy of the header at the end of each file ) in normal mode and only the data itself in mode 2 form 2 ... but we are implementing this in the basic specs, as well as the possibility to split the whole file into several portions ( like 2 - 4 CDs, whatever ). Please check the MCF specs, technical section, on http://mcf.sourceforge.net for more info.

How to implement this :

Well there is a very good working implementation for S-VCDs we could use. A few things have to be done though :

- Create a program that can do CUE/BIN from a MCF file on HDD, so its ready to be burned with Nero/CDRDAO/EZ etc. ; this should respect the header specific mode burning, being well possible i was informed.

- Implement reading support into the MCF parser or another DirectShowFilter, so MCF files can be played directly from disc ! Please note that reading S-VCDs is not a basic feature of any Windows OS ( not sure about 98SE/ME ) !!! A virgin Win2k installation can not read S-VCDs directly, only after installing WinDVD ( or whatever ) the OS gets these capabilities ! This is not true for VCDs it seems, at least on my box it is working to copy/read the data directly from the CD to HDD, maybe because i have old version of PowerDVD installed, dont know.

- Create an extraction tool similar to VCDimager/VCDgear to be able to copy/read the data from disc to HDD.

It is well possible it seems, but requires some work to be done. If anybody is interested in joining the MCF dev team for this purpose, please PM me here.

[Tronic]

What we're currently going to do with MCF:

All our sectors are Mode 2 sectors with 2336 octets of user-data.

Mode 2 sector structure: (total size 2352 o)
-Sync (00 FF FF FF FF FF FF FF FF FF FF 00) (12 octets)
-Minute/Second/Frame (3 octets)
-Mode (02) (1 octet)
-User data (2336 o)

In our track,
-Sector 0: MCF-CD Header (contains X1 and X2 + some other data).
-Sectors 1 - X1: MCF (headers), with some kind of error _correction_.
-Sectors X1 - X2: MCF (data), no error correction, just detection (per-sector + per-cluster).
-Sectors X2 - END: MCF (headers), with ECC.

This would give us even more storage space than SVCD has. It seems that there is no reason to include any filesystem on the disc.

Writing stuff to a BIN-image is pretty easy. The only tricky part is calculating the ECC.

Reading stuff from CD or image, without ECC, is REALLY easy. Reading with ECC is somewhat tricky.

Todo:
-Figure out how TOC works: how to instruct players on how to read our track.
-Decide how to do that ECC.

[movmasty]

reading S-VCDs is not a basic feature of any Windows OS ( not sure about 98SE/ME ) !!! A virgin Win2k installation can not read S-VCDs directly, only after installing WinDVD ( or whatever ) the OS gets these capabilities ! This is not true for VCDs it seems
this cause windows comes with a built in mpg1 driver,
old mpg1 is free,while you have to pay for mpg2.

Would be easy to make one driver for both mpg1 and 2,
but isnt done for commercial reasons.

as avi mpg could contain almost every codec(and in fact the mpg1 format is very less simple than many think,old mpg1 drivers cant read many new mpg1)
only you cant just add a dll with the new codec,but have to rewrite all the driver,
so would be possible,i think,also to implement any mpg4 codec in a mpg1+2+4 or mpg1+4 driver,
with the advantage that it will be supported immediately from existing players and editors.

[ChristianHJW]

Originally posted by movmasty this cause windows comes with a built in mpg1 driver, old mpg1 is free,while you have to pay for mpg2. ... yes, maybe, but pls. be aware that the MPEG1 decoder itself isnt necessarily capable of reading the data from the disc directly also ... but its likely M$ built in such support, as it will allow all M$ OS to view VCDs with Mediaplayer ( does it ? ), bein main purpose for MPEG1 ..

I'd be interested to leanr if the elecard MPEG2 decoder has the ability to read/parse S-VCDs also ... time to ask Blacksun about that i guess ...

[theReal]

but its likely M$ built in such support, as it will allow all M$ OS to view VCDs with Mediaplayer Windows is able to read VCD movies as normal files, Windows has also been able to read Audio CD's as normal files for a short while - but then the audio industry pressured MS to remove the direct Audio-CD reading (and now that they have done so, noone can copy audio CD's anymore, LOL)

I guess if it was only up to MS, Windows would be able to read & copy everything directly.

[DeXT]

I see a potential problem... when Windows reads a DAT file linked to the Mode2/XA area, it appends a RIFF header at the start of the file. This header is not present in the Mode2 video track (I've tested it). Also, the file is being read in RAW mode, i.e. it contains a sector header (sync,addr, mode) as well as subheader and EDC, every 2352 bytes.

So if you burn, say, an AVI file in the second track and link it from the first ISO track, you'll end with a AVI file with an additional RIFF header attached to it and lots ot sector headers to play with.

So any software reading this should bypass this extra data. This seems OS dependent (I don't know if Linux does the same when reading such files). Perhaps this is only related to "M" flag files...

Any thoughts?

[int 21h]

The biggest point you guys are missing is that Windows reads those file systems because they are declared standards (Whitebook, Greenbook, Yellowbook, etc) with Philips for Compact Discs.

Putting anything besides Mpeg2 in the Data area of a SVCD disc violates the standard, and its very likely it won't work correctly.

[movmasty]

burning mpg in mode 2 (http://www.vcdhelp.com/faq.htm#650)

[oddball]

Elecard reads S-VCD fine. Just have to open the MPG file from the folder on the S-VCD and it reads it as normal.

[ChristianHJW]

@oddball,

thanks mate, this is very helpful info .... by chance BlackSun has very good contact with the elecard guys ... so i know wht he has to ask them tomorrow :D !!

BTW : TRonic could create first 800 MB MCF CDs recetnly ( test only ) and updated the MCF CD specs accordingly.

Now its time to fix the header specs ...

[ChristianHJW]

Discussion about this subject is going on here if anybody is interested :
http://www.powerdivx.com/phpbb2/viewtopic.php?t=367

If somebody had any input on how to make this possible using a DirectShowFilter or whatever to make direct reading from CD possible for playback we would appreciate a lot ...

[ammer]

Go to a cd-r burning site to find out more about this subject. They are more ahead on this subject usually. If you need a site, one could be sited. Post a message for me and i'll get a link to for more about over-burning.

[MaTTeR]

@ammer

Overburning isn't really the goal here. Were looking into an entirely different method of burning a format such as MCF to a standard 80min CD but at 800MB instead of the average overburn 710MB.

The focus is how to read the 800MB of data from the CD using standard Windows system calls, etc. It appears a non-M$ driver might be needed for such a beast.

[ammer]

Wouldn't you imagine cd-r "burners" have tried to burn 800mb files on 80min or even 74min cds already?...even a/v files for that matter.

[ammer]

And error corrections can become a matter also. I'd hate to watch a movie that ends 1/3 way through the movie.

[evilhomer]

Have you read any of the rest of this thread??? Don't jump in a thread on page 4 and start babbling about stuff already discussed on pages 1,2 and 3...

EDIT: sorry if that sounds harsh, but its so frustrating when I get an e-mail 'cause i've been following a thread and someone posts something addressed a week ago.

[ammer]

No, not really. Just don't want you guys reinventing the wheel and such. Unless its got steel belts and water treading treads though.

[MaTTeR]

@ammer

Please! Enough posts from you in this thread. Your totally off topic. Read the thread and then also read the MCF specs and then come back with advice.

[Hanty]

I'm getting very mixed information about this subject, and despite my private research I have been unable to find a source that really goes into this. But from what I've learned it feels like we are starting out in the wrong end.

Should we not also ask ourselves what new properties the format of the 700Mb+ stuff we want to burn will recieve by effectively changing it to such a degree that it to the computer resembles (for example) an image or mpeg2?

To me I find that a change in the files structure itself in order to achieve the 700Mb+ burnable ability is more realistic than picking apart the fat32/NTFS and messing with hardware configs. (That's my logic as a hardware guy ;) software people might totally blast me for such a notion)

[movmasty]

>change in the files structure itself

as i said, mpg only video dont need error correction, with audio they need
i guess that to burn SVCD in mode2 the audio/video syncro was done in a different way,but dunno exactly

[ChristianHJW]

Originally posted by Hanty
To me I find that a change in the files structure itself in order to achieve the 700Mb+ burnable ability is more realistic than picking apart the fat32/NTFS and messing with hardware configs. (That's my logic as a hardware guy ;) software people might totally blast me for such a notion) .... in fact, yes ! If i understand you correctly, could you please tell me how we could distribute the firmware upgrades for all CD drivers out there to make them support MCF ;) ?

The normal CD specs give us enough space on normal discs without any overburning or changing hardware specs .. al we need is to do the same as the S-VCD guys are doing, get rid of all the redundant error correction stuff, movies are no files/software.

Mode 2 Form 2 is part of the standard CD specs, every CD drive does support it, and you can easily bring 800 MB on any normal 700 MB CD without overburning.

Of course ( once again ) you cant do this with good old AVI standard because its not prepared for this, but both MCF and Ogg can do it as they have internal error correction in the format system.

We cant use the same tools as the S-VCD guys do, because these are no MPEG2 streams but MCF of Ogg streams we are writing, but we could have a close look how they were doing things .... happening right now !! ;) ...

... stay tuned, there is more to come from the 'new A/V formats' ..

@ammer : rant#3 ... please read a thread and try to understand it before replying .. you obviously missed the point completely here ! Enforcing Overburning is a very very bad thing for a new movie format, especially if you want to achieve enhanced compatibility as MCF is targeting ...

[temporance]

If Ogg and MCF have sufficient error correction to fix CDROM read errors, then they will have similar ECC overhead to M2F1 and M1. So, the extra 100Mbyte capacity available from M2F2 will be occupied by the file format's own error correction. --> no benefit.

IMHO, Ogg/MCF have error detection, not error correction. This is not needed in M2F2 as M2F2 already has built-in error detection.

Remember, the key point is to work like SVCD and audio CDs, that is to use a combination of error detection and error concealment (skipping frames/muting audio) instead of error correction.

And the AVI format is not useless for this application if handled right. It might be a good idea to start experimenting with a known file format and a new disk format rather than trying to work with two relative unknowns. :)

EDIT: Is there any opensource CD burner project that this could be hooked into?
EDIT2: Yes, see http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/cdrecord.html for burning, not sure about reading

[Hanty]

If i understand you correctly, could you please tell me how we could distribute the firmware upgrades for all CD drivers out there to make them support MCF ;) ?

By FTP, http, IRC or even mail if push comes to shove ;)

Well seriously, like I said I'm a hardware guy and I wont pretend that I fully understand the problems that face the software solution.

But if you allow me to digress and come back to ogg, what will it take to fit 800Mb of .ogg and .ogm on a CD?

[ChristianHJW]

Originally posted by Hanty
But if you allow me to digress and come back to ogg, what will it take to fit 800Mb of .ogg and .ogm on a CD? ... well, probably the best would be to talk to Tronic on IRC, openprojects , #mcf ( he is there quite often ) and ask him how he is planning implementation for MCF ... then reading the reply from the elecard developer here http://www.powerdivx.com/phpbb2/viewtopic.php?t=367 and finally talk to Tobias Waldvogel in the 'New A/V formats' forums here at Doom9 ( see my signature ) or on http://tobias.everwicked.com to see if he could modify his Ogg splitter accordingly .. this is how i would do it if i could code :D !!

[ReferenceDivx]

How will the new hardware players read divx and Mpeg4 files? Maybe they are thinking of making a new cd format. Maybe someone that has info on how divx players are going to support divx and Mpeg4 can respond. Maybe the people from Divx Networks are thinking of a new CD format?

[DeXT]

I think it's still too soon for a standard specification for MPEG-4 on DVD, latest news suggest that the DVD Forum may be thinking in dropping MPEG-4 due to its high licensing costs and adopt a different format... http://www.eetimes.com/sys/news/OEG20020329S0023

I think Whitebook is the way to go. I.e. create a ISO-Bridge CD-ROM/XA format with a first ISO M2F1 track linking to several M2F2 tracks with the actual movie data. This is already supported by Windows and all that is needed is a DirectShow filter that is able to read RAW track data and discard the RIFF header and unneeded sector stuff. MS MPEG-1 Spliter and Elecard MPEG-2 Splitter already do this (as pointed by Peter from Elecard).

[temporance]

Hello DeXT,

"ISO M2F1 track linking to several M2F2 tracks with the actual movie data."

Yes, that is what I was thinking too. But, to me it seems that Yellow Book (CDROM) has all we need without worrying about any VCD aspects of White Book (VCD).

BTW, anyone have a copy or synopsis of the colored books?? Any useful information in electronic form?

It looks like Linux would be the best platform for experimenting, but it's going to take me some time to get Linux up and running :-(

It's time to start collecting some resources - is this forum the best place to discuss? BTW, I am agnostic on file format - just fascinated by the potential in the CD technology :-)

EDIT: also does anyone have any info on the file formats bin/cue and iso?

[ChristianHJW]

Originally posted by temporance also does anyone have any info on the file formats bin/cue and iso? [/B]

@temporance :

This problem is solved already, Tronic had already been able to create CUE/BIN files for 800 MB MCF CDs AFAIK.

@DeXT, temporance :

what can i do to motivate you to join the MCF coder team :D ?

You should try to talk to Tronic, ingo and robux4 sooner or later if you're really interested in this subject and helping with giving 800 MB Cds to the community.

Best way is IRC, openprojects server, #mcf or #mediaxw . In the #powerdivx ( same server ) forum you can meet ingo by time.

Or you send mail to the MCF developers mailing list, mcf-devel at lists dot sourceforge dot net ..... a first impression of MCF is to be found here : http://mcf.sourceforge.net .

[DeXT]

Thank you very much for your invitation ChristianHJW, the only problem is, I have very little knowledge about programming, althought I've gained some good knowledge about physical CD formats and CD image formats (feel free to take a look at my small web cdirip.cjb.net).

Sorry about the confussion, when I talked about Whitebook I meant the physical format, i.e. CD-Bridge with ISO and RAW tracks. Tronic has done a good work but if I've understood well, he is just writting files as RAW tracks, creating a CD that is illegible for Windows, and that either must be extracted from a CD tool such as ISObuster for it to be viewed, or needs direct (complex) software support. I personally think this is not the best approach. What he lacks is a ISO-Bridge track which makes it all readable from within Windows (i.e. the same as VCD), and greatly simplifies things, only requiring a DirectShow filter which is able to discard RAW data from the movie file.

The good news is, this has already been done, and as Open Source: it's called VCDImager (www.vcdimager.org). With this tool you can insert RAW files inside the ISO structure, just like VCD tracks. The only problem is, since this is a VCD tool, it still wants at least one MPEG track and all the VCD stuff like PBCs etc that we just don't need. The goal would be removing all this stuff from the source code and create a simple tool which can do exactly what we want. This is the magic of Open Source :)

I've been digging at VCDImager sources but since I'm poorly skilled for this I think it will be very difficult for me to do this alone, at least in a clean way. If someone wants to help it would be great. For now I think I'll keep in contact with MCF team through IRC, just as a contributor.

Let's see if we can do something good here...

@temporance: take a look at ECMA-130 and ECMA-168 from www.ecma.ch, I think these are the only free CD-ROM standards available on the Net.

[ammer]

Theres some cd-r related links on general info there also. http://www.cdrinfo.com/

[oddball]

Some interesting info over at cdfreaks...

"How to get 750 MB on 74min/650MB CD-R"



http://www.cdfreaks.com/news2.php3?ID=3894

[DeXT]

Pretty neat... burning data as Audio tracks. The only problem is, good data extraction (the file must have built-in error correction and you need a really good CD drive).

It's cool seeing someone mentioned VCDs in that thread, too. Yesterday I just finished a tool to do that. You can burn ANY file as VCD DAT files (M2F2). It creates a CD/XA Bridge CD that can be read from within Windows. I posted it on this thread:

http://forum.doom9.org/showthread.php?s=&threadid=23384

Just a quick note, burning as RAW audio gives you a 1.2% more space than M2F2. Burning as pure Mode2 gives you only 0.3% more space. I personally think these are not worth the pain since you lack Error management with these. M2F2 is THE way to go, IMHO.

[oddball]

Well I tried the method mentioned on CDFreaks forum. I could burn it just fine. But not read it back. The reason? I am using XP. The modified CDFS.VXD is Win98 only.

[ppera2]

Try mcfcdmaker - see URL at news.

[oddball]

Yeah fathomed this already. Waiting for the on the fly filter/driver so we can read back data from the CD directly without having to convert it back 1st. :)

[ChristianHJW]

First draft of MCF-CD specs : http://mcf.sourceforge.net/mcf-cd.htm

[mustaneekeri]

In the future would it be possible to add a normal data track in which u could store player program, external sub file and autorun function, into this kind of disk?

[Tronic]

Ingo's version of MCF-CD *requires* that you have ISO-9660 track on the disc.

Tronic's version allows having ISO-9660 or any other filesystem on the disc.

Btw, the MCF-CD page is updated.