Conversion Tool from DivX3/4 to MPEG4 ISO, both based on Windows and Linux - paid job

[ChristianHJW]

Hi guys,

i am in contact with Tan Ju Hian, the assistant sales Director of neuston ( http://neuston.com ). Similar to KiSS with their DP450 unit they are about to launch a DivX compatible hardware standalone player ( http://neuston.com/dvx_1201.asp and http://neuston.com/dvx_1307.asp ) and i am talking to them about the potential problems to be exptected with AVI as container. They are investigating if alternative containers such as MP4, OGM or maybe even matroska could be used as AVI replacement and therefore want to include a conversion software tool to help customers with AVIs that play without sync, as they fear the impact on their service departments if playback problems with AVI files will occur ( and these are very likely to happen as you know ).

I am trying to convince them that such a tool should be based on the GPL and sources be distributed with the CD, as it could use many existing code, making the job a lot easier.

Tan Ju said in his last email to me :


Your ideas and concepts seem like good ones and make a lot of sense. The inclusion of a 'conversion tool' being bundled with the player seems like almost a necessity as unfortunately, there is no hardware available whereby it can support DivX 4 and below on a stand alone piece of equipment. Software that can convert any .avi to work with the DVX-1201.

I would be more than appreciative with any more ideas and people I can get in touch with in regards to the conversion tool. I admire your passion and knowledge in the field, especially since it is a hobby of yours!

Kind regards,
Yu Hian

Tan Yu Hian
Asst. Director, Global Sales
neuston corporation
41 Science Park #04-07/7A
The Gemini Science Park II
Singapore 117610
Tel : (+65) 68733312
Fax : (+65) 68733304
Mobile : (+65) 98750846
E-mail : yuey@neuston.com



Main job was :

- parse AVI files
- read DivX3/Angelpotion/SMR/MPEG4V3 and convert to MPEG4ISO without reencoding !!
- detect VBR MP3 and/or AC3 audio
- send out warning signals about VBR audio not being AVI specs compliant, also potential AC3 audio probs ( cut in frames by nandub )
- convert file into a new AVI that can be read by player in sync in any case, i.e. is compatible with players AVI parser
- probably convert file into another container supported by Neuston player ( much later -- 2nd step ), preferably MP4

If anybody feels like he could take such a job, drop me a mail to christianhjw at users dot sourceforge dot net .

Best regards

Christian HJ Wiesner

[Ewi]

I posted a bug regarding CBR muxing in VirtualDubMod and in contact with the fixing developer(Stream9) I made the suggestion to include code to identify VBR MP3 files so they are not CBR-muxed without any comment (as VdubMod is doing it at the moment). He said this would be a good idea and I should post it as a feature request in CVS RFE, so perhaps someone is really going to implement this and you can use it in your tool...

Otherwise this tool is a good idea...

[-h]

I can't say with certainty but I believe the only potential obstacle to a MSMPEG4v2/MSMPEG4v3/WMV1 -> MPEG4 convertor is the storage of the DC component for intra blocks (MS may be more precise than MPEG4 allows - 9 bits versus 8). Everything else I found has a suitable lossless replacement in the MPEG4 spec, and everything that's needed is in ffmpeg already.

I had plans to work on "something" over this month, perhaps I should finally do this convertor instead.

You knew this would happen to things I work on didn't you Nic :)

-h

[ChristianHJW]

Originally posted by -h I can't say with certainty but I believe the only potential obstacle to a MSMPEG4v2/MSMPEG4v3/WMV1 -> MPEG4 convertor is the storage of the DC component for intra blocks (MS may be more precise than MPEG4 allows - 9 bits versus 8). Everything else I found has a suitable lossless replacement in the MPEG4 spec, and everything that's needed is in ffmpeg already.
mate, pls. gimme a shout if i should make contact once you know how to do it.
I had plans to work on "something" over this month, perhaps I should finally do this convertor instead.
Dont ask me what such a tool would be worth for the, we havent talked about numbers yet ...
You knew this would happen to things I work on didn't you Nic :)
-h
Lazy Nic ? First time the guy would have had a clue :P ...

[-h]

mate, pls. gimme a shout if i should make contact once you know how to do it.

Oh I'll do it for free, I meant to do it a while ago as it's just a cut-n-paste job with ffmpeg's existing codebase. Thus it will be LGPL'ed regardless.

-h

[sam_b]

Oh I'll do it for free

Bravo -h. But if they are offering... they are a profit-making company and seem willing to cope with a copyleft license. Even knowing that it's primarily a code cut-and-paste, it still takes some skill to do, and I'm sure they'll appreciate that.

[-h]

Hm. Well, things are looking up. Everything seems to be compatible (DC scale tables are the same for everything < quant 25 {yay}, but hopefully no one will be using quants that high {or won't miss a tiny bit of detail with such high quantization anyway}). That's for MSMPEG4v3 (DivX) anyway, MSMPEG4v2 (SMR?) and WMV1 appear to have different DC scale tables and are unlikely to survive conversion to MPEG-4 without significant artifacts (re-quantization of DC value). Also MSMPEG4v2 has halfpel interpolation rounding behaviour that is not compatible with the MPEG-4 spec (subsequent frames must alternate rounding between 0 and 1, MSMPEG4v2 is always 0).

I may go on a binge tonight and put something together, but ffmpeg is a right pain to get intimate with, all due respect.

-h

[Teegedeck]

-h, if you feel that making such a proggie wouldn't mean pressure for you, please take their money and even if only for building a career (a paid programming job still looks better on an application IMHO)! We all know your skill, but your future employer should feel honoured when you apply for a job ('oh, the guy who made that MPEG4-converter that comes with Neuston-DVD players!') :)

If Christian and you can perhaps talk them into making it a GPL'd piece of software we'd be very happy, too, of course.

[tangent]

Originally posted by ChristianHJW
Tan Yu Hian
Asst. Director, Global Sales
neuston corporation
41 Science Park #04-07/7A
The Gemini Science Park II
Singapore 117610
Tel : (+65) 68733312
Fax : (+65) 68733304
Mobile : (+65) 98750846
E-mail : yuey@neuston.com

Hmmm.. Singapore... I wonder if they're hiring...

[-h]

Well I just made a list of everything that needs to be changed to allow lossless conversion. There's no way I can decouple the necessary parts from ffmpeg any time soon, so it'll just be a command-line tool that accepts a DivX avi file and outputs an MPEG-4 one with CBR MP3 audio.

Here's the result of 2 hours of searching and yawning over ffmpeg. I'll see whether implementing these changes is as simple as it sounds tomorrow :)

decoding:

h263dec.c
- h263_decode_frame()
dump frame info before decode_slice()
- decode_slice()
dump block info after "ret= s->decode_mb(s, s->block);"
skip "ff_draw_horiz_band(s);"

mpegvideo.c
- MPV_decode_mb()
add skip check in "if (!s->mb_intra) {" above "/* motion handling */"

encoding:

mpegvideo.c
- MPV_frame_end()
don't draw edges
- MPV_encode_picture()
set frame type around "if(avctx->force_type){"
- encode_picture()
set vectors/type in "/* compute motion vector & mb_type and store in context */"
set additional vectors with set_p_mv_tables()
set fcode
skip I_VOP complexity stuff around "if(!s->fixed_qscale){"
skip scene change check at "if(s->scene_change_score > 0 && s->pict_type == P_TYPE){"
skip best_fcode()/fix_long_p_mvs()
set rounding, qscale and frame_qscale before "switch(s->out_format) {"
- encode_mb()
skip all code from "if (s->mb_intra) {" until "/* huffman encode */"
find last non zero coefficient for block_last_index
- MPV_decode_mb()
add skip check in "if (!s->mb_intra) {" above "/* motion handling */"


-h

[-h]

-h, if you feel that making such a proggie wouldn't mean pressure for you, please take their money and even if only for building a career (a paid programming job still looks better on an application IMHO)!

Well, I won't know whether it's possible until I try it, and once I've tried it the work will be done! :)

That doesn't mean I have to release it into the wild. Perhaps once I've decoupled it from the rest of ffmpeg I'll ask them for a "service charge". Of course they could just refuse, since I'd release it with source anyway for everyone here to use :)

-h

[temporance]

Also MSMPEG4v2 has halfpel interpolation rounding behaviour that is not compatible with the MPEG-4 spec (subsequent frames must alternate rounding between 0 and 1, MSMPEG4v2 is always 0).
MPEG-4 does not require alternate rounding (otherwise, why would rounding type need to be specified explicitly for each P-VOP?). I think alternate rounding is recommended in an informative part of the spec.

I looked into this sort of conversion a while back and decided it wasn't possible to do for all DivX 3.11 content. From memory there are some issues with edge padding that would cause prediction drift. There's also an issue with filesize due to different VLC tables and prediction algos. If I have a 650MB DivX 3.11 file, can I convert this to a 650MB MPEG-4 compliant file?? Probably not - it would finish being 700MB or 620MB or perhaps even 800MB!

Good luck to -h and others attempting the challenge! It's interesting to see a company here offering $$$ and people offering to do their work for nothing! At least bill them for support :)

[-h]

I think alternate rounding is recommended in an informative part of the spec.

Very true, I can't actually find the recommendation anymore.

I looked into this sort of conversion a while back and decided it wasn't possible to do for all DivX 3.11 content. From memory there are some issues with edge padding that would cause prediction drift.

I haven't seen anything in libavcodec that indicates a difference in edging between MS and MPEG, but if there is one it'll kill this conversion pretty fast. I know suxen_drol considered writing a similar convertor a while back, and didn't mention anything that would prevent him from doing so (apart from time, obviously).

There's also an issue with filesize due to different VLC tables and prediction algos. If I have a 650MB DivX 3.11 file, can I convert this to a 650MB MPEG-4 compliant file?? Probably not - it would finish being 700MB or 620MB or perhaps even 800MB!

Yes. I should think it will increase file size by around 10%, if not more.

-h

[temporance]

-h,

Looking at the code, I think there is another potential problem: iDCT accuracy. The MS encoder uses a fast but not accurate iDCT.

Playback using a MPEG-4 decoder's compliant iDCT will cause drift artifacts to accumulate.

Perhaps it's possible to correct these while doing the conversion, but it would be quite a headache. Or hopefully they are not too bad.

[ChristianHJW]

Originally posted by temporance If I have a 650MB DivX 3.11 file, can I convert this to a 650MB MPEG-4 compliant file?? Probably not - it would finish being 700MB or 620MB or perhaps even 800MB!

This could be a problem for sure, but XCD and mode2 form2 could certainly help here i guess, for either OGM or matroska, giving us 100 MB extra on every disc. In fact, whats looking like a disadvantage at first sight could turn out to be a major argument for any hardware company to adapt either OGM, MCF ( mf ;) ) or matroska in their players, as all three have CRC32 as EDC coming with them while AVI has not, making it not very well suited for mode2 form2 burning. Sure, we could use MP4 via XCD also, but this would deinitely be outside MP4 specs, so why at all use a proprietary, IMHO inferior container if its main advantage, being to procude files compatible with an industry standard, is not given anymore ??

Forgive me my arrogance, but its somehow nice that some of the projects i have ben contributing to over the last 2 years may get a REAL chance to become widely adopted. Its all in our hands now guys. Thanks so much -h !!!

I will point Tan Yu Hian to this thread here. I am convinced he will be impressed by the professionalism of the members of this forum here, and the immediate reaction to start working on him. As -h was pointing out he doesnt want any payment but plans to release the code as GPL or even LGPL ( i would NOT recommend LGPL for various reasons, but accept his decision here of course ) i will make the following recommendation to Neuston Corp. :

- the tool has to be released as open source under a GPL license, as otherwise major code rewrites were necessary instead of being able to simply use existing and tested code.

- a decent 'consultation' payment for -h , to be paid on a monthly basis, in return -h will work on the project as administrator until every existing DivX3 AVI can be converted into a OGM or whatever file that plays in perfect sync on the Neuston player

- a more than decent one time donation to this place, to FFMPEG project and maybe to corecodec.org ( BlackSuns and Betaboys upcoming open source community, specific for audio and video compression ), where the project should be hosted ( i will beg -h not to use sourceforge ;) )

Regards

Christian

[NuclearFusi0n]

Say, -h, you might be able to get that laptop you wanted for your train rides now. :)
...and what container format is the neuston player going to use?

[temporance]

Originally posted by ChristianHJW
In fact, whats looking like a disadvantage at first sight could turn out to be a major argument for any hardware company to adapt either OGM, MCF ( mf ;) ) or matroska in their players, as all three have CRC32 as EDC coming with them while AVI has not, making it not very well suited for mode2 form2 burning. Sure, we could use MP4 via XCD also, but this would deinitely be outside MP4 specs, so why at all use a proprietary, IMHO inferior container if its main advantage, being to procude files compatible with an industry standard, is not given anymore ??
ChristianHJW,

Mode 2 Form 2 already has error detection so, in theory at least, the CRC32 in OGM and matroska is redundant. Personally I believe the future lies with mp4 - with MovieDataAtom(s) written as Mode 2 Form 2 and everything else as Mode 2 Form 1. Alternatively a new type of data reference URL/URN could be used to refer to data that is present in Mode 2 Form 2 on the same disk as the .mp4.

EDIT: It is feasible that a file can have more than one format. For example, if we were clever enough, we could have a large matrovska file in Mode 2 Form 2 sectors (most of the CD) and a stub .mp4 file in the ordinary CDROM filesystem. The .mp4 would contain only a MovieAtom and would refer to the raw samplw data in the Mode 2 Form 2 segments. Ergo: both formats. Everyone's happy (with a few MB more overhead) Perhaps we should continue this discussion in another thread if there's more to be said.

There is enough weight behind mp4 to ensure that it will become a widespread standard for stand-alone players. It would be nice to see mp4/CDROM done well.

I agree with you that it is very nice to see the Open Source / DivX scene getting some recognition. Great things have happened in the last year or so.

[-h]

Looking at the code, I think there is another potential problem: iDCT accuracy. The MS encoder uses a fast but not accurate iDCT.

Where do you see that? I am seeing identical iDCT inits for all MPEG-based decoders (DCT_common_init()). You're right though, this is the most likely source of problems and some kind of artificial fix may be needed.

There will probably be errors regardless, with existing MPEG4 content, due to the excessive distances people have specified between I-frames.

-h

[-h]

As -h was pointing out he doesnt want any payment but plans to release the code as GPL or even LGPL ( i would NOT recommend LGPL for various reasons, but accept his decision here of course ) i will make the following recommendation to Neuston Corp. :

LGPL'ed code can be relicensed to GPL, and I will license the modifications I make as GPL anyway which will automatically remove LGPL'ed freedoms from redistribution. Though I would like to see the code added to ffmpeg itself some day, which requires LGPL.

- a decent 'consultation' payment for -h , to be paid on a monthly basis, in return -h will work on the project as administrator until every existing DivX3 AVI can be converted into a OGM or whatever file that plays in perfect sync on the Neuston player

Ouch. Sounds nice, but debugging hundreds of AVI sync and mismatch issues? :( I'll see what I want to commit to once I have an idea as to how problematic the conversion will be.

- a more than decent one time donation to this place, to FFMPEG project and maybe to corecodec.org ( BlackSuns and Betaboys upcoming open source community, specific for audio and video compression ), where the project should be hosted ( i will beg -h not to use sourceforge ;) )

My initial thought was to pop a zip file onto geocities :)

Certainly a donation should be made to ffmpeg if the tool works, and if they still have money left to give away I wouldn't object to them doing so.

-h

[Doom9]

I checked out the features of the two mentioned players and one point struck me as being rather important: GMC and QPel are not supported. This has been an issue for both the XCard and the Kiss player. Especially DivX5 users often use one or both of these features and once XviD 1.0 becomes a reality there'll be even more people using this features.. and all of them can't use any currently available hardware player (unless you count an Xbox with modchip and xmp or the PS2 equivalent as hardware players).

[ChristianHJW]

Originally posted by Doom9 I checked out the features of the two mentioned players and one point struck me as being rather important: GMC and QPel are not supported.

All the hardware products you were listing here are using the same decoder chip, the EM 8500 from SIGMA, and this cant play material with GMC and Q-PEL .... no idea if they can get this done with a firmware upgrade, probably not, we all know about the implications on CPU usage when using any of the 2 .... very likely the EM 8500 doesnt have enough processing power to make it.

[ChristianHJW]

Originally posted by -h :
ChrisHJW said : As -h was pointing out he doesnt want any payment but plans to release the code as GPL or even LGPL ( i would NOT recommend LGPL for various reasons, but accept his decision here of course ) i will make the following recommendation to Neuston Corp. :
LGPL'ed code can be relicensed to GPL, and I will license the modifications I make as GPL anyway which will automatically remove LGPL'ed freedoms from redistribution. Though I would like to see the code added to ffmpeg itself some day, which requires LGPL.

GPL was much better in this case, otherwise many closed source software developers would simply steal your code and offer a similar functionality in their ( unfree ) tools.

- a decent 'consultation' payment for -h , to be paid on a monthly basis, in return -h will work on the project as administrator until every existing DivX3 AVI can be converted into a OGM or whatever file that plays in perfect sync on the Neuston player
Ouch. Sounds nice, but debugging hundreds of AVI sync and mismatch issues? :( I'll see what I want to commit to once I have an idea as to how problematic the conversion will be.

Mate, i certainly dont want to push you into something you dont like to do ;) .... but if Neuston was prepared to pay some money they would probably expect anything they could really give away to their users for sure ...

- a more than decent one time donation to this place, to FFMPEG project and maybe to corecodec.org ( BlackSuns and Betaboys upcoming open source community, specific for audio and video compression ), where the project should be hosted ( i will beg -h not to use sourceforge ;) )
My initial thought was to pop a zip file onto geocities :)

NOOOOO !! Lets find a decent name and found a real open source project on http://corecodec.org !! Please !! I will assist as project admin and handle it all, you do the coding ;) ....

Certainly a donation should be made to ffmpeg if the tool works, and if they still have money left to give away I wouldn't object to them doing so.

Agreed !

[ChristianHJW]

All the talking above should not deduct us from the technical aspects that are to clarify here :

How do you think is the job done best :

- a VfW conversion plugin for Vdub, accepting DivX3 and outputting XviD ? Possible ?

- an AVISynth filter ?

- a Dshow filter ? ( best way IMHO )

- an exe parsing AVI, doing the conversion and outputting AVI ?

Thanks for clarifying ....

[-h]

GPL was much better in this case, otherwise many closed source software developers would simply steal your code and offer a similar functionality in their ( unfree ) tools.

I wouldn't mind that, it would mean more people developing around libavformat/libavcodec which is good for everyone :)

How do you think is the job done best :

Certainly a standalone exe that reads a DivX movie, attempts a lossless conversion of the video, falls back to a traditional re-encode to MPEG-4 at a given bitrate, converts the audio to CBR MP3 if it isn't already, and spits out an AVI file. Fortunately ffmpeg can already do all this.

Just looking at mplayer's demux code, the MP4 format appears to be very simple if it's just video/audio going in, so it probably won't be long before ffmpeg can output MP4 streams. I have no idea how the OGM format works, or Ogg streams in general, but since ffmpeg already supports Ogg Vorbis it wouldn't be too hard to output OGM either.

-h

[sh0dan]

Just out of curiosity - is there no way the VBR mp3 could be maintained? Us folks hanging out at hydrogenaudio are already shivering at yet another transcode. ;)

Isn't this just an AVI limitation?

[mf]

Christian asked the people of #XviD for a name for the converter, and I came up with one.


DivX 3.11 To DivX 4.x
3 11 2 4


Which makes: 31124

Now, read that as eleet (31337) text. That makes Ellza. Doesn't sound bad, and isn't hard to remember either.

So, what do you think ?

[trbarry]

Now, read that as eleet (31337) text. That makes Ellza. Doesn't sound bad, and isn't hard to remember either.

Isn't Eliza the name of the old AI Turing Test wannabee psychiatrist simulator? I would hate to start encoding with the wrong one and have it start cross examining me about my feelings for my parents or my sexual fantasies or something.

But I always have to have 1337 text explained to me very slowly anyway. Clever idea though. ;)

- Tom

[-h]

Just out of curiosity - is there no way the VBR mp3 could be maintained? Us folks hanging out at hydrogenaudio are already shivering at yet another transcode. ;)

It is an AVI limitation. I guess the thing supports the MP4 format too, but I don't know if they'll allow VBR MP3 inside the container. It's a moot point at the moment, since there's no MP4 muxer in libavformat yet (and I'm not keen on writing one just now).

I'm just starting to add code to ffmpeg, hopefully something will come from it in an hour or two.

-h

[ChristianHJW]

Originally posted by sh0dan Just out of curiosity - is there no way the VBR mp3 could be maintained? Us folks hanging out at hydrogenaudio are already shivering at yet another transcode. ;)
Isn't this just an AVI limitation?

Originally posted by -h

How do you think is the job done best :

Certainly a standalone exe that reads a DivX movie, attempts a lossless conversion of the video, falls back to a traditional re-encode to MPEG-4 at a given bitrate, converts the audio to CBR MP3 if it isn't already, and spits out an AVI file. Fortunately ffmpeg can already do all this.

If output format was OGM, matroska or MP4 the MP3 could be VBR, no problem. In case of AVI, well, KiSS claim the DP 450 can play VBR MP3 in AVI correctly, but i doubt this is valid for all different muxing modes out there. Maybe it was possible to maintain if it is made certain that some rules ar enforced, so that the hardware AVI parser can keep sync. But as AVI will run into probs when being used from mode 2 form 2 discs it maybe shouldnt be AVI output at all, and then i can also see no reason to trancode MP3 to CBR !

GPL was much better in this case, otherwise many closed source software developers would simply steal your code and offer a similar functionality in their ( unfree ) tools.

I wouldn't mind that, it would mean more people developing around libavformat/libavcodec which is good for everyone :)

I knew you would see it that way :D ....

Just looking at mplayer's demux code, the MP4 format appears to be very simple if it's just video/audio going in, so it probably won't be long before ffmpeg can output MP4 streams. I have no idea how the OGM format works, or Ogg streams in general, but since ffmpeg already supports Ogg Vorbis it wouldn't be too hard to output OGM either.
-h

Now, as OGM joined Xiph, we can maybe expect a working lib and some nice specs .. we`ll see ;) ...

[-h]

If output format was OGM, matroska or MP4 the MP3 could be VBR, no problem. In case of AVI, well, KiSS claim the DP 450 can play VBR MP3 in AVI correctly, but i doubt this is valid for all different muxing modes out there. Maybe it was possible to maintain if it is made certain that some rules ar enforced, so that the hardware AVI parser can keep sync. But as AVI will run into probs when being used from mode 2 form 2 discs it maybe shouldnt be AVI output at all, and then i can also see no reason to trancode MP3 to CBR !

I don't know how it treats MP3 audio in MP4 streams, I'm guessing the same way MPEG4IP writes it? If it says it supports VBR MP3 in AVI then great, though I'm not sure how well ffmpeg will mux it back in.

Now, as OGM joined Xiph, we can maybe expect a working lib and some nice specs .. we`ll see ;) ...

Hopefully. Hacking this conversion in is starting to suck, at every point (even allocating an empty picture) there are rings to jump through to prevent anything from breaking. I'll work through tonight though, I start work this weekend and don't want to have it lingering then.

-h

[Nic]

You knew this would happen to things I work on didn't you Nic
;) I started looking at it too, ffmpeg isn't pretty. Reading the stream in looked ok, but reordering & writing a correct bitstream looked a nightmare. Good luck mate :)

Lazy Nic ? First time the guy would have had a clue :P ...
WTF you mean by that Chris?

-Nic

[Atamido]

Originally posted by -h
It is an AVI limitation. I guess the thing supports the MP4 format too, but I don't know if they'll allow VBR MP3 inside the container. It's a moot point at the moment, since there's no MP4 muxer in libavformat yet (and I'm not keen on writing one just now).

I think it most depends on what their timeframe is. Do they need it in the next 2 weeks? Or, do they need it in the next 2 months? If they don't need it for 2 months, the Matroska parser should be done and tested by then.

[ChristianHJW]

Originally posted by Nic quote:
--------------------------------------------------------------------------------
Lazy Nic ? First time the guy would have had a clue :P ...
--------------------------------------------------------------------------------

WTF you mean by that Chris?

-Nic

Sorry mate, was ment to be a joke, but obviously i failed to use the right words. You know how much i admire your work, so just ignore it please .... :(

Apologies to all about the OT

[-h]

;) I started looking at it too, ffmpeg isn't pretty. Reading the stream in looked ok, but reordering & writing a correct bitstream looked a nightmare. Good luck mate :)

All I've managed to do is to enlarge the picture buffer that the frame would normally be decoded to, and write block modes / vectors / coeffs to it instead, which are then read by the compressor bypassing ME/DCT/quant/dequant/iDCT. It's hardly clean (or safe), but the structure of libavcodec didn't leave many alternatives.

If this works, I want to strip everything out and make a clean lib from scratch parts - it should really be encoding each macroblock as it's being decoded, none of this coeff memcpy'ing and rle'ing. But that will take a while longer.

-h

[mpeg4everybody]

Originally posted by Teegedeck
-h, if you feel that making such a proggie wouldn't mean pressure for you, please take their money [...]

Thats when you take the money that you begin feeling the pressure, usually to get the money you've to sign a contract, and then you have to report to them etc... Being a contractor is sometime even more of a headache than being an employee...

Of course you can still break the contract, not do the job (and of course not the money) but that's not very ...professional....

[ChristianHJW]

@ -h: any news :D ??

@ all : NO replies from Neuston when i was pointing them here. Seems they were just polite, and not more ....

[jcsston]

Would this tool be able to convert MS MPEG-4v1?

[-h]

Status: nearly done, debugging is a pain though. What is the easiest way to do a step-through with something that's only compilable via gcc?

MSMPEG4v1 uses a different set of values to scale the DC component, thus it could not be losslessly converted to MPEG-4. The same goes for MSMPEG4v2.

There is perhaps a workaround - DC components are only scaled for intra blocks, and to get around the scaling problem you could decrease the quantizer for any frame that includes an intra block such that lossless DC storage is possible again. File sizes would increase immensely, perhaps 80 to 100% bigger. I don't think I'll ever consider implementing something like that though.

-h

[DaveEL]

Originally posted by -h
Status: nearly done, debugging is a pain though. What is the easiest way to do a step-through with something that's only compilable via gcc?


Try insight (comes bundled with cygwin or from http://sources.redhat.com/insight/) i have not used it much but it seemed quite good.

DaveEL

[ChristianHJW]

Originally posted by -h Status: nearly done, debugging is a pain though. What is the easiest way to do a step-through with something that's only compilable via gcc?

You're a wizzard -h !! :D :D !!

MSMPEG4v1 uses a different set of values to scale the DC component, thus it could not be losslessly converted to MPEG-4. The same goes for MSMPEG4v2.
There is perhaps a workaround - DC components are only scaled for intra blocks, and to get around the scaling problem you could decrease the quantizer for any frame that includes an intra block such that lossless DC storage is possible again. File sizes would increase immensely, perhaps 80 to 100% bigger. I don't think I'll ever consider implementing something like that though.

Support for M$ MPEG4 V1/2 is certainly not even to be found on the end of the list of priorities for this project ;) ... would have been nice, sure, but not necessary at all IMHO ...

[temporance]

and now your next mission...

lossless conversion of wmv9 to MPEG-4 ;)

[-h]

Try insight (comes bundled with cygwin or from http://sources.redhat.com/insight/) i have not used it much but it seemed quite good.

Good form. It's no VC6, but it does the job I need it to.

Those not eager to download all of cygwin can get a win32 build of insight from here (ftp://ftp.kiarchive.ru/pub/.1/windows/programmer/insight-5.1-win32.zip). You just open the _g build of whatever you made - in ffmpeg's case, I just opened ffmpeg_g.exe.

-h

[DaveEL]

Originally posted by -h
Try insight (comes bundled with cygwin or from http://sources.redhat.com/insight/) i have not used it much but it seemed quite good.

Good form. It's no VC6, but it does the job I need it to.



Glad to be of some help. Can't wait to test this utility out. Any idea on file size differnce between the source divx3 and the output mpeg-4 yet?

DaveEL

[ChristianHJW]

-h !!!

lol .... any updates on debugging yet ? Wonna get my dirty hands on the baby :D :D ..

[-h]

:)

-h

[NuclearFusi0n]

Originally posted by -h
:)

-h
:confused: :confused:

:o

[Lobuz]

:eek:

Lobuz

[NuclearFusi0n]

:stupid:









:p

[fluss]

Have you loooked in to what the HW chips likes best in encoding ?
I believe that it likes an I frame a least every 24 frames for PAL.

Hoping for an almost perfect solution :-)
I have the Xcard and can be an alfa tester if needed :-).

Yours ...

[bond]

any news, any possible releases?

[gldblade]

Well, I don't have any DivX 3.11 rips (actually, I don't have any, I'm fully legit), but I'm sure someone out there is wondering: this tool will be free, right?

:)

*Edit*
Well, relatively legit. I do decrypt DVDs, so technically I've broken the law.

[-h]

Yeah it'll be free. My target finish date is tomorrow or the day after, since it's bugging me. I had to install a bunch of extra software to debug it, and I don't like it being on my computer.

-h

[dan_vegas]

Any chance I can beta test this ?
I own a Kiss DP-450 and would love the ability to convert my old rips from Divx 3.

[ChristianHJW]

Before the Beta there comes always an alpha first ;) ...

This proggie seems to bug -h A LOT !! I hope he doesnt loose interest in making it :( ...

[dan_vegas]

I dont mind if I have to Delta test it, I just want to get my hands on it :sly:

[-h]

Nah it doesn't bug me, I just don't have much free time anymore :(

-h

[dan_vegas]

I'll come round and do your laundry for you if it makes this tool available any quicker !!

:p

[mpeg4everybody]

Originally posted by -h
Nah it doesn't bug me, I just don't have much free time anymore :(

-h

Just put your source code somewhere, i'm sure someone with free time will take it over.

[NuclearFusi0n]

Originally posted by mpeg4everybody
Just put your source code somewhere, i'm sure someone with free time will take it over.
-h might be worried about somebody using the code to sell to the original company for this "paid job"

[-h]

Eh? I'm not selling code to anyone, it's going to be released freely.

It's actually not that much code, it's just debugging the traps in libavcodec that's painful.

-h

[PowerMacG4]

Yeah, it can be a pain to play with ffmpeg.

He can't not release the source and then get paid for his work, since he is using code written under the GPL (or is it LGPL) license.

[easyfab]

I've discovered this project today and i'm very surprised.
To "convert" divx3->mpeg4 without reencoding? whao, would be nice :)
Few questions:
-Is the conversion just an index modification or more ?
-Will the quality through the mpeg4 decoder be the same as the more specific divx3 decoding ?
-Will the transcoding file have ~the same size as the original ?

I wish you good luck for this useful tool :)
Thanks again -h

[-h]

Quality will be identical, output size will be (perhaps significantly) larger, and the modification involves decoding all motion vectors, block modes and dct coefficients, then encoding them into MPEG-4 format.

-h

[easyfab]

Thanks -h for answering.

Finally the conversion is not so easy/direct that i meant.

[dattrax]

-h

What are you doing about VOP edges? If I remember correctly the make_edge() is different between 3.11 and 4/5/xvid. Also aren't there some chroma rounding differences of motion vectors?

Jim

[dan_vegas]

I dont mind an increase in file size so much.
I have been re-encoding my 1 disk(700MB) divx3 rips to 2 disk divx5 so I can keep most of the original quality.

Will it take much time to convert from divx3 to 5 using this method?

[ChristianHJW]

Originally posted by dattrax
What are you doing about VOP edges? If I remember correctly the make_edge() is different between 3.11 and 4/5/xvid. Also aren't there some chroma rounding differences of motion vectors?

- h,

with more experts entering the scene on this issue, wouldnt it be worth considering to release the sources you have now, so some people could make suggestions ? Of course, only sources, no binaries yet ...

Christian

[dan_vegas]

Agreed, you know what they say, two heads being better than one and all that.

[dan_vegas]

has anyone heard any news on this from -h yet ?

[ChristianHJW]

Dont put pressure on him .... lets try to be patient :D

I am convinced he wants this tool to be out as badly as we want it ;) ...

[Acaila]

ChristianHJW wrote:
with more experts entering the scene on this issue, wouldnt it be worth considering to release the sources you have now, so some people could make suggestions ? Of course, only sources, no binaries yet ...
dan_vegas wrote:
Agreed, you know what they say, two heads being better than one and all that.
If I were a software developer I would probably be offended by those remarks. It's like being told, "you're not good enough, you're not fast enough, let someone else take over what you started."

Of course you two are correct that more people working on a project can increase the quality and speed of development, but it still feels like coming short when you're being told this.

In short, let -h work on it how/when he wants to and stop bugging him about it.

Sorry if I stepped on anyone's toes here...

[elcabesa]

hy
maybe i'm wrong, but looking this situation in a FREESOFTWARE way, christianHJW is right, if there are other people that can help why not make them able to help? it's my point of view and it's not perfect ,not right not wrong, only my point of view =)

to -h
please make it the best=)

[ChristianHJW]

Originally posted by Acaila If I were a software developer I would probably be offended by those remarks. It's like being told, "you're not good enough, you're not fast enough, let someone else take over what you started."

Remember Acaila, -h is from 'down under' ... those guys are quite robust, i'd be surprised if he would feel offended from my suggestion to release what he has got right now ... ;)

[easyfab]

I agree with Acaila, let -h have the time to do his tool and let him have fun to do that.
It's cool to share with other but it's more exciting to succeed a hard job alone.

Have fun -h :) and give us a good tool

[dan_vegas]

Originally posted by dan_vegas
has anyone heard any news on this from -h yet ?

Lets stop this already, I only asked if anyone had heard any news.

You are all right in one way or another.......except easyfab! :p

[cinghio]

Hi -h and all other guys,

Would it be feasible to find an optimized algorithm for a direct transcode MPEG2 => MPEG4 without any resize/filter/crop (i.e. NULL transformation)?
In this case iDCT and then DCT stages are, obviously, to be avoided, but is there any other process that could also be avoided?
Quantization can be bypassed?

It would be very useful for storage of DVB recorded material (varios resolutions, form factor needed).

Is the approach feasible and convenient?
Or is there any unavoidable obstacle?

Thanks for your attention

[PowerMacG4]

You cannot transcode MPEG-2 to MPEG-4.

[mpeg4everybody]

Originally posted by PowerMacG4
You cannot transcode MPEG-2 to MPEG-4.

You may not be abble to do a completely lossless transcode, but you can certainly increase the speed by using information from the mpeg2 stream to encode.

For example, you could use the motion vectors used in the mpeg2 encoding to accelerate the motion estimation steps. I'm also pretty confident you can find enough information in the mpeg2 stream to enhance the bitrate control mechanism...

Off course it may not be optimal, but i think it could potentialy greatly speed up the encoding.

[sysKin]

Originally posted by mpeg4everybody
For example, you could use the motion vectors used in the mpeg2 encoding to accelerate the motion estimation steps. I'm also pretty confident you can find enough information in the mpeg2 stream to enhance the bitrate control mechanism... No this wouldn't work - you would have to use the same P/B decisions as in mpeg2, and this would look bad in mpeg4.

[jamieuk]

good luck with the work you are doing on this tool -h, this tool will meen a lot to alot of people on the net, cant thank you anuff 4 the work you are putting in 2 this.

will donate some money to you if you set up a pay method when you have finished, as i think every 1 else should

[dan_vegas]

bump

[ChristianHJW]

Hey -h !!

I bet a bottle of finest Scotch ( Macallan 18 yrs. , Dalwinnie, whatever ) that the matroska team will release their DShow parser and matroskadub earlier into public beta status than you can do with your tool .... accept the bet ??? :D :D ...

[NuclearFusi0n]

Originally posted by ChristianHJW
Hey -h !!

I bet a bottle of finest Scotch ( Macallan 18 yrs. , Dalwinnie, whatever ) that the matroska team will release their DShow parser and matroskadub earlier into public beta status than you can do with your tool .... accept the bet ??? :D :D ...

omg :D

[dan_vegas]

FAO ChristianHJW, what will matroskadub do please explain ?
I need to be able to convert my old divx 3.11 movies to divx 5/xvid so I can watch them on my hardware Kiss DVD player, will matroskadub be able to do this ?

If so when will it be released ?

[bond]

seems like -h disappeared!?

[zulu]

Originally posted by dan_vegas
FAO ChristianHJW, what will matroskadub do please explain ?
I need to be able to convert my old divx 3.11 movies to divx 5/xvid so I can watch them on my hardware Kiss DVD player, will matroskadub be able to do this ?

If so when will it be released ?

hehe..no, matroskadub won't convert your old divx3 movies for you.
it's a vdub modification supporting the new upcoming matroska container format.

[dan_vegas]

i guessed you would say that.

well then, what are the chances of getting matroska container support for my DVD player ??

I think that was what was mentioned at the start of this thread

[ChristianHJW]

Originally posted by dan_vegas well then, what are the chances of getting matroska container support for my DVD player ?? I think that was what was mentioned at the start of this thread

Well, this question is extremely hard to answer. I guess its all depending of the pressure that the buyers of 'DivX' PLayers put onto the manufacturers, with respect to their inability to play DivX3 movies.

Sure, all of them will explain in the docs that DivX3 is NOT supported in their units, but i am not sure everybody will read this before buying the unit. I expect that the service departments of KiSS and Neuston, if such are existing, are having a pretty hard time right now .. lol :D !!

The only feasible solution i could see for them, was to make or distribute a tool that would allow the customer to convert his DivX3 AVIs into MPEG4 ISO files, and to a container that will allow to support
- VBR MP3
- AC3 audio
- mode2 form 2 ( 800 MB )

Only 2 containers come to mind, being OGM and soon matroska.

To answer your question about matroskadub, i dont think that -h's conversion tool could be integrated into matroskadub easily, but maybe a command line tool with a nice GUI could do this for you ? ;) ...

I suggest you guys spam KiSS and Neuston with support requests for matroska .. LOL :D :D !!

[dan_vegas]

sorry it has come to this, i wish someone could prove me wrong.....-h ????

"Conversion Tool from DivX3/4 to MPEG4 ISO, both based on Windows and Linux - paid job" == VAPORWARE

[dan_vegas]

sorry it has come to this, i wish someone could prove me wrong.....-h ????

"Conversion Tool from DivX3/4 to MPEG4 ISO, both based on Windows and Linux - paid job" == VAPORWARE

[sam_b]

How the hell can software to be released under GPL being coded by one person FOR FREE IN HIS OWN TIME be vapourware? Twice.

-h has a job, as have most people who code for XviD etc.... If he does not have time to finish it, or the debugging just plain does his head in, you have not lost anything. He owes you and has promised precisely nothing.

P.S. Sorry about by rant, I've just been programming TI DSP code for 9 hours straight....

[seewen]

Sure, all of them will explain in the docs that DivX3 is NOT supported in their units, but i am not sure everybody will read this before buying the unit


Kiss will be able to read DivX 3.11 soon. The public firmware will be released in a few days/weeks .

http://www.kiss-technology.com/news/content.asp?id=52

( and actually, Kiss is alredy able, since firmware 2.6.3, to read "GMC" ).

[avc]

Originally posted by sam_b
How the hell can software to be released under GPL being coded by one person FOR FREE IN HIS OWN TIME be vapourware? Twice.

-h has a job, as have most people who code for XviD etc.... If he does not have time to finish it, or the debugging just plain does his head in, you have not lost anything. He owes you and has promised precisely nothing.

P.S. Sorry about by rant, I've just been programming TI DSP code for 9 hours straight....

You are absolutely right. -h might be fired by his boss if he keeps thinking the debugging all night long. And his promise is nothing though.

I saw some one in Divx forum said the motion vector in Divx3.11 is quite different with MPEG-4 too (extended UMC). because of the dpcm coding of motion vector, re-doing motion search is necessary. Then it is another pain for -h.

[dan_vegas]

hey look guys, i apologise for complaining. i understand about the problems that cause projects like this to take longer than anticipated.

[ubow]

Originally posted by dan_vegas
hey look guys, i apologise for complaining. i understand about the problems that cause projects like this to take longer than anticipated.

Well, I didn't interpret your query as a complaint, although the use of the term "vapourware" could have been a little more tactful, since the word seems to elicit surprising defensive responses from developers :)

I think it's a valid question to pose though... what happened to this project? Several weeks ago -h said "my target finish date is tomorrow or the day after" so obviously the project has reached an impasse of some kind.

There's no need to treat -h with kid gloves though, as "what's the status of the project" is a question that he will hear ALL his life if he works as a developer.

Even though -h has stated that he's doing it for free, there is still a "paid job" being offered by Neuston for this project so perhaps someone else could step up to the podium and tackle this problem if ..... and I repeat IF... -h has abandoned it.

[ChristianHJW]

Originally posted by ubow
Even though -h has stated that he's doing it for free, there is still a "paid job" being offered by Neuston for this project so perhaps someone else could step up to the podium and tackle this problem if ..... and I repeat IF... -h has abandoned it.

Sorry for not having adressed this problem earlier, but the guys simply dont reply to my emails anymore. Things start to make sense of you go to the SIGMA designs homepage and see that they obviously have found a way to make their EM 8500 play DivX 3.11 ... dont ask me what deal they have with M$, maybe they promised to implement WMV9 in return, i have no other explanation ....

... in short : somebody change the subject of this thread and cancels the 'paid job from Neuston' ... it seems its not valid anymore :( ....

[tiki4]

Anyway, has someone seen -h recently? Seems he's vanished totally. What a shame, he's been a nice guy round here. I surely miss him.

tiki4

[Nic]

I sent him a mail not too long ago...he normally replies pretty much straight away even when he wasnt on the forum much. But no reply this time :(

Hope he's ok and enjoying himself :)

-Nic

[MarkCoolio]

Just found this thread on my bookmarks...
Any news from -h?

[BoNz1]

Yes, see this thread, http://forum.doom9.org/showthread.php?s=&threadid=55087 as for any progress on the tool, well only -h would know. If it does get done then that would be great but I guess it is less of an issue now since a lot of these players now support 3.11, however not properly, but thats another topic, ;)

[tiki4]

Well, these players are of course a quite intersting thing, but my interest in that tool was more an compatibility aspect. More and more players and thingies play MPEG4 (and hopefully one day also MP4) content. It would be quite interesting to get old DivX 3.11 movies into MP4 container and play them as any other ISO-MPEG4 stream. I'm thinking of streaming and other purposes.

tiki4

[dimzon]

waitind for DivX3.11 -> MPEG4 transcoding tool!

[zulu]

well, better don't hold your breath in doing so ;)

[bond]

*bump*

hehe, i now stumbled over this nice old thread and imagined how nice it would be if i could convert all my good old divx3.11 babies in nice new 100% spec compliant .mp4 files :D

jokes aside, anyone ever felt the need for writing such a tool again?
but beaware this tool already made some people vanish ;)

[Doom9]

... in short : somebody change the subject of this thread and cancels the 'paid job from Neuston' ... it seems its not valid anymore ....This thread should've been left resting and go the way of all other inactive threads.. disappear.