GPL command line M2V transcoder

[RB]

Hi,

recently, Metakine (http://www.metakine.com/) released a DVD transcoder similar to DVD2One etc. for Mac OS X. Their transcoder engine is based on the GPL'd mpeg2enc and libmpeg2 libraries and consequently, they provide the source code (http://www.metakine.com/files/M2VRequantizer.tgz) to their engine under GPL. You can read more about this (in German) at heise.de (http://www.heise.de/newsticker/data/vza-25.09.03-000/).

I've taken the liberty to port this to Win32 so it can be compiled with MSVC++ and does not require the ugly Cygwin environment. Will run on Win9x/ME/NT/2K/XP/2003. I really don't deserve much praise for this as the port was fairly easy :)

Download the port (source included] from http://home.t-online.de/home/340044300675/requant.zip

It's a command line program and takes as input an quantization factor and an input and output M2V file (MPEG2 elementary stream). For example:
requant 2.0 input.m2v output.m2v
This would transcode input.m2v to output.m2v, reducing average bitrate by a factor of 2.0 (note the factor is a floating point value). Quality is pretty good to my eyes, even at factor 2.0, speed is similar to DVD2One.

This post is only meant to put the ball in the game, please don't request any additional features/improvements from me :). Anybody with enough spare time and programming skills please just take the source, obey to the GPL and take it to the next level(s). I'll post a note on this in the Development forum. Have fun!

[mrbass]

thanks for the port to win32 RB.
So is int 21h Jan 2003 proposal now a reality?
http://forum.doom9.org/showthread.php?&threadid=44321

[int 21h]

I browsed that article a few days ago, but since I didn't know german, and the translation was bad.. I couldn't make heads or tails of it... I'll take a look at this now :)

[atreides93]

I am in awe.

This 100k program transcodes mepg2 files??

[CropsyX]

I've only done a quick test but this looks extremely good so far. It's great to have a program that can work on plain mpeg2 video files. I do quite a bit of work with Digital TV captures and sometimes they're just a bit too big to fit on a DVD, this will be perfect for solving that problem.

Thanks!

[RB]

Originally posted by atreides93
I am in awe.

This 100k program transcodes mepg2 files??

Sure :). With a few compiler optimizations, you can get it down to 17k.

[int 21h]

I think I am going to work on making this oO C++, and perhaps making a shiny GUI for it...

First I have to finish a simple DCT compression project for work though.. I'll post progress as I make it.

[DVDRFreak]

Tested this little proggie on my favorite test subject NARC.

Wanted to decrease with the same value as recode and shrink so calculated 71% - factor 1.41.

Speed: 10 and 23 seconds to transcode (3.5x as fast as shrink or recode).

Compliancy of MPEG2 stream is no problem. Could author it without any problem at all.

Used PowerDVD to watch the resulting DVD. I mus say the quality is for this movie very good (read much better the recode or shrink).

The first scene is amazing. Recode and Shrink would cause a lot of macro blocking (even visible when just watching the scene)) because of the high motion. This little tool causes almost no macro blocking at all (even when freezing the picture I could not detects any aditional macro blocks).

Will do some further testing on other movies but until now I am impressed with the speed and the quality.

Thanks a lot RB for porting this little proggie.

[Rombaldi]

Thank GOODNESS! I've been hoping someone would do this!

Must give it a whirl...

[Guest]

What do you people use to demux M2V from VOBs?

[int 21h]

DVDDecrypter if its on DVD...

[RB]

If the VOBs are already on your hard disk, use Vobedit or vStrip.

[zyzzle77]

@RB:

What compiler optimizations did you make to get it down to a paltry 17 kbyte? Can you make this version available to download? Would be much appreciated.

-z

[RB]

I only have MSVC++ Standard Edition which does not allow for any optimizations, but I also have the Windows DDK which comes with an own compiler that has several optimizations on by default. So I simply compiled it in the DDK build environment (yes, it can build user mode apps) and it came out at 17.5 kb. I'm attaching the executable to this message. This may probably not run on Win9x/ME.

[DVDRFreak]

Recompiled the source using VC6 enterprice edition at work. Used optimalization setting for speed and code generation set to pentium pro.

Testing new exe on AthlonXP 1900+ 1GB memory

Did some more testing:

Movie: Gangs of new york
Compression: 1.492538 (67%)
Transcode time RB exe: 15:17 min
Transcode time optimized VC6 for speed: 11:00 min
Quality: Very Good

Shrink and recode caused a jumping pixel effect in the last scene where the two guys are fighting. This little proggie did not.

[zyzzle77]

What an excellent piece of work. In some cases, this even beats out Shrink in quality (eg, on R1 FIGHT CLUB and WEIRD SCIENCE. Elegant in utility yet very small code footprint...

RB: I couldn't download your 17.5 kb binary compile. Am I missing something, or did you upload it to another server or forget to include it as an attachment?

DVDRFreak:

Very interesting and promising results. Compiling with VC Enterprise edition & optimized for speed & P6 results in a 25% boost in speed! As I don't have Enterprise edition, could you also somehow make your optimized binary available for download?

Thank you both for such positive insight and promising results.

[Rombaldi]

This is quite an AMAZING piece of work, I've been trying it out and not only is the quality quite execellent, but the speed is great. I had been archiving TV episodes that were 'just a little too big' to fit three on a disc, so I ended up authoring them and then Shrink/IC'ing them ... I swear that it seems to cut the process in half by shrinking them BEFORE authoring..

Should someone GUI'ify this.. I can think of two features that would be a blessing..

1) be able to specify the 'shrinkage' either in a percentage (that way you don't have to find a calculator all the time) and/or as a 'target file size' (let's say the file is 1546mb, you want a target file of 1400mb, let it calculate the proper shrink percentage).

2) what else... BATCH MODE! I've got a pile of files stacked up that I'll need to run thru this.. would be lovely to set an input/output/percentage (ie. an 'add to batch' button above) for a pile of files and then let it process them all...

no matter if anyone does or not... GREAT STUFF!!!!

[DVDRFreak]

Originally posted by Rombaldi
1) be able to specify the 'shrinkage' either in a percentage (that way you don't have to find a calculator all the time) and/or as a 'target file size' (let's say the file is 1546mb, you want a target file of 1400mb, let it calculate the proper shrink percentage).


I modified my version so it will except a percentage as parameter. Works great and the speed/quality is also awsome. I have a new favorite tool to do movie only backups.

Just 64KB big and it compressed the matrix_reloaded i just did buy in 8:42 min.

Quality is asome. I do not see the difference between the original and the compressed version when playing. If I freeze frame I can find one or two macro blocks in the title scene.

imho better then recode or shrink for speed and aslo for quality.

[Rombaldi]

DVDRFreak...

how about letting us peons without programming skill at your modified version????

[DVDRFreak]

Originally posted by Rombaldi
DVDRFreak...

how about letting us peons without programming skill at your modified version????

I do not have a FTP server to upload it to.

[zyzzle77]

DVDRFreak,

Since you don't have an FTP server, could you possibly E-mail me your optimized version. Here is my email address:

zyzzle77 *at* yahoo dot com.

Looks like this tool is now the best solution to movie-only backup. Quality continues surpass even Shrink's output.

-z

[Rombaldi]

DVDRFreak

check your PM's

[DVDRFreak]

I just received an offer from MrBass to host my version. I just send him a mail including the optimized exe and modified source files.

I hope he will post a link here soon.

[mrbass]

http://www.mrbass.org/M2VRequant.rar

[mrbass]

http://www.metakine.com/products/dvdremaster/
M2VRequantizer:
A module which simply requantizes the video leaving the motion vectors unchanged. This module is very fast and provides relatively good quality.

M2VDownsizer:
A modules which recompresses the video stream completely (including motion vectors). This will usually yield slightly better quality, at the cost of time.

what about the M2VDownsizer: better quality?

here's that article (http://translate.google.com/translate?u=http%3A%2F%2Fwww.heise.de%2Fnewsticker%2Fdata%2Fvza-25.09.03-000%2F&langpair=de%7Cen&hl=en&ie=UTF-8&oe=UTF-8&prev=%2Flanguage_tools) translated from German to English by Google.

[int 21h]

To port that second one to Win32, it looks like you have to port libmpeg and mpeg2enc... however, I seem to remember that Nic ported these both awhile back? Perhaps he can supply us with the needed code fragments, because porting pthreads would not be fun, (to me anyways).

[Nic]

Excellent news (Thanks for pointing this out to me Int21h). I'm crazy busy right now with work, but this deserves my attention :) Ill try and port M2VDownsizer now.

-Nic

ps
Edit: Well, already got it compiling, now just got to get all the 24 unresolved errors removed. I haven't put in the MMX code or anything yet, expect the first release to be slow as hell.
Edit2: Although it might not help a jot, i did build a ICL 7.1 compiled Requant http://nic.dnsalias.com/Requant.zip
Edit3: Got it compiled as an exe, now gotta work out how to use it ;)
Edit4: Uggh, it's one of these apps that uses stdin and stdout for the input/output...that's just wrong, im on windows afterall ;) soon change that.
Edit5: Well it's running, using 99% cpu so I suppose it's going ok. If I can't get it working then Ill at least release the source code, got to get back to work now. It's all done in VC6 using ICL 7.1.
Edit6: Well now its running correcly ;) Very very slowly making output...really need to get the mmx code in, may not put it out till then.
Edit7: Wow! It works...did my first little test. Slow as hell though.
Edit8: By the way, ill make a little GUI for Requant and make it so it can accept PS/TS/ES Streams as well (although it will only output ES)

[Rombaldi]

Edit8: By the way, ill make a little GUI for Requant and make it so it can accept PS/TS/ES Streams as well (although it will only output ES)

you were reading my mind.. I was about to make the suggestion that a good enhancement would be to let it accept something more than a M2V and DEMUX the output streams while it went (ie. input a MPG program stream, output being the 'requanted' M2V and the AC3/MPA stream... ready to feed into the authoring program..)

[rasta21]

...what we need is a standalone trancoder like rempeg for example...without all those "iwanttocopytonsofdvds" features like reading vobs, get sectors, create images.

...then you can use that trancoder standalone or in tools like numenu4u for example.

[int 21h]

I'd rather have the VOB reading/writing feature so that it could easily handle multi-pgc titles. After this weekend I anticipate having alot more free time to look into this project further.

[Rombaldi]

I see good use for both variants (M2V/MPG and VOB).

One over looked area (but the IC/DVDShrink/et.al. approaches) are the
home recorder (growing by leaps and bounds). I've had more than one instance where I might have (for ex) three episodes that won't quite fit on a disc.. yet one is just a bit bigger than the other two.. till now, I had to author the disc image, process the WHOLE image and the burn... now I can shrink down one ep a touch (or all three) and THEN author/burn... much quicker not having to process the whole disc.

[jdobbs]

By the way -- this is pretty exciting. The output looks very good. With VOB reading capability this could really be nice -- we would finally have a ReMPEG2 replacement that gives good quality and is fast. If it read a .VOB you could simply take the output and reintegrate into the orignal stream using IFOEDIT's remux function.

Note: The compression factor isn't very accurate -- depending upon the input I've had things that were told to be compressed at 3.0 come out to be less than 2.0.

[Nic]

Im just about complete on my little GUI app (ive called it ReJig for now) that lets you load in all types of MPEG-2 file(s), see all the streams (demux if wanted) and then re-encode to m2v. After I've finished that i'll look on making the compression a bit more accurate :)

Looking through the code, it's real simple, should have pulled my finger out ages ago and done it myself :rolleyes:

I'll post back later :)

-Nic

[mikegun]

Originally posted by jdobbs
By the way -- this is pretty exciting. The output looks very good. With VOB reading capability this could really be nice -- we would finally have a ReMPEG2 replacement that gives good quality and is fast. If it read a .VOB you could simply take the output and reintegrate into the orignal stream using IFOEDIT's remux function.


afai remember, it's not that easy to remux, at least it is not possible to remux using cce encodet streams.

mikegun

[jdobbs]

That's because CCE doesn't necessarily create an identical GOP stream (IPB, etc) -- this one does!

[jdobbs]

Looking through the code, it's real simple, should have pulled my finger out ages ago and done it myself

I was thinking the same thing. The quality isn't CCE -- but it's not bad either, and this is really, really fast.

[Nic]

So,

http://nic.dnsalias.com/ReJig.zip
(Sourcecode: http://nic.dnsalias.com/ReJig_src.zip )

It needs a little work (bit rough and ready at present), but I think any suggestions will help motivate me.

Takes Vob/M2V/TS input, shows the streams along with audio delay etc, allows you to demux the audio streams, and it uses ReQuant built-in to compress the video.

It's only a beginning, but it should be a good start (I hope :) )

-Nic

[DVDRFreak]

Looking great NIC. It still has the speed of the original command line util but now it has a nice GUI.

I was testing a little bit and found some strange things:
1) When I add VOB files to the left listview and press the arrow keys the files disapear.
2) When I add files to the left list view and delete them again the right listview is not updated anymore. Even when I press the refresh button.

Some suggestions:
1) A possibility to define the compression ratio per stream would be nice.
2) The percentage I enter now is a integer it would be nice if it would be a float again so I can enter something like 77.8.
3) A way to select the video stream I want to transcode from a VOB would be nice. Sometimes a VOB has more then one stream if it contains logo's and extra's in different programs.
4) Now it looks like the program transcodes the stream per VOB it would be nice if it could transcode a complete movie (maybe I did something wrong will test a little bit more what happens if I add all VOB files).
edit1:
It is worse, the program seems to combine all the streams of all the VOBS that where add in the left listview and combines the streams in random order. Ended up with a movie starting at the end scene then jumping to the beginning, etc...

edit2:
Question.
When the program detects a AC3 stream with a delay does it correct the delay so I have a stream I just can add to my authoring program without any problems?


For a first version ReJig looks prommising.

[jdobbs]

Wow, Nic. You must be the fastest programmer on the block! Thanks!

[Nic]

@DVDRFreak:

Thanks for the bugs ill fix those now.
1) Per Stream? Not quite sure what you want there.
2) Ok :)
3) Hmmm, Ok, Ill add that :)
4) Could you explain exactly what it should do? I've never authored a dvd in my life, so Im not sure what the desired effect should be. And it shouldn't have been random order, but instead the order in which the vobs were added. (ill check that)

No it doesn't correct the AC3 file, merely informs you what it should be. I suppose I could add it as a feature, not exactly sure how.

Well I'll fix that, and also I've added a reasonable amount of the MMX code to that M2VDownsizer. It works ok, but it is very very slow (MPEG2Enc from mjpegtools is a slow encoder even though heavily optimised). Ill release a commandline version of that with source tomorrow in case any one wants to take a look.

Any other feature requests welcome,

Cheers,
-Nic

[DVDRFreak]

Youre welcome always fun to test a new tool :)

1) What I mean is when a DVD has multiple video stream it would be nice to be able to select which video streams to transcode and define the compression per stream. Also you will need to define the output file per stream or make it so that it adds a stream numnber to the global output filename.

4) related to the first option in fact. Now it just takes the VOB'S added and combines all streams in them to one big M2V file. This is not ok when authoring a DVD you would need one big M2V containing the movie and one AC3 file for audio. For the extra's you would need the same thing again.

An example:
DVD x contains:
VIDEO_TS.VOB Menu VOB
VTS_01_0.VOB Menu VOB
VTS_01_1.VOB Main movie
VTS_01_2.VOB Main movie
VTS_01_3.VOB Main movie
VTS_02_0.VOB Menu VOB
VTS_02_1.VOB Extra's with programs so multiple streams (example 3)
VTS_02_2.VOB Extra's with programs so multiple streams (example 3)

If I would want to transcode VTS_02 I would want to see someting like a selection saying

PGC 1 77.8%
PGC 2 50.0%
PGC 3 67.0%

Resulting in 3 sperate M2V files. To do this you would proberly want/need to parse the IFO files to get the correct stream info. I know that there are other open source programs available that do this already so maybe you can get some inspiration there. Also vstrip can split per stream I think.

AC3 correction is pretty common stuff so running google with AC3fix source should do the trick ;)

[ChrisBensch]

All I can say is WOW, it took about 15 minutes to 'downsize' Braveheart to 70%. DIF4U -> Rejig and it looks pretty nice. If this tool could reach multipass CCE quality, the revolution that followed would be great!

Thank you to all who is helping with this and all the other tools

[DVDRFreak]

Hi NIC,

Another idea. Make it process a bunch of M2V files in batch. The same way DoCCE4U is doing it. If you make it possible to just read the CCEdata.txt file for the files to compress we would have a great conbination with DoItFast4U (which does all the ripping stripping demuxing already flawless.

So left listview shows a list of M2V files to process (with a menu option to import this list form the file CCEdata.txt). I can click the file and set some settings like compression ration maybe filename.

This option would make it a great (much faster) alternative replacement for DoCCE4U.

[Nic]

Time for me to download DoItFast4U methinks ;)

-Nic

ps
Please email me a standard CCEData.txt you would use as well :)
nic@nic.dnsalias.com

pps
Looks like ac3 delay correction isn't that common, most seem to do it with AC3 Delay Corrector. Looks like I'll port that from VB to C++, Ugh ;) (ac3fix just fixes ac3 files, doesn't add delay)

[DVDRFreak]

Originally posted by Nic
ps
Please email me a standard CCEData.txt you would use as well :)
nic@nic.dnsalias.com

pps
Looks like ac3 delay correction isn't that common, most seem to do it with AC3 Delay Corrector. Looks like I'll port that from VB to C++, Ugh ;) (ac3fix just fixes ac3 files, doesn't add delay) [/B]

CCEdata.txt is in the mail. With a nice little description of the structure.

Woops :o my mistake just search google and concluded to fast the source was usable.

[Nic]

Well I'll release this for now, and now begin looking at the CCEdata.txt for the next version.

This version does a series of Jobs, if you want to add multiple files into a single job you can select multiple files in the Open File dialog. Hopefully it will all be self expantory and I've tried to incorporate some of DVDRFreak's ideas (as I've perceived them at least :) )

http://nic.dnsalias.com/ReJig.zip
(Sourcecode: http://nic.dnsalias.com/ReJig_src.zip )

-Nic

ps
Oh, and I haven't ported the AC3 Corrector yet, so no correction yet. Haven't received that email yet, if it's easy for you to send please also post to my other address nic@everwicked.com

[Exo]

Hi Nic,

there seems to be a little bug in ReJig.
50 frames or more are missing at the end
of the output file.

btw, when using ReQuant on the same file this
doesn't happen.

[Nic]

was it an M2V or Program Stream (.vob/.mpg/etc ? )

@DVDRFreak: Ahhh, CCE uses AVS files, I forgot. Could you send me the avs files it creates...they may be simple ones that just use MPEG2Source in which case I could just grab the MPEG FileNames from there.

-Nic

[DVDRFreak]

Originally posted by Nic
was it an M2V or Program Stream (.vob/.mpg/etc ? )

@DVDRFreak: Ahhh, CCE uses AVS files, I forgot. Could you send me the avs files it creates...they may be simple ones that just use MPEG2Source in which case I could just grab the MPEG FileNames from there.

-Nic

You can use the filename the same as the .AVS file the M2V has the same name only the extension is different.

so "C:\MYMOVIE\1.AVS" in de CCEData.txt would translate to "C:\MYMOVIE\1.M2V" for the ReJig tool to transcode.

[Nic]

@Exo: Excellent work, I really didn't spot that bug. Now fixed.
ReJig v0.2c
http://nic.dnsalias.com/ReJig.zip

@DVDRFreak: Genius. Ill do that ASAP.

-Nic