GPL command line M2V transcoder

[DVDRFreak]

@INT 21h
Thanks for the profiling. I modified the code a little (changed the refill function so it will do the while loops) and now it performs better the my original compiled requant C code. Will see if I can do the same without the while loop.

I have now as results:

Test file 5.12 Gb at 71%.

Original C code 521 seconds.
Untuned OO code 540 seconds.
Tuned OO code 517 seconds (including jumping fix).

[int 21h]

@DVDRFreak:

Can you post the modified oO Refill_Bits function? I'm messing with changes also, so I'd like to make sure I'm on the same page.

[int 21h]

Also, I'm considering changing the macros to functions because the profiler will not report their time (due to the nature of macros).

[DVDRFreak]

Jup I can I changed it like this:



void CReQuant::Refill_bits(uint n)
{
assert((rbuf - cbuf) >= 1);
while (n > inbitcnt)
{
inbitbuf |= cbuf[0] << (24 - inbitcnt);
inbitcnt += 8;
SEEKR(1)
}
}

void CReQuant::Flush_Bits(uint n)
{
assert(inbitcnt >= n);

inbitbuf <<= n;
inbitcnt -= n;

assert( (!n) || ((n>0) && !(inbitbuf & 0x1)) );

if (unlikely(inbitcnt < 24)) Refill_bits(24);
}




In the main loop I changed
Refill_bits()
Refill_bits()
Refill_bits()
Refill_bits()

to

Refill_bits(32)


Changing the macro's to functions/procedures will proberly degrade performance because it will get the overhead of calling a procedure/function. But you could do it for profiling purposes of course and see what happens.

[TheJez]

With this profiling you will know where to gain the speed ! You might even consider changing the bits that consume the most runtime to assembly code. I don't know how efficient the compiler that you are using is, but generally things can be done (much?) more efficient in assembly.

Unfortunately I am not able to help you out with the programming, but I support your efforts and follow this thread with great interest ! Keep up the good work !

The Jez

[Nic]

I already did quite a bit of profiling on this using AMD's CodeAnalyst (I really like that for profiling...). The functions can't be increased much by Assembly, but some could be done.

@PurpleMan: Thanks for the info, yup that should be easy to implement, Ill probably only do it for the IFO mode (because things are easier that way ;) is that going to be Ok?).
@all: Please point these things out to me, because I've not authored a DVD before and may miss even the very obvious.

@int21h & DVDRFreak: Keep up the good work...my office keep calling me and I had to go into work yesterday, so haven't made the progress I would like, but should be able to work on it today. There's no rest for the wicked... ;)

[timecop]

Wow!
A .ts demuxer!
Is it fast?
Faster than TSDemux?
What the freaking hell is it with this program being overrun by pirates using it to copy DVDs?
I want to use it to demux .ts files into m2v + aac audio.
But when I load my .ts files, it only shows the video PID.
Where's the audio PID?

Oh hell half the time (er, 1 out of 10 files I tried it on)
it doesnt even find the video PID.
blah.
I guess I stick with tsdemux

[Nic]

Cut out a bit of a TS stream and send it to me & ill fix the code to show your PIDs. Also it probably isn't showing the AAC audio, because Im not detecting it ;) I've only got one .ts file to test with...your lucky it worked at all ;)

[timecop]

Originally posted by Nic
Cut out a bit of a TS stream and send it to me & ill fix the code to show your PIDs. Also it probably isn't showing the AAC audio, because Im not detecting it ;) I've only got one .ts file to test with...your lucky it worked at all ;)

Hm, tempting.
I'll bring some sample streams with me tomorrow to work and pm you a url with that. Or perhaps I should just be unlazy, look at the source, and see how you are demuxing it and fix it, but hah, that would actually mean working.
I'll get back to you with a bunch of sample streams.

[int 21h]

Thanks DVDRFreak.

Does anyone know what the purpose of this is?


#define likely(x) (x)
#define unlikely(x) (x)

[unixfs]

Originally posted by Nic
Cut out a bit of a TS stream and send it to me & ill fix the code to show your PIDs. Also it probably isn't showing the AAC audio, because Im not detecting it ;) I've only got one .ts file to test with...your lucky it worked at all ;)

Nic,
sometimes AAC in TS shows as MPA layer-4
(in the first 2 bytes of the PES payload you will find
0xfff[89] (layer bits are set to 00)).

[Nic]

@unifxs: Interesting...I hate these minor workarounds/hacks used with MPEG streams broadcasters do, the BBC do weird things also...

@int21h: They were specific to the Apple version and for the compiler. Those #defines you posted do nothing, it's the same as if they didn't have the likely/unlikely macro around the code...it's just there so the likely/unlikely commands didn't need to be added everytime for the Apple compile.

[DVDRFreak]

Originally posted by int 21h
Thanks DVDRFreak.

Does anyone know what the purpose of this is?


#define likely(x) (x)
#define unlikely(x) (x)


It's for the GCC compiler in the original it looks like this:


// gcc
#ifdef HAVE_BUILTIN_EXPECT
#define likely(x) __builtin_expect ((x) != 0, 1)
#define unlikely(x) __builtin_expect ((x) != 0, 0)
#else
#define likely(x) (x)
#define unlikely(x) (x)
#endif

[unixfs]

Originally posted by Nic
@unifxs: Interesting...I hate these minor workarounds/hacks used with MPEG streams broadcasters do, the BBC do weird things also...


Me too, but the worst thing is that these hacks are part of the
AAC standard (ADTS syntax) :(

[crimson]

Thanks for the helpful program.

I have one problem with it -- when I transcode my mpg2 file, the audio is split off separately and I cannot remux it together with the transcoded file. I have tried every program I could think of (TMPG, M2, DVTOOLS, etc.) and it always gives me illegal mpg stream on the transcoded m2v file.

Any suggestions for different remux programs that work or is there any way to have Rejig not strip off the audio?

[int 21h]

I have successfully tested remuxxing with IfoEdit version 0.6 (available at the IfoEdit website)

[E-Male]

after reading along for soem time, i decided to post aswell:

GREAT WORK

i started a little test comparing the quality (and speed *g*) to CCE
my first test with "Das Boot" (superbit, bar scene) impressed me quite a bit

keep up the good work

I'll soon release some results of further tests

CU
E-Male

[digitalman]

Crimson,
Is this MPEG file from a commercially produced DVD? Like a blockbuster movie? If it is, someone had similar issues when trying to author a DVD with a ReJig or Requantizer file. The post is here:
http://www.mmbforums.com/modules.php?name=Forums&file=viewtopic&t=2378&start=0&postdays=0&postorder=asc&highlight=

[crimson]

Not a commercial DVD mpeg. It's pulled from my ReplayTV unit and these types of units are known for weird MPEG2 formats. I can't get IFOedit to work on remuxing without having any VOB, IFO, or BUP files. I'll take another look at IFOedit and see what I can do. Thanks.

[mrbass]

Originally posted by crimson
Not a commercial DVD mpeg. It's pulled from my ReplayTV unit and these types of units are known for weird MPEG2 formats.

go read http://www.mmbforums.com/modules.php?name=Forums DVDLab forums which discussed TIVO, ReplayTV mpeg issues in great length. I think Oscar implemented some workarounds with DVDLab as far as remuxing them, etc.

[mmgrover]

Just Curious, what ver we at?

mike

[Nic]

Even I can't remember, sorry for the lack of updates...Got the flu...sat here shivering crazily. Got most of the IFO parsing code done. (You can select Chapters/Vob-ID/Cells, select multiple streams at the same time, etc etc). Once finished that I begin work on the rate control :)

-Nic

[mmgrover]

@nic

Sorry to hear about the flu :-( I just got over it myself!
Can I download this anywhere?
Curious to check it out...

mike

[horseflesh]

This is pretty exciting, I need a tool that will drop the bitrate on my out-of-dvd-spec MPG2s.

Because I am new to authoring, I am not sure what to do next though. None of the guides out there seem to have me in mind.

Start with: weird high-bitrate MPG2
Rejig to DVD-friendly MPG2 in M2V format
Then... ?

I hit Demux to give me the MPA file too, but what's next?

I have Nero but I can't get it to recognize the MPA and M2V files properly. It just wants to put them in the root of the DVD-video image.

Thanks!

[jhmac]

I too, would like to see a guide or a brief "How To" on using this. I haven't had any luck with it but it is probably because I am not using it correctly... Any help would be appreciated!

[Nic]

@mmgrover: Thanks...feeling a little better today...Ill try and put up a version today, it won't be fully working, but you'll see what I've been upto at least :)

@Horseflesh: Get IFO-Edit and you can reauthor with that, it's pretty straightforward and should work fine. Doom9 even has a guide on using it.

@jhmac: Will be a little while before documentation is done. If you've got any specific questions, ask away :)

-Nic

[jhmac]

OK when you demux a mpg file shouldn't you have a .mpa, and .mpv after demuxing? I always get just one so I have to run the process again.

should you only compress the .mpv or can you compress the .mpg with out demux.

Can rejig also mux in future release or do you recommend another program.

[E-Male]

Made another test:
Fightclub RC1 at 64% (enough to put it on a DVDR with only the English 5.1 audio)
---------------------------------
perfect result & impressive speed
---------------------------------
I'm really starting to like this a lot
and i could imagine some interesting way to use this code [GPL rules!!]

i'll post more when i'm less tired

CU

[E-Male]

REQUEST:

could a program be made that "looks like" docce4u, but instead of cce uses this transcoder or it's code?

so it could be tested in full backups with the big 3 or with numenu4u or other tools using docce4u

thx a lot in advance
E-Male

[daxab]

It seems like people are successfully using ReJig (very, very cool) in two ways:

1. IFOEdit only, with remuxing of the video stream.

2. DoItFast4U.

Can someone post a quick step-by-step HOWTO for both these approaches? It's also a little confusing as to what you can keep -- subs, angles, etc. -- and what you lose with each approach. Also, is menu compression possible?

[darkseid]

I think a simple how to guide of how us noobs who have been following this thread with interest (and coders bewilderment).
Can somehow use rejig (which is fast with good quality) can perform a full backup of a DVD 9 to DVD 5

[mmgrover]

Just Load the CCEData.txt file with rejig and recompress :-)
(Pick a desired compresion level)
Folow all other steps as usual with the big 3.

Oh, I put a version of rijig on a site here that has my last
CCEData.txt parser change. nic, If you want me to remove offered down load, let me know.

http://rejig.mmgrover.com

mike

[mmgrover]

@E-Male

I've been pondering the idea of dll version of rejig with
a callback function for the transcode progress.
Nic, this could allow it to be used very easily in other
Apps? What ya think?

mike

[Nic]

@mmgrover: Cool :) I meant to release a new build last night, but still ill, fell asleep early. I did right the commandline code into ReJig so people could use it from other apps even easier than a DLL. But there's been no interest in it so far...

@darkseid: If I finish the changes I've made soon, IFO Mode should make backing up DVDs a lot easier and obvious, and hopefully that will be the main mode for DVD backup.

-Nic

[E-Male]

if teh program itself does all the job thats of course fien with me
but integration with the known tools, just replacing CCE, would be great

I think many people would agree, if they knew this thread :)

[echooff]

@Nic

Hope you feel better soon. Love ReJig.

[Nocturno]

Great work guys,

i'm certainly interested in either commandline or dll.
like many others i like to make my own gui's for this kind of stuff,so commandline/dll versions are always appreciated. :p

[Eyes`Only]

I've found this app (thanks E-Male) and will be integrating it shortly into DoItFast4U! (thanks Nic!)

[Eyes`Only]

After Phase I integration and some testing by myself as well as others, we've determined the app is almost useable with Big3 in it's current state. The problem is that it seems to be ignoring the bitrates in the ccedata.txt file and just using 70% for everything, which will obviously create oversized dvdrs. Can we get this fixed? Optimum behavior would be to determine the % by analysis of the video file and the bitrate, but even if it would accept different % for each file that would help. 70% across the board isn't workable, for obvious reasons.

The Big3 method and it's calculations gets closer to it's target size on a regular basis than ANY one-button app does, though some are finally getting close. The ability to keep this accuracy is highly important!

[JvD]

Originally posted by mmgrover
Just Load the CCEData.txt file with rejig and recompress :-)
(Pick a desired compresion level)
Folow all other steps as usual with the big 3.

Oh, I put a version of rijig on a site here that has my last
CCEData.txt parser change. nic, If you want me to remove offered down load, let me know.

http://rejig.mmgrover.com

mike

This file givs me an error asking for "MFC70.DLL". Is it me or the file? Whar revision of Rejig is available?

If I would like to integrate this app into the Big 3, what parts and where do i change my standard process.

[Eyes`Only]

JvD: I'm trying to do this for you all but the ccedata files are being encoded at 70% which won't give you dvdr size. With Nic's help we can hopefully get thru that issue. Use Nic's build if you dont want that VC7 error stuff. Newer VC compilers require users to install extra crap! I thought you could statically compile the dlls with VC7 though...

[MedicineMan]

Great. Now that Eyes`Only is working on integrating Rejig into the Big 3, we'll soon have a fully working and freeware (well, almost; we still need Scenarist) solution to our backups.

And it will solve my problem of not being able to get DoCCE4U to work with the Big3.


Keep the good work. We all in this board will be glad to help and test whatever you want.


Sincerely


MM

[Rombaldi]

Please don't forget the 'plain' M2V stream processing! I've been using the heck out of this on making discs of old TV I'm archiving.. being able to give a light 'squeeze' on the M2V streams before authoring is worth it's weight in GOLD!

[Eyes`Only]

What do you mean, "forget it"? Doesn't it already work?

[Eyes`Only]

Nic:

Found another issue. For NTSC m2v files that don't need pulldown afterwards, the filename structure is *.nopull.avs. Inside the AVS, the actual d2v/m2v file is *.m2v though. Rejig is looking for *.nopull.m2v for some reason, and skipping all nopull files, causing NTSC to severely fail.

mmgrover: I saw that your version DOES compensate for the nopull files (thanks!), but since it's making everyone find a vc7 dll, I can't really use it for integration yet (I don't like to require my users to use framework or external dlls, can't it be compiled with the dll static?). Tried to compile it myself with MSVC6 so ppl could use it but kept getting, "H:\rejigmm\IFOModeDlg.cpp(332) : error C2520: conversion from unsigned __int64 to double not implemented, use signed __int64". I read earlier that RB made a function to fix this, but I'm not sure how to implement it, and it's too late so I gotta get to bed. :( I'll delve further into the fix tommorrow, if there's no way to compile VC7 dlls static.

It's still not doing anything with the bitrates from the ccedata.txt file is it? I couldn't find any code that led me to believe it was. When you are doing the big3 method, aren't your dvdrs coming out oversized? Normal bitrate for extras is around 1850, which results in a file about 35% of the original. I'm not sure what that would look like transcoded, I'm assuming it would look terrible, but I'd at least like to get this all implemented so I can confirm or displace my fears. :)

[Nic]

@Eyes: mmgrover's code is in my version at home, want to get a version of the ifo parsing working and finished before releasing. There is no real bitrate control as yet, that's next on my list. Probably best for you, if you want to that is, get involved after that's implemented.
I use VC6 at home, and going from unsigned __int64 to double doesn't cause me issue, but RB's function does it well (can be found in RequantEngine.cpp I think). I'm slowly getting over the flu, so ill release a new build soon :)
Shout me if you've got any questions or anything I should implement I haven't thought of :)

-Nic

[Wilbert]

Do you still need a sample of a ts containing an AAC stream (for demuxing it)? Or is it already capable of doing this?

[mmgrover]

I put the visual studio run time library's on web page.
Sorry, I forgot about that...

http://rejig.mmgrover.com/vs7runtime.zip

I had put file's there temporarly be cause nic was sick :-(
(Hope you are feeling better :-) )



But I can leave other builds there if nic wants me to?


mike

[TEB]

Awsome app. It reminds me of the RT hardware from TandbergTV and Scientific Atlanta are doing in RT transrating domain. (as its called in the DVB/ATSC sattellite area) A full transcoding is way to expensive still in RT.
Anyway ive tested many of my SPTS test streams (single program transport streams) and not one of them could be read or demuxed by rejig, or dvd2avi either (nics version)..(just sent ya a email Nic)

A open question: Is it possible in this process to change from VBR to CBR? Say change from a mpeg2 dvd-constrained mpeg2 to a 6mbit cbr stream. In such a process AFAIK transrating should be enough since the bitrate reduction isnt that large from dvd quality to 6mbit. But where the crossline is for going down to a full transcode is i dunno...
Ill test with procoder later to check vs rejig.

best regards teb

[digitalman]

Originally posted by Rombaldi
Please don't forget the 'plain' M2V stream processing! I've been using the heck out of this on making discs of old TV I'm archiving.. being able to give a light 'squeeze' on the M2V streams before authoring is worth it's weight in GOLD!

Yeah!! Don't forget about the single .m2v file encoding. Keep on improving the transcoder aspect of Rejig or Requant. Thanks!!!!