GPL command line M2V transcoder

[JvD]

OK, this tools seems promising, BUT, is there a way to keep certain subpictures (subtitles) intact OR is ReJig _JUST_ a transcoder (a not-so-many-features-application)? I would like to use it but I MUST HAVE SUBTITLES?

[Nic]

Well, I'm keeping myself busy...been adding IFO parsing support. (when I say adding, I mean hacking VStrip into ReJig ;) )

@JVD: I'll make it so you can demux subtitles as well. The tool is only small and simple now, but it will grow. Got to buy me a dvd burner too...anyone gotta spare? ;)

@Rombaldi: I like the name ReJig :P as Garf/Monty once famously said:
"I Like The Name. I Wrote the Software. The Name Stays."
http://www.hydrogenaudio.org/index.php?act=Search&CODE=show&searchid=273b48fb1549e0cd0d8a36e855883703&search_in=posts&result_type=posts&highlite=the+name+stays

@DVDRFreak: Harder to do if it's not a raw M2V file, because a vob say isn't a true representation of the size of the m2v. I've got lots of ideas for the engine once I do all this periphery stuff. One of which is rate control. Then either two pass, or the ability to select certain time periods for quant boosts/decreases (i.e. boost on credits, lower the quant in difficult sections, etc)

It's growing, should be able to release a minor IFO parsing version (not fully functional, just an example) over the weekend, got to fit in seeing Kill Bill as well, obviously.

-Nic

[zeus163]

@Nic

Thanks for the fantastic update. This little tool has worked wonders for me on two DVD's I've authored now. I wasn't looking forward to the hours of re-encoding several .m2v streams I had to fit onto DVD's I wanted to author and was able to compress them to 90% and have them fit and I can't even tell.

I look forward to 2 pass on this tool. Great job!

[mmgrover]

Nic, I made a couple of changes to the "ReadInCCEData" Function.
I would loose the output file name on the last file and I changed
code to allow for same output file names as if they were run
through CinemaCraft Encoder and pulldown.

thanx, mike

ps: Nice app, Compiles well under Visual Studio .net



void CReJigDlg::ReadInCCEData(LPCSTR szCCEData)
{
FILE *fin = fopen(szCCEData, "r");

list<CString> lstM2VFiles;
list<CString>::iterator iter;

if ( fin )
{
char szPath[500];
char *p = szPath;

bool bInQuotes = false;
while ( true )
{
int c = getc(fin);
if ( c == EOF )
break;

if ( c == '"' )
{
bInQuotes = !bInQuotes;

if ( !bInQuotes )
{
// Then we have the filename
*p = 0;

// Convert to a .M2V
if ( strrchr(szPath, '.') )
strcpy(strrchr(szPath, '.'), ".m2v");

lstM2VFiles.push_back(szPath);
p = szPath;
}
}
else
{
if ( bInQuotes )
{
*p = c;
p++;
}
}
}
fclose(fin);

for ( iter = lstM2VFiles.begin(); iter != lstM2VFiles.end(); iter++ )
{
// Add the Jobs in
CJob Job;
Job.csInputFile = *iter;
Job.fCompressionFactor = m_fCompressionFactor;

// Create suggested output filename
Job.csOutputFile = *iter;
int nDot = Job.csOutputFile.ReverseFind('.');
if ( nDot != -1 )
{
Job.csOutputFile = Job.csOutputFile.Left(nDot);
}

//Job.csOutputFile += "_ReJig.m2v";
int nNopull = Job.csOutputFile.Find("nopull");
if ( nNopull != -1 )
Job.csOutputFile += ".mpv";
else
Job.csOutputFile += ".mpv.m2v";



m_JobList.push_back(Job);
AddString(*iter);

//Save the last input before refreshing
// mike
m_csOutputFile = Job.csOutputFile;

// Select this one on the input list
m_lstInputList.SetCurSel(m_lstInputList.GetCount()-1);
m_PreviousJobSelection = m_lstInputList.GetCurSel();
OnRefresh();
HorizScrollBox();
}
}
}

[Nic]

Sounds fine to me :) Ill put it in. Thanks :)
(you can put text like that in using square brackets with the word "code" in, then you won't loose the formating and the post is smaller)

some code
etc :)

[mmgrover]

Thanx, I didn't know that :-( my bad lol

[int 21h]

Direct VOB output needs what.. update of VOB navigation packs and correction of sectors in IFO table?

[DVDRFreak]

Originally posted by Nic
Well, I'm keeping myself busy...been adding IFO parsing support. (when I say adding, I mean hacking VStrip into ReJig ;) )

@DVDRFreak: Harder to do if it's not a raw M2V file, because a vob say isn't a true representation of the size of the m2v. I've got lots of ideas for the engine once I do all this periphery stuff. One of which is rate control. Then either two pass, or the ability to select certain time periods for quant boosts/decreases (i.e. boost on credits, lower the quant in difficult sections, etc)

It's growing, should be able to release a minor IFO parsing version (not fully functional, just an example) over the weekend, got to fit in seeing Kill Bill as well, obviously.

-Nic

Wow. with IFO support this little proggie will realy Rock :cool: . True about the filesize thing did not think of that.

Thanks Nic for all your time and affort to make this little proggie realy shrinktastic.

[JvD]

Yeas, the is Tooo hard for me. Why not manual XXXXMB? In ReJig that is. (dont know how to estimate/calculate the % shrinkage, including sound, etc.)
Pls linify me or explain it.

PS. After a couple of tests I can say that the Quality IS REAAALLY great for an app this fast. DS.

[girv]

@Nic: thanks :)

@all: anyone done any quantitative tests comparing ReJig quality to any of the other transcoders out there?

[Grunter]

@JvD Why don't you use ifoedit for authoring m2v output from ReJig with audio and subtitle from original dvd? It works well ;)

[unixfs]

Hi,
how much can you compress your videos?
I tried to set a factor of 12.5% on an 8 GB video expecting to have
a compressed 1 GB file (conscient that quality would suck), but it was 2.5 GB.

What's the typical ratio of quants size/whole movie?

Thanks.

[Nic]

(I believe im getting ahead of myself and already am looking into the reauthoring side...even though it's so tempting just to automate ifo-edit ;) )

[int 21h]

Originally posted by Nic
(I believe im getting ahead of myself and already am looking into the reauthoring side...even though it's so tempting just to automate ifo-edit ;) )

Resist the temptation! ;)

[JvD]

Grunter, how do I do that. I know hoe to author (mux) video and sound together, but how do I add the subpictur/subs that I want. PLS, enlight me.

[EDIT]OK, I think I got it!! ;-)

[DVDRFreak]

Originally posted by JvD
Grunter, how do I do that. I know hoe to author (mux) video and sound together, but how do I add the subpictur/subs that I want. PLS, enlight me.

[EDIT]OK, I think I got it!! ;-)

You can also make a VOB job with ReJig 0.3a to transcode the video to wathever size you want and the remux it with IfoEdit back into the original VOB. This way you keep the menu and all the extra's you want to keep.

[JvD]

Dont get it! I forgot to do my homework , sorry! Have to read som more about this! Any tips! Enlight me MOOORE! I have been living in a shelter with all those easy oll-in-one-solutions ;-)

[mmgrover]

Has anybody been successful at remuxing a video file from ReJig
using IfoEdit?

I keep getting pauses and jerks during playback?


mike

[DVDRFreak]

Originally posted by mmgrover
Has anybody been successful at remuxing a video file from ReJig
using IfoEdit?

I keep getting pauses and jerks during playback?


mike

Jup just did shrek. Tried it with Recode and Shrink but the picture quality was blurred and I did get visible macroblocks.

Used IfoEdit 0.95 to strip and ReJig to transcode the M2V file directly from the VOB files (just click first VOB (VTSxx_01) and ReJig will add the rest of the requierd files. After ReJig is finished open the ifo for that title set and remux the video. When this is finished delete the original VTS_x_01 files and move the new ones into the place. Press get VTS sectors and presto. Plays perfect.

The backup I have now is almost perfect with all the extra's included. No visible quality loss imho. Only thing stripped is one language track and all menu's that where not in english.

[JvD]

Thanks DVDRFreak!
Will this method keep the possibility of subs?

[int 21h]

It will not...

I think you'll need VOB output from Rejig before you can deal with the subs.. (without reauthoring a DVD that is..)

[DVDRFreak]

Originally posted by JvD
Thanks DVDRFreak!
Will this method keep the possibility of subs?

This method keeps the original DVD structure in tact. So all the subs yoy do not strip with IfoEdit will remain. By the way if you want to reauthor a DVD with subs the easyest way is to use DoItFast4U to demux all the stuff you want and the compress it with Requant's CCEdata.txt imput feature.

[JvD]

Thanks again! I have tried what you descibed in earlier post and it works really great. Will do some reading on DoItFast4U to see what i can do and HOW. "Ill be back!"

[DVDRFreak]

Hi Nic,

Found on sourceforge some sources to author a DVD maybe these will be also useable for this new tool. Don't now however how good they are. Project is called DVDAuthor.

[JvD]

Originally posted by DVDRFreak
This method keeps the original DVD structure in tact. So all the subs yoy do not strip with IfoEdit will remain. By the way if you want to reauthor a DVD with subs the easyest way is to use DoItFast4U to demux all the stuff you want and the compress it with Requant's CCEdata.txt imput feature.

OK, tried it! Ripped the complete DVD, rund the main-movie-vobs trough ReJig, Demuxed, the sound. Then Muxed it togheter in Ifoedit creating new vobs. Replaced the old ones with this new set of vobs. In Ifoedit, got VTS sectors, and saved it. When playing it, THERE ARE NO SUBS? WHAT DID I DO WRONG? I didnt stripp anything. The picture is amazing, with hte speed in mind. Sure, CCE makes perfect copies. The day this transcoder is placed under som great GUI, in a smart all-in-one-solution, this product WILL rock (it allready does). Keep on working guys!

[Exo]

Hi JvD

You can either rip the subtitles with vobedit to sup files
and use those sup files in Ifoedit's DVD Author mode or
you can use Ifoedit's remux function to simply replace the
video in your original vob with your ReJigged m2v (and
keep all original audio and subpicture tracks, also the menus
should work as in the original).

[Nic]

@DVDRFreak: Way ahead of you ;) Already ported DVDAuthor to Win32...with a combination of mplex -f 8 and dvdauthor all built in, things may go quite well :) Ill also add code to automate IFOEdit, as that is the tried and tested method (even if it's not opensource :( )

[DVDRFreak]

Originally posted by Nic
@DVDRFreak: Way ahead of you ;) Already ported DVDAuthor to Win32...with a combination of mplex -f 8 and dvdauthor all built in, things may go quite well :) Ill also add code to automate IFOEdit, as that is the tried and tested method (even if it's not opensource :( )

Wow :cool: can't wait to test this sucker. Way to go Nic.

[layer3maniac]

This is REALLY exciting. Kudos on all your work, Nic. It's hard to believe how far this has come in such a short time. You are amazing! Keep up the good work, and BTW, thank you!

[DVDRFreak]

Hi Nic,

in your code for the engine in 0.3a there is still the printing of the percentage for the command line version removing it will save some time. Removed the all printf commands in general and it was 1-2% faster.

[JvD]

OK, thanks for all help. Now I got it to work. I used Vobedit to rip subpicture streams (is this neceserry?). The joind them all together (m2v, ac3 ond sup) in Ifoedit (do I really need to do all three?). Replaced the original VOB-files (movie) with these new ones. VOILAT! It worked fine. THANKS MATES! But i AM SOOOO bad at calculating the size (in %) for the moviefiles. The movie ended up too big (100MB:-) Yeas, i AM really lousy ;-)

[DVDRFreak]

Originally posted by JvD
OK, thanks for all help. Now I got it to work. I used Vobedit to rip subpicture streams (is this neceserry?). The joind them all together (m2v, ac3 ond sup) in Ifoedit (do I really need to do all three?). Replaced the original VOB-files (movie) with these new ones. VOILAT! It worked fine. THANKS MATES! But i AM SOOOO bad at calculating the size (in %) for the moviefiles. The movie ended up too big (100MB:-) Yeas, i AM really lousy ;-)

For a movie only backup this could be a method (so a DVD without extra's or menu's). You indeed need a substream a AC3 stream and a M2V stream for a DVD with sound and subtitles.

If you want just to shrink a DVD to fit on one DVDR do the following.

1) Rip it to harddisc with DVDDecryptor.
2) Strip all stuff you do not want out with IfoEdit (there are somegood guides on the doom9.org site) or browse the ifoedit forum.
3) Check the size of the stripped projects (for example it is 4.8 GB). So you need to shrink the total by 4.8GB-4.37GB=0.43GB.
For now there is no way to tell with ReJig how big the M2V sreams are in a VOB so to be able to see this demux all VOB's to M2V to see the size. Then calculate a percentage to save the amount of space needed.
4) ReJig the M2V's with the calculated percentage.
5) Start IfoEdit and select a ifo belonging to a Rejig M2V. Under the menu VOB extra's select Remux M2V. In the dialog select your new M2V.
6) When IfoEdit has finished replace the original VOB/IFO files with the new ones created by ifo edit.
7) Repeat step 5 and 6 for all Rejiged M2V files.
8) When all is done open IfoEdit once more and press GET VTS sectors.

Now we have a fully working DVD with menu's subtitles extra's. That will fit on a DVDR disc.

[Nic]

Well i'm working loads on ReJig, but this version doesn't allow you to do much more, I'm away tuesday-thursday, so I thought I'd release this and fix little things since 0.3a (i.e. has mmgrover's code, jsoto's ac3 correction, dvdrfreak's suggestion of removing printfs, also no one had noticed the audio delay always reported 0ms ;) etc)

But there's lots of new code inside that hasn't been turned on really yet :) For example, With the code that's in there you can rip a ifo parsed PGC video straight from DVD to a M2V file (it's very unpolished at present, but works fine). So even though it seems not much of a step forward with 0.3b, the next proper version (0.4), should have a lot more...

(...spent the last hour with a hex editor seeing how .sup files work for IFOEdit. Easy to add, that will be in the next version so you can demux to sup)

http://nic.dnsalias.com/ReJig.zip
http://nic.dnsalias.com/ReJig_src.zip
(Sourcecode is always updated with every release and always at that link, sometimes I'm just too lazy to type the link with every release ;) )

-Nic

ps
Thanks for all the testing and feedback, helps loads! :)
Special thanks goto maven, because i've done a really shameful hack to get VStrip to do all the work for me, one day ill neaten it up (but it works ;) )

[DMagic1]

It would be nice if there was something to aid people in calculating their %.

[Macanudo]

I remember reading somewhere that it was possible for Transcoders to change the resolution. I think it is an option in ReMpeg2. If it is possible, I think it would be a great feature to add to Rejig.

You could reduce the resolution to 352x480, which as you know is still DVD complaint, and eliminate blocking at low bitrates. This might allow you to keep some extras at low bitrates that you would otherwise delete.

Just a suggestion,
Macanudo:D

[int 21h]

Currently with this engine it is not possible as it would require subsampling of the motion vectors... also, unless you had some specific filtering applied to the subsampled vectors, you would introduce a good amount of distortion.

[jhmac]

Sorry for the newbie questions...
When I demux should't I get 2 files a .mpa & .mpv files? Also the 2 recompress buttons what are the difference between them? What is the best way to mux the files back together? I am playing with this tool but really don't know how to use effectivly.

OK first I load the vob files then demux the audio then recompress the video now I think I mux and convert to vobs with ifoedit but not sure if this is correct or how to do this. Anyone feel like giving a crash course in ReJig'ln a movie!

Thank You!

[jsoto]

Hi Nic
(...spent the last hour with a hex editor seeing how .sup files work for IFOEdit. Easy to add, that will be in the next version so you can demux to sup)
I've spent more than one hour and I was unable to find the clue about timestamp (the first bytes of each frame). Could you explain it? Well, I can wait because it will be included in the sources...
BTW, I have written a source with subs extraction, but, obviously, timestamp is not working. I am trying to include this feature in mdvdauth plugin, without using additional SW.

jsoto

[Nic]

Ahh, the timestamp is a PTS, so divide by 90000 to get the number of seconds. But strangely (well almost) it is not the PTS of the Sub Stream in the MPEG, but it is offset, I think, by the video stream PTS.

So (from memory) you have 'SP' (which is the start of sup sub) then the PTS (which I think is a QWORD) then pretty much the size of the sub minus it's header and that's about it. (I think there was one more bit, but I can't remember (im at work now))

Hope that helps :)

-Nic

ps
jhmac: I'll write a mini-guide for the next release, but for now read through mine and DVDRFreak's posts in this thread, and that should give you an idea of how things work. But it sounds like your doing it right anyway :)

[noir]

Originally posted by Macanudo
I remember reading somewhere that it was possible for Transcoders to change the resolution. I think it is an option in ReMpeg2. If it is possible, I think it would be a great feature to add to Rejig.

You could reduce the resolution to 352x480, which as you know is still DVD complaint, and eliminate blocking at low bitrates. This might allow you to keep some extras at low bitrates that you would otherwise delete.

Just a suggestion,
Macanudo:D

i also noticed that for a perfect dvd compilant the 352x480(576)stream must be 4:3 not 16:9, this introduce a complex resize problem

[jsoto]

Ahh, the timestamp is a PTS, so divide by 90000 to get the number of seconds. But strangely (well almost) it is not the PTS of the Sub Stream in the MPEG, but it is offset, I think, by the video stream PTS.
Many thanks, Nic.
I'll play with it when I get some free time.
jsoto

[mmgrover]

Nic, I have an idea for a a new feature.
I just did Star Wars Episode I, but at the podracer scene movie was
blocky :-(. I then demuxed by Cell ID. I rejiged VobID 6,cells 6-9
at 80% and dropped the other cells by 1 percent to end up with same
final output size. I then glued all the cell back together to a single m2v stream and remuxed. pod racer scenes were much better looking!

I wonder if it would be worth adding individual compression
settings for the multiple file input?

just an Idea, mike

[Nic]

"I wonder if it would be worth adding individual compression
settings for the multiple file input?"

Could you explain this a little more...I was going to add a feature in to allow you to specify times in the input for the quant to be boosted/decreased...are you thinking of a similar thing?

Anyway I'm just about to catch a plane to see a client, be back thursday/friday...

-Nic

[DVDRFreak]

Hi Nic,

Some ideas for the engine:

When in Ifo mode it would be nice if you could select the compression per chapter. In file mode a time selection would be an option.

Also you could make the engine smarter by detecting somehow high motion scenes. These normaly compress not so good and would therefore need a lower compression ration then the rest of the mpeg. Maybe it is possible to analyse the bitrate or the motion vectors so those scenes could be detected.

[int 21h]

Originally posted by DVDRFreak
Hi Nic,

Some ideas for the engine:

When in Ifo mode it would be nice if you could select the compression per chapter. In file mode a time selection would be an option.

Also you could make the engine smarter by detecting somehow high motion scenes. These normaly compress not so good and would therefore need a lower compression ration then the rest of the mpeg. Maybe it is possible to analyse the bitrate or the motion vectors so those scenes could be detected.

You could do this by applying a bias... instead of applying a straight X% quantizer increase across all frames, you apply X% quantizer increase on certain frames, and a lower Y% on frames that are more complicated. You could detect 'complicated' frames by looking at the coefficients I think... As far as I can tell, currently the ReQuant engine just blindly increments the quantizer scale...

[DVDRFreak]

@INT 21h

Jup, thats what I saw in the source to. It increases the quant scale and decies whether to compress I/P/B frames depending on the fact if it already compressed the video enough (react_time constant). Do you know how to determine the coefficients (I think you mean the motion vectors) ? Is it already in the source somewhere.

This could help Nic allot to build in this new feature.

[bwalerud]

Handle a mpeg2-stream contained in an .avi?

[duartix]

From what I know about DVDShrink before deep analysis, it was using Constant Frame Quantization, i.e. if you choose 70%, it would reduce the frame to 70% of it's original size.

Since it just works on DCT data, lets say the frame was:
20% MV
80% DCT
it would have to reduce the DCT to:
(70% "desired reduction" -20% "non reductable MV")/(80% "DCT data")
which is (0,7-0,2)/0,8 = 62,5%

But what if the frame was:
50% MV
50% DCT ?
then the final rate of the DCT data would be:
(0,7-0,5)/0,5 = 44%

In a higher motion scene like this:
65% MV
35% DCT
the final rate of the DCT data would be:
(0,7-0,65)/0,35 = 14%

Well I think you get my point: As the amount of motion in a frame is getting higher, so is the reducion of the DCT values. This was why DVDShrink performed so poorly in high motion scenes (before deep analysis) When the amount of Motion Data is beyond the reduction rate it is impossible to reduce the frame further without going into MV reduction. That's probably where a bit reservoir mecanism takes place.

So, instead of Constant Frame Quantization what is needed is Constant Quantizer Reduction. But in order to do that, a first pass is needed to calculate that global quantization value.
Either that or motion vector reduction also.

I don't know how Rejig is working (in fact I don't know either how DVDShrink is working now that is has deep analysis) but wanted to share my toughts...

[int 21h]

You can't touch motion vectors, or you shouldn't anyways..

For transcoding within the compressed domain, there are two main ways to do it.

The first is increment quantizer scale, you uncompress until you get to quantized coefficients, which you then dequantize, and requantize on a higher scale.

That method is the most inefficient but also the simplest to implement.

The second method is to selectively drop coefficients. You can do this several ways also. You can always drop coefficients over block position X, or you can drop after X amount of 0s. The best quality method is to adaptively do both of these.

This method is the most efficient but also more difficult to implement.

ReQuant (the ReJig engine) uses both of these methods. On I-Frames, we increment the quantizer scale. On P-Frames, the last coefficient is dropped, if there is more than one coefficient. On B-Frames, the quantizer scale is multiplied by a computed factor.

So how do we make this more efficient and better quality?

That's hard to say. In general, if you look at a graph with PSNR on the Y-axis, Rate on the X-axis, and graph the regular MPEG compression process, its a positively sloped curve. Then, graphing the transcoded process, at high rates, you'll see the two curves very near each other, when the curve for the regular compression begins to drop off, the transcoded curve maintains a higher PSNR longer, then more rapidly drops off. This tells us that if we reduce the rate too much, we're going to end up with a worse PSNR from transcoding than you would re-encoding. (i.e. worse picture quality)

Also in general, in higher motion scenes, we should be able to either increase quantizer scale more.... or be more liberal with the coefficients we drop.

Right now, ReQuant is not setup to consider motion vectors in the overall scheme of things, but to improve quality, we could further improve the decision process of which method to use, instead of blindingly applying it by frame type...

[jhmac]

Holy crap what did he just say?