GPL command line M2V transcoder

[DVDRFreak]

Originally posted by jhmac
Holy crap what did he just say?

He explains how Requant works and gives some good tips how to improve it :D .

@int 21h
Good explanation int 21h hope this helps. I myself was digging through the source and found a variable called delta that is calculated based upon motion (at least I think it is). When I run it through the debugger it seems to get higher values when fast motion scens are processed and low values when normal scenes are processed varying between 10-255 or so.

Can we maybe use that value and adapt the fact_x based upon that ? Or did I not understand this calculation correctly ?

[int 21h]

Delta also appears to me an indicator of motion... but it appears that the functions dealing with motion vectors are currently only used around line 1271:

case MC_FRAME:
MOTION_CALL (motion_fr_frame, macroblock_modes);
break;
case MC_FIELD:
MOTION_CALL (motion_fr_field, macroblock_modes);
break;
case MC_DMV:
MOTION_CALL (motion_fr_dmv, MACROBLOCK_MOTION_FORWARD);
break;


The only problem with changing fact_x once you begin processing, is that ReQuant has no rate control... when you start ReQuant you tell it you want to apply a blind X% reduction across all frames, if you change fact_x while execution proceeds, you're no longer getting a reduction of X%, so you would need to implement more sophisticated rate control...

[int 21h]

Perhaps we could change it like you suggest, so that on higher motion values we do less of the requantization process, and we make a sort of debt buffer that we try to equalize out later by applying even more compression on lower motion scenes... I dunno, I get confused when I start to stare at the code too long, its so messy.

[DVDRFreak]

So if we want to use this we would have to do a two pass. First pass build a info file with motion information. Second pass calculate fact_x dynamicly based on this info file and some kind of bias setting (1-255) that tells Requant when to lower and when to increase fact_x.

[int 21h]

Instead of, or perhaps in addition to, manipulating the factor, I think you should manipulate the methods used, i.e. you only drop coefficients on high motion, increase quant on low motion, etc.

[RB]

Originally posted by int 21h
I dunno, I get confused when I start to stare at the code too long, its so messy.
Simply couldn't resist an "me too" here :) Why is it that great minds always write such higly obfuscated code with no comments?

[duartix]

Simply couldn't resist an "me too" here Why is it that great minds always write such higly obfuscated code with no comments? I can't possibly agree with that as I know one of these people, which is now taking a Master Degree in genetic programming, and I've never seen clearer code than his. We were both in Computer Science (Artificial Inteligence) and I've seen Math students copying his projects over the years because of it's simplicity, cleanness and structure. It was like looking at at manual of Structured Programing using OO.

Anyway, @DVDFreak:
Isn't it easier than this? Second pass calculate fact_x dynamicly based on this info file and some kind of bias setting (1-255) that tells Requant when to lower and when to increase fact_x.Isn't it enough to make a first pass where you calculate the ratio of MV and DCT in the total amount of data, and then you just calculate the effective DCT reduction rate by:
effective DCT reduction rate = (desired reduction - MV ratio)/(DCT ratio) ?

This way we could have a constant DCT quantizer. Doesn't this mean constant quality?

Or perhaps I am oversimplifing things?
I must confess that int 21h post has put me on my knees. :o

[Sbofen]

Originally posted by jhmac
Holy crap what did he just say?

Just when I thought I was starting to understand a little about DVD structure and programming...

Ummm, I've got cookies and a twinkie - if it'll help.

All I can say is that I'm glad there are people that do fully understand all this and can make a great app out of it for me!

Man I feel dumb
:stupid: :eek:

[int 21h]

I'm hesitant to really recommend tying too much to motion because motion is not always a precise indicator of scene complexity... it reminds me alot when me and a friend bothered nando until he put those motion based DRF (really quantization) boxes into Nandub, and it made it a little better, but about 5x more complicated (i.e. you had to know alot more about the content, etc.) Whereas, coefficients could indicate it a little more accurately (in my opinion)...

I regret I don't have a lot of time to dive into alot of changes to the Requant engine, I really would like to oO the source, but there are some parts I just cannot understand :( Maybe Nic can help me out when he returns.

[Mug Funky]

hey, first of all, thanks you guys. your work is _very much_ appreciated.

one thing i've noticed is that rejig/requant seems to be excessively hard on b-frames. perhaps percentages could be specified separately for i, p, and b frames? this would make file size prediction hell (i think), but it would make requant/rejig much more versatile.

[duartix]

Ok, my doubts on DVDShrink's deep analysis compression method were answered here (http://forum.digital-digest.com/showthread.php?s=&threadid=26603).

So what's it all about?
DVDShrink performs a first pass where it calculates the engine's maximum and minimum compression for every frame in the source. That's why it can take as long as half a(n)? hour.

With that information, it's very easy to set the compression uniformly across all frames on the second pass.
That easy! :cool:

[Nic]

Very useful thread duartix thanks, doesn't sound too complex at all. Should be able to implement that, just came back and now have a week off (assuming the software I wrote for the client doesn't fall over ;) ). So I should be able to work on it lots over the next week...Lots of interesting/thought provoking discussion, since I've been away, keep it coming... :)

-Nic

[vidcel]

I have tried using this a few times to trascode my mpegs that I have accidently created to large, this all goes fine except when I import it into DVDAuthor it tell me that it's max is 10200kbps and the DVD max should be 9800kbps, even if I do it at 20% still the same. So something a bit odd.
I tried an old bitrate program as well and the same was reported.
:confused:

[mikegun]

hi,

did you try dvdpatcher ?

rgds.,

mikegun

[vidcel]

thanks for that.
That worked.
Cam

[Nic]

@int21h et al: I converted ReQuant into C++ code...but it's quite a bit slower than the C :( also allowed me to do some tests with the Intel Compiler...which makes it a helluva lot faster than VC6.

http://nic.dnsalias.com/ReQuant_src.zip

is the C++ version and the old C version, for you to play with, I moved some of the more interesting functions to the top of ReQuant.cpp
The C++ version won't make it into ReJig, but it may be useful for testing quality...
(I did a fc and the C++ version and C version report the same output, which is good, if you can make the C++ version the same speed let me know).

Back to IFO parsing stuff, then onto rate control ;)
(Surprised no one has tried double clicking the right mouse button in ReJig to see what I was upto before I went away... ;) )

-Nic

[JvD]

OH! I totaly missed that part :eek: I normaly right-dubbel-click everything :D Nice "interface"!

*DEMUXING AS WE SPEAK*

[Nic]

LoL...don't expect it to work...but it's a start. Thought someone would have picked up on it ;)

[JvD]

SORRY to tell but it doesnt work. Tried two times and in the end, when it all was at full (100%) I get an error msg.
When I pick an IFO (first one) then get an m2v that is way bigger then the original. (used an allready shrinked set of files, movie only) The m2v itself should be appr 3.6GB.

[int 21h]

Originally posted by Nic
@int21h et al: I converted ReQuant into C++ code...but it's quite a bit slower than the C :( also allowed me to do some tests with the Intel Compiler...which makes it a helluva lot faster than VC6.

http://nic.dnsalias.com/ReQuant_src.zip

is the C++ version and the old C version, for you to play with, I moved some of the more interesting functions to the top of ReQuant.cpp
The C++ version won't make it into ReJig, but it may be useful for testing quality...
(I did a fc and the C++ version and C version report the same output, which is good, if you can make the C++ version the same speed let me know).

Back to IFO parsing stuff, then onto rate control ;)
(Surprised no one has tried double clicking the right mouse button in ReJig to see what I was upto before I went away... ;) )

-Nic

Interesting that its so much slower in C++, have you profiled it to try and see why?

[Nic]

@int21h: Haven't profiled it, could be that now it doesn't use static variables everywhere, etc the compiler can't optimise as well, etc...take a look, ill just keep with the C code for now, my C++ code doesn't help the readability (<--is that a word ;) ) much.

@JvD: Well it's only the test stuff, so I doubt it would work, but it has worked fine for me, any more info you could give me would be great (like the error msg it said, the percentage you set it to before hitting reencode, the buttons and the order in which you pressed them etc etc). Thanks :)

[mmgrover]

Nic I had to add another line of code, mike


for ( iter = lstM2VFiles.begin(); iter != lstM2VFiles.end(); iter++ )
{
// Add the Jobs in
CJob Job;
Job.csInputFile = *iter;
Job.csInputFile.Replace("nopull.",""); //<- mike
Job.fCompressionFactor = m_fCompressionFactor;

// Create suggested output filename
Job.csOutputFile = *iter;
int nDot = Job.csOutputFile.ReverseFind('.');
if ( nDot != -1 )
{
Job.csOutputFile = Job.csOutputFile.Left(nDot);
}

//Job.csOutputFile += "_ReJig.m2v";
int nNopull = Job.csOutputFile.Find("nopull");
if ( nNopull != -1 )
Job.csOutputFile += ".mpv";
else
Job.csOutputFile += ".mpv.m2v";


If any body needs, I can compile Under Visual Studio 7 to test.


mike

[m1482]

@Nic:

I don't know if others had notice, that when transcoding with this tool, the transcoded file is somehow JUMPY. I carefully check it when running the ".m2v" file with Power DVD. I notice that every 3 or 4 seconds, the movie tend to jerk or jump. You can notice this specially when the camera is moving horizontally. First I noticed it when playing the movie in my Sony DVD player. I confirmed it, playing it through Power DVD, and using also other movie.

Both transcoded movie are NTSC, and the compression was adjusted to 58% and 60% respectively. Maybe the framerate is altered or changed, however no audio sync problem is detected...

[Nic]

It is a bug that was kind of noticed before, but obviously does not happen often, or we would have heard more about it...I need to get it to reoccur here for me to find the problem. What was the name and region of the DVD (or material) you were transcoding?

(if it happens often enough, maybe it would be easy enough for you to cut a bit out of the original source and send it to me?)

Cheers,
-Nic

[mmgrover]

@nic

I have a couple of movies that I also have this problem with.
One is StarWars Episode I. Another one that might be a problem is
"The Core" I recompressed the video with rejig and Horizontal
camera movement seems fuzzy and jerky. I just finished
running it through IC8 and I am going to compare it now.
I can send you a segment of these if you want?


mike

[DVDRFreak]

Originally posted by mmgrover
@nic

I have a couple of movies that I also have this problem with.
One is StarWars Episode I. Another one that might be a problem is
"The Core" I recompressed the video with rejig and Horizontal
camera movement seems fuzzy and jerky. I just finished
running it through IC8 and I am going to compare it now.
I can send you a segment of these if you want?


mike

Did also the core with rejig (movie only) I did not notice any jerky or fuzzy camera movement. Did the demuxing with DoItFast4U maybe that is the difference. Or it just happens at higher compression ratio's. Movie only I just needed a compression of 86% I believe.

[Nic]

Yup, that's an interesting point...anyone that gets jerkyness, could you try demuxing it to M2V first (with DVD Decrypter or whatever) and then ReJiging it. That way ill be able to pinpoint where the problem is...

If you can send me any problem clips that would be great mmgrover.

Cheers,
-Nic

[DVDRFreak]

@Nic

It's the engine I think. When you compress at high compression levels it happens to me also. Tested it on NARC. Used decryptor to rip first chapter compressed it at 77% it is Ok. Compressed it at 50% still it is Ok. Compressed it at 40% and the picture starts to go jumping around like it is in fast forward.

[m1482]

@Nic and DVDRFreak:

I discovered something...

First I demultiplex with VStrip and transcode again using 58 anf 70%. The problem persist!!! Then when I was concluding that the problem was with the transcoder I check the demux m2v file created by VStrip, and the problem begins there!!!

So, is the demuxing program use in Rejig the same or similar as VStrip???

[dragongodz]

just some thoughts following on from Int21's comments.

instead of tying variable requant to motion you could base it on original GOP size compared to average VBR for whole mpg/vob. that is if the original GOP is say 10% larger than the average then requant less than target value and if the GOP is 10% smaller than average requant a bit more. then use the values as a payback system to hit the target average.

this would need to be limited in payback amount to try and keep to target closely.asuming the original compression VBR ratios were based on quality for all motion etc this would help maintain quality with high motion scenes, especially using large compression amount.

[Nic]

@m1482: Well if you used normal mode (i.e. not IFO mode) then the demuxing is based on libmpeg2's demux code, which is supposedly flawless (along with VStrip which is also flawless?!). What is the name of that DVD, ill go buy it and look into it...

@DVDRFreak: Sounds like an integer is looping round or going negative when it shouldn't...looks like the ReQuant engine isn't flawless, i'd already patched little bits of it to make it less susceptible to errors (i.e. if you used it on neuron2's infamouse bitch.vob, it used to crash)

@dragongodz: Interesting points...which rises a question:

What is currently seen as the better/quickest way to determine the PSNR of two MPEG-2 clips? Running them through AviSynth? Because the difference in quality isn't that large i'll need to rely on increasing PSNR, rather than my eyes, to raise the quality, and make sure the changes I make actually improve the video etc
(I know PSNR isn't the best guide, but it is a rough easy well known one)

-Nic

[int 21h]

In regards to C++ version vs C version... here are my findings with a release build of the C++ version, a release build of Rejig, and the original executable provided by RB.

Release builds compiled with Intel Compiler 5.0


Test File - 67MB NTSC video file

requant (original) - 100% completed, 10 seconds, 6633 kb/sec.
requant (C++) - 100% completed, 5 seconds, 13267 kb/sec.


Test File - 407MB NTSC video file

requant (original) - 100% completed, 52 seconds, 7654 kb/sec.
requant (C++) - 100% completed, 28 seconds, 14215 kb/sec.
ReJig - 100% completed, 37 seconds


I didn't think of timing the 67MB file in Rejig until I started on the 407MB file.

[DVDRFreak]

Originally posted by int 21h
In regards to C++ version vs C version... here are my findings with a release build of the C++ version, a release build of Rejig, and the original executable provided by RB.

Release builds compiled with Intel Compiler 5.0


Test File - 67MB NTSC video file

requant (original) - 100% completed, 10 seconds, 6633 kb/sec.
requant (C++) - 100% completed, 5 seconds, 13267 kb/sec.


Test File - 407MB NTSC video file

requant (original) - 100% completed, 52 seconds, 7654 kb/sec.
requant (C++) - 100% completed, 28 seconds, 14215 kb/sec.
ReJig - 100% completed, 37 seconds


I didn't think of timing the 67MB file in Rejig until I started on the 407MB file.

I think you made a typo 52/28/37 seconds for a complete DVD is not possible. Speed of the C++ version looks promissing almost 50% faster then the original.

[Nic]

@int21: To do a fair test use the C++ source and enable and disable the the C++/C versions by changing which main function is run...the C source is the DVDRFreak version(using percentages)...and the C++ version is based on that. A problem with the C++ version is that it reports a different average bitrate to the C version even though they produce exactly the same file...didn't look into why.

The C++ version, which I spent a good hour or so converting, was slower in all tests, there is no reason for me to lie...seeing that I wasted my time creating the C++ version...

-Nic

ps
Just finished my code for converting sub streams to .sup files for use in IFOEdit...should be in the next release for tonight.

[Logiqx]

I'm not entirely sure whether this is a problem with ReJig (0.3b), DVD2AVI (1.77.3dg1.0.0rc2) or MPEG2DEC3 (1.77.3dg1.0.0) but somewhere along the line I seem to be gaining an extra frame of video in my test. It's not the most significant problem by any means but it is interesting as I'd expect no change at all.

I am using Minority Report (Region 2 - PAL) as a test and the original demuxed M2V shows 208899 frames in Bitrate Viewer. Also, loading the M2V into VirtualDub (1.5.7) via AviSynth+MPEG2DEC3 I can see the same number of frames (i.e. 208899).

After running the M2V through ReJig (reducing to 73%), Bitrate Viewer still reports 208899 frames but VirtualDub (via AviSynth+MPEG2DEC3) is showing 208900.

Does anyone have any ideas what is going on? I'm not completely sure whether it is ReJig or DVD2AVI/MPEG2DEC3 that is causing it. I can provide the D2V files if they are of any use in diagnosis.

Logiqx

[int 21h]

Originally posted by DVDRFreak
...I think you made a typo 52/28/37 seconds for a complete DVD is not possible...

I didn't typo... it was only a 407MB file...

[int 21h]

Originally posted by Nic
@int21: To do a fair test use the C++ source and enable and disable the the C++/C versions by changing which main function is run...the C source is the DVDRFreak version(using percentages)...and the C++ version is based on that. A problem with the C++ version is that it reports a different average bitrate to the C version even though they produce exactly the same file...didn't look into why.

The C++ version, which I spent a good hour or so converting, was slower in all tests, there is no reason for me to lie...seeing that I wasted my time creating the C++ version...

-Nic

ps
Just finished my code for converting sub streams to .sup files for use in IFOEdit...should be in the next release for tonight.

I wasn't saying you were lying, I was just curious :) You are right however, tested against the C source you included, its slower (Something like 12-15%?).


407MB NTSC Test File

100% completed, 35 seconds, 11372 kb/sec. (C++)

100% completed, 29 seconds, 13725 kb/sec. (C)

[DVDRFreak]

Originally posted by int 21h
I didn't typo... it was only a 407MB file...

I see now, sorry did not read correct.:o

[RB]

Originally posted by Nic
@DVDRFreak: Sounds like an integer is looping round or going negative when it shouldn't
Nic, maybe you want to review the ui64ToDouble() function I added to the original code. It looks OK to me, though.

I added it because at some points the code assigned a unsigned __int64 to a double and VC++ 6 does not support this conversion. IIRC newer versions can so you can probably get rid of the function.

[Nic]

@RB: Your ui64ToDouble() looks absolutely fine. However, have you got Service Pack 5 installed on your vc6 ? Because mine can go from unsigned __int64 -> double with no problems ? So ill remove your function and let vc6 do it for me, which might be slightly quicker...

[RB]

Yes, I've got SP5 installed. Maybe it's because I have only the Standard Edition.

[digitalman]

I know everybody knows this, or there wouldn't be ten pages of posts, but the results of this program are amazing. I like to put two movies per 1 DVD. So I have been setting the compression at 62-64%, and their is no pixellation, even on fast moving scenes. There may be some loss in color, but nothing the average layman would notice. I only had a class years ago in C++ for DOS, so I don't know how NIC and you guys do your magic, but whatever it is, keep it up.
Thanks to:
Makira for M2VRequantiser (on which the compression code is based)
DVDRFreak, RB, Int 21h, mmgrover, jsoto (AC3 Delay correction code), maven (VStrip),
Videolan.org (libdvdcss DVD Authentication)
And everyone else at Doom9.
-from NIC's readme file in the rejig.zip.

[int 21h]

Originally posted by Nic
@RB: Your ui64ToDouble() looks absolutely fine. However, have you got Service Pack 5 installed on your vc6 ? Because mine can go from unsigned __int64 -> double with no problems ? So ill remove your function and let vc6 do it for me, which might be slightly quicker...

Can you do that with both compilers? It seems to me when I'm compiling code for work, the Intel compiler is a little more relaxed on casting than the Microsoft compiler...

[DVDRFreak]

@Nic

Was testing the OO version and noticed something interesting. Used Rejig 0.3b and the OO version you made for requant. Compiled both without the conversion function ui64ToDouble().

Tested both on the first seen of NARC (700MB at 40%). The Rejig 0.3b version came out 284MB the OO version 302MB.

But now the interesting part. The Rejig version has the jumping fastforward problem. The OO version is working OK.

Edit:
Another thing is that the OO version will not compress the file more then 44%. If I choose anything lower it just comes out 44%. Maybe this is the difference.

Edit3:
Tested ReJig engine 0.3b / Requant original and both engines allows me to compress the video until a maximum of 39%. With the motion bug of course.

Edit2:
Performance related the OO version takes 90 seconds for 677MB compared to 83 seconds for the requant I compiled in the beginning.

Edit4:
Figured it out why the OO did not compress as much as the ReJig engine. Forget to enable option REMOVE_BYTE_STUFFING. Removed some compiler warning and enabling this option the time to comprees at 40% is now 86 seconds compared to 83 for my original release.
Only problem now the motion bug exists also in the OO version. So the bug is in the REMOVE_BYTE_STUFFING option.

The remove_byte_stuffing option is used for CBR streams since we are transcoding DVD's and most of them are VBR we should copile without this option.

Edit5/6:
Tried disabling the option and the quality realy sucks. So enabled it and enhanced check to check for 8 zero bytes instead of 6. Seems to work for NARC and the quality is much better again.

Disabling the option caused the first scene in NARC to be as blocky or even worse as the rest of the transcoders at 71%. Checking on 8 0 bytes solved the issue for the first scene. Need to do some more testing.

By the way enabling the option makes the OO code almost as fast as the C code 537 second OO code against 521 seconds C code for complete movie. So that is 3% this speed difference could be due to the fact that I now check for 8 zero bytes.

[Nic]

@DVDRFreak: Excellent work, that makes alot of sense. I'll look into that too. I only tested it once on a 2 minute file and they both outputted exactly the same file, so I assumed all was ok.Doh!
It will be great if that fixes the "jerky" bug (I havent been able to replicate it yet). Let me know of any progress or what you think should be changed :)

@int21h: Yup, that's with normal VC6 SP5 compiler, on my Athlon 1800xp it takes ages to compile with the intel compiler so for testing I always use vc6 (infact with the last version of ReJig released that was done with just vc6 as the intel one was taking too long ;) )

Cheers,
-Nic

[DVDRFreak]

Originally posted by Nic
@DVDRFreak: Excellent work, that makes alot of sense. I'll look into that too. I only tested it once on a 2 minute file and they both outputted exactly the same file, so I assumed all was ok.Doh!
It will be great if that fixes the "jerky" bug (I havent been able to replicate it yet). Let me know of any progress or what you think should be changed :)

Cheers,
-Nic

Will mail you my version of the OO code so you can take a look what I did change. The remove_byte_stuffing stuff is not very clear to me so I did a trial and error. Changed the number of 0 bytes to check before removing one 0 byte. Don't know if this works for all the movies it did for NARC.

[int 21h]

After profiling...

27% of CPU time is spent in Flush_Bits
21% of CPU time is spent in Put_Bits
10% of CPU time is spent in Refill_Bits
8% of CPU time is spent in putAC
6% of CPU time is spent in get_intra_block_B15

The rest is divided among the other functions...

What does this tell us? I interpet that to mean I/O is our main bottleneck here... I profiled running across both of my IDE channels with source/destination.

Comments?

[Vapor]

Just a quick test that might be handy, not IO or CPU related as far as I can see.

Rejig 0.3b - 70% ratio - ntsc/demuxed/stripped m2v from a dvd vob.
Test file: 834mb (15.1 mbytes/sec read)
Out file: 584mb (10.6 mbytes/sec write)
Time taken: 55 secs

Cpu usage: 30-33% (dual mp2900 - 50% = 1 whole cpu)
Source file on: 15000rpm scsi drive
Out file to: 2x10000rpm scsi drives (raid0)

Normal disk transfers (read/writing scenario as above) are usually around 50mbytes/sec sustained.

Great tool btw, keep up the good work.

[PurpleMan]

Dear NIC

For this software to be efficiently used for DVD backup purposes, there is a very important feature that noone has mentioned yet.
When inputting a VOB file, have the program list the VOBID/CELLID's inside of it (you can use vstrip sourcecodes I believe) and make it possible to transcode just a single cell or ID. Or rather, the ability to specify a different compression ratio for each CELL/ID.

If you do that, we can practically write a new DVD Shrink (easily too).

I'm here if you have any questions regarding my suggestion and/or future development of the abovementioned.

PurpleMan

[int 21h]

Originally posted by Vapor
Just a quick test that might be handy, not IO or CPU related as far as I can see.

Rejig 0.3b - 70% ratio - ntsc/demuxed/stripped m2v from a dvd vob.
Test file: 834mb (15.1 mbytes/sec read)
Out file: 584mb (10.6 mbytes/sec write)
Time taken: 55 secs

Cpu usage: 30-33% (dual mp2900 - 50% = 1 whole cpu)
Source file on: 15000rpm scsi drive
Out file to: 2x10000rpm scsi drives (raid0)

Normal disk transfers (read/writing scenario as above) are usually around 50mbytes/sec sustained.

Great tool btw, keep up the good work.

I get 13 mbytes per second using IDE, so it seems to me the actual I/O processes themselves are where the bottleneck is, not the devices, I should have made that more clear, sorry.