QMatOp for DVD-RB : QMat Optimization

[SAPSTAR]

Originally posted by genericuser
I am getting ready to try version .11. I have downloaded it and installed it. I am not doing cartoons. I am wondering a couple of things:

Do I have to mess with the matrix setting used in dvd-rb itself? I just have it set to the default matrix.

I am using cce 2.7 and I am planning on turing on the QATMAT option in cce too. I think people in this thread have said that was ok as well.

If I read the thread correctly, I just have to set up my .ini file to point to the dvd folder that gets created, set the sample rate and precision and such.

I think that using .11 I "prepare" in dvd-rb and everything flows from that. Is that correct thinking?

Thanks!

Adapt the QMapOp.ini file to your DVD RB work directory, activate or deactivate what you wish(Splitperiod, tolerance, ...), then open DVD-RB without One-Click mode (in the next release I`ll check that), deselect/select all your stuff. You can choose any setting you wish in DVD-RB. Then click Prepare, start QMatOp....and that`s it.

I think you fully understood what to do.

[Axlemar]

Originally posted by SAPSTAR
:cool: Thank you !!! I'm to release a new version with a function to auto adjust the Quality Prec for each cell, of course, it's useful only with CCE.....

Auto prec? Sounds like interesting stuff. Well, I have compared the different versions for a while and it does seems like the newest one is the best so far. Please continue to add or use any features that you think will improve the quality (even though it seems good now). Thanks.

[Carpo]

prob a stupid question but - will this work with MS java - i have a pc here that i use for compiling sources and it has C++ 6.0 on it which installs ms java machine

also any updates on new version of qmatop ? :)

[SAPSTAR]

Originally posted by Carpo
prob a stupid question but - will this work with MS java - i have a pc here that i use for compiling sources and it has C++ 6.0 on it which installs ms java machine

also any updates on new version of qmatop ? :)
About MS Java....well, Frankly, I have no idea ! About, the next version with auto Qprec + auto VbrBias, I should release it tonight, if my own test is successful, and if I receive a feedback from FishMan0919 who tested it on Bandits.

[SAPSTAR]

Read carefully the RTF file, as three new parameters were introduced :
QualityPrecAuto
CCESP26X
VbrBiasAuto

CCESP26X must only be activated if you are using CCE SP 2.6x, those parameters are useless if you are not using CCE.
Let me know the results...I really need some feedbacks in order to improve QMatOp if needed.

[Hostile_Monk]

SAPSTAR:

I'd like to personally thank you for the work you've done. I've been following this project for a few days now; I've used v.11 on a lot of my movies with simply amazing results. Auto quality prec and vbr_bias sounds awesome, I can't wait to try it out! Is there any specific way you'd like me to report my results, other then viewing with my naked eye and posting if it's significantly better? I know one dvd9 that has given me some trouble, Pulp Fiction, I can't get it to emulate the original(quality wise) no matter what I try, this will be a good test! Once again, thanks for all the good work.

[Carpo]

would the vbrbias in qmatop conflict or cause issues with the same setting in dvd-rb?

atm in dvd-rb i have vbrbias as 0 - was told this gives better results - and from the 20 films i have done i have never had any probs

if in doubt i can do a test encode - just it would save me time :)

[Fishman0919]

QMatOp will change your setting of VBR Bias and Quality Prec no matter what you have them set to in DVD-RB. It will Optimize them for each cell.

[Carpo]

will have to do a few test encodes to see if it helps - although the prog to date has helped so i cant see it making things worse :P

[SAPSTAR]

@Hostile_Monk : Thank you for the support ! :D And please let me know about the result of your test with Pulp Fiction, I would be interested to know.

@Carpo : Fishman0919 is right (as usual ;) )....whatever your settings are in DVD-RB, they will be changed, if you activated the the function in QMatOp of course....FYI : 0 as Vbr Bias is the settings of all other encoders (HC, ...). So if you really want to use the full power of CCE, just play with Vbr Bias :). By the way I heard that hank315 was to implement the equivalent in HC. Anyway, thank you for your support and let me know your results, I'm really interested in them, as you are used to another kind of value.....
@FishMan0919 : Thank you for all the testings you did ... and the good feedbacks.

[raddygast]

So for regular NTSC movies, what split is recommended? 15? or 12?

I don't know if my stuff is interlaced or not -- but it's just regular studio movies. I assume non-interlaced?

[hutch1711]

Originally posted by SAPSTAR
Rule of thumb : FrameRate / 2 is a good ratio
=> PAL 12 should be good
=> NTSC 15 or 12 depending on interlaced or non interlaced DVD.

This should help.

[raddygast]

Yeah, I had read that already. Was just looking for someone to tell me whether to use 15 or not, since QMatOP defaults to 2/12 anyway.

Hey, how long is it taking you guys? I'm running QMatOP right now with 2/12 settings, and it seems to be only about 25% of the way into disc 1 of Fellowship of the Ring extended edition.

It's been running an hour now. With settings of 2% tolerance, 12 split, should it really be taking like 4 hours on something under 1h30m of video?

My computer isn't crazy fast, but it's not garbage either. It's a Pentium 4 2.4 GHz. Am I doing something wrong, or is this QMatOP only for the totally obsessive? It looks like it's going to take as long as or even longer than the encode itself!

[Fishman0919]

If you are doing a Movie (something that has 23.976 fps).... 12 should work great...if you are doing a season disc (some kind of TV show (29.97 fps or something like that))) 15 would be better.


Edit: I did Lord of the Rings - The Two Towers with 12fs ans 2%t and it took about 1 hour on my AMD64 3800+ OT to about 4000+

[raddygast]

Ok cool. So 12 for non-interlaced (movies) and 15 for interlaced (TV), right? Or maybe I got that backwards regarding the interlaced status.

By the way -- it's not supposed to take this long with 2-12, is it? I'm thinking it's gonna take over 2 hours to complete the run for Fellowship disc 1. At this rate I'm gonna have to think of QMatOP as something that I only run on really really special sources.

Also, a suggestion for SAPSTAR:

For future releases, I wonder if you can build in some integration with DVD-RB's ini file so that QMatOP can be used in batch mode. That may not be too easy to do, though.

The next best thing would be for QMatOP to have its own "batch" protocol. But the first step there is to be able to define multiple working dirs. I use DVD-RB in batch mode so my working-dir is different for each project. Currently the way I do things is I set up each project and run a prepare on it, then tweak settings in RBOpt or whatever. When all the projects have been prepared, I simply add each project file to a batch, and then run the batch from inside DVD-RB. That encodes and rebuilds each DVD before moving onto the next. It works great.

At the moment I see no way to work that in with QMatOP, though. And especially because it seems to take an eternity to run, automation is really advantageous. You made a great first step with the AutoEncode and AutoRebuild options, but I'm wondering if you can think of a way to add batches too.

The way RB does it, I believe, is simply through project files. Each project file seems to reset the rebuilder.ini. Maybe there's a way that QMatOP could poll that file?

But I don't see a way to build QMatOP into RB's batches; that would require DVD-RB to have an option to call QMatOP externally.

So the best thing I can think of is to abandon DVD-RB batch mode altogether. Instead, run prepare on each and every movie in DVD-RB, setting up each movie's working-dir. Then have some kind of working-dir list in QmatOP.ini, but with the autoencode and autorebuild options on. Then after each optimize, it would trigger RB to encode, then rebuild, and after that was done it would move onto the next film and start the optimize.

Still, I think RB might need its own project files, which is a bit of a nuisance I guess.

[Fishman0919]

Ok cool. So 12 for non-interlaced (movies) and 15 for interlaced (TV), right? Or maybe I got that backwards regarding the interlaced status.

Yes ;)

[SAPSTAR]

Originally posted by raddygast
Ok cool. So 12 for non-interlaced (movies) and 15 for interlaced (TV), right? Or maybe I got that backwards regarding the interlaced status...

The rule of thumb is FrameRate/2...So 12 will work in any case (PAL, NTSC, interlaced, non-interlaced) but it's not fully time-optimized...


By the way -- it's not supposed to take this long with 2-12, is it? I'm thinking it's gonna take over 2 hours to complete the run for Fellowship disc 1. At this rate I'm gonna have to think of QMatOP as something that I only run on really really special sources.


It can be long, on my Athlon XP 2600, it can run for 2 hours on regular movies....but if you want the best quality on highly compressed movies, the time is not really important, is it ?


The next best thing would be for QMatOP to have its own "batch" protocol. But the first step there is to be able to define multiple working dirs.

I'll think about it....There may be a way, my only issue is I don't know how to locate DVD-RB, maybe another parameter ?

[raddygast]

If you say framerate/2, then why does interlace vs. non-interlaced play a role? I thought NTSC was always 30 fps. Maybe that's just TV sources. I think I read somewhere about film being 24.

About locating RB, yeah.... just use another parameter. But the problem is that RB isn't going to call QMatOP, so maybe QMatOP needs to be running the show. Is there a way you can get QMatOP to force DVD-RB to load another project? I'm wondering how you get it to do the encode and rebuild in the first place.

About the time, yeah -- I guess it's not too important as long as I can automate things. But I was thinking about running QMatOP as part of the preparation work, and for that it takes too long, unless I want to encode films over several computer sit-down sessions. :)

[Axlemar]

Does QMatOP take into account the different values used to calculate the bias and prec for different CCE versions? If I don't use the 2.6x switch will it work correctly for the newest 2.7?

[Fishman0919]

Yes, if you are using CCE SP 2.70 set the switch to CCESP26X=0

[Trevorontario]

Yes, if you are using CCE SP 2.70 set the switch to CCESP26X=0

If you are using CCE SP 2.50 should the switch be set to zero as well?

[SAPSTAR]

Originally posted by raddygast
If you say framerate/2, then why does interlace vs. non-interlaced play a role? I thought NTSC was always 30 fps.
Wrong...Interlaced movies are 23.976fps.....

[SAPSTAR]

Originally posted by Trevorontario
If you are using CCE SP 2.50 should the switch be set to zero as well? Yep !

[Carpo]

2 things

1 - is CG classed as animation - cus im going to do a backup of final fantasy and am not too sure

2 - when i have used dvd2avi to check if a source is progressive or interlaced it has given 29.97 (something like that) are you saying that its wrong and its really 23 - if so i will need to redo a film with qmatop

i know i have been asking many stupid questions but when re-encoding it takes time and its always best to ask questions than to waste time :P


v0.12 kicks ass btw - extra 3 options really do work well (did on the film i tested it on anyway :))

[raddygast]

So wouldn't it be better to use 2/15 for most movies (except interlaced ones), and 2/12 for interlaced sources (like most TV)?

It's just that I think Fishman gave the opposite recommendation.

[Carpo]

from what i understand for pal use 2 - 12

and for ntsc use 2 - 15

DGMPGDec - show most of the interlaced stuff i have used as 29.97 - so thats why im a bit confused atm

[onesoul]

Wrong...Interlaced movies are 23.976fps..... I wouldn't call it interlaced but progressive film instead which with use of pulldown flags, interleaves repeated fields to achieve 29,97fps.
DGMPGDec - show most of the interlaced stuff i have used as 29.97 - so thats why im a bit confused atm If it reports 29,97 interlaced then it is almost certain pure interlaced ntsc.

[SAPSTAR]

@carpo,raddygast : There is no bad choice....It's just a question of time passed to split, just use 2-12, an it will work in any case, when I say work I mean : 25/12 ~= 2, 23.976/12 ~= 2, 29.97/12 ~= 2.5. So the only diff is in the last case...you will have more info, the result will be more accurate.... It's just a question of time/quality. more time => more quality....and there is a good compromise at Framerate/2.....that's it.

[Carpo]

reguarding my 1st Q - do you have to use the animation flag on CG stuff like final fantasy ?

also i use 2 - 12 for pal and 2 /15 for ntsc - and have never had any probs so i'll stick with them :P

[SAPSTAR]

Originally posted by Carpo
reguarding my 1st Q - do you have to use the animation flag on CG stuff like final fantasy ?

also i use 2 - 12 for pal and 2 /15 for ntsc - and have never had any probs so i'll stick with them :P

Well, about CG, frankly I'm not sure as I never really tested them....But if I were you, I would try with Animation flag first....as when I designed it, I also thought about the CG stuff, without being able to test it.....

[Fishman0919]

I did Titan AE (a pretty good looking CG)with the animation flag on and it looked great

[Carpo]

what does the ccd26x setting actually do? and is there a poss that you could put an option for cce 2.7 (when all the bugs are ironed out of course) :p

[SAPSTAR]

Originally posted by Carpo
what does the ccd26x setting actually do? and is there a poss that you could put an option for cce 2.7 (when all the bugs are ironed out of course) :p
:sly: Hmmm you should read the RTF file coming with each release....In CCE SP 2.6x, the range for Quality Prec was 0-64 instead of 0-100 like the other CCE versions....That's why I had to add this parameter, you have to set it ONLY if you're using CCE SP 2.6x...any other CCE version will work with CCESP26X=0.

[Carpo]

still trying to work out if i should use (buy as well) cce 2.6 or cce 2.7 - (seems that version is hit or miss tho) :mad:

[alfixdvd]

I have Sun JRE 1.5.0_02 installed, and QmatOp 012, and Dvd_Rb 0.85 free.

I get the following error:


C:\DvdTools\QMatOp>QMatOP.exe
Execing cmd.exe /C .\DVDRBInterface.exe WAITPREPARE "C:\DvdTools\QMatOp\.\WAITDV
DRB"
Execing cmd.exe /C del /Q .\frames\Intra*.ppm
No se encuentra C:\DvdTools\QMatOp\frames\Intra*.ppm
Execing cmd.exe /C .\avs2avi .\temp.avs .\avs2avi.log -o l -c MSVC

AVS2AVI v1.31 (c) 2002,2003 Christophe PARIS (christophe.paris@free.fr)
Contributions by: int21h
Additional code (c) 2002,2003 David Leatherdale (dave@leatherdale.net)

Source:
* Filename: .\temp.avs
* Bit depth: 32
* FOURCC: none (RGB)
* Frames: 240
* Resolution: 564x80
Compressor:
* Name: Microsoft Video 1
* FOURCC: MSVC

Pass 1/1 frame 0/240, 0.00fps, ETA: , size: 0.00MB
Pass 1/1 frame 88/240, 295.30fps, ETA: 00:00:00.514, size: 0.00MB
Pass 1/1 frame 176/240, 294.31fps, ETA: 00:00:00.217, size: 0.00MB

* Frames: 0
* Key frames: 240
* Size: 0.000MB
Pass 1/1, compression time 00:00:00.897 (0.00fps).
Finished
Execing cmd.exe /C del .\temp.avs
Execing cmd.exe /C del /Q .\frames\Inter*.ppm
No se encuentra C:\DvdTools\QMatOp\frames\Inter*.ppm
Execing cmd.exe /C .\avs2avi .\temp.avs .\avs2avi.log -o l -c MSVC

AVS2AVI v1.31 (c) 2002,2003 Christophe PARIS (christophe.paris@free.fr)
Contributions by: int21h
Additional code (c) 2002,2003 David Leatherdale (dave@leatherdale.net)

Source:
* Filename: .\temp.avs
* Bit depth: 32
* FOURCC: none (RGB)
* Frames: 240
* Resolution: 564x80
Compressor:
* Name: Microsoft Video 1
* FOURCC: MSVC

Pass 1/1 frame 0/240, 0.00fps, ETA: , size: 0.00MB
Pass 1/1 frame 88/240, 295.30fps, ETA: 00:00:00.514, size: 0.00MB
Pass 1/1 frame 175/240, 290.97fps, ETA: 00:00:00.223, size: 0.00MB

* Frames: 0
* Key frames: 240
* Size: 0.000MB
Pass 1/1, compression time 00:00:00.897 (0.00fps).
Finished
Execing cmd.exe /C del .\temp.avs
Execing cmd.exe /C del /Q .\intra\*.* > NUL
Execing cmd.exe /C del /Q .\inter\*.* > NUL
Execing cmd.exe /C del /Q .\pics\*.* > NUL
Execing cmd.exe /C .\Calculate.bat 34 23
Exception in thread "main" java.lang.ArithmeticException: / by zero
at QMatOp.AverageMatrix(QMatOp.java:529)
at QMatOp.ECLCalculate(QMatOp.java:289)
at QMatOp.main(QMatOp.java:61)

C:\DvdTools\QMatOp>

What's wrong ?

Where is calculate.bat ?

Please help me.

Best regards

[gilligan2]

Something is wrong before QmatOp.Look at your resolution,564x80. You have a problem in you avs script or somewhere else that may or may not be your main problem but it should read 720x480 or 720x576 or another DVD Standard size. Check your AVS script in notepad and see what is going on.

g:)

[SAPSTAR]

@gilligan2 : :thanks: I didn't see it....I was to read my code! there must be something wrong in the AVS....DVD-RB wouldn't have work either !

@alfixdvd : gilligan2 found your pb !!! Check your DVD-RB config...then use QMatOp, once everything is working correctly without QMatOp.

@all : I'm currently testing (with Fishman0919 as beta tester) a new function of Auto GOP Structure.....

[alfixdvd]

@sapstar and @gilligan2

Thanks.

When I edit an avs file, I see that I make a mistake: I missed to specify the path to Dgdecode.dll at DVD-RB config.

I'm sorry

Best regards.

[gilligan2]

No problem guys,glad to help !!

g:)

[TheSeeker]

Sorry if this has been answered already I, admittedly, did not read the whole thread. But what makes this tool different than, say, using CCE's AQM features? The fact that it changes QMat's on a per cell basis?

Im always looking for ways to tweak the quality of my encodes. And this looks like the next great add on for DVD RB.

[Carpo]

taken from the RTF file


I – Purpose
This software is designed to optimize the Quantization Matrix of each cell in the ECL file generated by DVD-Rebuilder. It's using dctune2.0 from the NASA, and also avs2avi from Christophe PARIS in order to sample each cell and determine what is the best matrix according to the current average and max bitrate.



III – Usage
The QMatOp.ini file must be edited according to the D2VAVS directory you are using.
Some other parameters exist:
tolerance: it controls the error tolerance for each frame size after compression (see IV – How it works). The default value is 5 (meaning 5%)
framesplitperiod: it controls the period used for splitting each cell into analysed frames. The default value is 30 (meaning every 30 seconds)
AutoEncode: it controls if you want QMatOp to start the encode after the processing.(You have to keep DVD-RB open)
AutoRebuild: it controls if you want QMatOp to start the rebuild after the processing.(You have to keep DVD-RB open)
Animation: it controls whether the movie to encode is an animation or not, this function is still being tested. I would recommend to test first without it. The default value is 0, meaning not activated.
RangeLimit: It allows you to change the range for the matrices, with CCE must be 127, which is also the default value; with HC you can use 255 at your own risks.
QualityPrecAuto: It activates or deactivates the auto Quality prec function, useless if you are not using CCE.
CCESP26X: To be activated, if QualityPrecAuto is activated and if you are using CCE SP 2.6x, otherwise must be deactivated.
VbrBiasAuto: It activates or deactivates the auto VBR Bias function, useless if you are not using CCE.
IV – How it works (Quick explanation)
The REBUILDER.ECL is read, for each cell the following is done:
- Extraction of N intra frames in PPM format (raw format) into the Frames directory
- Extraction of N inter frames in PPM format (raw format) into the Frames directory
- Calculation of the intra matrices of each frame, according to the max and average bitrate of the current cell. (Intra directory for the matrix file, Pics directory for the compressed frames)
- Calculation of the inter matrices of each frame. (Inter directory for the matrix file, Pics directory for the compressed frames)
- Processing of the average intra and inter matrices (it also takes into account the initially chosen matrix in the ECL file, if any)
- Normalization of the previous result. (Matrices are moved if not compliant to the MPEG2 standard) + not compliant to the CCE specs
Then the REBUILDER.ECL file updated according to the results. A copy of the ECL file is created REBUILDER.BAK

[jdobbs]

Originally posted by Carpo
still trying to work out if i should use (buy as well) cce 2.6 or cce 2.7 - (seems that version is hit or miss tho) :mad: I get random reports of problems with 2.70 (missing M2V files)... but honestly I've only had it fail once for me on an HBO whitescreen section. I then tweaked the calculation of the maximum bitrate a little and it hasn't reappeared. I've been using it on a lot of discs, too, because I want to see if I can make it happen... but it just keeps doing fine. It's possible there may be a questionable hack out there somewhere. I really can't explain it.

[SAPSTAR]

Originally posted by TheSeeker
Sorry if this has been answered already I, admittedly, did not read the whole thread. But what makes this tool different than, say, using CCE's AQM features? The fact that it changes QMat's on a per cell basis?

Im always looking for ways to tweak the quality of my encodes. And this looks like the next great add on for DVD RB.
AQM is just an internal function from CCE which adjust a little bit the QMat from a pure compression point of view, but if the given matrix is not good for what your are encoding the result will be mixed. What QMatOp does, is calculating a VISUALLY optimum matrix for each cell. It's using DCTune from the NASA in order to keep the best visual quality and compresses as much possible. And it's working with other encoders than CCE SP 2.7 !!

@By the way, yesterday I did an experiment with HC, and I chose the range value 0-255 in QMatOp...the quality was really good !!! Better than CCE SP 2.70. (It was a an interlaced movie, with a very high compression level so low bitrate)

[Carpo]

Originally posted by jdobbs
I get random reports of problems with 2.70 (missing M2V files)... but honestly I've only had it fail once for me on an HBO whitescreen section. I then tweaked the calculation of the maximum bitrate a little and it hasn't reappeared. I've been using it on a lot of discs, too, because I want to see if I can make it happen... but it just keeps doing fine. It's possible there may be a questionable hack out there somewhere. I really can't explain it.

ye i had it once with cce 2.7 - so thats why i am umming and ahhing about which to get - still undecided yet :/

[raddygast]

Wait a second. I've never heard of a case where CCE was *worse* than any other encoder. This is really alarming. I thought the problems with CCE had to do with obscure bugs (occasionally) -- not with quality.

Isn't CCE the undisputed king of quality?

[SAPSTAR]

Originally posted by raddygast
Wait a second. I've never heard of a case where CCE was *worse* than any other encoder. This is really alarming. I thought the problems with CCE had to do with obscure bugs (occasionally) -- not with quality.

Isn't CCE the undisputed king of quality?
Here you are wrong...on very low bitrates, it is known that CCE SP is not so good...interlaced materials are also a big deal for it, for example Procoder is better on Interlaced stuff than mant other encoders....Don't think that CCE is the ultimate encoder.....

[TheSeeker]

If you take into account a quality/speed ratio it IS the king. And for the most part, for most people CCE is usually the best bet. Great quality on the most sources, with great speed. Procoder is pretty much ONLY good for interlaced low bitrate sources, yet HC is starting to encroach on CCE quality territory for sure.

[Fishman0919]

I think CCE is a B+/A- encoder... it not the best at everthing but it real good all around (with some tweaking you can get it pretty good on most sources). For most people who want speed and quality you can't go wrong with CCE.... but to to say CCE the undisputed king of quality is really incorrect.

[raddygast]

Alright. So what is the undisputed king of quality? Is it this HC thing I've heard talk about? I thought that was free?

I was under the impression that CCE was what was used by the big-time studios for DVD authoring. Certainly the Pro version of CCE costs enough to suggest that it is a "big league" encoder.

[SAPSTAR]

Originally posted by raddygast
Alright. So what is the undisputed king of quality? Is it this HC thing I've heard talk about? I thought that was free?

I was under the impression that CCE was what was used by the big-time studios for DVD authoring. Certainly the Pro version of CCE costs enough to suggest that it is a "big league" encoder.
In fact each encoder has its own strength...If you're looking for the best ratio speed/quality then CCE is your king ! But if speed is not a problem for you, but just the quality, then depending on the source you may choose a different encoder.
[EDIT] I think that HC is the king for the ratio Quality/price !!!
[EDIT] Maybe we should talk about this in another thread, as this one was originally for QMatOp...not for the eternal fight of CCE vs others. :thanks: