AutoQMatEnc - a new free MPEG2 encoder

[SAPSTAR]

AutoQMatEnc 0.28b - SAPSTAR - WWW.Doom9.org
MPEG2 (Video Only) Encoder

Characteristics :

- Use same ECL Files than CCE SP Trial
- Read any AVS script by using directly avisynth.dll
- Built-In advanced QMatOp technology - Activated by adding adjust_q_matrix=1 in ECL
or CCEAQM=1 in the REBUILDER.INI file for DVDRB users.
- MultiPass enabled - Above 4 passes is close from useless but can be performed
- Part of CCE Simple Settings (Animation/CG/Natural) taken into account
Recommended for Anime (Animation) and old movies (Natural)
- QMatOp phase not reprocessed but detected if encode cancelled once finished.
- Pass 1 not reprocessed if already done.
- MPEG2 Encoder using a part of avcodec.dll : libavcodec is released under the LPGP.
You can get a copy of the LGPL at http://www.gnu.org/copyleft/lesser.txt
and download the source from the ffmpeg homepage http://ffmpeg.sourceforge.net/download.php.

Installation/Configuration :
Download it from : The web Site provided by BadServo : http://www.autoqmatenc.com/
Install it anywhere then use AutoQMatEnc.exe instead of CCE SP Trial.

DVDRB users : just make DVDRB point to it instead of CCE SP.

NON DVDRB users : To create your own ECL file you can use CCE SP Trial with ECLCCE.

Greetings :
- FishMan0919 for all his support, ideas, help in debugging, tests.
- BadServo for his tests, ideas, comparisons and file hosting.
- Axlemar for his very detailed tests on high speed animations.
- Carpo for his tests and ideas.
- Doom9 for his help in writing properly the licensing text.

Changes History:
0.28b - 2005/08/05
- Corrected a major bug in the credits "type" scenes
- Corrected the rate control, 2 passes are enough for 90% of the scenes,
3/4 passes for the rest.
- Improved the sizing - Should be ok now.
For DVDRB Users, the default target size for DVDRB is 4.27Gb,
I suggest CCETargetSectors=2260000 or 2280000 when using AQE.
- Recompiled avcodec, I'm now using the very last version of it...
(some overflows corrections were brought with it)
0.27b - 2005/07/20
- Added a low motion detector to tweak the rate control
- Improved the rate control
- Updated the averaging in QMatOp
- Corrected an error on very big files for the Nth path(N>2)
- Minor tweakings and errors.
0.26b - 2005/07/13
- Corrected the PATH variable, added a ; at the end.
- Corrected the rate control once again...a new approach was implemented.
- Corrected a bug when the rate control was totally wrong,
the Nth pass was ending asking for pressing a key.

[Rockas]

Great!
Will it do "one pass vbr" encodes?

[jptheripper]

just to make sure, is this a seperate encoder (i.e. dont need cce) that does the custom matrices of QMatOp internally? if so happy to check out with my nastiest source real soon.

[SAPSTAR]

@Rockas
Currently no...but could be a future dev. :)
@jptheripper
It's real encoder ... just using the same format than CCE for an easy integration with DVDRB

[Fishman0919]

just to make sure, is this a seperate encoder (i.e. dont need cce) that does the custom matrices of QMatOp internally? if so happy to check out with my nastiest source real soon.

Yes, just point to AutoQMatEnc instead of CCE SP (new) and put CCEAQM=1 in the REBUILDER.INI file under options

[Rockas]

Sorry... my first test... didn't work.
Rebuilder says

"ERROR: File V0100000(...) shpws completed, but no .m2v file exists. Aborted"

[Carpo]

must be using the cce 2.7 engine ;D

[Fishman0919]

Sorry... my first test... didn't work.
Rebuilder says

"ERROR: File V0100000(...) shpws completed, but no .m2v file exists. Aborted"


Yes Rockas, I get the erorr sometimes too... if you have CCE Sp 2.70 (or any CCE Sp 2.6#) point to it and start the encoding, then stop dvd-rb befrore cce finishes the segment... repoint to AQE and it should work.

I was just talking to SAPSTAR about this and we are tring to work it out... but this is a temp solution

[Hostile_Monk]

Cool that was finally released to the public; it has looked great on all of my sources, including animation!

[BadServo]

Hmmm...how odd. I never encountered this error. However, I haven't tried an encode with 0.20a yet. I expect it's s minor bug.

[Rockas]

I've done what Fishman0919 suggested and its working fine now :)

[SAPSTAR]

After testing and talking with FishMan...it may come from the length and/or the presence of spaces in the AQE path...try installing it in C:\AutoQMatEnc for ex....it seems to avoid the pb...I'll try to remove this stupid bug !

[jptheripper]

am i pointing to cce sp, cce sp (trial), or cce (2.5)

[SAPSTAR]

am i pointing to cce sp, cce sp (trial), or cce (2.5)
In setup you point to CCE SP(new), then you use CCE SP Trial mode in the CCE settings.

[jptheripper]

okay got it running, says
interlaced movie
/[mpeg2video @ 00676E18]rc buffer underflow
\
Done

is this okay?

[Fishman0919]

okay got it running, says
interlaced movie
/[mpeg2video @ 00676E18]rc buffer underflow
\
Done

is this okay?

Yes, the first frames always underflow and is corrected in the next pass

[jptheripper]

as a side note, i love the ratio size to expected size, although i wish maybe it would write this to a log file or something?

[jptheripper]

odd, i keep crashing on pass 6, gonna lower it to 4 pass

[Rockas]

@SAPSTAR
Man... you really got to make it faster :D

How long does it take (average) to encode a DVD? - I know, I know, it depends of a lot of things lol... but an answer like... the movie X took me Y hours and I have a pentium 166 (lol) and 8MB of memory (double lol).

[jptheripper]

oh another thing, can i use filters (i.e. undot and deen) with it

[Fishman0919]

@SAPSTAR
Man... you really got to make it faster :D

How long does it take (average) to encode a DVD? - I know, I know, it depends of a lot of things lol... but an answer like... the movie X took me Y hours and I have a pentium 166 (lol) and 8MB of memory (double lol).

ah, The Hulk movies only on an AMD 64 3700+ took 9 hrs with 4 passes and QMatOp on.... same movie without QMatOp took about 3 hrs with 4 passes

[BadServo]

Ah, your killing me Rockas. I deleted all my DVD-RB logs from when I ran the comparison encodes literally minutes before this post. However, after going back over logs of conversations with Sapstar, I did find one comment I made about speed.

Encoding a 2-hour peice of material took AQE 761 minutes, using QmatOp. This was using 6-pass mode. Machine speed is an Athlon XP 2400+ overclocked to a 142 Front Side Bus. I'd say it's equivelent to about a 2700+ PR Rating. The machine has 1GB or RAM.

I'll be running some more tests this week doing comparitive encodes, so I'll turn the FSB down to standard level and report the encoding times of each encoder (HC and AQE). Based on testing I've done in the past, I see absolutely no reason to waste my time with CCE any longer.

[Fishman0919]

oh another thing, can i use filters (i.e. undot and deen) with it
Sure, I was going to test undot and deen with it but didn't get a chance ...it should be OK... might not be need with QMatOp on :)

[Rockas]

I tested one segment... 2 passes, QMatOp On and the result seems pretty good... the segment I tested is not an easy one... I've taken it from "Magnolia" a three hours movie ;)

[Fishman0919]

odd, i keep crashing on pass 6, gonna lower it to 4 pass

Did 4 passes work for you???

[hutch1711]

...try installing it in C:\AutoQMatEnc for ex....it seems to avoid the pb...I'll try to remove this stupid bug !
Installed in the simplest manner possible C:\AutoQMatEnc.exe and still received the error. Did this workaround correct the error for anyone else? :confused:

[BadServo]

Personally, I have the files installed in C:\Program Files\AutoQatEnc\ and have never encountered this error. Very strange. I wonder if there's some other aspect we're missing.

[hutch1711]

Thanks Badservo. Strange indeed. I'll sit on the sideline until Sapstar solves the problem in his usual efficient manner. :)

[Fishman0919]

Installed in the simplest manner possible C:\AutoQMatEnc.exe and still received the error. Did this workaround correct the error for anyone else? :confused:

hutch1711 try installing it like SAPSTAR say... C:\AutoQMatEnc\AutoQMatEnc.exe... some of the files AutoQMatEnc uses are temp put in the root dir C:\ ..... that may be a conflict

Edit: I said it in an earlier post to Rockas

Yes Rockas, I get the erorr sometimes too... if you have CCE Sp 2.70 (or any CCE Sp 2.6#) point to it and start the encoding, then stop dvd-rb befrore cce finishes the segment... repoint to AQE and it should work.

I was just talking to SAPSTAR about this and we are tring to work it out... but this is a temp solution

[gilligan2]

Same error here also. Tried the work around and that didn't even help.

g:)

[BadServo]

I don't knwo if this could have anything to do with it, but is there plenty of available free space on the drive that AQE is being run from? I know that I had drastically underestimated the size of it's temp working files when initially using it and ran into problems as a result, though I don't recall this specifc error message.

In my expirience, using AQE requires several hundred megs of free space to make calculations and pic comparisons in the folder it's run from, even if the source and output folder are on other drives.

Just a thought.

[SAPSTAR]

@all : Sorry for not being there yesterday evening...one of my kid was sick. In order to debug the errors, I would need more info from you guys :
- DVDRB version
- Full Path where you installed AQE + D2VAVS Path
- Memory Config
- Windows version
Something you can do, which could be really useful : There should be an item.ecl file in the D2VAVS dir. copy it in the AQE dir, then open a dos windows in the AQE dir and type : autoqmatenc.exe -ecl iem.ecl
Give me the log if it doesn't work....

I'll work on it with FishMan0919, BadServo and hostile_monk the beta testers of the early hours......

[gilligan2]

Thanks guys and SAPSTAR for trying to help !!

OK,i have plenty of HD space (over 15 GB) on the C: Drive where i have AutoQmatEnc installed.

DVDRB Version = 93.2

Full Path of AQE-Have tried it both in just C:\AutoQmatEnc.exe and in the folder C:\AutoQmatEnc\AutoQmatEnc.exe and neither works

Full Path of D2VAVS path = D:\NEW\D2VAVS

memory 2 GB's

Windows XP SP-2

Have the Mode as CCE,Chose CCE SP Trial in CCE options.


I didn't get a log when i put the item.ecl in the same directory as AutoQmatEnc. I got an error message saying "Could not access item.ecl" then sometimes i get "Press Any key to continue" after it gives me the details on the program.

Hope this helps

G:)

Here is my item.ecl:

; CCE SP Trial Version -- Encoder Control List
; Created by DVD Rebuilder

[item]
title=V01000000001001
aud_out=0
vaf_file=D:\NEW\D2VAVS\V01000000001001.vaf
aud_file=D:\NEW\D2VAVS\V01000000001001.mpa
file_focused=0
packet_size=2048
width=720
height=480
frame_rate_idx=1
cbr_brate=6000
vbr_brate_avg=2818
vbr_brate_min=500
vbr_brate_max=9800
seq_endcode=0
dvd=0
half_width=0
half_height=0
lum_level=0
adjust_q_matrix=1
aspect_ratio=2
gop_m=3
gop_nm=5
gop_hdr=15
seq_hdr=1
all_closed_gop=0
fix_gop_length=0
samples_per_sec=44100
stereo=2
brate_idx=7
crc=1
progressive=0
alternate_scan=1
intra_dc_prec=2
intra_dc_precision_9_max=113
intra_dc_precision_10_max=113
aud_mode=0
tc_ref_frm=0
drop_frame=0
fix_vbv_delay=0
letter_box=0
pulldown_detect=0
offset_line=0
create_new_vaf=1
credits_tweak=0
credits_start=0x00000
credits_brate=1000
h_filter=0
h_filter_idx=8
dither=0
dither_max=8
quality_prec=19
timecode=0x0000000
q_char_f=19
tc_offset=0
v_filter=0
v_filter_val=6
pict_name=
pict_type=3
pict_level=255
video_type=16
vid_file0=D:\NEW\D2VAVS\V01000000001001.m2v
vid_file1=D:\NEW\D2VAVS\V01000000001001.m2v
vid_out=1
vaf_out=1
opv_q_factor=20
opv_brate_min=0
opv_brate_max=7800
vbr_bias=0
vbr_pass=2
use_filter=0
filter_val=6
non_linear=1
top_first=0
mpeg1=0
mpeg1_cps=1
qmat_idx=-1
q_scale_type=1
qmat=
08 16 19 22 26 27 29 34
16 16 22 24 27 29 34 37
19 22 26 27 29 34 34 38
22 22 26 27 29 34 37 40
22 26 27 29 32 35 40 48
26 27 29 32 35 40 48 58
26 27 29 34 38 46 56 69
27 29 35 38 46 56 69 83
16 17 18 19 20 21 22 23
17 18 19 20 21 22 23 24
18 19 20 21 22 23 24 25
19 20 21 22 23 24 26 27
20 21 22 23 25 26 27 28
21 22 23 24 26 27 28 30
22 23 24 26 27 28 30 31
23 24 25 27 28 30 31 33

[file]
name=D:\NEW\D2VAVS\V01000000001001.avs
frame_first=0
frame_last=3718
encode_first=0
encode_last=3718

And my rebuilder.ini file:

[Options]
CCEAQM=1
vts_min_size=1
CCETargetSectors=2256400
HCTargetSectors=2256400
Mode=1
OneClick=0
ReduceOpt=1
NoWarn=1
AdditionalOutput=0
LogFile=1
QuEncHQ=1
EncoderMinimized=1
RemoveDTS=1
Skin=Rockas Original
HC_Quality=3
HC_Matrix=1
SkinVersion=5
CCE=3
HalfD1=00
Convert_16_9=00
DisableInterlace=00
iDCT=3
MovieOnly=1
MainMatrix=MPEG Standard
AVSFilter01=
Completed=3
AudioDub=0
DCPrec=3
[Paths]

QuEnc=C:\Program Files\DVD-RB PRO\Encoders\QuEnc\QuEnc.exe
ReJig=C:\Program Files\DVD-RB PRO\Encoders\ReJig\ReJig.exe
HC=C:\Program Files\DVD-RB PRO\Encoders\HC Encoder\HCbatch.EXE
ProCoder=C:\Program Files\DVD-RB PRO\Encoders\EclPro\EclPro.exe
DECOMB=C:\Program Files\DVD-RB PRO\Filters\Decomb521.dll
MPEG2DEC=C:\Program Files\DVD-RB PRO\DGDecode.dll
Working=D:\NEW\
Output=D:\NEW\
CCENew=C:\AutoQMatEnc\AutoQMatEnc.exe
Source=K:\VIDEO_TS\
[Audio]
Selected=1
[Subpictures]
Selected=
[CCEOptions]
VBR_bias=0
Quality_prec=12
eclPasses=3

[Fishman0919]

Well, we found a temp fix for the error... DVD-RB seems to be calling AQE in the dvd-rb matrix dir... so by putting AutoQMatEnc.exe and dctune2.0.exe in the dvd-rb matrix dir... that should fix the problem for now

[SAPSTAR]

Well, we found a temp fix for the error... DVD-RB seems to be calling AQE in the dvd-rb matrix dir... so by putting AutoQMatEnc.exe and dctune2.0.exe in the dvd-rb matrix dir... that should fix the problem for now
FishMan0919 is absolutly right...after many tests he did for me, we found the issue....I'll release a new version correcting this bug soon...most probably tomorrow. I'll have also improved the speed of the "QMatOp" part.....
Thank you FishMan0919 for all your tests it was a tricky one...DVDRB sometimes calls the encoders from its own dir but not in the encoder dir itself ?!

[hutch1711]

Well, we found a temp fix for the error... DVD-RB seems to be calling AQE in the dvd-rb matrix dir... so by putting AutoQMatEnc.exe and dctune2.0.exe in the dvd-rb matrix dir... that should fix the problem for now
Thank you Fishman0919 the temp fix works like a charm :) :thanks:

[gilligan2]

Thanks guys,that did the trick. Its working now so thanks again !!

g:) :)

[Carpo]

SAPSTAR - as asked i will test your encoder - joss stone mind body and soul sessions - nice little dvd so i can use yours and cce on it doing batch encode over night :)

will let you know what i think in the morn :)

[Carpo]

could you make another standalone version with the advanced qmatop for those who like to use it alongside other encoders ?

just an idear

also as pointed out in the pm i now have another error msg

Warning, QMAT_SHIFT is larger then 21, overflows possible
/[mpeg2video @ 00676E18]rc buffer underflow
/[mpeg2video @ 00676E18]rc buffer underflow
/[mpeg2video @ 00676E18]rc buffer underflow

[Fishman0919]

also as pointed out in the pm i now have another error msg

Warning, QMAT_SHIFT is larger then 21, overflows possible
/[mpeg2video @ 00676E18]rc buffer underflow
/[mpeg2video @ 00676E18]rc buffer underflow
/[mpeg2video @ 00676E18]rc buffer underflow

Yes they are normally.../[mpeg2video @ 00676E18]rc buffer underflow... should only be in the first pass and should be corrected in the 2nd pass

[Carpo]

ah ok - but i have to close the window for it to carry on which is a pain in the arse - cus i like to leave qmatop running while i have a little blast on the xbox

[SAPSTAR]

Yes they are normally.../[mpeg2video @ 00676E18]rc buffer underflow... should only be in the first pass and should be corrected in the 2nd pass
Exact !!! And about the message starting by "Warning"...as said this is a warning, saying that overflows could occur...it's not an error...
And no...for now, I don't plan to create the advanced QMatOp feature standalone.....I have no time to develop it...sorry...maybe once AQE will be more stable...

@all : I have now a very good new version, I'm still waiting for tests from the beta testers before releasing it, but my own tests were really good...I'm rather proud of the new rate control (better quality) + the new speed/quality of the qmatop phase... I'll release it really soon be patient !!!

[BadServo]

I'm running a series of quality comparisons on this new version as I write this. Although I'm still testing and will be for many hours yet to come (slow PC), I must say this is quite impressive. Our ol' buddy Sapstar is giving Hank's HC Encoder a run for it's money.

I'll have a page of comparison shots ready as soon as all the necessary encodes are completed.

[Carpo]

could u add an option for more verbose output either on screen or in log ?

and idealy how many passes should u use when doing a backup - i know hc only does 2 and with trial versions of cce sp trial u can do more

but it more really better or just a time wasting exercise ?

i have a few backups i would like to do and 5 passes takes too long if you have 2+ movies to do :( but i have always been told to do 5

[Fishman0919]

but it more really better or just a time wasting exercise ?

i have a few backups i would like to do and 5 passes takes too long if you have 2+ movies to do :( but i have always been told to do 5

5 passes with any encoder is a waste of time.... 2 or 3 is more..more then enough...

The idea behind more then two passes, is in short, to take a little from here where it might not be needed and put it over here where it MIGHT look better. It's not a magic wheel that the more you turn it the better it gets, the enocoder can't magicaly pull bits out of nothing. CCE (or any encoder) gets ALL the info it need to encode the video in the first pass (.vaf pass) and applies it in the second pass.. it simple math (well maybe not real simple) ... the 1st pass gives the encoder the info it need and then distribute the bits to where they are needed on the 2nd pass,.... the 3rd, 4th, 5th pass and so on are really just 2nd and 3rd guesses and not every guess is going to be better... In some test I did 6 or 7 passes look worst then 2 or 3 and in 1 test 100 passes look worst then 2 passes.

Hey, you can do as many passes as you want but it not always the best... I do 2 passes on just about everything and for something long (like LOTR)... 3 passes wouldn't hurt

LOL 360 posts... does that mean I'm back where I started

[Carpo]

2 passes it is then :) - i have done tests in the past where u couldnt really tell the diff between a 2 pass encode and a 5 pass (old eyes u see :)) plus i have been impressed with HC and that only uses 2 passes

and seeing as im not that knowlegdeable on the subject i will take any help given :P


right time to start up sapstars encoder - just wish cce didnt have the stupid way u have to select the passes ;)

:stupid:

[SAPSTAR]

I finally chose to release it :

Things remain to be tweaked or improved...but I think a new version can be released :

Changes History:
0.22a - 2005/05/16 :
- Corrected a bug when DVDRB calls AQE from its own dir.
- Improved the QMatOp part - faster&better
- Corrected a bug when alternative scan block was set.
- Improved the rate control curve - Should correct some artifacts
(However the sizing could be less accurate to - I'm waiting for feedback on it)

I had amazing results on low bitrates encode, on the HalfD1 segs it beats anything I have seen before...even CCE SP.

[Carpo]

was fast didnt even get time to beta test it ;D

little edit : does the advanced qmatop still change the matrix u select or does it use what it thinks is best over writing the matrix u have choosen ?

also will teach me for being late :D

[gilligan2]

Same error problem again,except now it still gives the error even if i put the proggie in the Matrix folder like the work around before.

g:)

[Carpo]

no probs here and im running it from D:\