"qcce", utility to automate One Pass VBR starting from an AviSynth file

[jonny]

Current version: 1.02 (27th January 2004)

Download:
http://jonny.leffe.dnsalias.com/qcce/rar/qcce_1_02.rar

Screenshot:
http://jonny.leffe.dnsalias.com/qcce/images/qcce.png

Source code (can be interesting if you are a Perl programmer ^^'):
http://jonny.leffe.dnsalias.com/qcce/rar/qcce_1_02_source_pack.rar


The app is tested with:

CCE 2.66.01.07
CCE 2.67.xx.xx (last version checked: CCE 2.67.00.23)

EclCCE 1.7b




You can run:
- Normal encodes (opv only)
- Size prediction
- Q finder (to find the Q needed to hit your target size).


For the Q finder, i use a simple binary search (so there are a lot of rooms for improvement in this direction).

+ you get full save/load settings function, joblist, full save/load joblist functions, you can use custom matrices and (optionally) shutdown at the end.

I've done all this stuff mainly to learn CCE a bit.
It's useful for me, maybe can be useful for someone other...


Cheers
jonny

[DDogg]

jonny, this certainly looks like a lot more than a learning exercise! For a first version it is packed with features. Do you plan to continue it and are you open to suggestions?

BTW, people may have been confused when you said "The app works only with CCE SP Trial 2.66.01.07". I think you mean this is what you have tested with? It seems to work fine with higher versions of CCE and ECLCCE in the very limited testing I have done.

Great job!

DD

[jonny]

jonny, this certainly looks like a lot more than a learning exercise! For a first version it is packed with features.

hihi, the trick is that i've used a lot of code from my previous apps :)

Do you plan to continue it and are you open to suggestions?

sure!

on my side, things i'd like to add are:

-Conditional encoding (encoding after q finding only if the Q is <= *something*)

-Conditional stopping the joblist:
you fill the joblist with "Q Find" jobs, the 1st with unfiltered source and good matrices, the others using progressively more aggressive filtering and/or matrices, after running all, the process will stop after finding the first combination that give a Q <= *something* (and maybe a jobs-grouping, so the feature can be used with more movies in the same joblist)

-Maybe a resizing pass (i'm still not sure is needed, on my tests with full movies, i've got always undersized by 40-80MB, i'd like to hear experiences from others)

-Optimization of the Q Finding process

-Calculator for the target size

++Other ideas/suggestions i find interesting (so i can learn more ^^)



BTW, people may have been confused when you said "The app works only with CCE SP Trial 2.66.01.07". I think you mean this is what you have tested with? It seems to work fine with higher versions of CCE and ECLCCE in the very limited testing I have done.

I've tested it only on 2.66.01.07 (i've preferred to start with the latest nonbeta version)
I use a fixed "ecl template", if the ecl format is the same between 2.66/2.67, it will work (i'll verify this in this week and let you know - anyway i think it will be not a big pain to eventually add 2.67 support)
Atm you can use the menu function: "EclCCE->Save Ecl+Run EclCCE" to verify if all the parameters are set correctly with 2.67.xx.xx


Thanks for the feedback

[Amnon82]

Jonny great program! I'll upload it now to my page so You have a mirror. If You want, I can translate Your program into german. PM if You want this.

I've the latest Version of CCE 2.67.00.23 and I'll test Your program with it in the next days with full PAL Movies.

I think maybe You can beat Kwag's CQMatic (only for TMGEnc) of KVCD.NET.

Go on with Your proggy.

;)

[Amnon82]

Here is the mirror: DVDRPAGE.DL.AM (http://rj-elektronik.de/avideo/files/qcce_1_00.rar)

Have Fun

:D

[jonny]

many thanks Amnon82 :)

I've done testing on 2.67 and added support for it:

http://jonny.leffe.dnsalias.com/qcce/rar/qcce_1_01.rar
http://jonny.leffe.dnsalias.com/qcce/rar/qcce_1_01_source_pack.rar

(only a little change was required, selecting "UFF" in my interface will now result in "Offset Line"=1 when you use CCE 2.67)

To make a summary, i've tested v1.01 with:

2.66.01.07
2.67.00.13
2.67.00.23

(i'll check for compatibility every time a new beta will be out)

EDIT: you was too fast for the mirror :)

[ADLANCAS]

Hi Jonny!
Great job! I was waiting for such tool!

I noted that you use same different settings in CCE:
GOP Sequence= 3;4 (not "3;5")
Luminance level=16 to 255 (not "0 to 255")
Restrict auto I frame insertion selected (not "unselected")
-> ("my settings")

My settings comes from a template of RB. Maybe he can comment about it.

Another thing. Maybe you could make a little change for searching faster a Q (and better?).
In your app., I´ve made a test and it was necessary to encode 4 times to find a Q.
Using the same source, manually, I needed encode only 2 times.

I´ve just encoded something today.
My sample test (including line "SelectRangeEvery(750,15)":
File desired=29.555KB
Q53 -> results 31.172KB -> error=+5,47%
Q57 -> results 29.572KB -> error=+0,06%
Qfinal=56

Encode settings:
Q56 (500-9000kbps)
QC (17)
DC precision (9)
Aspect Ratio 4:3
Progressive frame flag (NO)
Upper field first (NO)
Scanning (alternate)
Frame rate 29.970
Quantization matrices (Standard)

Final:
File size results=1.458.481KB
File size desired=1.477.744KB
Error=-1,3%
-------------------------------
Calculations:
->NextQ=Q+(1,5*Q*(FileSize-DesiredSize)/DesiredSize)
->if Abs(ActualError%)<=2% then, Qfinal=Q-1+LastError%/(LastError%-ActualError%)/2
This is nearly the same what Tylo uses in his plugin. It works very good to me.

Regards,

Alexandre

[jonny]

Hi Alexandre :)


GOP Sequence= 3;4 (not "3;5")

I've got it from the doom9 guides, do you think i should add an option for "3;5" (or maybe are you referring only on size prediction)?


Luminance level=16 to 255 (not "0 to 255")

If i'm not wrong it doesn't make difference if the source is YUY2. (i dunno if many people uses AviSynth RGB output, what is your opinion about this)?


Restrict auto I frame insertion selected (not "unselected")

I guess you are always referring to size prediction (with a normal encode you should find this option off)
This option seems to reduce the error on the single size prediction, you can find some tests here:
http://forum.doom9.org/showthread.php?s=&threadid=62446&perpage=20&pagenumber=3



Calculations:
->NextQ=Q+(1,5*Q*(FileSize-DesiredSize)/DesiredSize)
->if Abs(ActualError%)<=2% then, Qfinal=Q-1+LastError%/(LastError%-ActualError%)/2
This is nearly the same what Tylo uses in his plugin. It works very good to me.

yep i've already seen something like this, before starting to write the app (DDogg pointed me to some really interesting resources)
i've preferred to start with something simple :) (reducing the number of passes for the Q finder it's probably the next thing i'll go to do)



Great job! I was waiting for such tool!

I'm happy you like it :)

[ADLANCAS]

Hi,
...do you think i should add an option for "3;5"
yes, should be better.

i dunno if many people uses AviSynth RGB output, what is your opinion about this)?

I don´t use. Maybe if someone make a request...

Restrict auto I frame insertion selected (not "unselected")
I saw you page. It seems good.
I´ll make some tests with this.

Regards,

Alexandre

[Amnon82]

Here is the mirror (new Version 1.01): DVDRPAGE.DL.AM (http://rj-elektronik.de/avideo/files/qcce_1_01.rar)

Have Fun

:D

[jonny]

I saw you page. It seems good.
I´ll make some tests with this.

I'm really interested on compares with full movies (my tests are done with small clips)
Actually the test was fully generated with my app (the html generator and other options are disabled in the binary release)
If you have ideas on improving the single size prediction let me know, i can compare methods using a lot of automation.

Cheers
jonny

[RB]

Originally posted by jonny
Atm you can use the menu function: "EclCCE->Save Ecl+Run EclCCE" to verify if all the parameters are set correctly with 2.67.xx.xx

Looking through my code, the only difference between the 2.66 and 2.67 ECLs that EclCCE expects is that "top_first" (2.66) is replaced by "offset_line" in 2.67. Additionally, the "Make room for USCC" option (whatever that is, don't know) is recored as "uscc=1/0" in the 2.67 ECL.

[jonny]

Thanks RB, this confirms that 1.01 works with all the 2.67.xx.xx versions.

I've a little question for you, actually i'm passing to EclCCE an ecl file that contains all the 3 options:

...
top_first=...
offset_line=...
uscc=...
...

so i don't need to make differences between 2.66/2.67
Seems to work nicely (i see no strange effect on both 2.66/2.67), am i doing something wrong?
(i don't know how you handle internally this situation, so i have this little doubt)

[RB]

You are fine.

[Amnon82]

Hi Jonny,

I want only tell you that I test the version 1.01 now with my DRP 2.2 Settings on full movies ... THX for adding all the answers I need in Your app!

I'll tell You if I reached 3.300 GB as I wanted ...

P.S.

Your PM-Mailbox is full...:D

BTW: How to read Your Log-Files (*.enj)? Do You have a ENJ-Viewer?

[jonny]

THX for adding all the answers I need in Your app!

So i guess you no more need answers about different modes :)
(i've tryed to document the main stuffs directly in the interface, usually you'll get a popup for every parameter that need to be explained)



BTW: How to read Your Log-Files (*.enj)? Do You have a ENJ-Viewer?

:D
"Load job list" or "Append job list" from the joblist menu

Generally you can save 2 kind of files:
"ens" contains settings (basically it's one job)
"enj" contains jobs


Your PM-Mailbox is full...

hihi, give me some time to reorganize it :D

[Amnon82]

Done with the test and it was fantastic!!!!

here is the result screenshot:

http://rj-elektronik.de/avideo/images/DRP22FIN.PNG

It was amazing! Realy I started Your app with Blue Crush (its a Surfermovie with a lot of water...). I wanted the size 3.450 GB and I got 3.415 GB, it is only 34 MB less as I wanted!!!

How I tested it:

I select the movie, entered my DRP 2.2 Settings with AVAMAT4 and entered 3.450 GB as wanted size. 10% of the movie for calculating and run Q Find without encoding.

After that I entered the final Q 21 and run the encode mode to see the difference of the wanted and getting file.

This app is fantastic and I'll start writting now the guides for it.

Read the PM form me about DRP2.2 ... THX


Guide for the QCCE Program:

Avalon's DRP 2.2 Guide (http://rj-elektronik.de/avideo/main.php?set=drp22guide)

[jonny]

10% of the movie for calculating and run Q Find without encoding

I think you can go safe with a 2% (on my tests with full movies i've never surpassed this value)
Soon i'll do some tests with full movies to show what kind of error one should expect using different % (something really similar to what i've done for XviD/DivX)

[Amnon82]

You realy mean 2% of the movie...

...K, let see it.

I run a test with 2% of a movie I've already encoded with Q22.
Let see if QCCE will get this value also with 2%...

http://rj-elektronik.de/avideo/images/2POM.PNG

As You can see it works also with ONLY 2% of the movie! FANTASTIC!!!
Precition ONLY in 9 min!!!! CQMatic for TMPGEnc take over 30 min to get this!!! GREAT done jonny!

... and thanks for advice ...I change this in my guide

Guide
http://rj-elektronik.de/avideo/main.php?set=drp22guide
(many thanks to Avalon for this!!!)

I thank You for the app.

And it is the only reason that I start now learning pear, cos I want this app german :D

I looked into the PL and I felt home! I translate it and create a DRP-Version with my settings as default;)

I hope I can compile it...

I've troubles with compiling it. I send you a PM about the 2 errors...

[Amnon82]

:( Bugga!!!

I compiled my edited qcce.de.pl-file somehow and got a exe.
I tried runing it and shot so my system!!!

Jonny I'll send you a PM with the PL-file. Maybe I did something wrong.
If You have time for it take a look into it and try to compile it.
I think I did MANY mistakes editing it...:mad:

I got a 2 MB NOT runing file...
... and it shots my system...

Make a GHOST-Backup of C:\ before runing the compiled qcce.de.pl.

THX anyway!

[Amnon82]

Hi jonny,

did I something wrong with the translation of the PL-file?


I'm start writing a manual for the app.

[jonny]

i'm checking the pl now :)

[Amnon82]

THX for helping me out!

And I write today the manual for YOU!!!!

[jonny]

you run like a train!
thanks for all

[Amnon82]

Released german version

download QCCE.DE 1.01.RC1 (http://rj-elektronik.de/avideo/files/QCCE.DE.EXE)

German Support Page (http://rj-elektronik.de/avideo/main.php?set=qccede)

[Amnon82]

Hi Jonny,

I wrote you a PM about the german version and about myself.

Read it and send me a return PM

[needle]

@jonny

qcce is a useful program that in practice saves computation time.

I suggest to add two aditional features:

First, the support of CCESP 2.50, since this version is faster than 2.66 and 2.67.

Second, in order to obtain frame acurate chapters is necessary to set the
chapter points in CCE when encoding the full movie. A frame which
is set as a chapter point in CCE is configured as a I-frame and the
authoring program can place the start of the chapter in exactly that frame.
The following example shows how to set the chapter points in the ecl
file. Assume that the the user need chapters at frames 73, 3397, 11965 and
17533. Then, the file section of the ecl file that qcce submits to CCE should be as follows:

[file]
name=
type=0
frame_first=0
frame_last=xxxx
encode_first=0
encode_last=xxxx
chapter=0:
chapter=73:
chapter=3397:
chapter=11965:
chapter=17533:

Thanks.

[jonny]

Hi needle

For the CCE 2.50 support, you'll probably don't get this soon (i'd like to first finish other things).
For the chapter support, yep, it's a good suggestion!

[jonny]

I've got a PM about adding the process priority of CCE.
I respond here so everyone can read:

There is no way to set process priority with EclCCE (via command line).
The best thing is to set the preferred priority running EclCCE manually, so this priority will be used for all the succesive encodes (personally i set it to idle)

[buzzqw]

ciao jonny :p

Could be possible an automatic creation of .mpv, .vaf based on avs script, putted in a predefinited directory based on avs ' name?
Ex. pippo.avs is in c:\vob, predefinited directory c:\temp .
Should be created an c:\temp\pippo\qcce_pippo.mpv and c:\temp\pippo\qcce_pippo.vaf ?


I could explein better in italian...

Grazie/Thanks a lot(s)

BHH

[jonny]

Ciao BHH!

I think this will lead in some confusion, and it's a too specific request.
In the settings, there is no reference to the original .avs file (actually i store the full the script).
Plus, one could decide to write the script from stratch, or to copy/paste it instead of using the load button.

An option to append automatically some settings in the output filenames will be probably better (so you'll find the "AviSynth title" in the final names, giving you a similar result).
What you think about this?
(i was thinking to something like this, but i've still not taken a decision)

Ciauz

PS: your english is ok, i hope you understand MY english :)

[tyee]

Originally posted by ADLANCAS
Calculations:
->NextQ=Q+(1,5*Q*(FileSize-DesiredSize)/DesiredSize)
->if Abs(ActualError%)<=2% then, Qfinal=Q-1+LastError%/(LastError%-ActualError%)/2
This is nearly the same what Tylo uses in his plugin. It works very good to me.

Regards,

Alexandre

Hi Alexandre
Do you have a link to the theory on this equation for Q? I'd very much like to read how it was determined.

tyee

[RB]

Originally posted by jonny
There is no way to set process priority with EclCCE (via command line).
It's recorded in the "Priority" value in the INI file. You could change it in the INI file and reset it afterwards. The supported priority values are as follows:


-1 Default. CCE gets run with whatever priority EclCCE.exe was run
with (use CreateProcess() to run EclCCE.exe and specify the
priority in the dwCreationFlags parameter).

32 Normal

64 Idle

128 High

256 Realtime

[jonny]

@RB:
Ops, i've missed this :)
I think i'll follow your suggestion adding the priority option in the "EclCCE Menu"


@tyee:
Those are my resources about Tylo's plugin:

This is an xls from r6d2:
http://www.angelfire.com/droid/r6d2/

Original thread:
http://forum.doom9.org/showthread.php?s=&threadid=60191

This explain things a bit more (bottom of the page):
http://forum.doom9.org/showthread.php?s=&postid=361071#post361071

A general overview of the "The Newton - Raphson Method":
http://www.sosmath.com/calculus/diff/der07/der07.html

[Amnon82]

Thx for Your imput RB...:D

[needle]

@jonny

Thank you for your reply.

I have been thinking about differences between ecl's of CCE versions 2.50 and 2.67. I think that in order to add support for CCE 2.50 only is necessary to add the following four lines into the [item] section of 2.67's ecl file:

encode_mode=x
vmode=x
fast_mode=x
top_first=x


where the precise values 'x' for each line may change depending of the specific movie being encoded. 'top_first' is related with 'offset_line' and is present in 266's ecl. 'vmode' should be related with 'video_type'. I do not know the meaning of 'encde_mode' and 'fast_mode'.

May be someone in the forum would be so kind to help us to identify the precise meaning of each line.

I have experimented adding these lines to an ecl of CCE 2.67 and there was no problem when open in both CCE 2.50 and CCE 2.67.

[jonny]

I've still not checked how 2.50 saves parameters.
If i'll find the thing not problematic, i'll add support in the next version :)

[jonny]

v1.02 out (the first post is updated with new links)

Changes:
- Chapters support
- CCE process priority (global option in the EclCCE menu)
- Both % of the movie and ERR % can be non-integer
- GOP size can be 12 or 15
- Conditional encoding (after the Q finding), MinQ and/or MaxQ can be excluded from encoding (this is mainly to prevent big oversized/undersized encoding)

The Q finder is still binary (work/testing in progress here, expect something new on the next ver).

CCE 2.50 will be integreted later as a separate binary (i don't like the idea of mixing 2.66/.67/.50 in the interface).

Talking about pure CCE parameters, i think all the options needed are in the interface now ( if i'm still missing something, suggestions are wellcome! :) )

[tyee]

Hi Jonny
I've been playing around with a similar program to yours that I'm writing just for learning about how to program using scripting. It's nowhere close to yours but it's still fun to try to write it and have it work.
I have one question regarding the .ecl file which is loaded by eclcce.exe. I can write into the .ecl file, Q Factor, the required output directories for the .m2v, .vaf, etc., but how do you enter the last 4 lines of the .ecl file that consists of the frames to encode? I can write the name of the .avs file just above those 4 lines, but that doesn't tell CCE which frames to encode does it?
I suppose, in your program, that when we load the .avs file, somehow the frames are read, correct??

tyee

[jonny]

Hi :)

I read AviSynth script info via vfw. The source is in "video::codec" and is written in C (i use a Perl module called InLine, it permits to mix Perl code with other programming languages in an easy way).

The base function is "video::codec::AviSynthInfo" and is invoked from "sub AviSynthInfo" (qcce.pl).

To make it work you must install all as described in the source docs (you need VC6 to do this)

[Amnon82]

I updated the mirror:

Download QCCE 1.02 at dvdr-page.tk (http://rj-elektronik.de/avideo/files/qcce_1_02.rar)

Nice and realy good changes! Nice to see the new version working!
Go on jonny.:D

[Chibi Jasmin]

Just wanted to say thanks for the tool, it's a pleasure to use :-)

[jonny]

It's a pleasure for me to read this ^^

[lighty]

@jonny
Qcce is great tool indeed but (as always) there is a room for improvements.

1. Why don't you add an option to select number of passes? Some of us like to have 5 passes for example.;)

2. It would be great if you could add an option to set lower bitrate for credits. It is a part of eclcce and it is well documented in eclcce.pdf. I guess it could be called after a Single pass VBR is finished and *.vaf is created- before the Multipass VBR starts.
I know that in eclcce.pdf it is stated that this option is available only for new vaf creation but I don't see a reason why existing vaf shouldn't be used since that option is clearly there and I use it all the time.
I don't know if that one is easy to implement but it does help to set much lower bitrate for credits thus adding more bits to distribute over the other parts of video.

3. Q finding tool is extremely helpful but the Q value that is founded is automaticaly entered into current job and it would be much more visible in some part of the main dialogue.

4. Option to load matrices extracted by ReStream maybe works and maybe it doesn't. How can one tell? There is no indication anywhere. Would it be possible that one can import matrices and that it should become a matrix listed in the list? (also if one matrix is selected there isn't any kind of indication, even for the ones you already have hardcoded in the list).

5. I also tried you method (http://forum.doom9.org/showthread.php?s=&threadid=68596) for importing matrices but when I select Save ecl option it turns back "Set Mode to Encode" message and I am not sure if ecl file is even created or not and if matrix is created or not. So- is it bug or not?


Anyway- I know I made a lot of requests here but it is what I personaly would want to see in Qcce if I would to consider it "ultimate" CCE GUI and automation tool. ;) ;) ;)

But hey it's great as it is right now! Good work!:cool:

[jonny]

hi lighty :)

1) Atm my target is opv encoding, i could add a multipass option later, but i'd like to first concentrate myself on a faster Q prediction (i usually do things in little steps ^^).

2) I'm interested on this too, but there is a big problem: will be hard to make size prediction with this option activated (it would be possible to encode credits first and predict size of the movie after, taking into account credits size... but this means reimplementing the function, i have to think a bit before going into this)

3) The problem of this is that the main dialog and the jobs are completely separated, how i handle this if you run 2 "Q Find" jobs?

4) First, if the load matrices operation fails, you get an error.
The operation fails if more or less than 128 numeric values are found in the input file (you can separate this values as you want).
(the range of each value is checked when you add a new job... i see no possibilities to do mistakes).

I tell you how all the matrices stuffs should be used:

Save your matrices in the "matrices" folder, using a name that describe the matrices (example):

very_good_mat.txt
very_bad_mat.txt
very_lol_mat.txt

When you load the matrices, for example "very_good_mat.txt", you'll see that "very_good_mat" is showed in the title (so you know what is the current matrices, this happens with hardcoded matrices too)
If you change values inside matrices manually, you should change the title too (example: "very_good_mat_mod").
At this point, if you save the matrices, "very_good_mat_mod.txt" will be proposed as file name.

Seems good and easy to me, isn't it?

5) "Set Mode to Encode"... you have to "Set Mode to Encode" before using this function :) (the title of the window is "Error", if i'm not wrong)


Thanks for the suggestions.

Multipass will be probably added (but before i have to be satisfied by the Q finder and i'd like to add a "Target Size" calculator)
I've added the "Save ecl" function mainly for this (at least you can run additional passes manually and in an easy way / given that you calculate the bitrate correctly).

The credits option add a good level of complication (at the moment you could use heavy AviSynth filtering on credits to obtain a *similar* result)

[lighty]

1. Well said. Develop in little steps.;)

2. Isn't it possible to somehow directly tweak vaf file in the way it is done natively by CCE when using a credits tweak option in multipass mode? But I guess it will be a problem since using this option means that a new vaf has to be created in a way that would mark some parts of video as already "fixed bitrate". Basicaly the same option can be done in a more complicated way after a first pass with the existing vaf file by manualy set Bit alocation of credits to some limit.

So if you plan to add multipass mode it would be great if you could open vaf file after the first pass and add "bitrate fix" markings directly in it- the same way that Bitrate allocation tweaker of CCE does. In that way after a first pass Qcce could automate Bitrate tweaking process that is otherwise done manualy and start later Multipass part automaticaly. It would effectively automate complete 1 pass VBR + Multipass process and would also include credits tweak in the easiest possible way.

Mind you- I never even tried to open a vaf file so I am not even sure in what format it is written so... please consider this as an uneducated guess. :D

3. Ummm, never thought of that...:o

As for Avisynth filtering- what filters are you suggesting one should use because except for the heavy Convolution filtering I don't have any idea what could help compresability? By using the credits tweak one can simply specify maximum bitrate for credits and that's all one has to do and is much more efective in the terms of bits alocation.



Anyway- those are all sugestions. You already did a great job as far as 1 pass VBR goes so I guess you will eventualy come to consider these requests also- but hey, keep it easy for yourself and take your time. :cool: :cool: :cool:

[jonny]

2. Isn't it possible to somehow directly tweak vaf file in the way it is done natively by CCE when using a credits tweak option in multipass mode?

Yep, but the main problem is still handle this in the size prediction (i should do some testing).
( + i'll never go to modify the vaf file, what happen if the format of this file change in newer CCE versions? from experiences in the past, better to stay away from this :D )


As for Avisynth filtering- what filters are you suggesting one should use because except for the heavy Convolution filtering I don't have any idea what could help compresability?

Take a look at the AviSynth docs:
SpatialSoften / TemporalSoften (CoreFilters)
Spatio-Temporal Smoothers (ExternalFilters)
Spatial Smoothers (ExternalFilters)
Temporal Smoothers (ExternalFilters)

There are a lot of ways, make some experiments to see what you like more.
In the past, with DivX, i was using heavy spatio-temporal smoothing and an high quantizer (it's similar to using an high Q with CCE).

Now i like 2 ways:
1-encoding credits with no modifications
2-cut out credits :)

[r6d2]

Originally posted by jonny
Yep, but the main problem is still handle this in the size prediction (i should do some testing).I think the easiest way to encode credits with lower quality using OPV is this,
[list=1]
Let the user select the cut point
Encode the credits entirely with a high Q, and obtain the size used by them.
Do the movie encode with normal sampling, taking into consideration the credit size (smaller target).
Merge the 2 MPVs afterwards.[/list=1]If this can get automated, you can have best of both worlds.

As per good compression, albeit rather slow, UnDot()+Deen() (aka KISS) is the way to go for me.

[jonny]

Let the user select the cut point
Encode the credits entirely with a high Q, and obtain the size used by them.
Do the movie encode with normal sampling, taking into consideration the credit size (smaller target).
Merge the 2 MPVs afterwards.

Yep, i think too is the best way.

[Boulder]

jonny,

thanks for this excellent program! I used to do all the steps manually but now it's a lot faster and easier to encode several clips per one day/session:)

I second the request for a resize pass as I've noticed that the quantizer gets lower when an OPV and then a resize pass is performed so I've been doing that in all my encodes. I've also noticed that CCE produces slightly undersized files every time with the predicted Q value so this would also help me fill the discs to max capacity.

[SiliconSoul]

i noticed that it does do undersize often and im experimenting with higher % of the movie/video to see if that helps. the quality results, that rival multipass, are great for the time required to encode. i normaly did 4-6 pass encode before depending on the content.

i noticed that it comes defualted at 1% im testing 5% now... i would like to stay under 10% if possible....

what % of the movie are you using?

[jonny]

Got it, multipass added in the wish list :)

Atm i'm using 2% and ERR % = 0
I've not tested this deeply, i have to find some time to give an average figure of the errors with differents % (the automatization for this is ready, i have only to prepare a couple of movies for the test and run the jobs).

[SiliconSoul]

with 2% of the movie and always 0% err. i had some undersize by 50 megs or so...
im doing a backup of sopranos s1d2 right now... so it is 4 episodes of different lengths. the first one was 50 megs off at 2% and at 5% it is -17.01 megs off. second episode at 2% was about 1 megs off and at 5% was -0.72 meg off. third episode i forgot to record how far it was off and its encoding second time at 5% right now. fourth episode at 2% was about -34 megs off and at 5% it was -50.64 megs.

this is not too bad. considering the time saved once i get used to how it works. im thinking about not encoding at first see what value it gives and then manually entering in the Q from the set of data. im noticing that maybe if i drop the err to 1-3% it might get closer to the estimated megs.

so if i select 1-3% err. does it give me 1-3% too big or 1-3% too small? or it just choses which is closest to the 1-3%.

with it set at 2%
so if i have 1.1% to big
and -1.2% small
it will select the -1.2%?
does it select the one closest to 2% or less? or the one that is closest to 2% that is bigger than 2% ?
just trying to understand how it works with the err better...

[jonny]

i had some undersize by 50 megs or so...

yep, this is quite normal, what was your target size?


so if i select 1-3% err. does it give me 1-3% too big or 1-3% too small? or it just choses which is closest to the 1-3%.

It works with the absolute error, the closest value is used (so the sign is ignored when i compare with ERR%)

[SiliconSoul]

Originally posted by jonny
yep, this is quite normal, what was your target size?


It works with the absolute error, the closest value is used (so the sign is ignored when i compare with ERR%)

target size was 922 mb
so i just put the movie % upto 20% last night and encoded that one episode and now it is just -14MB

maybe an option for it to use - or + when it find the err.

okay now i have a problem. i was using the big3 but i decided to use qcce for the encoding part. used RA to create the scenarist script and when i try and use that script and import it i get this error.
am i going to have to do this step manually? a setting i set wrong when encoding?


Warning This Video [ C:\Winfast_workarea\dvdripping\Soprnaos.s1d2.try2\SOPRANOS2\VTS01\VTS__01_P01.16~9_1.MPV.M2V ] has different duration.
Error C:\Winfast_workarea\dvdripping\Soprnaos.s1d2.try2\SOPRANOS2\VTS01\VTS__01_P01.16~9_1.MPV.M2V
Error : Number of fields (38) in GOP(# 2) or between sequence_header_code should be equal or smaller than 36.

Error Importing failed

[jonny]

maybe an option for it to use - or + when it find the err.

You can obtain something similar keeping ERR%=0 and slightly lowering/rising the target size


okay now i have a problem. i was using the big3 but i decided to use qcce for the encoding part. used RA to create the scenarist script and when i try and use that script and import it i get this error.
am i going to have to do this step manually? a setting i set wrong when encoding?

Try to import the m2v manually in scenarist ( i think it's not a qcce issue... anyway i've never used big3, RA, scenarist ^^' )

[SiliconSoul]

okay strange.... i still ge this error then i do it manually....

Error C:\Winfast_workarea\dvdripping\Soprnaos.s1d2.try2\SOPRANOS2\VTS01\VTS__01_P01.16~9_1.mpv.m2v
Error : Number of fields (38) in GOP(# 2) or between sequence_header_code should be equal or smaller than 36.


anyone else have this problem? anyone else use the big3 or scenarist manually?

well just to test something im encoding the same 4 video files with batchccews with 1 pass vbr and the Q that qcce gives and all the other settings the same and ill see what happens in 4 hours. would be quicker but one of the episodes needs
Telecide()
Decimate(cycle=5)
as per the avs script doitfast4u produced.

just trying to help out... i want this to work! speed savings will be great! :)

[jonny]

The m2v plays fine with PowerDVD/WinDVD?

Could this be related to GOP size=15? (maybe you could try 12)

Can you post the full .avs and the settings used in qcce?

:EDIT:
What CCE version are you using?

[SiliconSoul]

yeah the m2v plays great in powerdvd and looks great.
okay ill try the GOP size of 12... i also noticed that when encoding with batchccews the GOP stays closed the whole time... is this related to the GOP size? with qcce it changes across the encode from closed to open.

same script works fine in batchccews with multipass just takes forever for it to encode.
full avs

LoadPlugin("C:\Winfast_workarea\avsfilters\2.5\mipsmooth\MipSmooth.dll")
import("C:\Program Files\DoItFast4U\new.avs\addaudio.avs")
LoadPlugin("C:\Program Files\DoItFast4U\new.avs\mpeg2dec3.dll")
Mpeg2Source("C:\Winfast_workarea\dvdripping\Soprnaos.s1d2.try2\SOPRANOS2\VTS01\VTS__01_P01.16~9_1.d2v",idct=0)
AddAudio()
MipSmooth(preset = "MovieLQ")
ConvertToYUY2(interlaced=true)

qcce settings
gop 15
min 0 max 9000
QC 30
DC 10
AR 16:9
interlaced PFF *this might be setup wrong im not sure*
BSO ZZ *im encoding progressive material*
FR 23.976
MB 922
SP 5%
qfind err% 0.0 minq 1 max 60q
min check and encode no max

cce 2.67.00.23

[SiliconSoul]

Q Find
[1 - 60] [30] [1,287,301,758] [966,787,072] [320,514,686] [33.15%]
[30 - 60] [45] [951,420,794] [966,787,072] [-15,366,278] [-1.59%]
[30 - 45] [37] [1,101,655,842] [966,787,072] [134,868,770] [13.95%]
[37 - 45] [41] [1,019,731,130] [966,787,072] [52,944,058] [5.48%]
[41 - 45] [43] [984,005,697] [966,787,072] [17,218,625] [1.78%]
[43 - 45] [44] [967,369,422] [966,787,072] [582,350] [0.06%]
Encode (44)

aoky this is with 5% and err 0
so it chose 44
i did it again manually with 43... -25MB
might even be able to do it with 42 and hit the mark...
so how do i ensure that it choses 42-43? or not encode and chose myself?

[SiliconSoul]

Originally posted by jonny
The m2v plays fine with PowerDVD/WinDVD?

Could this be related to GOP size=15? (maybe you could try 12)

Can you post the full .avs and the settings used in qcce?

:EDIT:
What CCE version are you using?

GOP size of 12 fixes the problem. :)
thanks!

im still learning! :)

[SiliconSoul]

sopranos s1d2 922 megs for each episode
okay here is some SP ive done...
i encoded fully at 5% and only SP at 2%
at 5% encoded
vid1 Q 44 = -50MB about
vid2 Q 39 = -14.04MB
vid3 Q 56 = 0.91MB
vid4 Q 38 = -14.25MB

since it was so small i reencoded manually with
vid1 Q 43 = -26MB about

2% SP
vid1 Q 41 should be close
vid2 Q 38 should be close
vid3 Q 54 going to be big
vid4 Q 41 going to be small

im going to reencode 1 and 2 with the 2% prediction and 4 with Q 37

how can i get better prediction?
do i need to up it to 10% movie so that i do not have to worry about wrong size as much?
do i need to go as much as 25% ?

[jonny]

GOP size of 12 fixes the problem

Ok :)
I've found some resources about the problem:
http://forum.doom9.org/showthread.php?s=&threadid=48993&highlight=sequenceheadercode+importing
http://forum.doom9.org/showthread.php?s=&threadid=22069&highlight=Number+of+fields+in+GOP
http://forum.doom9.org/showthread.php?s=&threadid=31748&highlight=sequenceheadercode+importing

If you try to search in the forum for:
sequence_header_code importing
or
Number of fields in GOP
you should get more resources (could be interesting for you to learn more about the problem).


About the %:
There is no general rule, you'll always get an error, no matter how much you rise the %
Going with an higher %, increase the possibilities of getting a better prediction (but this doesn't mean you'll get always a better one)
Plus, using a really high %, you lose the target: speed (at this point a conditional resizing pass would be better instead of trying to get the nearest Q with many tries)

[r6d2]

Originally posted by jonny
Plus, using a really high %, you lose the target: speed (at this point a conditional resizing pass would be better instead of trying to get the nearest Q with many tries) @SiliconSoul, Remember that in the general case you have to do roof(log(60,2)) = 6 passes, so if you use a high %, like 5% for example, you're in fact doing 1.3 passes.

jonny has increased prediction accuracy a lot, but it doesn't always work perfectly. Besides, Q is an integer, so one step variation may produce a huge difference in output size, making it impossible to get 100% target fit in every case.

[SiliconSoul]

Plus, using a really high %, you lose the target: speed (at this point a conditional resizing pass would be better instead of trying to get the nearest Q with many tries)

can qcce do a conditional resizing pass? what exactly is that?
is that back to normal multipass?

jonny has increased prediction accuracy a lot, but it doesn't always work perfectly. Besides, Q is an integer, so one step variation may produce a huge difference in output size, making it impossible to get 100% target fit in every case.

yeah that is a problem. and that is why a little under is good.

[r6d2]

Originally posted by SiliconSoul
can qcce do a conditional resizing pass? what exactly is that?
is that back to normal multipass?The guide on my sig can give you the whole picture. On the Table of Contents, go to "D2Sroba unveiled".

[HypFra]

Hi Jonny!!!

We didn't meet @ SMAU (remember?:) ), but I can always find u on the web!!!
I will try this interesting software and I'll let you know. Here or on QCIPC, obviously.

See ya!!!

HypFra

[jonny]

Hi!!!

This year will be better ( i hope :D )
I've got 2 phones :)

[r6d2]

jonny, are you planning to support target sizes in a more "natural" way, like number and type of media, for instance, and allow for 1 or 2 audio tracks and their BR?

/Add
Or even simpler: Target BR?

[jonny]

yep, this is what i mean with "ts calculator" :)
(anyway i'll be a little out for sometime, i've got dsl connection and i'm a little "distracted" for this ^^)

[burnix]

Hello. I'm using your soft, in the latest version 1.02. I use too, cce 2.67.00.23 and eclcce 1.7b.
For encoding mpeg2 file (d2v) there is no problem and all works well.
But i experience problem with avi file and directshowsource.

I use this script :
---------------------------------------------------------

directshowsource("f:\fini\SaturdayNightFever.avi",25)
LanczosResize(720,448)
addborders(0,64,0,64)

-----------------------------------------------------------

And i have this result :

----------------------------------------------------------

[1 - 300] [150] [912,143,794] [1,992,294,400] [-1,080,150,606] [-54.22%]
[1 - 150] [75] [1,338,506,235] [1,992,294,400] [-653,788,165] [-32.82%]
[1 - 75] [38] [2,251,633,179] [1,992,294,400] [259,338,779] [13.02%]
[38 - 75] [56] [1,649,889,476] [1,992,294,400] [-342,404,924] [-17.19%]
[38 - 56] [47] [1,891,372,443] [1,992,294,400] [-100,921,957] [-5.07%]
[38 - 47] [42]00:21:47 [Error] Evaluate: Unrecognized exception! (tmp\qcce.avs, line 2) QF 1.0% (ERR=0.0% MinQ=1 MaxQ=300 Enc MinQ=On) 1900 MB "Sat" F:\TempTransfert\Avs\sat.mpv F:\TempTransfert\Avs\sat.vaf (0-9000) GOP=15 QC=16 DC=10 AR=1:1 FR=25.000 PFF BSO=zz "Standard" C=0 C:\Program Files\Custom Technology\CCE SP Trial Version\EclCCE.exe
Q Find
[1 - 300] [150]00:00:01 [Error] DirectShowSource: Could not open as video or audio. Video returned: "DirectShowSource: couldn't open file f:\fini\Saturn3.avi: ClassFactory ne peut pas fournir la classe demandée " Audio returned: "DirectShowSource: couldn't open file f:\fini\Saturn3.avi: ClassFactory ne peut pas fournir la classe demandée " (tmp\qcce.avs, line 1) QF 1.0% (ERR=0.0% MinQ=1 MaxQ=300 Enc MinQ=On) 1900 MB "Saturn3" F:\TempTransfert\Avs\saturn3.mpv F:\TempTransfert\Avs\saturn3.vaf (0-9000) GOP=15 QC=16 DC=10 AR=1:1 FR=25.000 PFF BSO=zz "Standard" C=0 C:\Program Files\Custom Technology\CCE SP Trial Version\EclCCE.exe

------------------------------------------------------

The problem was the same with avisynth 2.54 and 2.55alpha.

The same script with mpeg1 file (vcd) works great.

So if you have an idea, i thank you a lot.

[jonny]

The problem seems related to AviSynth and/or DirectShow.

With avi source, should be better to use:
OpenDMLSource
AviFileSource
You could try both (AviFileSource have 2GB limitation on the input and use the vfw interface, OpenDMLSource have no limitation and use the same code of VirtualDub).

Let me know if this fix the problem.

[Wilbert]

1) Use AviSource instead of DirectShowSource (for avi).

2) If you still want to use DirectShowSource (for some reason), open the avi first in graphedit to see what dshow filters are used to play it.

[r6d2]

@jonny, did you have fun with your DSL yet? :) If you're about to implement Newton into Qcce and need any help, just let me know.

[burnix]

First i use directshowsource in my script because with avisource and opendmlsource, i have audio sync problem. I can play with avisynth, but when i encode i have delay on audio. With directshowsource, no delay (if someona have some idea).

If i do a "render file" with graphedit, it use classical filter and works well.

The strange thing was that, if i put my avs script directly in cce, i can encode it normally in cbr mode.

I think there is a problem on qcce or eclcce. In line 2 of my script i sue LanczosResize, so can try bicubic to see.

One last thing. Why a directshowsource.dll file in plugin directory of avisynth, and how to use it ?

I keep your informed. Thanks.

[jonny]

During size prediction, a lot of seeking is done in order to extract snips (with selectrangeevery), in the case of directshowsource, this is done with directshow. @Wilbert, could this be the problem?

Anyway, is the problem still present using avisource?
(you could use avisource and demux/process the audio saparately)

[Wilbert]

One last thing. Why a directshowsource.dll file in plugin directory of avisynth, and how to use it ?
It is turned into a separate plugin, due to compiling reasons (only important for the ones who are compiling AviSynth). The use is exactly the same as before :)

Maybe the audio is VBR MP3? In that case you should do what jonny advices (demux/process audio separately).

[burnix]

So i made test. Opendmlsource works great. Now i have to see, if there is no problem with audio.

[ADLANCAS]

Hey jonny!

Are you trying to make a faster Q prediction ?
Should be very very nice to have this approach!

[jonny]

I've a faster approch "in progress", but i have really reduced free time in this period ( maybe something new will be out in the "ester eggs" week :) )

[ADLANCAS]

Nice to hear that:D

[Boulder]

jonny,

any new versions coming out? ;)

If you're fiddling with one at the moment, it would be great to have an option to alter the bias value too since I feel that for my encodes the default 30 is too large.

Thanks!

[ADLANCAS]

jonny
Are you there ?:(

[jonny]

Hi

I'm still alive and it's all ok :)
I'm sorry but i'm really busy (for a couple of different reasons) and i'm rarely visiting the forum/working on the encoding side :(

[Amnon82]

new version online: http://forum.doom9.org/showthread.php?s=&threadid=77825

[Amnon82]

QCCE and QCCE CyberEdition are RIP now. If You need a Predictiontool for CCE use the new AutoQ.

AutoQ Thread @ Doom9 (http://forum.doom9.org/showthread.php?t=87764)