Other matrices in BatchCCEWS

[LeXXuz]

I'd like to use other matrices in BatchCCEWS than the normal ones implemented.

How can I do this ?

[Trahald]

Im looking into that actually .. this week im going to tackle matrices.. (ie for quenc/hcenc and cce) im trying to see if i can use the same location (within the gui) for all 3.. although people may want different matrices for each encoder.. ive never really played with custom matrices, usually the ones offered have sufficed ( i only ever used ultra low on some svcd stuff and an occasional half d1 dvd ).. i'll see what happens as a experiment... there is actually commented out code for matrices.. so the groundwork is there.

for now to answer your question i imagine things would have to be done manually.

[LeXXuz]

...ive never really played with custom matrices, usually the ones offered have sufficed...


Not to me. I usually get better results by using the same matrices for quantisation as of the original movie.


If there's still no way to tweak a custom matrix into BatchCCE
I guess I still have to use my CCE manually :(

[Trahald]

well like i said.. even if i may not use it.. im still going to add it.. so custom matrix support should be in batchcce by the end of the week. (for the cce versions that directly support it)

[LeXXuz]

well like i said.. even if i may not use it.. im still going to add it.. so custom matrix support should be in batchcce by the end of the week. (for the cce versions that directly support it)

That would be nice. Thanks in advance. :)

[Zeul]

@LeXXuz
Let me get this straight. When you do a re-encode you always determine the matricies from the original, and then use the same in cce/hc?
How are you determining the matrix? I know its written in the video stream.
Does using an identical matrix make that much difference?
I'm with Trahald, i never play with matricies - always use the default ones, however, that may change ;)

[influenza]

Well assuming the original matrix is the optimal one (and that that still holds ground with a different average bitrate) it would seem to make sense. I'm also curious how you would determine that matrix though.

If things could be automated it would be a nice feature to implement of course. :)

[mpucoder]

This is interesting. From what I can figure, the original matrix would not have to be the optimal matrix for this to improve the quality of the encode. It is because it is the same matrix that quantization errors are reduced. When you multiply a number (e.g. quant x coefficient during decoding) and then divide it by the same number (encoding) there is no rounding error.

BUT... mpeg-2 can change the matrix at the frame level, so how do you coordinate that?

[influenza]

I'm already sorry I replied :D. This is getting over my head :)

I have absolutely no idea if a 'regular' encode uses the same matrix for the entire encode or that this will change at frame/cell/vobid level. I would be interested to know though.

[Guest]

I'm also curious how you would determine that matrix though.

If things could be automated it would be a nice feature to implement of course. I've released a CLI quant matrix dumping utility that could be used to automate the process:

http://neuron2.net/dgmpgdec/quantdump10.zip

It detects and reports any changes to the matrices encountered in the stream, including both at the sequence and picture levels.

mpucoder correctly points out that the quant matrix extension allows the matrices to be changed on every picture. In practise, however, I've never seen it used. I've seen only changes at the sequence level. mpucoder, have you ever seen a quant matrix extension?

[influenza]

Ok gonna play around with that for a bit, thanks.

[mpucoder]

@neuron2 - Honestly, no, I've never seen any encoder do this (but I haven't looked as hard at it as you have).

@anyone - Does CCE allow for more than one custom matrix, or is it the same custom matrix for the entire encode?

[Guest]

Ok gonna play around with that for a bit, thanks. The dumper doesn't explicitly say if it is a sequence or picture extension, but you can deduce it from the headings it prints. If it says "Luma and Chroma..." then it is sequence level, because that extension cannot change the luma and chroma individually. if it says "Luma ..." or "Chroma...", then it is picture level.

[Angelus]

@anyone - Does CCE allow for more than one custom matrix, or is it the same custom matrix for the entire encode?

I'm not sure but the newer cce 2.70 and up have a setting where you can enable or disable adaptive q-matrix switching. So I'm guessing this analyzes the stream and implements what it thinks is the best matrix for that?

[Zeul]

wow
All the heavyweights are here :D
On a film source this is the output:

Default Luma and Chroma Intra Quant Matrix:
8 16 19 22 26 27 29 34
16 16 22 24 27 29 34 37
19 22 26 27 29 34 34 38
22 22 26 27 29 34 37 40
22 26 27 29 32 35 40 48
26 27 29 32 35 40 48 58
26 27 29 34 38 46 56 69
27 29 35 38 46 56 69 83

Default Luma and Chroma Non-Intra Quant Matrix:
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16

Luma and Chroma Intra Quant Matrix:
8 8 9 11 13 13 14 17
8 8 11 12 13 14 17 18
9 11 13 13 14 17 17 16
11 11 13 13 13 17 18 20
11 13 13 13 16 17 20 24
13 13 13 16 17 20 24 29
13 12 13 17 19 23 28 34
12 13 17 19 23 28 34 41

Luma and Chroma Non-Intra Quant Matrix:
8 8 8 9 9 9 9 10
8 8 9 9 9 9 10 10
8 9 9 9 9 10 10 10
9 9 9 9 10 10 10 10
9 9 9 10 10 10 10 11
9 9 10 10 10 10 11 11
9 10 10 10 10 11 11 11
10 10 10 10 11 11 11 11

I presume the 'default' is always printed (should have looked at the code b4 posting), and if the video differs from this then the new matricies are written. If we were to use this in cce/hc then the non-default ones would be used, correct?

[LeXXuz]

Wow what happened here? :eek: :D

@LeXXuz
Let me get this straight. When you do a re-encode you always determine the matricies from the original, and then use the same in cce/hc?
How are you determining the matrix? ...

Yes. I always use the matrices from the original for my recodes. I use ssh's Restream to readout the matrices and then paste them into the CCE manually.

Well assuming the original matrix is the optimal one (and that that still holds ground with a different average bitrate) it would seem to make sense.

... When you multiply a number (e.g. quant x coefficient during decoding) and then divide it by the same number (encoding) there is no rounding error....


That's exactly the idea behind all this. Assuming you don't change a single bit of information in the video stream (no colorspace conversion, resizing etc...) you would not get any "additional" quantisation errors by using the same matrices.

A long time ago I made quite a hard run on testing the results of different movies encoded with a bunch of different matrices. I let a lot of people view the results and also viewed the frame differences of original and copy in PhotoShop. What I saw was quite identical to the opinion of my "test audience". The recode with the same matrices for intra and nonintra made the run.

Example: the original movie is quite noisy and uses a nonintra matrix with "11"-quantisation (F*x Home Entertainment often uses these) and I made a recode with a "harder" quantisation (like the nonintra CCE uses as default) the background noise "changes" and looks much more disturbing than before.
On "clean" movies like the ones recorded with digital equipment I recognized another effect when using extremely different matrices. Very fine structures seem to begin what I call "swimming".
It's very hard to me to put this into the right words...
The bottom line after weeks of testing was: the recode always looked much more pleasant when using the same matrices and of course still having enough bitrate.

@neuron2 - Honestly, no, I've never seen any encoder do this (but I haven't looked as hard at it as you have).


Me neither. And I'm quite sure there is NO authoring/encoding studio out there using this feature.

[Guest]

I presume the 'default' is always printed (should have looked at the code b4 posting), and if the video differs from this then the new matricies are written. If we were to use this in cce/hc then the non-default ones would be used, correct? Yes, the MPEG2 default matrices are always printed first, because they are in effect unless a stream definition changes them. Then if the stream changes any of them, the changed matrix(s) are printed and the new one stored to detect further changes. So all *changes* to the matrices are reported. If the same matrices are sent in all sequence headers, I only print it the first time it appears. I don't know anything about what CCE will use. This utility just dumps the matrix changes. What you do with that information is up to you.

[Zeul]

thanks neuron2
I can see that the whole concept of using matricies needs more attention. It seems that cce/hc support 1 custom matrix of Intra and Non-Intra but there is no mention of differentiating between Luma and Chroma. So my best guess would be to use the dumped matricies when the matrix is Luma AND Chroma, otherwise default to Mpeg2 matrix. Of course Hank315 maybe interested in this thread - and then perhaps HC could be adapted to accomodate various matricies throughout the encode.

[hank315]

and then perhaps HC could be adapted to accomodate various matricies throughout the encode.When I started HC it was one of the things I wanted to implement but I skipped it because some hardware players will not play it.
It is however completely DVD-compliant but even CCE gives a warning in their manual that their adaptive quantization matrix switching isn't supported by some HW players.

One of the strange things of MPEG2 is you can give Luma intra and inter Q-matrices in the sequence header (for DVD this means every GOP) and separate Luma/Chroma intra and inter Q-matrices in the quant matrix extension (for every frame).
BTW, I also never have seen a quant matrix extension in a MPEG/DVD stream.
IMHO it would have made more sense to provide separate Luma/Chroma matrices per GOP.

The 4 matrices given by Zeul seem to be an example of the adaptive Q-matrix switching of CCE SP,
this method is used by CCE to keep the bitrate above a certain minimum but it can also improve quality in dark scenes and fades.

But I still like the idea of different matrices for different parts of a movie, dependend of high/low action, bitrate etc.
Implementing such a feature isn't very difficult but choosing the right matrix for every GOP is another story, requires alot of testing...

[Zeul]

Thanks for responding hank315
Is there any way HC could determine the original matrix from the original source on-the-fly whilst encoding?

[hank315]

No, that's not possible ATM.
The only thing HC gets from the source is some header info as width, height and fps.
And ofcourse the frames in YV12 format.
Then a whole new encode starts and because the bitrate is lower than the original bitrate other matrices could do a better job.

I think the next matrices should be a good starting point for most sources:
Intra
8 16 19 22 26 27 29 34
16 16 22 24 27 29 34 37
19 22 26 27 29 34 34 38
22 22 26 27 29 34 37 40
22 26 27 29 32 35 40 48
26 27 29 32 35 40 48 58
26 27 29 34 38 46 56 69
27 29 35 38 46 56 69 83

Inter
16 17 18 19 20 21 22 23
17 18 19 20 21 22 23 24
18 19 20 21 22 23 24 25
19 20 21 22 23 24 26 27
20 21 22 23 25 26 27 28
21 22 23 24 26 27 28 30
22 23 24 26 27 28 30 31
23 24 25 27 28 30 31 33

Intra=standard MPEG.
Inter="modified standard" MPEG, used in CCE and alot of other encoders.

Alot of people will have a different opinion about it but this combination is still one of the most used Intra/Inter matrices.

[Trahald]

ok custom matrix support added for hcenc, cce and quenc in batchccews 0916i unstable.

[LeXXuz]

ok custom matrix support added for hcenc, cce and quenc in batchccews 0916i unstable.

Many :thanks: for the fast work Trahald.
I'll give it a try asap. :)

[influenza]

Well let's see if someone can code the use of neuron's/donald's tool into scenaid/dif4u ;);)

edit: it could also be programmed into batccews of course...... ;)

[Trahald]

yeah.. i think i will add that as an option. i guess i will have to use the last matrix it finds in the video. i'll modify the source to maybe not check the entire file (which can take some time) ... 99% of the time there is only one matrix

[Zeul]

already available in numenu for next release :D

[D3s7]

/me glares at flu

yeah yeah... why not just come out say it.... we all know who you mean

[influenza]

;)

Well I think batchccews is a better place for it anyway. As long as there's a switch in scenaid to control it :D

and Thanks for the quick programming guys :thanks:

[jlobee]

Thx for the updates Trahald.

Theres one potential bug in batchccews 0916i unstable, but wasnt there in 0916h.

Upon starting a batch encode with CCE 2.70, in 0916i it just opens up the encode menu and the user needs to press the encode button for the encode to begin. Whereas in 0916h it opened up the encode menu and started the encode automatically.

If you can look into that, id appreciate it.

Thanks

[Trahald]

2.70 is my main use cce. havent experienced that issue. can you email me your .ecl and your batchcce log. wewk584 at users.sourceforge.net
thanks

[Zeul]

A couple of suugestions that came to light when i started the new guide. If a non cce app is to be used, can we turn off the eclcce warning.

After the settings screen is closed, enable just the template, so the encoder can be selected. At the moment this requires an avs file loaded to do this.

[LeXXuz]

ok custom matrix support added for hcenc, cce and quenc in batchccews 0916i unstable.


Works fine so far. :)

BTW: Can anybody explain to me why BatchCCE tries to do a pulldown on PAL-Material ?

00:02:54 Starting Job: VTS__03_P01.I-TFF.4~3_1
06:24:13 Pulldown Started
06:24:13 SOURCE = D:\kr3\20050730_115303\VTS03\VTS__03_P01.I-TFF.4~3_1.mpv
06:24:13 TARGET = D:\kr3\20050730_115303\VTS03\VTS__03_P01.I-TFF.4~3_1.mpv.m2v
06:24:13 - no .mpv file being made
06:24:13 - disabling 2:3 pulldown
06:24:13 - setting frame rate to 25.000000
06:31:54 EOF. Emptying buffer and exiting. (pulldown.cpp)
06:31:54 Total Frames = 69365
06:31:54 Total Fields = 173413
06:31:54 Total GOPS = 6028
06:31:54 - Average bitrate = 3980933 bits/sec
06:31:54 - (MPEG stamped @ 9000000 bits/sec)
06:31:54 Time taken = 461 seconds...
06:31:54 Creating: D:\kr3\20050730_115303\VTS03\VTS__03_P01.I-TFF.4~3_1.mpv.m2v.vif
06:31:55 Highest Field Count: 18 AtGop : 1
06:31:55 Time taken = 1 seconds...
06:31:55 Pulldown successfull.
06:32:40 Job: complete in 23386 seconds

Pulldown is disabled, but still runs through the whole file. Why ?
What's the purpose of the *.vif Files ?

[Trahald]

It will do that if you have the' always create vif' option on. 2:3 pulldown isnt actually done. basically it uses pulldown to parse the mpeg.. its what scenarist does.
the vif file tells locations of the syncs for gops and pictures and other stuff which scenarist uses during dvd compile. now if you are using muxman then you can turn that option off as muxman doesnt use scan files.. it figures out everything during compile. you can also turn it off if you use scenarist and just prefer it to do it on its own. (it creates the vifs during import)

i keep it on cause it can take a while and i do my encoding at night and my compiling during hte day.. so id rather it create the vif while im sleeping. although i rarely use scenarist now adays anyways, only for angle movies. i use muxman for everything else.

[LeXXuz]

Thanks for the info. :)

[Trahald]

I removed the warnings. or at least lightened them.. it simply states that there is no eclcce as in info. equivelant to giving the versions.

i put in a new button that will open edit mode and allow you to choose a template to edit without an avs file. you can add an avs file to that later on.

[Zeul]

sounds good. as soon as it is on sourceforge i will have a look.

[Trahald]

Thx for the updates Trahald.

Theres one potential bug in batchccews 0916i unstable, but wasnt there in 0916h.

Upon starting a batch encode with CCE 2.70, in 0916i it just opens up the encode menu and the user needs to press the encode button for the encode to begin. Whereas in 0916h it opened up the encode menu and started the encode automatically.

If you can look into that, id appreciate it.

Thanks

i tested the .ecl you sent on the command line and cce started right up encoding

can you open up a command line and do '*path to ccs*\cctspt.exe -ecl "*pathtoeclfile*\VTS__01_P01.P.16~9_1.ecl" -batch
put your paths and files of course

see if cce starts right to encoding .. if it doesnt.. see if you can cut out the info from the cmd window cce spits out and paste here.

[jlobee]

i tested the .ecl you sent on the command line and cce started right up encoding

can you open up a command line and do '*path to ccs*\cctspt.exe -ecl "*pathtoeclfile*\VTS__01_P01.P.16~9_1.ecl" -batch
put your paths and files of course

see if cce starts right to encoding .. if it doesnt.. see if you can cut out the info from the cmd window cce spits out and paste here.

cce does not start right to encode, and it does not spit out any info in the command line.

[Zeul]

what is the path to cce and the ecl that you have used. Are they surrounded by quotes?
ie "c:\program files\custom technology\cctspt.exe" -ecl .....

[Trahald]

jlobee - the version of cce you have (2.70.02.00) has a bug in it.. thats why your ecl worked for me since i use 2.70.01.05. cinemacraft fixed the bug in 2.70.02.01... but as a workaround in the 0916k i force cce to minimized when 2.70.02.00 is used which seems to fix it. use 0916k and you should be ok.

[jlobee]

thanks, works great, one thing that I noticed:

Using CCE (2.70.02.00), if you don't have the "Hide CCE Main window on execution" option checked upon starting the program, but later check that option without restarting batchccews, I guess it resets the "force cce minimize" option you had set for this version of CCE and CCE does not autostart.

This only happens if you follow the above steps, and isnt a big deal.

[Trahald]

@jlobee
thanx... i'll look at that..

btw .. i have setup the an option to use the original matrix .. (the 5th matrix option is 'OriginalM2VMatrix'... when this is set as the matrix , it will use quantdump by neuron2 (i compiled it into batchccews and just .cpp'd it a bit) to create a matrix file for the original matrix from the .m2v and use that in the ecl.. you can set original m2v matrix in your template or use a 'force original m2v matrix during import' option in settings to have the original matrix option automatically set when ccedata is loaded (you can change individual jobs later)

[LeXXuz]

btw .. i have setup the an option to use the original matrix .. (the 5th matrix option is 'OriginalM2VMatrix'... when this is set as the matrix...

Sounds good :D Will the upcoming version have this option ?

[Trahald]

yepper.. just testing at the moment. busy painting and stuff (domestic labor) so will be a few days b4 i put it out.

[Steel]

IM using CCE 2.70.02 and batchccews 916n. Trouble is, I used to use the robshot-bach template. But this no longer works. The template is there in batchccews. But when u encode, CCE does one pass, then no more. It never gets past the first one. ANy suggestions?

[Trahald]

can you post some of the log? i'll look into it..

[Steel]

which log?

[jel]

BatchCCEWS log file

File > Settings > Enable Logging.

O/T: @Trahald - love the new look mate, very nice :D also want to send out a big thanks for continuing development on 'the best durn CCE batch file processor around!"
YOU ROCK!

[salkku]

Hello and thanks again for this great program :)

Do you need to use ecl-file for the quantdump to work? I keep getting error saying something like "error getting matrix" and I am not using ecl.

[Trahald]

quantdump is called during ecl creation ... but i set the program to create ecls even when its not needed so quantdump is run. what encoder are you using... i'll check to see whats up.