Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 ASP

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th April 2004, 10:57   #1  |  Link
SoonUDie
Registered User
 
Join Date: Dec 2003
Posts: 147
DCTune Custom Matrix Discussion (thread-split)

I have decided not to continue work on matrices at the moment, because I think I have a better strategy: calculate a new matrix for every encode by using DCTune on about 200+ frames of the source and averaging the results. Tests have been encouraging.
__________________
Stuff was here at some point.

Last edited by Teegedeck; 1st May 2004 at 13:49.
SoonUDie is offline   Reply With Quote
Old 26th April 2004, 11:04   #2  |  Link
dimzon
BeHappy/MeGUI developer
 
dimzon's Avatar
 
Join Date: Oct 2003
Location: Moscow, Russia
Posts: 1,727
Quote:
Originally posted by SoonUDie
calculate a new matrix for every encode by using DCTune on about 200+ frames of the source and averaging the results
please, write small guide for beginners!
dimzon is offline   Reply With Quote
Old 26th April 2004, 11:05   #3  |  Link
iradic
Registered User
 
Join Date: Jan 2004
Posts: 332
@SoonUDie
Could you post your usage of DCTune... (like some little guide)?

Thanks

EDIT: dimzon was faster...
iradic is offline   Reply With Quote
Old 26th April 2004, 11:40   #4  |  Link
dimzon
BeHappy/MeGUI developer
 
dimzon's Avatar
 
Join Date: Oct 2003
Location: Moscow, Russia
Posts: 1,727
Quote:
calculate a new matrix for every encode by using DCTune on about 200+ frames of the source and averaging the results
Hmm. Anybody knows DCTune algorythm? Maybe best way is AVS2QMATRIX automation tool?
dimzon is offline   Reply With Quote
Old 26th April 2004, 15:56   #5  |  Link
SoonUDie
Registered User
 
Join Date: Dec 2003
Posts: 147
It would be awesome if the process could be automated, but I don't really have a clue how to do it at the moment. It's already automated for the most part (but in different steps).

My chain of processing goes like this:
1. Use Avisynth to create frames in native DCTune format (.ppm).

Example script:
Code:
SelectEvery(last, 1000, 1)
ConvertToRGB()
ImageWriter(last, file = "C:\dctune_raw\yoursource", start = 0, end = 0, type = "ppm")
This will output every thousandth frame in the form of yoursource000000.ppm, yoursource000001.ppm, etc.



2. Use BathEncoder to call DCTune on all of the images. Simply drag and drop the files into it, set your encoding switches, and hit "go". BathEncoder creates a batchfile which calls DCTune on every image. You can get it here. Be sure to put DCTune in the same folder.

The switches I normally use for DCTune are:
dctune2.0 -f <infile> -p 1 -inches 8 -dpi 133 -o <outfile>_quants_p1.txt -j <outfile>_p1.jpg

Note the -p value. This is basically your quality setting. -p 1 is normally perceptually lossless. For a high-bitrate encode, I normally use this to create my intra-matrix; then I run it a second time with -p 2 and change the outputs accordingly, and use those to create my inter-matrix.

In general, I use the following p-values:
High bitrate: 1, 2
Medium bitrat: 2, 3
Low bitrate: 3, 4

The high bitrate matrices also look good at low bitrate, but you have a higher average quantizer (so it looks a little ragged).



3. Average the results to get a good quantization matrix. I'm currently using a pretty simplistic program I wrote in Java to do this. I'm not really a programmer, so I have no clue how to export it at the moment. I *might* be able to turn it into an applet.
__________________
Stuff was here at some point.

Last edited by SoonUDie; 26th April 2004 at 15:59.
SoonUDie is offline   Reply With Quote
Old 26th April 2004, 16:20   #6  |  Link
dimzon
BeHappy/MeGUI developer
 
dimzon's Avatar
 
Join Date: Oct 2003
Location: Moscow, Russia
Posts: 1,727
Quote:
Originally posted by SoonUDie
It would be awesome if the process could be automated
Seems like I can write simple avs2qmatrix.bat to automate it
===================
Done!
Create such directory structure:
Quote:
c:\dctune\
c:\dctune\frames\
c:\dctune\trash\
c:\dctune\inter\
c:\dctune\intra\
place avs2avi.exe and dctune2.0.exe into c:\dctune\ folder

place extract.bat into c:\dctune\ folder
Quote:
@echo off
del /Q c:\dctune\frames\*.ppm > NUL
echo Import("%1") > temp.avs
echo SelectEvery(last, %2, %2/2) >> temp.avs
echo ConvertToRGB24() >> temp.avs
echo ImageWriter(last, file="c:\dctune\frames\x", start=0, end=0, type="ppm") >> temp.avs
echo crop(0,0,8,8) >> temp.avs
c:\dctune\avs2avi temp.avs -o n -c DIV3
place calculate.bat into c:\dctune\ folder
Quote:
@echo off
del /Q c:\dctune\intra\*.* > NUL
del /Q c:\dctune\inter\*.* > NUL
FOR %%i IN (c:\dctune\frames\*.ppm) DO c:\dctune\dctune2.0.exe -f %%i -p %1 -inches 8 -dpi 133 -o c:\dctune\intra\%%~ni.txt -j c:\dctune\trash\last.jpg
FOR %%i IN (c:\dctune\frames\*.ppm) DO c:\dctune\dctune2.0.exe -f %%i -p %2 -inches 8 -dpi 133 -o c:\dctune\inter\%%~ni.txt -j c:\dctune\trash\last.jpg
cscript c:\dctune\calculate.vbs
place calculate.vbs into c:\dctune\ folder
Quote:
Option Explicit
dim g_oFS ' FileSystem Object
set g_oFS = CreateObject("Scripting.FileSystemObject")

function ReadMatrixFromFile(sFileName)
dim aResult
dim aLine
dim oFile
dim s
dim i,j
redim aResult(63) ' 8X8, zero based
set oFile = g_oFS.OpenTextFile(sFileName, 1)
' Skip file header
oFile.ReadLine
oFile.ReadLine
oFile.ReadLine
oFile.ReadLine
oFile.ReadLine
oFile.ReadLine
oFile.ReadLine
oFile.ReadLine
oFile.ReadLine
oFile.ReadLine

for i=0 to 7 ' 8 lines total
s = oFile.ReadLine
s = replace( trim(s), CHR(32) & CHR(32), CHR(32))
s = replace( trim(s), CHR(32) & CHR(32), CHR(32))
s = replace( trim(s), CHR(32) & CHR(32), CHR(32))
s = replace( trim(s), CHR(32) & CHR(32), CHR(32))
s = replace( trim(s), CHR(32) & CHR(32), CHR(32))
s = replace( trim(s), CHR(32) & CHR(32), CHR(32))
aLine = Split(s,CHR(32))
for j=0 to 7
aResult(i*8+j)=CLng(aLine(j))
next
next
oFile.Close
ReadMatrixFromFile = aResult
end function

function CalculateAverageMatrix(sDirectory)
dim oFile
dim aResult
dim aSubMatrix
dim nCount
dim i
redim aResult(63) ' 8X8, zero based
for i=0 to 63
aResult(i)=CDbl(0)
next
nCount = 0
for each oFile in g_oFS.getFolder(sDirectory).files
aSubMatrix = ReadMatrixFromFile(oFile.Path)
nCount = nCount + 1
for i=0 to 63
aResult(i)=aResult(i) + CDbl(aSubMatrix(i))
next
next
if nCount>0 then
for i=0 to 63
aResult(i)=CDbl( aResult(i) /nCount)
if aResult(i) > 255 then aResult(i)=255
next
end if
CalculateAverageMatrix = aResult
end function

sub main
dim aIntraMatrix
dim aInterMatrix
dim aTotal
dim rMultiplier
dim oXml
dim oStream
dim i
redim aTotal(127)
aIntraMatrix = CalculateAverageMatrix("c:\dctune\intra")
aInterMatrix = CalculateAverageMatrix("c:\dctune\inter")
for i=0 to 63
aTotal(i)=aIntraMatrix(i)
next
for i=0 to 63
aTotal(i+64)=aInterMatrix(i)
next


'Note: The top left value (0, 0) must always be 8
' (may be defined so in the MPEG standard), and other values shall not be below.
rMultiplier = 8/aTotal(0)
for i=0 to 63
aTotal(i)=CLng( rMultiplier * CDbl(aTotal(i)))
if aTotal(i) > 255 then
aTotal(i) = 255
elseif aTotal(i) < 8 then
aTotal(i) = 8
end if
next

'Note: The top left value (0, 0) must always be 16
' (may be defined so in the MPEG standard), and other values shall not be below.
rMultiplier = 16/aTotal(64)
for i=64 to 127
aTotal(i)=CLng( rMultiplier * CDbl(aTotal(i)))
if aTotal(i) > 255 then
aTotal(i) = 255
elseif aTotal(i) < 16 then
aTotal(i) = 16
end if
next

for i=0 to 127
if aTotal(i)>15 then
aTotal(i) = HEX(aTotal(i))
else
aTotal(i) = "0" & HEX(aTotal(i))
end if
next
set oXml = CreateObject("MSXML2.DOMDocument.3.0").createElement("dummy")
oXml.dataType="bin.hex"
oXml.text=join(aTotal,"")
set oStream = CreateObject("ADODB.Stream")
oStream.Open
oStream.Type=1
oStream.Write oXml.nodeTypedValue
oStream.SaveToFile "c:\dctune\result.qmatrix", 2
oStream.Close
MsgBox "Done!"
end sub

CALL Main()
USAGE

1-st - extract frames
extract.bat <source_avs_file_name_and_path> <frame_denominator>
example: extract c:\dvdrip\gigli\gigli.avs 1000

2-nd - calculate qmatrix
calculate.bat <intra_quality>, <inter_quality>
example: calculate 2 3

3-td - got Your matrix c:\dctune\result.qmatrix


Last edited by dimzon; 27th April 2004 at 12:29.
dimzon is offline   Reply With Quote
Old 26th April 2004, 18:59   #7  |  Link
SoonUDie
Registered User
 
Join Date: Dec 2003
Posts: 147
I bow before your skills.

Another note: you should probably go through and manually check all the frames, and delete any ones that you don't want to use. You should definately delete blank frames (all black or all white), and you may want to cut out any credits as well.
__________________
Stuff was here at some point.

Last edited by SoonUDie; 26th April 2004 at 19:12.
SoonUDie is offline   Reply With Quote
Old 26th April 2004, 20:22   #8  |  Link
vaxis
Registered User
 
Join Date: Jan 2003
Posts: 6
This is really cool. I am in the middle of testing it on Iron Maiden - Rock In Rio, which is the hardiest dvd to encode I have ever seen.

I did have to change a couple of lines in the calculate.vbs to get it to work. In the function ReadMatrixFromFile, where the double for loop is, your way of dividing the numbers into the array aLine did not work, some post in it where just blanks and the CLng then returned an error. instead I simply made aLine to a string and implemented the inner for loop a little different and removed the lines with replace calls in the outer for loop.

So here is the loop solution that worked for me:

for i=0 to 7 ' 8 lines total
s = oFile.ReadLine
for j=0 to 7
aLine = trim(mid(s,j*4+1,4))
aResult(i*8+j)=CLng(aLine)
next
next


You should update your script with that.

Again, thanks for this much easier automatic solution.

Last edited by vaxis; 26th April 2004 at 22:05.
vaxis is offline   Reply With Quote
Old 27th April 2004, 00:41   #9  |  Link
SoonUDie
Registered User
 
Join Date: Dec 2003
Posts: 147
I hope it works out for you. In general, I've found that it seems to produce a very good amount of detail - more than HVS Best is most cases. The main plus side, in my mind, is that the matrix is totally adapted to the source. I think the only way you could improve upon this general method would be if XviD automatically calculated and used a different (optimized) quant matrix for each individual scene.
__________________
Stuff was here at some point.
SoonUDie is offline   Reply With Quote
Old 27th April 2004, 00:45   #10  |  Link
SoonUDie
Registered User
 
Join Date: Dec 2003
Posts: 147
Oh, I should mention that DCTune produces matrices with a lowest possible value of 2. You may want to implement a way to multiply all the other values accordingly, so that you always get 8 in the upperp left corner. If not, you can use excel and a little bit of math once you have the final matrix.
__________________
Stuff was here at some point.
SoonUDie is offline   Reply With Quote
Old 27th April 2004, 01:26   #11  |  Link
ObiKenobi
Guest
 
Posts: n/a
Just to make sure I'm doing this right, when DCTune outputs the txt file like the one below, I use the top matrix right?

# Quantization Matrices for G:\000000.PPM
# Using DCTune Version 2.0
# Quantization Table Calculated Using Perceptual Optimization
# Target Perceptual Error 2.000000
# Mean Luminance: 33.532667 Candelas/meter^2
# Peak Luminance: 67.162400 Candelas/meter^2
# Number Gray Steps: 256
# Pixels/Degree X: 18.666667
# Pixels/Degree Y: 18.666667
# Size of DCT: 8
4 4 4 4 4 6 255 255
4 2 4 2 4 4 255 255
4 4 6 4 4 6 255 255
4 4 4 6 10 255 255 255
4 4 4 10 255 255 255 255
6 4 255 255 255 255 255 255
8 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
# Quantization Table Calculated Using Perceptual Optimization
# Target Perceptual Error 2.000000
# Mean Luminance: 33.532667 Candelas/meter^2
# Peak Luminance: 67.162400 Candelas/meter^2
# Number Gray Steps: 256
# Pixels/Degree X: 9.333333
# Pixels/Degree Y: 9.333333
# Size of DCT: 8
18 255 255 255 255 255 255 255
24 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
# Quantization Table Calculated Using Perceptual Optimization
# Target Perceptual Error 2.000000
# Mean Luminance: 33.532667 Candelas/meter^2
# Peak Luminance: 67.162400 Candelas/meter^2
# Number Gray Steps: 256
# Pixels/Degree X: 9.333333
# Pixels/Degree Y: 9.333333
# Size of DCT: 8
8 255 255 255 255 255 255 255
8 255 255 255 255 255 255 255
8 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255
  Reply With Quote
Old 27th April 2004, 02:10   #12  |  Link
iradic
Registered User
 
Join Date: Jan 2004
Posts: 332
hi...

for me -o c:\dctune\inter\%%~ni.txt doesnt work... only -o %%i.txt
using winme

any help...

EDIT: problem solved

Last edited by iradic; 27th April 2004 at 04:14.
iradic is offline   Reply With Quote
Old 27th April 2004, 02:53   #13  |  Link
SoonUDie
Registered User
 
Join Date: Dec 2003
Posts: 147
Quote:
Originally posted by ObiKenobi
Just to make sure I'm doing this right, when DCTune outputs the txt file like the one below, I use the top matrix right?
Yes. I believe the bottom ones are for chroma (not an option with XviD... and even if it was, DCTune tends to suck at chroma except at -p1 or higher quality)
__________________
Stuff was here at some point.
SoonUDie is offline   Reply With Quote
Old 27th April 2004, 02:55   #14  |  Link
ObiKenobi
Guest
 
Posts: n/a
Is it just me or does that matrix seem a bit odd though? Compared to other matrices I've looked at it just looks so much different.
  Reply With Quote
Old 27th April 2004, 08:10   #15  |  Link
SoonUDie
Registered User
 
Join Date: Dec 2003
Posts: 147
Quote:
Originally posted by ObiKenobi
Is it just me or does that matrix seem a bit odd though? Compared to other matrices I've looked at it just looks so much different.
If I understand quant matrices correctly, each entry covers a specified frequency range. Since humans don't perceive these frequencies equally, an optimized matrix will increase the quantization of those frequencies. I believe that the MPEG quant works this way (but uses an older percptual model?), and I know that the HVS matrices (HVS = Human Visual System) work that way, too.

The difference between using a generalized matrix like HVS-best and calculating an average matrix using DCTune is that, theoretically, the averaging method will create a perceptual matrix adapted to the source's quality, and therefore better for purposes of compression, retaining the same amount of quality with less bits. Of course, this assumes that DCTune's alogrithms for creating an optimized matrix are valid.

As an example of the DCTune method, here is the average matrix I got for the SeaBisucit R1 (widescreen) DVD at p=1 (~400 frames sampled, using my program to calculate the matrix):
Code:
8	8	8	10	11	16	86	255
8	8	9	9	10	28	106	255
8	9	16	21	35	77	255	255
8	9	20	60	87	174	255	255
10	10	23	86	149	255	255	255
15	16	57	118	255	255	255	255
26	22	92	185	255	255	255	255
41	59	166	255	255	255	255	255
__________________
Stuff was here at some point.

Last edited by SoonUDie; 4th May 2004 at 04:12.
SoonUDie is offline   Reply With Quote
Old 27th April 2004, 09:31   #16  |  Link
dimzon
BeHappy/MeGUI developer
 
dimzon's Avatar
 
Join Date: Oct 2003
Location: Moscow, Russia
Posts: 1,727
Quote:
Originally posted by vaxis
I did have to change a couple of lines in the calculate.vbs to get it to work. In the function ReadMatrixFromFile, where the double for loop is, your way of dividing the numbers into the array aLine did not work, some post in it where just blanks and the CLng then returned an error. instead I simply made aLine to a string and implemented the inner for loop a little different and removed the lines with replace calls in the outer for loop.
It's bcz forum engine changed my source string:
Quote:
s = replace( trim(s), "<SPACE><SPACE>", "<SPACE>")
to
Quote:
s = replace( trim(s), "<SPACE>", "<SPACE>")
so i will change this string into
Quote:
s = replace( trim(s), CHR(32) & CHR(32), CHR(32))


anywhere Your solution works fine too (but i'm too lazy to analyze quantizers position into source string)

and some user nickname=Shade PM-ed me such info about DCTune (i have no idea why he PM-ed to me but not to post this to this thread so I repost it):
Quote:
DCTune can only calculate intra matrix. It's designed for still image(JPEG), not for motion pictures.
The second matrix generated by DCTune is for chroma. JPEG can use different matrix for luma and chroma, and both matrix are intra matrix. You must design inter matrix(for inter block in P/B-frame) by youself.
Intra matrix quantize intra-block, the coefficients in intra-block are "texture".
Inter matrix quantize inter-block, the coefficients in inter-block are "residual data after ME".
The properties of intra/inter matrix are different.


Last edited by dimzon; 27th April 2004 at 09:40.
dimzon is offline   Reply With Quote
Old 27th April 2004, 10:42   #17  |  Link
SoonUDie
Registered User
 
Join Date: Dec 2003
Posts: 147
Quote:
"DCTune can only calculate intra matrix. It's designed for still image(JPEG), not for motion pictures."
True. I don't really know as much about quantization as I want to, yet, but I've just assumed that inter-frames in general use a "lower-quality" matrix (which is where my use of 1 lower p value comes from).

Quote:
Guys, can you remember me? I'm the one who started this thread
Yea, sorry about that . I've somewhat hijacked the thread in developing my own method for creating matrices.

Also, you can stop using the Evil Matrix now - it's served its purpose as a test, and I think my current method far outstripes it in creating a quality encode. I would appreciate it if you try my new method

On another point, I went through the forums using search, and DCTune has been discussed in relation to XviD a few times before (though not in this capacity or detail), and some people came to the same conclusion I have - it would be awesome if we could use use DCTune to optimize individual scenes, or even frames. For example: call DCTune on each i-frame, so that each scene gets the most appropriate matrix. I know that XviD supports (or at least, used to support) changing quantizations like this, but I think it's not mpeg4 compliant.
__________________
Stuff was here at some point.
SoonUDie is offline   Reply With Quote
Old 27th April 2004, 10:58   #18  |  Link
dimzon
BeHappy/MeGUI developer
 
dimzon's Avatar
 
Join Date: Oct 2003
Location: Moscow, Russia
Posts: 1,727
Quote:
Originally posted by SoonUDie
Oh, I should mention that DCTune produces matrices with a lowest possible value of 2. You may want to implement a way to multiply all the other values accordingly, so that you always get 8 in the upperp left corner. If not, you can use excel and a little bit of math once you have the final matrix.
How does You thik what is the best:
1) multiply all the other values on (8/top_left_intra_value)
OR
2) increase all the other values on (8-top_left_intra_value)
OR
3) set values=8 where values is < 8
dimzon is offline   Reply With Quote
Old 27th April 2004, 12:34   #19  |  Link
dimzon
BeHappy/MeGUI developer
 
dimzon's Avatar
 
Join Date: Oct 2003
Location: Moscow, Russia
Posts: 1,727
I have just updated calculate.vbs in my post to follow MPEG specs.


That's my quantizer matrix for "KNOCKIN' ON HEAVEN'S DOOR" movie
Quote:
8,8,8,8,11,16,30,114
8,8,8,8,10,14,25,73
8,8,13,14,20,26,45,151
8,8,14,24,43,63,123,255
12,10,19,38,114,200,255,255
16,12,25,56,208,255,255,255
27,23,44,125,255,255,255,255
85,61,140,255,255,255,255,255

16,16,16,17,29,53,152,255
16,16,18,17,29,48,128,255
16,18,32,39,61,108,255,255
17,17,39,80,181,255,255,255
30,28,57,178,255,255,255,255
48,42,100,255,255,255,255,255
143,117,255,255,255,255,255,255
255,255,255,255,255,255,255,255
now I'm encoding...
dimzon is offline   Reply With Quote
Old 27th April 2004, 14:08   #20  |  Link
springl
Registered User
 
Join Date: Nov 2002
Posts: 32
@dimzon
How do I have to modify the bat files to make them work flawlessly under WIN98SE,'cause I get a "not valid option /Q" when I run extract.bat and calculate.bat. Moreover,due to calculate.vbs, I get
a runtime error of M$ Vbscript : "activeX is not able to create the
object 'MSXML2.DOMDocument.3.0'".And of course, result.qmatrix is not created.

Thanks.
springl is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 14:23.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2023, vBulletin Solutions Inc.