R4R_ENC - Batch tool to use XVID,AVS2AVI and OGGMuxer inside Robot4Rip

[bilu]

The Readme file:

R4R_ENC - Batch tool to use XVID,AVS2AVI and OGGMuxer within Robot4Rip
======================================================================
by Bilu. April 16, 2003. My first tool and Readme ;)

Usage: r4r_enc 0.65 video.d2v audio.ogg

where 0.65 is compressibility.

Compressibility is the parameter, it wasn't made with 2-CD fitting in mind. I hope this bring inspiration to someone, it works for me ( now encoding by just clicking Start on R4R !!! ) but may not work for you.

Syntax used on the batch file exists on XP and may exist fully in W2K, I don't know.
It doesn't work with previous versions of Windows.

Uses a nice command line search-n-replace utility called Switch to make some changes on Vobsub's .idx files.
(download from http://ped.deadartists.com/switch.zip)

To use you must first replace

set PLUGINDIR=D:\avs25\plugins
set ENCODES=D:\avs25\tests
set R4R_ENC=D:\avs25\r4r_enc
set AVS2AVI=D:\avs25\progs\avs2avi
set OGGMUXER=D:\avs25\progs\OGMuxer

with your own folders.


Features:

Permanent subtitles
=============
- corrects small bug in Vobsub batch ripping
(http://forum.doom9.org/showthread.php?s=&threadid=51019)

- automatically adjusts vertical offset based on resolution on .idx file

AVS Scripting
=============

- automatically generates AVS script with AutoCrop, AutoResize (512 width respecting AR), AutoDeinterlace (explained later on this file) and permanent subtitles in the language you chose on R4R (assuming you choose only one)

- automatically generates another AVS script that grabs the 1st pass size, makes compressibility calculations and generates the 2 pass size in hexadecimal to be exported to XVID registry settings
(Hex function credits go to WarpEnterprises, http://forum.doom9.org/showthread.php?s=&threadid=50862)

XVID Encoding
=============
- changes encoding settings (1st-pass,2nd-pass,null,2nd pass size) in registry
- AVS2AVI encoding with settings in registry

OGM Muxing
==========
- uses OGGMuxer to create OGM files, and splits (I think OGGMuxer can split on the keyframe, unlike Oggmux)

TO DO
=====
- use chapter info in OGM

AutoDeinterlace
===============

Tested on first chapters (haven't yet made a full rip) of:

- Monty Python's "And Now For Something Completely Different" (NTSC telecined)
- Gladiator (progressive PAL)
- Matrix (interlaced PAL)
- The Abyss (interlace PAL)

and it worked well. I'm no Decomb expert, I mostly use progressive PAL.
But I would like to see feedback on this on the forum :)

It's like this:

function AutoIVTC(clip c){
IVTC= (c.framerate > 25)? "Decimate(Telecide(c),cycle=5)":"Telecide(c)"
return eval(IVTC)
}

AutoIVTC()


Credits and copyrights of the valuable tools used belong to their respective developers/owners.


I hope this batch or at least some code can serve you well :)

Best regards,
Bilu

[bilu]

Hi,

I deleted the first attachment and uploaded a new one on this post.

This version also deletes (after OGM muxing) the Project and Rips folders made by R4R to allow job queueing without filling too much your HD.

(attach waiting for approval)

This post also gives you a small idea of what was done:

http://forum.doom9.org/showthread.php?s=&threadid=50862


OGGMuxer - http://cyrius.bunkus.org/tools.html

AVS2AVI - http://daveel.leffe.dnsalias.com/avs2avi.zip


Download: http://pagpessoais.iol.pt/bruno_henriques/r4r_enc.zip

Hope to get some feedback soon :)


Best regards,

Bilu

[SithUK]

I downloaded all the secondary files and set the proper command paths in the batch file. How exactly do I get R4R to interface with your batch file?

Your installation instructions (or rather complete lack of) are probably why there are no replies :-)

Sith

[bilu]

@SithUK

Quoting myself: "My first tool and Readme" ;)

That's why I need replies to know what is missing.
I'll do a nice installation file on the next days.

About the R4R integration: on R4R, right-click on "Finalize" and fill the "Custom Command" with

D:\avs25\r4r_enc\r4r_enc.bat 0.65 %d2vfile% %audiolist%

Replace D:\avs25\r4r_enc with your path.

@all

My AutoIVTC still needs some work. I'm trying to rip a BBC humor series called "The Fawlty Towers" which is a real pain :confused: . If you search for Fawlty Towers here on the forum you'll see others complaining too. The best method I've used so far was

Bob()
SelectEven()

I'll try to find if this is better than Telecide on other PAL interlaced movies.

I'm starting to learn what means interlaced problems and why it affects anime lovers so much: in anime EVERY edge counts.

Still everybody can just cut in half my batch file and make just AVS automatic generation to tweak out later ;)


C'mon guys! More feedback please! :p


Best regards,
Bilu

[TaZ4hvn]

Well, it does work !!
I may be the only one patient enough to manage it to work thou, mainly because of the lack of docs ...
Well after a couple of hours reading the files, I had all the filters necessary and mofified it to suit my needs.
With r4r it does a complete, stable, and working robot: congrats :)
Could you please modify it to target a specified filesize rather than a compresibility ?
Should be easy enought to get the desired size from command line with a -s option perhaps, like: D:\avs25\r4r_enc\r4r_enc.bat -s 700M %d2vfile% %audiolist%, and then calculate the video file size as beeing "desired minus audio" ?

[bilu]

@all

Two bugfixes and a note for the morning, I'll try to release a new and documented version today:

NOTE: You must disable manually the "Discard 1st pass" option on the codec in order to get the 1st pass size and make calculations.

Bugfix 1
========
Replace "reg import %r4r_enc%\2ndpass.reg" with "reg import %ENCDIR%\2ndpass.reg". It wasn't using the calculated filesize, just setting 2nd pass.

Bugfix 2
========
Replace in avs1_a.txt

function AutoIVTC(clip c){
IVTC= (c.framerate > 25)? "Decimate(Telecide(c),cycle=5)":"Telecide(c)"
return eval(IVTC)
}

with

function AutoIVTC(clip c){
IVTC= (c.framerate > 25)? "Decimate(Telecide(c),cycle=5)":"SelectEven(Bob(c))"
return eval(IVTC)
}

It seems to work better and faster than Telecide on PAL interlaced, doesn't seem to harm any kind of PAL movie with exception of a little shimmering mainly on static details, like the text intro of Gladiator.

But it did a wonderful job cleaning the blended fields on "Fawlty Towers"! :)

Blended fields were killing the codec's motion estimation on panning scenes, and the little shimmering (just on a very few frames) is a secondary effect much easier to handle with, both for encoding and watching the final work.

I'll try it on some more movies
but I think this may be the best generic AutoIVTC function, and feedback is allways welcome. :)


How R4R_ENC works:
==================

1) Parse variables;

2) Parse Vobsub's IDX files and modify vertical offset and language to the batch ripped one (offset 30 on PAL, 0 for NTSC);

3) Generates an AVS script with AutoCrop, permanent subtitles (it's the only automatic way to split and stay in synch that I know of) and AutoIVTC (as described above);

4) Generates a second AVS script that grabs the 1st pass size, makes compressibility calculations and generates the 2 pass size in hexadecimal to be exported to XVID registry settings. It runs the script with a BlankClip encoded with AVS2AVI in XVID null mode, since it exists only for calculations;

5) Sets XVID modes and desired size on 2nd pass in [HKEY_CURRENT_USER\Software\GNU\XviD] like this:

[HKEY_CURRENT_USER\Software\GNU\XviD]
"mode"=dword:00000005 --> 2nd pass Internal
"desired_size"=dword:0000094a --> 2378 KB

6) Encodes using
avs2avi.exe test.avs test.avi -p 0 -c xvid

-c <4cc> : Use codec having <4cc> with default settings
-p 0 is priority idle.


Default scripts - change to your preference
===========================================
Variables were replace with sample folders/values for better reading.

1st script

Loadplugin("d:\avs25\plugins\AutoCrop.dll")
Loadplugin("d:\avs25\plugins\avsmon25a.dll")
Loadplugin("d:\avs25\plugins\Decomb.dll")
Loadplugin("d:\avs25\plugins\MPEG2Dec3.dll")
Loadplugin("d:\avs25\plugins\VSFilter.dll")
Loadplugin("d:\avs25\plugins\Deen.dll")
Loadplugin("d:\avs25\plugins\TemporalCleaner.dll")

function AutoIVTC(clip c){
IVTC= (c.framerate > 25)? "Decimate(Telecide(c),cycle=5)":"SelectEven(Bob(c))"
return eval(IVTC)
}

function ResizeARW(clip c,int nwtemp,string resizefn) {
nhtemp=floor((c.height * nwtemp)/c.width)
nh=nhtemp - nhtemp%16
nw=nwtemp - nwtemp%16
return eval(resizefn+"(c,"+string(nw)+","+string(nh)+")")
}

mpeg2source("c:\movie\movie.d2v",idct=2,cpu=0)

AutoIVTC()

Autocrop(mode=0,wMultOf=16,hMultOf=16,threshold=20)

vobsub("c:\movie\movie")

TemporalCleaner (ythresh=5, cthresh=5)
ResizeARW(512,"LanczosResize")
deen("a3d",1,10,12)
LumaFilter(17,1.1)
Limiter()
MonitorFilter()

2nd script

Loadplugin("d:\avs25\plugins\CALL_25.dll")

function Hex(int n) {
v="0123456789ABCDEF"
tmp=midstr(v, 1+n%16,1)
n=n/16
tmp=midstr(v, 1+n%16,1) + tmp
n=n/16
tmp=midstr(v, 1+n%16,1) + tmp
n=n/16
tmp=midstr(v, 1+n%16,1) + tmp
n=n/16
tmp=midstr(v, 1+n%16,1) + tmp
n=n/16
tmp=midstr(v, 1+n%16,1) + tmp
n=n/16
tmp=midstr(v, 1+n%16,1) + tmp
n=n/16
tmp=midstr(v, 1+n%16,1) + tmp
return tmp
}


Import("d:\avs25\encodes\size.avs")
Newsize=floor((size/1024.)*0.65)
Newhex=Hex(newsize)

Call(BlankClip,"d:\avs25\plugins\Nicexe d:\avs25\encodes\2ndpass.siz "+newhex, "1")

Best regards,
Bilu

[bilu]

Originally posted by TaZ4hvn
Well, it does work !!
I may be the only one patient enough to manage it to work thou, mainly because of the lack of docs ...
Well after a couple of hours reading the files, I had all the filters necessary and mofified it to suit my needs.
With r4r it does a complete, stable, and working robot: congrats :)
Could you please modify it to target a specified filesize rather than a compresibility ?
Should be easy enought to get the desired size from command line with a -s option perhaps, like: D:\avs25\r4r_enc\r4r_enc.bat -s 700M %d2vfile% %audiolist%, and then calculate the video file size as beeing "desired minus audio" ?

About docs, do you think the information in the post above is enough? What do you feel that is missing?

About the filesize issue: I don't think it is possible to change the custom command on R4R for each job, if someone intended to use it that way. But I'm thinking about

- parsing the AVS2AVI info on a 10 frame encode to get the clip's total time ("Fawlty Towers" are approximately 30 minute episodes, I want each on half CD);

- do a compressibility check to get the final size

- lower resolution and extra denoising (deen("a3d",2,10,12)) if needed. I won't replace Lanczos with anything ;)

- do another compressibility check

- if it gets near 50% compressibility at least then do a 1st pass and then second

The best setting wouldn't be a filesize, but an average bitrate.
But I'll try to have the compressibily mode still available for those who want it :)


I also thought about using a command line stats reader if it were available (didn't check) to spare the HD from the 1st pass. But then again the old method may be more reliable :)


Best regards,
Bilu

[TaZ4hvn]

First of all thanks again for your work.
Yes, additional information is welcome, as long as I seem to be the only one testing it and it works for me, docs is well enough :D

I found a couple a bugs, I think:
- "echo Newhex=Hex(newsize) >> %ENCDIR%\compcalc.avs
echo. >> %ENCDIR%\compcalc.avs
echo Call(BlankClip,"%PLUGINDIR%\NicEcho.exe %ENCDIR%\2ndpass.siz "+newhex, "1") >> %ENCDIR%\compcalc.avs"

Should be changed by

"echo Newhex=Hex(Newsize) >> %ENCDIR%\compcalc.avs
echo. >> %ENCDIR%\compcalc.avs
echo Call(BlankClip,"%PLUGINDIR%\NicEcho.exe %ENCDIR%\2ndpass.siz "+Newhex, "1") >> %ENCDIR%\compcalc.avs"

looks like vars are case sensitive...

- In line:
"%OGGMUXER%\OGMuxer.exe -f --split 640M -v -o %ENCDIR%\%OGMFILE% %ENCDIR%\%AVIFILE% %OGGFILE%"
I had to remove the "-v" for it to work, don't even know what it was intended for as I wasn't able to find documentation about it ?

A piece of advice for other users:
If, like me, you don't want subs support just give a wrong path in "set SWITCH=D:\avs25\progs\switch" and everything goes right, else you'll get a verbose output from SWITCH wich interrupt the process and so break the point of the batch !

And now about the wishes:
- you're certainly aware most of people out there doesn't want to encode regarding compressibility but final size, just to get it to fit on a CDR, so you really should implement such a fonction.
It does seem a lot easier to implement than the compressibility check you did, even if you can't get the argument in the commandline we can always change it in the script.
Actually I almost succeed doing it, when you know how I do suck in batch coding is a performance!
I can input size via commandline, I can get the size from the audio file, but I don't know how to do a minus operation !! (you see I suck :D )
If you have a clue for me ...
- cropping is not satisfactory enough to me: I'd like a smart cropping like GKnot does, ie autocrop then crop left/right to respect the AR.
- about compressibility VS file size, I think the best of the two worlds would be, ask for compressibility, ask for final size and then make a resolution decision to fit the desired size with the desired compressibility ... oups, looks difficult enough :)

Regards

[bilu]

@TaZ4hvn

1) Case-sensitive variables? Shouldn't be :confused:
From http://www.avisynth.org/index.php?page=ScriptGrammar

"Avisynth ignores case: aViSouRCe is just as good as AVISource." I'm using the last AVS 2.5.1 from CVS (check Sh0dan's sig) and haven't had any problems before.

2) About OggMuxer's -v switch: I'm using v1.1a5 (09-03-2003)

-v/--video "truncate" other streams when video ends\n

3) To disable subs: replace
echo vobsub("%SUBFILE%") with echo #vobsub("%SUBFILE%"). This way it doesn't give error messages while still adapting subs if you intend to use them with DirectVobSub.

4) About filesize calculations: change or create a new version of the compcalc script. It's the best way for math calculations.

5) Cropping: from the Autocrop Readme, don't know if it is what you want:

aspect - Float - default 0 - Aspect is ignored

-1 - Maintain aspect ratio of the source clip
0 - Aspect is ignored
>0 - Aspect ratio will be set to this. For example for a 4:3 aspect ratio you would pass
4.0/3.0 or 1.33333333333. Note 4/3 won't work the .0 is need to tell AVISynth the value is
a float.

6) Quoting you:" about compressibility VS file size, I think the best of the two worlds would be, ask for compressibility, ask for final size and then make a resolution decision to fit the desired size with the desired compressibility "

If I ask for filesize instead of using parameters it wouldn't be completely automated :)

My idea is something like this: defining a minimum compressibility and an average bitrate. This way I can assure that a 30 minutes episode and a 90 minutes movie get the same treatment without user intervention. What do you think?


Best regards,
Bilu

[TaZ4hvn]

1) That's what I though too :D Perhaps it isn't true for vars, or more probably I changed something else I don't remember :)

2) Didn't see it ! Strange it doens't work for me, but not a big point.

3) That's to disable sub if you ripped them off the DVD, but else SWITCH is claiming for its .idx files, that's why I simply ommited its path.

4) LOL, that's exactly what I was doing when I got your answer :)

5) That's what I read but doesn't interest me that much since -1 would lead to keep 720:576 and >0 neither since I can't predict the AR after cropping.

6) When I say ask, I mean trough the command line !! I'm dumb for sure, but not on this :D

[bilu]

Originally posted by TaZ4hvn
When I say ask, I mean trough the command line !! I'm dumb for sure, but not on this :D

When I meant no user intervention I was thinking about queueing jobs, and not be asked in the middle of the night for a filesize :D

Bilu

[TaZ4hvn]

ok, ok, let's clarify it !
I'm mainly doing 1CD XCD rips, so that the only filesize I want is 800Mo (except for rare 2 Cds of course) so that I won't have to change it ...
So that I would fill r4r with something like:
D:\avs25\r4r_enc\r4r_enc.bat 800M %d2vfile% %audiolist%
One time for all :)

[bilu]

Hmmm... I think I've understood now.

Maybe the best is to specify as parameters:

- 1 CD size
- 2 CD size
- minimal resolution ( resolution value would be width*height to deal with different ARs)
- minimal compressibility

If we couldn't achieve 1 CD size with minimal compressibility and minimal resolution we would aim for 2 CD size. Would this fit your needs?


Bilu

[bilu]

Originally posted by TaZ4hvn

5) That's what I read but doesn't interest me that much since -1 would lead to keep 720:576 and >0 neither since I can't predict the AR after cropping.

How about autocropping twice? After first crop you would get ratio.
It may be possible to compare that ratio with different profiles (1.3333, 1.7777, 2.35),choose the nearest one, and then crop again with the aspect switch. :rolleyes:


Bilu

[TaZ4hvn]

Well, you got it perfectly this time :D
This behavior would perfectly suit my need!!


Originally posted by bilu
How about autocropping twice? After first crop you would get ratio.
It may be possible to compare that ratio with different profiles (1.3333, 1.7777, 2.35),choose the nearest one, and then crop again with the aspect switch. :rolleyes:
Bilu

That's exactly what I'm aiming to.
Actually I used the Aspect=1.611 option of AutoResize as you kindly suggest to suit PAL anamorphic 2.35:1 movies (90% of my encodes)
And by this I get a perfect aspect ratio very close to the one GKnot predict, that's great :)
Now I'm doing the same for 1.33 1.77 1.85 ...

The profile idea is great but, the problem is some formats are very close once cropped: for instance 1.33 non anamorphic and 1.85 anamorphic once cropped both give near 1.33 ...

[bilu]

Originally posted by TaZ4hvn
The profile idea is great but, the problem is some formats are very close once cropped: for instance 1.33 non anamorphic and 1.85 anamorphic once cropped both give near 1.33 ...

I never felt this problem because I choose between Zoom Player's AR profiles. Do you have a reason for not doing the same? :confused:


Bilu

[TaZ4hvn]

Originally posted by bilu
I never felt this problem because I choose between Zoom Player's AR profiles. Do you have a reason for not doing the same? :confused:
Bilu

Well, no big thing, I don't use ZP has I dislike it :)

[bilu]

It seems to me that we cannot do better cropping automatically, is as good as it gets :D Maybe you can find another nice player that supports AR profiles.

About compressibility and size parameters: I was thinking about using different weights on the "number of CDs" decision.

Some people may have less problem going for a 2-CD encode instead of 1-CD than going to a 3-CD encoding instead of 2-CD. What do you think?


Bilu

[TaZ4hvn]

That's a good idea :)
How I see things, you have to make a kind a decision algorithm to set:
- Number of CD
- Resolution
- Resizefilter (if you can stand dropping Lanczos :D)

Parameters should be:
- Targeted pix/frames (better than resolution, as you can't predict AR)
- Compressibility
you can add preference weighs for number of CDs too.

Ok that's the idea but not the algo :) I'm trying to determine how decision should be make ...

[bilu]

I'll try to look at it today (even before doing docs :D )

I already made a pix/frames resize function @home, I'll post it when I get there. Syntax is ResizeARP(150,"LanczosResize") where 150 means resolutions around 150000 pixels.

I made a sample test with "Fawlty Towers" (ratio=1.22222) with this ResizeARP(150,"LanczosResize") and it resizes to 416x336. Tried Bicubic and Bilinear on the same resolution and it looks too soft and washed :rolleyes: Also tried in some dark scenes on the first chapter of Matrix and Lanczos beats the hell out of Bicubic.

I'm encoding with the last uManiac build with these settings:

H.263, Chroma Motion, B-frames 3/150/100, the rest is default.

I love H.263 + Lanczos :D

Bilu

[TaZ4hvn]

Ok, first idea:

To me, the importance of parameter are as follows:
1- number of cd
2- resolution
3- resizefilter

A kind of algo could be:

set cd_number=1

set res=high (like 200k pix/frame)

set resize=lanczos
do comp check
if comp<min_comp set resize=bicubic
do comp check
if comp<min_comp set resize=bilinear
do comp check
if comp<min_comp set res=med

for res=med (like 160k pix/frame)

set resize=lanczos
do comp check
if comp<min_comp set resize=bicubic
do comp check
if comp<min_comp set resize=bilinear
do comp check
if comp<min_comp set res=low

for res=low (like 120k pix/frame)

set resize=lanczos
do comp check
if comp<min_comp set resize=bicubic
do comp check
if comp<min_comp set resize=bilinear
do comp check
if comp<min_comp set cd_number=2

for cd_number=2

idem


and so on ... :)
Of course all bounds are indicative.
You could as well omit the filter routine as this become to make a big amount of comp check !!
The most beautiful is this way the main key remain compressibility :D

[TaZ4hvn]

Originally posted by bilu
I'll try to look at it today (even before doing docs )


Well be sure I'm waiting for this new version :)

Originally posted by bilu
I made a sample test with "Fawlty Towers" (ratio=1.22222) with this ResizeARP(150,"LanczosResize") and it resizes to 416x336. Tried Bicubic and Bilinear on the same resolution and it looks too soft and washed :rolleyes: Also tried in some dark scenes on the first chapter of Matrix and Lanczos beats the hell out of Bicubic.
Bilu

That may be cause you're using heavy pre-processing, I don't like it :)
Usually simpleResize perform rather well and is a good alternative to sharper filter + preprocessing.

my two cents :D

[bilu]

I had no time last night and probably won't on the next two days :o

I'm @work now and all I can do is have ideas and look for stuff here ;)

About resize filters: I´ve searched (http://forum.doom9.org/showthread.php?s=&postid=244626&highlight=simple+AND+bicubic#post244626) and I'm now convinced to integrate SimpleResize :rolleyes:

Soft Bicubic is too soft. Neutral and Sharp require extra parameters :(
but then I could create functions like NeutralBicubic and SharpBicubic and use them, like:

ResizeARP(150,"NeutralBicubic")

function NeutralBicubic(clip c,string width,string height){
return eval("BicubicResize(c,"+width+","+height+",0,0.5)")
}

function SharpBicubic(clip c,string width,string height){
return eval("BicubicResize(c,"+width+","+height+",0,0.75)")
}

I can't check if syntax is correct at the moment.

About the decision algorithm: haven't given enough thought yet.
I think we can have better looking Lanczos low-res than Bilinear hi-res, have no opinion yet. Also it could be that

Bilinear Hi-res worst than Lanczos Med-res
Bilinear Med-res better than Lanczos Low-res

and this logic applied to every other resize filter to be included.
Maybe I'll make weights on this as well and put my defaults, so everyone can change them if they like. And weights should be applied to combinations, like:

MedLanczos=1
LowBilinear=2

or whatever method that I come up with (or someone suggests).
Also tests should be made with compressibility tests at 1%. As you can see here (http://jonny.leffe.dnsalias.com/whywork.htm), you have errors at any size, and we will compare every test at 1%.

(to be continued after lunch :D )

Bilu

[jonny]

Also tests should be made with compressibility tests at 1%. As you can see here, you have errors at any size, and we will compare every test at 1%.

The page is quite old (and the tests are made only with 1 movie)

It's better to refer to this for DivX:

http://jonny.leffe.dnsalias.com/sptests.htm

and this for XviD:

http://jonny.leffe.dnsalias.com/enc/sp/xvid.htm

(Some suggestions:
1% is not safe. sometime 2% is not safe too
4% can be a good compromise between speed and accurancy)

[bilu]

@jonny

My idea is to make 1% compressibility tests over different resize/denoise combinations (maybe I don't need to do every combination) and check which combinations come within a 10-15% of at least 50% compressibility. I think 1% is safe for comparing because we'll do every test over the same frames.

And five 1% tests take the same time as one 5% test :D

Then we would do a 5% test on the nearest combinations and choose.

Do you think comparing several 1% comp. tests can give accurate results in terms of difference in compressibility?

This would mean checking every result within a 50-65% compressibility range. I'll create profiles with a reasonable difference in terms of compressibility in order to avoid making too much comp.tests

Bilu

[jonny]

Do you think comparing several 1% comp. tests can give accurate results in terms of difference in compressibility?

Yes, if your target is the difference, the results will be good.
The only thing you should take care is that the absolute error can be high (like 10%).
(probably giving an option to the user for the % of the movie to analyse will be the best solution imho)

[bilu]

My concept for CD fitting:

(CD size will be a parameter)
1% > 0.00 CD => 50% > 0.00 CD => 1 CD
1% > 0.02 CD => 50% > 0.01 CD => 2 CD
1% > 0.04 CD => 50% > 0.02 CD => 3 CD
1% > 0.06 CD => 50% > 0.03 CD => 4 CD

Can be translated into this formula:
floor((comp.test size / CD size) * 100 * 0.5) +1

I think it is safe to make just Lanczos 1% tests to find this, since they're more demanding and none of the other resizers seems to double the compressibility (the minimum required to change decision if less than 4 CDs).
And we will probably want more than just 50% compressibility after all.

Then we would:

- do several 1% comp.tests over resize/denoise combinations;

- select those between 50%-65% compressibility, then 5% test over these to select one. This will need to deal with weights in order to define if you prefer to have better resolution or compressibility. I'll use 65% as maximum.


@jonny
By choosing between 50%-65% compressibility results to make a 5% test later, I think it can be safe ;)


Bilu

[TaZ4hvn]

Originally posted by bilu
I'm now convinced to integrate SimpleResize :rolleyes:


Well, finally I have been usefull for something :D

Originally posted by bilu
Neutral and Sharp require extra parameters :(
[/B]

Not a big point as your fonctions should work,
otherwise you could stick with Lanczos/Bicubic/Simple were Bicubic is default, i don't remenber if it's (0,0.5) or the worse (0.333,0.333) ?

Originally posted by bilu
Maybe I'll make weights on this as well and put my defaults, so everyone can change them if they like.
[/B]

Of course that's what has to be achieved, but it seems we do not have the sames tastes:
I'm usually aiming at a bigger resolution (~180k) and higher compressibility (~70%).
You're aiming to sharper picture and, I guess, lower filesizes.
I won't debate these anymore: this is really a matter of choice, and once the algo is working it will be piece of cake to change settings.
I think you should first concentrate on doing a simple working algorithm which fits your needs: anyone will be able to tweak it to his likes afterward :)

Originally posted by bilu
Do you think comparing several 1% comp. tests can give accurate results in terms of difference in compressibility?
[/B]

I'm afraid only intensive testing will show right value here, but to me 1% sounds too low :(
*** oups sorry, you posted while I was typing, ok sounds good now ***

[bilu]

Originally posted by bilu
About the decision algorithm: haven't given enough thought yet.
I think we can have better looking Lanczos low-res than Bilinear hi-res, have no opinion yet. Also it could be that

Bilinear Hi-res worst than Lanczos Med-res
Bilinear Med-res better than Lanczos Low-res

and this logic applied to every other resize filter to be included.
Maybe I'll make weights on this as well and put my defaults, so everyone can change them if they like. And weights should be applied to combinations, like:

MedLanczos=1
LowBilinear=2

or whatever method that I come up with (or someone suggests).


My point is that a sharper resizer at lower resolution could look better than a softer resizer at higher resolution and vice-versa.

Maybe it's pointless to soften every resizing before lowering resolution if sharper resizer can retain more detail.

I think I'll create all combinations and a flag to exclude the ones I like less and also the ones with look equally good but have very little compressibility difference.

Others can change the flags later, and the idea is also to avoid making a lot of 1% and 5% compressibility tests on highly compressible movies, where a lot of combinations could fit the 50%-65% range.


Bilu

[bilu]

Originally posted by bilu
About compressibility and size parameters: I was thinking about using different weights on the "number of CDs" decision.

Some people may have less problem going for a 2-CD encode instead of 1-CD than going to a 3-CD encoding instead of 2-CD. What do you think?


Regarding the CD fitting strategy and the resize/denoise combination weight, does anyone still think we should use different weights per CD?

I won't have time to code until Thursday at least, but I'm trying to get clear ideas.


Bilu

[TaZ4hvn]

Sadly at the moment I'm out of the game:
I got the same error as Beave in this thread http://forum.doom9.org/showthread.php?s=&threadid=36768&perpage=20&pagenumber=7
I tried everything advised there and a couple of others and it's still broken :(
As I'm not the only one I guess avs2avi may not be that stable :D
Did it happen to you ? Have a clue ?

[bilu]

Sometimes I had a problem if I aborted AVS2AVI because it created resume.codec files. My workaround was

del /q d:\avs25\avs2avi\resume.codec
del /q resume.codec

the last one because file is created on current directory.

I hadn't posted this before because I was using version 1.26 (avs2avi-test) and not 1.25, which I haven't tested with R4R_ENC yet.

Besides that I only noticed cosmetic issues (not displaying the correct FPS at the end of an encode). But it can't be noticed in the middle of a batch file ;)

Hope it solves it for you.

Bilu

[TaZ4hvn]

Well I shearched for a resume.codec in all my drives and wasn't able to find one: that must be the reason why avs2avi failed to resume !
But, anyhow, I do think something must be wrong with resume support as i din't launched a resumable encoding neither I paused it: I just tried to redo an already complete task, for testing purposes, with the same name.
I indeed 'fixed' the problem by using a new name for each task I launch ... well, not a fix, but I least I can test more :D
I will search a way to totally disable resume support, as we don't need it and it can only bring trouble to us.
I will ask for it in the developpment forum.

[TaZ4hvn]

Well I tested more with the 1.26 version of avs2avi (according to author resume is badly broken, I don't care as I try to avoid it).
This time I do get resume.codec and resume.stats during the first pass, they are located in the current directory.
If I delete them both everything is back to normal, so I included a del /q resume.codec and del /q resume.stats before the second pass is launched.
I think you should at it to your batch: if you don't want resume support it's safer, if you want it, do it anyways until it works correctly ...

[TaZ4hvn]

Well I just tested a bit more with the 1.26 version of avs2avi (support of resume is badly broken in this version, according to the author ... I don't care as I actually try to avoid it to kick in)

I now get two files in the working directory: resume.codec and resume.stats (it was a first pass).
I just deleted them, and things are back to normal.
I also added del /q resume.codec and del /q resume.codec in the correct location to your batch to prevent further hassle.
I think you should do the same for the next version, it's safer until the resume support is finalized in avs2avi (or until I understand how it works :D)

**oups, sorry for the double post, the server had some heavy loads problem**

[DaveEL]

try adding -r to the command line then it will allways try to start a new resumable session which stops it trying to resume an old one.

DaveEL

[TaZ4hvn]

I had an idea for your batch about adding chapters info to ogm file.
You must already know, but, in case ...
Zulu has just added a CLI version of his keyframe enforcer:
http://zuludev.de.vu/ and thread
http://forum.doom9.org/showthread.php?s=&threadid=39687&perpage=20&pagenumber=2
by adding a command like "kfec -i input.stats -o output.stats -cl list.txt -fps 25" before the second pass, you can force keyframes at start of each chapters during the second pass.
I already used it numerous times (not the CLI version though) and it works as expected.

Regards

[bilu]

Well, it says

"There is also a little drawback:
This won't work with XviD 0.9 Stable, this build simply ignores the new keyframes of the 1st-pass statsfile.
Currently this is only working with the latest builds with bframes _disabled_ (-1), because there is a bug/feature in XviDs 2nd-pass IPB-Decision code.
Probably this will be fixed in the future. "

But then again (http://xvid.hopto.org/alpha)

XviD.Alpha.20.03.2003.0940.exe(~352kb)
U xvidcore/src/encoder.c (rev.1.96) syskin:

- some cleanups ; revised p/b decision with sensitivity control ; max iframe interval works again

so I guess this is a good idea :)

About PAL deinterlacing on AutoIVTC(), I think this is the feedback I wanted :)

"If you deinterlace by discarding a field, this precaution is not necessary because there is no residual combing"

http://forum.doom9.org/showthread.php?s=&postid=300317#post300317


Best regards,
Bilu

[bilu]

@TaZ4hvn

And it's a pity we can't use this too ;)

http://forum.doom9.org/showthread.php?s=&threadid=40343

Chapter Grabber (ogm support tool)
==================================
This program will take an IFO file and a web address to the bn.com scene index of your favorite DVD to create a chapters text file with useful chapter names for use with an ogm muxer.


Bilu

[DaveEL]

ok i finally put together a new avs2avi version so if you want to try it its at http://daveel.leffe.dnsalias.com/avs2avi-126.zip

DaveEL

[TaZ4hvn]

@bilu

Well I didn't read far enough :p
I think you should include it anyways, disabled by a rem by default and let people decide to use it or not.
As I don't use B-frames (still too much compatibility breaches) and stick with dev builds, I'll include it to my own copy.
I think most people may think as I do.

I'm also wondering why you did it as a r4r "finalizing" tool and not doing things CLI from the beginning to the end ?
Every tool r4r uses has CLI support so that they could be chained.
You can even finalize with mode2cdmaker for those doing XCD rips, and why not burn with cdrdao.

@DaveEL

After your advice, I did testing with the "old" 1.26 (avs2avi-test.zip) and found that it systematicaly try to resume manually stopped encodes EVEN without the -r switch.
So did I misunderstood, is it intended to work like this ?

[bilu]

Originally posted by TaZ4hvn
@bilu

I'm also wondering why you did it as a r4r "finalizing" tool and not doing things CLI from the beginning to the end ?
Every tool r4r uses has CLI support so that they could be chained.
You can even finalize with mode2cdmaker for those doing XCD rips, and why not burn with cdrdao.


Huh? :confused: Why did I what?

I haven't changed my tool since I posted it, even the bugfixes haven't been added to the ZIP ( lazy me :) )...

My purpose with this tool is to apply your own tastes on a automatic way.

Humm... I understood you now :)

There are this in r4r that I don't know if I can do - haven't checked yet. Those include choosing the language on ripping subs, the audio track...

Some say open source is made to scratch one's itch. Well, I haven't felt THAT itch yet ;)



Bilu

[TaZ4hvn]

Ok I rephrase it:

Your batch is intended to come after r4r prepared the work.
I just ask why you don't get rid of r4r and just add DVDdecrypter, DVD2avi, Besweet ... command line at the beginning of the batch so that r4r is not needed.

Was it clear enough this time ?

[bilu]

Well, when (and if we can) do everything we're proposing here, then we can go for it. I think we should focus on compressibility / resizing / denoising decisions first and the weights concept.

Then we can try to replace R4R functionality and pack everything needed, so users don't have to download GKnot.

Mode2CDMaker and CDRDAO should be the last things to work on.

I can tell you one thing: I won't do anything for the rest of the week, because my wife is in a very "demanding" phase... ehehe :D and I'll got out for the weekend.

Meanwhile why not volunteering to make some experiments on the OGM chapters and the KF Enforcer yourself? ;)

This is an open project and you have knowledge to share :)

Best regards,
Bilu

[DaveEL]

Originally posted by TaZ4hvn

@DaveEL

After your advice, I did testing with the "old" 1.26 (avs2avi-test.zip) and found that it systematicaly try to resume manually stopped encodes EVEN without the -r switch.
So did I misunderstood, is it intended to work like this ?

the command line for resume is a bit funky (i didnt design it basically got done this way cos its easiest).

If you want to start a resumable session you use -r if you don't use -r and a resume.codec is present it will try to resume the session.

Anyway the new version has no resuming code at all so it should work fine for you.

DaveEL

[TaZ4hvn]

@DaveEl: thanx a lot, I tested it and it now perfectly suits my needs (until a find another problem :p )

@bilu: Well, by reading your post I think you had the feeling I urged you, this is absolutly not the case !
I just give you anything crossing my minds that can be useful to you, you are of course completely free to use it or not !

"you have knowledge to share" ... *sigh* if only ...
Well as I'm willing to help, I'll try to implement by myself the others tools support I proposed, and let you focuse on the main thing.
If I succeed we can merge later.

"my wife is in a very "demanding" phase..." well, this things happens and it's your duty to calm her down :D

regards.

[youngster]

You all have been awfully quiet lately. I am hoping you haven't lost interest in further development of r4r_enc.

I am most interested in more flexibility for resolution choices. I want to be able to encode with bigger width than 512.

@Bilu

I hope you are out of your wife's doghouse :D

[TaZ4hvn]

@youngster:

I'm working on it by now. Bilu's work gave me ideas and the taste to code something different from this.
Purpose is to do a fully automated rip from DVD to Xvid/Vorbis in Ogg container and then burn to XCD and mount on Daemon.
Actually, I DOES WORK, but partially, I have finalized some parts:

- multi languages support: auto find chosen languages -Works
- autocropping, and autoresizing -Works(for 2.35 only at the moment).
- autoset disk number based on a given min compressibility -Works
- mux to ogm, splits to parts. -Works
- create .bin .cue .toc -Works
- automount on Daemon -Works

I still have to implement:

- correct sizing for 1.85 and 1.33 movies
- ivtc/deinterlacing stuff (bilu's function is good, but does tend to blur)
- auto find credits to reduce compression, don't know if I could be done ... perhaps based on luma check

Well I'm far from done yet, but i'm confindent I will achieve t write something usefull.
I hope I'll can post a version soon :)

[bilu]

Originally posted by TaZ4hvn
"you have knowledge to share" ... *sigh* if only ...


Seems to me you're getting the job done, my friend :p

About my AutoIVTC() function: it does not tend to blur, it was really killing half the vertical resolution :o

I've been reading a bit more about IVTC and interlaced here:

Decomb Guide
http://www.doom9.org/decomb.htm

Decomb parameter overview
http://www.doom9.org/decomb-params.htm

IVTC & Deinterlacing tutorial (read THE DREADED HYBRIDS )
http://www.doom9.org/ivtc-tut.htm


I'm still making tests, but it seems I'll replace AutoIVTC() with:

function AutoIVTC(clip c){
IVTC= (c.framerate > 25)? "Decimate(Telecide(c),mode=1,threshold=50)":"Telecide(c)"
return eval(IVTC)
}

It can deal with NTSC hybrids and PAL interlaced without harming resolution, and doesn't harm progressive streams.

I'm also replacing LumaFilter and Limiter with

ColorYUV(off_y=17,gain_y=32,opt="coring")

But that's a matter of personal taste :)


Working resizer functions (I sure do need to reorganize my files :) ):

function NeutralBicubic(clip c,int width,int height)
{return BicubicResize(c,width,height,0,0.5)}

function ResizeARP(clip c, int resolution, string resizefn){
ratio = float(c.width)/float(c.height)
res = float(resolution*1000)
nhtemp = sqrt(res/ratio)
nwtemp = nhtemp*ratio
nh = round(nhtemp) - round(nhtemp)%16
nw = round(nwtemp) - round(nwtemp)%16
return eval(resizefn+"(c,"+string(nw)+","+string(nh)+")")
}

Usage: ResizeARP(150,"NeutralBicubic")

Resizes regarding AR to resolutions near 150000 pixels (width*height)


As for weights and resolutions: I'm thinking about using just Lanczos and a preferred resizer for encoding, Lanczos for CD fitting decision and when a movie barely couldn't fit in one CD and has lots of space to grow on the 2nd CD. In this case it would be used after surpassing a certain resolution in the decision process.

I have to agree with Karl from RV9 that NeutralBicubic looks almost as good as Lanczos as it's hard to tell the difference. Nevertheless it may get noticeable on higher resolutions, that's why I think parameters should be:

-Minimal resolution (insert your taste limits here :) )
-Minimum compressibility (same as above)

-Good resolution (applying Lanczos after this one)
-Maximum compressibility (just for resizing decisions)
-Maximum resolution (we have to think about the players too :) )
-Preferred resizer (will be NeutralBicubic by default,you can change)

After choosing Lanczos it would be a permanent choice for further upsizing if needed. Now I really need to make tests on this to see what sort of stepping in the resize process should be made, we wouldn't want compressibility tests too close on results because it would be a waste of time :rolleyes:

[TaZ4hvn]

That's excelent you worked exactly on parts I didn't look at yet :)
Actually my version has some serious limitations:

- no ivtc, no interlaced material treatment (waiting for your function :D )
- no compcheck is made at all: I just keep first pass, compute real compressibility for 1,2,3 ... cds and choose first one exceeding a given min_comp. That's effective but perhaps not enought: I dunno still have to test it.
A good thing i did is to set a min comp for each cd, for instance:
mincomp_1cd=0.65
mincomp_2cd=0.8
mincomp_3cd=0.6
means:
switch to two cds if comp for one is under 0.65 (resonable quality)
switch to three if comp for two is under 0.5 (be very reluctant to go over twp cd)
switch to four if comp for three is under 0.8 (don't matter if three or four, given i'm over two)
This is my vision of the weighted decision we wanted.
- no resize method choice: set to simple for instance.
- resolutions are set one time for all, near 180k pix/frames:
640*272 for 2.35, 576*320 for 1.85, 512*384 for 1.33

So, as you see, the only parameter I'm playing with is cd number, no resolution decision, no resizing decision, so I guess your code is pretty welcome !
I have plenty of time tonight and tomorrow to work on it so I will clean up code and fix some bugs and send you my code.
When we are both finished, we can finally merge :)



function ResizeARP(clip c, int resolution, string resizefn){
ratio = float(c.width)/float(c.height)
res = float(resolution*1000)
nhtemp = sqrt(res/ratio)
nwtemp = nhtemp*ratio
nh = round(nhtemp) - round(nhtemp)%16
nw = round(nwtemp) - round(nwtemp)%16
return eval(resizefn+"(c,"+string(nw)+","+string(nh)+")")
}

it look pretty clever, I will test it, but we have yet to find proper display aspect ratio: with resizeARP only evry character will be a "coneheads" for sure :D


-Minimal resolution (insert your taste limits here )
-Minimum compressibility (same as above)

-Good resolution (applying Lanczos after this one)
-Maximum compressibility (just for resizing decisions)
-Maximum resolution (we have to think about the players too )
-Preferred resizer (will be NeutralBicubic by default,you can change)


Everything can be usefull or not, it does depend on how you use them: i will wait to see what you will do, as i'm not sure how you will use them.
We also have to keep user configurable variables as small as possible for the final user, or it won't be understandable !
I already use lots of them, I'll try to be less consuming.


About my AutoIVTC() function: it does not tend to blur, it was really killing half the vertical resolution


I didn't say it wasn't good for sure ! I didn't try for myself but I trully trust you on this.
But you must admit that killing half the resolution does finally look blurry :p

Finally I hope we are not both wasting our time, and it will end up with a usefull proggy ... wait n see :)

[bilu]

Originally posted by TaZ4hvn
it look pretty clever, I will test it, but we have yet to find proper display aspect ratio: with resizeARP only evry character will be a "coneheads" for sure :D


Links of interest (about the DAR flag)

http://forum.doom9.org/showthread.php?s=&threadid=38081

http://forum.doom9.org/showthread.php?s=&threadid=45315

http://forum.doom9.org/showthread.php?s=&postid=219002&highlight=flag+AND+DAR#post219002

http://www.doom9.org/index.html?/aspectratios.htm

http://forum.doom9.org/showthread.php?s=&postid=147559&highlight=flag+AND+DAR#post147559

http://mpucoder.kewlhair.com/DVD/mpeghdrs.html#seq

http://mpucoder.kewlhair.com/DVD/ifo.html#vidatt

If you look at a DVD Decrypter rip, specifficaly into VTS_01 - Stream Information.txt, you'll see something like:

0xE0 - Video - MPEG-2 / 720x576 (PAL) / 4:3 / LBA: 1 / PTS: 00:00:00.159 / Delay: 0ms

or

0xE0 - Video - MPEG-2 / 720x576 (PAL) / 16:9 / Letterboxed / LBA: 3649 / PTS: 00:00:00.132 / Delay: 0ms

In Tools -> Settings -> IFO Mode you would have to select Stream Information. But that's on a manual rip, R4R with the DVD2SVCD internal routines doesn't do it. Maybe we should try to check if DVD2SVCD uses this AR flag info :confused:

http://forum.doom9.org/showthread.php?s=&threadid=21859

DVD2SVCD Official FAQ , read Q11.

[TaZ4hvn]

I'm laughting out loud !! Really.
This is exactly what I implemented to my batch: Remember I now have a version fully automated from DVD rip to XviD/Vorbis with XCD burning that implies finding the right audio track(s).
I do decrypt thru DVDdecrypter, so this is exactly the way I find language(s): by parsing the "Stream Information.txt" done.
Believe me or not it does rock this way !

Actually for languages treatment I coded it this way: you have two variables Lang1 and Lang2.
Lang2=none by default but can be changed to target bilanguaged rips.
Lang1=Français, for instance, since I'm french.
The algo works that way: Parse the file to find %Lang1%,
- If one or several tracks are found, I keep the one with the maximun channels amount (to target DD AC3 rather than stereo or mp2... dunno if it's better with the downsampling, but anyway this is necessary to avoid "Comments" tracks).
- If %Lang1% not found, I seek for "English" with the same goal: maximun channels possible. In this case, if %Lang2% was English set it to "none".
- If neither %Lang1% nor "English" are found I use first stream 0x80 and display Language and channels of it.
I tested this on the *only* 22 DVDs I have and it never failed :D

Tonight I added better algo for cd decision and also clean up my code a bit.
What's funny is I planed for tomorrow to look at this sizing problem and find if using info contained in "Stream Information.txt" are enought to make a resize decision each time :D

I'll give you the code when I'm finished with this. I'm already rather satisfied with it, the only thinks I left are the ones you're working on:
- subs (didn't test your first version for this, does it work well ?)
- ivtc/deinterlace ... well it seems it's your task :p
- better algo than mine to include sizing/filter to the number of cd decision: I won't code it, at least now, since it was really painfull to just consider cds ... and I'm satisfied with mine, at the moment.

So, if you can finalize subs+ivtc+deinterlace then ... we have a first fully working tool :D

[TaZ4hvn]

Well ! I wasted my afternoon to that but I finally succeded at writing what I wanted.
I now autocrop and autoresize correctly the source from my batch.

Here is how it work:
First set the kilopix/frame as in your resizeARP function.
As expected, parsing the Stream Information.txt does give everything I do need. First I retrieve:
- DVD type, PAL or NTSC
- Resolution
- 16:9 or 4:3 ie anamorphic or not.
Then I use type and resolution to determine the correct pixel aspect ratio (PAR) against an ITU-R BT.601 compliant table.
Then I set ANA=1 for 4:3 and ANA=1.333 for 16:9.
Finally I pass the Distortion=%PAR%*%ANA% value to an avisynth script I named PerfectCropResize (I'm rather proud of it :) ) whose goal is not only to resize with correct aspect to a resolution close to kpix/frame but also to do a 'smart' crop according to the decided resolution:

function PerfectCropResize(clip c, int resolution, float distortion, string resizefn){
cropclip = eval("AutoCrop"+"(c,mode=0,wMultOf=4,hMultOf=2,threshold=20,samples=10)")
targetratio = (float(width(cropclip)) / float(height(cropclip))) * distortion
res = float(resolution*1000)
nhtemp = sqrt(res/targetratio)
nwtemp = nhtemp*targetratio
nh = round(nhtemp)-round(nhtemp)%16+round(round(nhtemp)%16/16)*16
nw = round(nwtemp)-round(nwtemp)%32+round(round(nhtemp)%16/16)*32
recropratio = (float(nw)/float(nh)) / distortion
recropclip = eval("AutoCrop"+"(c,mode=0,TopAdd=1,BottomAdd=1,LeftAdd=1,RightAdd=1,wMultOf=4,hMultOf=2,threshold=20,samples=10,aspect = "+string(recropratio)+")")
return eval(resizefn+"(recropclip,"+string(nw)+","+string(nh)+")")
}

What it does:
- First Autocrop, calculate aspect of it.
- multiply target by 'distortion': this is the correct AR name 'targetratio'
- Then determine best resizing resolution to both fit 'targetratio' and 'resolution'=kpix/frame. I change your maths here, because this way I first find closest multiple of 16 to nhtemp and not the one just below. Note I keep the 'round(round(nhtemp)%16/16)' in the nw calc to ensure both width and height are rounded the same way.
- Calculate the new crop dimension, 'recropratio' according to choosen resolution then crop again (from original) with aspect=recropratio. The Add=1 are just here because I hate when it remains a black single line as a border.
- Finally resize to given AR.

Only testing can tell if it is good enought. I did some, but not so much, and outputed resolution are often close or equal to the one given by GKnot. I will tweak to close to it even better.

That's it. Finally my tool is perfectly working, perhaps not for the IVTC/Deinterlace stuff (I copied your AutoIVTC but didn't check as I don't have NTSC DVD). Anyway I only use PAL DVDs :)
Well, I will finish some cleaning stuff and perhaps add some features (bilanguage is not working that well) and post a first version tomorrow :D

[bilu]

@TaZ4hvn

I haven't had the time yet to look at your work, but can you tell me if you had any success creating Stream Information.txt by using DVD Decrypter with command line switches? I couldn't, and reported it here:

http://forum.doom9.org/showthread.php?s=&postid=303951#post303951

and here:

http://forum.doom9.org/showthread.php?s=&postid=305088#post305088

It does create the file if selected in IFO Mode, I just didn't had used DVD Decrypter on the GKnot directory but a newer version on the default directory :rolleyes:

[bilu]

About IVTC:

When doing NTSC encodings you could get an hybrid clip. For those who don't know what it means, hybrid clips are a mix of progressive (video) and telecined (film) material, read "The Dreaded Hybrids" in

http://www.doom9.org/ivtc-tut.htm


From Decomb's 4.1b2 help file:

Mostly Video Clips (mode=1)

Now let's consider the case where the clip is mostly video. In this case, we want avoid decimating the video portions be smooth. For the film portions, we want to leave them at the video rate but change the duplicated frame into a frame blend so it is not so obvious.

Decimate(mode=1) is useful for hybrid clips having mostly video. It allows you to leave the frame rate at video rates (to get smooth video sequences) but also to ameliorate the effect of duplicate frames that are emitted by Telecide() for film sequences (frames that are normally removed with mode=0)

Mostly Film Clips (mode=3)

Let's first consider the case where the clip is mostly film. In this case, we want to decimate the film portions normally so they will be smooth. For the nonfilm portions, we want to reduce their frame rate by blend decimating each cycle of frames from 5 frames to 4 frames. Video sequences so rendered appear smoother than when they are decimated as film.

Decimate(mode=3) is useful for hybrid clips having mostly film. It allows you to decimate the clip to film rates and treat the film normally while doing a blend decimation of video sequences to retain their smoothness.

For a generic IVTC function to be prepared for hybrids, we need to know if it's mostly film or video to get the smallest side effect (blended frames).

I think that hybrids having mostly video (by other words, mostly NTSC content) published on DVDs must be TV series, and hybrids having mostly film (mostly telecined from 24 fps to 30 fps) published on DVDs must be real movies.

I also think that most TV series have 4/3 ratio and most films have widescreen ratios. If so, it is safe to use Decimate(mode=1) or Decimate(mode=3) based on Aspect Ratio! ;)


I've posted it on the General Discussion thread, waiting for feedback.

http://forum.doom9.org/showthread.php?s=&postid=305350#post305350


Best regards,
Bilu

[TaZ4hvn]

Ok first about my PerfectCropResize function, I paste here the extra explanation I sent to bilu via private as I think it will help everybody understand it:

Ok let's have some examples:

say round(nhtemp)=278 then round(nhtemp)-round(nhtemp)%16=272
that's a good decision since 278/16=17.375 so the closest 16 multiple is 272. So OK for this. Then my additionnal part will be:
round(round(nhtemp)%16/16)*16
= round(278%16/16)*16
= round(6/16)*16
= round(0.375)*16
= 0*16
= 0
So that the good 272 is kept :) (well that's a required condition)

Now let's say round(nhtemp)=284 then round(nhtemp)-round(nhtemp)%16=272
that's a BAD decision since 284/16=17.75 so the closest 16 multiple is 288. So this not good in this case. Then my additionnal part will be:
round(round(nhtemp)%16/16)*16
= round(284%16/16)*16
= round(12/16)*16
= round(0.75)*16
= 1*16
= 16
So that the good 288=272+16 is set !


Actually I think I discovered why I get different resizing than in GKnot:

Note I keep the 'round(round(nhtemp)%16/16)' in the nw calc to ensure both width and height are rounded the same way.

It seems GKnot doesn't crop this way but to closest (MOD32 or MOD16) for both width and height, so I changed my function according to this to stick with GKnot choice as this is a reference for everyone.
So rather use:

function PerfectCropResize(clip c, int resolution, float distortion, string resizefn){
cropclip = eval("AutoCrop"+"(c,mode=0,wMultOf=4,hMultOf=2,threshold=20,samples=10)")
targetratio = (float(width(cropclip)) / float(height(cropclip))) * distortion
res = float(resolution*1000)
nhtemp = sqrt(res/targetratio)
nwtemp = nhtemp*targetratio
nh = round(nhtemp)-round(nhtemp)%16+round(round(nhtemp)%16/16)*16
nw = round(nwtemp)-round(nwtemp)%32+round(round(nwtemp)%32/32)*32
recropratio = (float(nw)/float(nh)) / distortion
recropclip = eval("AutoCrop"+"(c,mode=0,TopAdd=1,BottomAdd=1,LeftAdd=1,RightAdd=1,wMultOf=4,hMultOf=2,threshold=20,samples=10,aspect = "+string(recropratio)+")")
return eval(resizefn+"(recropclip,"+string(nw)+","+string(nh)+")")
}

Actually wich method is best is discutable since it will depend on the source and rounding decision: that's pretty tricky to code but I will try to improve my function (will certainly grow in complexity:D )
Anyway the function does really work well :)


I think that hybrids having mostly video (by other words, mostly NTSC content) published on DVDs must be TV series, and hybrids having mostly film (mostly telecined from 24 fps to 30 fps) published on DVDs must be real movies.

I also think that most TV series have 4/3 ratio and most films have widescreen ratios. If so, it is safe to use Decimate(mode=1) or Decimate(mode=3) based on Aspect Ratio!

I agree with this but i'm far from beeing a decomb expert :p
Also there is certainly some real film material in NTSC with 4/3 aspect ratio ...

Well, I will finish some cleaning stuff and perhaps add some features (bilanguage is not working that well) and post a first version tomorrow

Actually I won't at the very moment, since even if working my decision algorythm based on compressibility and cd numbers oonly is not accurate enought:
I did an encode wich ended 2cd with only 500MB on the second (so quant2 only is used) because the 1cd compressibility was 0.57 versus bound set to 0.60 !!
Nobody wants this, it's clear that sizing a bit less and using a less sharpening filter would easily end with a 0.65 compressibility on 1cd.
So I'm working on it by now.


Finally does anybody knows of a way the emulate variables delayed expansion introduced in XP.
I have to use them but do want the prog beeing usable on others OS so I have to avoid them ...

[TaZ4hvn]

One more thing:

I haven't had the time yet to look at your work, but can you tell me if you had any success creating Stream Information.txt by using DVD Decrypter with command line switches? I couldn't, and reported it here:

Funny that's a question I was about to post this evening :p
I followed the links but I didn't get something:
Say the user disabled creation of Stream Information.txt via the GUI, will call DVDDecrypter in IFO mode via CLI enable it again ?

EDITED:
Silly me, I just have to test !
I did and it this not :( that's really bad for my tool: if I can't find Stream Information.txt every rechearch will fail !

[bilu]

http://forum.doom9.org/showthread.php?s=&threadid=52321

No need to worry :)

[TaZ4hvn]

and whatever is selected in the gui applies to the cli also.

No :( ... test it and see.

[bilu]

I already did it today :D using R4R switches on the GKnot version which it uses.

It would be scary if all that work and 22 DVD testing would get useless :scared:

[bilu]

Originally posted by TaZ4hvn
I did an encode wich ended 2cd with only 500MB on the second (so quant2
only is used) because the 1cd compressibility was 0.57 versus bound set to
0.60 !!
Nobody wants this, it's clear that sizing a bit less and using a less
sharpening filter would easily end with a 0.65 compressibility on 1cd.
So I'm working on it by now.


My concept (will try to code it later)



Parameters:
-MinRes = Minimal resolution
-MinComp = Minimum compressibility
-GoodRes = Good resolution (applying Lanczos after this one)
-MaxComp = Maximum compressibility (just for resizing decisions)
-MaxRes = Maximum resolution (we have to think about the players too )
-PrefRsz = Preferred resizer (NeutralBicubic by default,you can change)

-AudioSize = Clip duration in seconds * (AudioBitrate/8)
-Margin = defined by the user in KBytes

(assuming 20 > MaxComp - MinComp > 10)


NewRes=MinRes;
PrefRsz=NeutralBicubic;
Comp1= (1% Comp.test with NewRes + PrefRsz);
Comp5= (5% Comp.test with NewRes + PrefRsz);
step=2 (means 2000 pixels in ResizeARP, about 44 width per 44 height)

1) Comp1;

NrCD_1p= (100*Comp1)/(CDsize - Margin - Audiosize)
NRCDmax_diff=abs(NrCD_1p*MaxComp - round(NrCD_1p*MaxComp))
NRCDmin_diff=abs(NrCD_1p*MinComp - round(NrCD_1p*MinComp))

if NRCDmax_diff < NRCDmin_diff then NRCDs= round(NrCD_1p*MaxComp)
else NRCDs= round(NrCD_1p*MinComp)

2) DesiredSize = NrCDs * (CDsize - Margin) - AudioSize
Comptest1 = (100 * Comp1)/DesiredSize
Comptest5 = (100 * Comp5)/DesiredSize

3) if Comptest1 > MaxComp then NewRes = GoodRes; Comp1

4) if Comptest1 < MinComp then set PrefRsz = NeutralBicubic; NewRes = NewRes - step ; Comp1; goto 4)
else if Comptest1 > MaxComp then
if PrefRsz <> Lanczos set PrefRsz = Lanczos ; Comp1; goto 4)
else if (MaxRes > NewRes + step ) then NewRes = NewRes + step ; Comp1; goto 4)

5) nrdiff=0;Comp5; Compdiff= abs(Comp5 - (5 * Comp1))/Comp5; resdiff=ceil(step * (1+Compdiff));

6) nrdiff=nrdiff + 1;
if Comp5 < MinComp then NewRes=NewRes - nrdiff*resdiff; Comp5; goto 6)
else ENCODE


Haven't given much tought to weights yet. First let's make this one work :)


Bilu

[TaZ4hvn]

Well i didn't understand strategy very well ... I will read it again :D
Anyway I will try to implement something this night I think, will tell ya.

[bilu]

Great news: I found a reliable way to deal with NTSC hybrid movies!

As referenced in http://www.doom9.org/ivtc-tut.htm , there is a tool called ParseD2V http://shelob.mordor.net/dgraft/tools/ParseD2V.zip that can be used like this:

ParseD2V Movie.d2v >log (with a space between ParseD2V and Movie.d2v and >log, where "Movie" is replaced by whatever name you're using)

"Then you can inspect the log file and you'll see frame numbers. If it says "Pattern break" next to the number then it is outside of 3:2 pattern, otherwise it is inside."

The output is like this:

0 -- 2 [3:2 pattern break]
1 -- 2 [3:2 pattern break]
2 -- 3
3 -- 0
4 -- 1
6 -- 2
7 -- 3
8 -- 0
9 -- 1
11 -- 2
12 -- 3
13 -- 0
14 -- 1
16 -- 2
17 -- 3

Since every time we get "[3:2 pattern break]" means it is Video and not Film, I found a way to count and check if a potentially hybrid NTSC movie is mostly Video or mostly Film, and then making a safe choice between Decimate(mode=1) and Decimate(mode=3)!!! :D :D :D

Made an example batch file for counting Video and Film Events.


====== Example batch file. Usage: batch movie.d2v ============
@echo off
set D2VFILE= %1
parsed2v %D2VFILE% > %D2VFILE%.LOG

for /f "usebackq delims=: tokens=3" %%I in (`find /c "[3:2 pattern break]" %D2VFILE%.LOG`) do set Video=%%I
for /f "usebackq delims=: tokens=3" %%I in (`find /c /v "[3:2 pattern break]" %D2VFILE%.LOG`) do set Film=%%I

echo Video=%Video%
echo Film=%Film%

====================================================


Will integrate into code later.

Bilu

[bilu]

Resize/Compressibility Algorithm Explained :


Parameters:

-MinRes = Minimal resolution
-MinComp = Minimum compressibility
-GoodRes = Good resolution (applying Lanczos after this one)
-MaxComp = Maximum compressibility (just for resizing decisions)
-MaxRes = Maximum resolution (we have to think about the players too )
-PrefRsz = Preferred resizer (NeutralBicubic by default,you can change)

-AudioSize = OGG file in KBytes
-Margin = defined by the user in KBytes

(assuming 20 > MaxComp - MinComp > 10)

NewRes=MinRes;
PrefRsz=NeutralBicubic;
Comp1= (1% Comp.test with NewRes + PrefRsz);
Comp5= (5% Comp.test with NewRes + PrefRsz);
step=2 (means 2000 pixels in ResizeARP, about 44 width per 44 height)


If any doubts in the parameters part, please ask.


1) Comp1;

NrCD_1p= (100*Comp1 + Audiosize)/(CDsize - Margin)
NRCDmax_diff=abs(NrCD_1p*MaxComp - round(NrCD_1p*MaxComp))
NRCDmin_diff=abs(NrCD_1p*MinComp - round(NrCD_1p*MinComp))

if NRCDmax_diff < NRCDmin_diff then NRCDs= round(NrCD_1p*MaxComp)
else NRCDs= round(NrCD_1p*MinComp)

2) DesiredSize = NrCDs * (CDsize - Margin) - AudioSize
Comptest1 = (100 * Comp1)/DesiredSize
Comptest5 = (100 * Comp5)/DesiredSize

3) if Comptest1 > MaxComp then NewRes = GoodRes; Comp1

4) if Comptest1 < MinComp then set PrefRsz = NeutralBicubic; NewRes = NewRes - step ; Comp1; goto 4)
else if Comptest1 > MaxComp then
if PrefRsz <> Lanczos set PrefRsz = Lanczos ; Comp1; goto 4)
else if (MaxRes > NewRes ) then NewRes = NewRes + step ; Comp1; goto 4)

5) nrdiff=0;Comp5; Compdiff= abs(Comp5 - (5 * Comp1))/Comp5; resdiff=ceil(step * (1+Compdiff));

6) nrdiff=nrdiff + 1;
if Comp5 < MinComp then NewRes=NewRes - nrdiff*resdiff; Comp5; goto 6)
else ENCODE



In the first part we make a 1% 1-pass encode with MinResolution and
NeutralBicubic and then calculate the number of CDs needed for a
1st pass, in float, based on defined CD size, a defined free space
CD (margin) and audio track size.

Then calculate number of CDs for maximum and minimum compressibility.
The NrCD_diff part: since we can get different number of CDs from Max
and Min compressibility results in float ( example: 1.35 for Min and
1.60 for Max )it checks which result is closer to a number of CDs ( in
this case 0.35 for Min and 0.40 for Max ).

Result would be one CD . Obviously resolution would decrease under
Minimum until achieving Min compressibility and then a 5% comp.test.
If less than minimum on the 5% comp.test it would reduce even more
the resolution and then do the 5% test again, then ENCODE.

If result had been 2 CDs on this example the 1% comp.test would
probably have exceded the Max compressibility.We would then change
from Min resolution to Good resolution, and then a new 1% comp.test.
If compressibility on this comp.test was lower than minimum then
decrease resolution by a step of 2000 pixels (about 44 width per 44
height, if square - just to get the idea) , do a new comp.test and
evaluate again. Resizer forced to Neutral is explained on next
paragraph.

If compressibility was larger than maximum and resizer was Neutral,
change to Lanczos, do a new comp.test and evaluate again ( that's why
if after changing to Lanczos I force Neutral if compressibility
becomes lower than minimum). If still larger than maximum then
increase resolution by 2000 pixels.

BUG HERE: if (comp.test Lanczos - comp.test Neutral ) > (MaxComp -
MinComp) it will loop, because if Lanczos gets comp lower than
minimum,and Neutral gets comp higher than Maximum (that was the reason
why we chose Lanczos) , and given that after forcing
Neutral I'll reduce resolution, then I'll exceed MaxComp even more. And then
Lanczos again because it was Neutral before: Eventually stop with
result being Lanczos at a very low resolution, but within comp.range :o

Correcting the bug:

3) if Comptest1 > MaxComp then NewRes = GoodRes; Comp1;Lanczos=1

4) if Comptest1 < MinComp then
if Lanczos=1 set Lanczos=0; PrefRsz = NeutralBicubic; Comp1; goto 4)
else NewRes = NewRes - step ; Comp1; goto 4)

else if Comptest1 > MaxComp then
if PrefRsz <> Lanczos and Lanczos=1 set PrefRsz = Lanczos ; Comp1; goto 4)
else if (MaxRes > NewRes + step ) then NewRes = NewRes + step ; Comp1; goto 4)


Added a flag to be able to use Lanczos on step 3). If it gets
compressibility lower than minimum then it will stop trying to use
Lanczos,and then do a new comp.test and reduce or increase resolution as
needed.
But if Lanczos comp.test exceeds compressibility it will increase
resolution until getting on comp. range. (MaxRes > NewRes ) is needed
to avoid growing resolution larger than permitted.

About 5) and 6) : Compdiff is a percentage, the difference between 1%
and 5% comp.test results.Resdiff is step amplified my Compdiff, if
there is a signifficant difference.If compressibility is lower than
minimum then resolution is decreased by a Resdiff step until it fits
into the Min-Max comp range, then ENCODE.

TODO: - maybe replace the fixed resizing step by one larger when
difference between Actual compressibility and Minimum or Maximum is
big, something like what's used on 5) and 6).

-Use weights at least on my NrCDdiff decisions: we shouldn't have gone
for 1 CD with MinResolution and NeutralBicubic on this example on a
1.35 result, maybe just for something smaller than 1.10 or 1.15

Waiting feedback :)

Best regards,

Bilu

[bilu]

Now it make sense for both TODO issues :)


1) Comp1;

NrCD_1p= (100*Comp1 + Audiosize)/(CDsize - Margin)
NRCDmax_diff=abs(NrCD_1p*MaxComp - round(NrCD_1p*MaxComp))
NRCDmin_diff=abs(NrCD_1p*MinComp - round(NrCD_1p*MinComp))

if NRCDmin_diff < 0.15 then NRCDs= round(NrCD_1p*MinComp)
else NRCDs= round(NrCD_1p*MaxComp)

2) DesiredSize = NrCDs * (CDsize - Margin) - AudioSize
Comptest1 = (100 * Comp1)/DesiredSize
Comptest5 = (100 * Comp5)/DesiredSize

3) Lanczos=1; Compdiff1= abs(Comptest1 - (MaxComp or MinComp))/(MaxComp or MinComp);
resdiff1=ceil(step * (1+Compdiff1)); if Comptest1 > MaxComp then NewRes = GoodRes; Comp1

4) if Comptest1 < MinComp then
if Lanczos=1 set Lanczos=0; PrefRsz = NeutralBicubic; Comp1; goto 4)
else NewRes = NewRes - resdiff1 ; Comp1; goto 4)

else if Comptest1 > MaxComp then
if PrefRsz <> Lanczos and Lanczos=1 set PrefRsz = Lanczos ; Comp1; goto 4)
else if (MaxRes > NewRes) then NewRes = NewRes + resdiff1 ; Comp1; goto 4)

5) nrdiff5=0;Comp5; Compdiff5= abs(Comp5 - (5 * Comp1))/Comp5; resdiff5=ceil(step * (1+Compdiff5));

6) nrdiff5=nrdiff5 + 1;
if Comp5 < MinComp then NewRes=NewRes - nrdiff5*resdiff5; Comp5; goto 6)
else ENCODE


if NRCDmin_diff < 0.15 then NRCDs= round(NrCD_1p*MinComp)
else NRCDs= round(NrCD_1p*MaxComp)

On former example, does 1 CD if MinComp results are less than 1.15, regardless of MaxComp result.


Compdiff1= abs(Comptest1 - (MaxComp or MinComp))/(MaxComp or MinComp);
resdiff1=ceil(step * (1+Compdiff1))

Increases step based on difference between comp.test value and Min ou Max compressibility.
Will speed resizing until getting into comp. range

NOTE: This is not code, just algorithm design!

Bilu

[bilu]

Project is not dead, me and TaZ4hvn have been working on it :)

TaZ made a version that supports aspect ratio correction based on Stream_Information.txt generated by DVD Decryptor and other features he can describe better because I hadn't time to check it yet.

I made a complete rewrite of the current functionalities and inserted ParseD2V, now IVTC should not be a problem unless under extreme cases.
It even supports hybrid streams like Star Trek TNG and Babylon 5, because it chooses based on statistics the correct procedure between Decimate(mode=1) and Decimate(mode=3).

I've already coded the CD number decision today, but haven't tested it. Still missing is the resize code.

After testing I'll release this version, but there are plans for the future, namely MP4 muxing (easily doable) and AutoAnamorphic for those with widescreen output. I'm also looking for a way to use Vobsub Cutter command line to split and stay in synch, for those who want subs through DirectVobsub.

Last but not least, after everything is done we'll try to make the batch file compatible with older versions of Windows.

Hope to bring news soon,
Bilu

[TaZ4hvn]

Project is not dead, me and TaZ4hvn have been working on it
Far from it ! I'm working on it hours a day ... if I didn't post recently is because I'm so close to a first release I can't find time to write :p

TaZ made a version that supports aspect ratio correction based on Stream_Information.txt generated by DVD Decryptor and other features he can describe better because I hadn't time to check it yet.
Well I added so much stuff since the version I sent you I can't even remenber ... as I will post very soon I won't comment by now.

It even supports hybrid streams like Star Trek TNG and Babylon 5, because it chooses based on statistics the correct procedure between Decimate(mode=1) and Decimate(mode=3).
Well good to ear you achieved it to work, I didn't find on this at the moment, I still hope we can merge later in a complete achieved tool.

I've already coded the CD number decision today, but haven't tested it. Still missing is the resize code.
Why don't you use my AviSynth function, it's good enough ??

Last but not least, after everything is done we'll try to make the batch file compatible with older versions of Windows.
Looks like you ahd headaches with delayed expansion too :p

[bilu]

Originally posted by TaZ4hvn
Why don't you use my AviSynth function, it's good enough ??


I was talking about resizing/compressibility function. In the meantime I've finished it and will test it for the first time :)

About resizing functions, I want to have 3 different methods:

1) Crop All - all bitrate goes to movie, none to borders. For those with players that change AR.

2) "PerfectCropResize" - your function :D . It gets AR info from Stream_Information.txt file created by DVD Decryptor, and corrects image based on a PAR table. Uses borders. (I'll try to change it later to add borders at the end of processing, to make script faster and easier to compress)

3) Anamorphic - whenever possible stretch movie height by 33% to reduce border and improve image on widescreen TVs. If you have on of those, it's a must. ;) Frame will have less borders, which means more bitrate for movie.


Looks like you ahd headaches with delayed expansion too :p


If you're speaking about CMD environment variable delayed expansion, I'm not using it.
What I really meant was to increase the universe of potential users of our tools! :D :D :D


Bilu

[TaZ4hvn]

If you're speaking about CMD environment variable delayed expansion, I'm not using it.
Damn you're stronger than me :( ... I'de like to avoid 'em but didn't find a way yet ... if you find a way to achieve delayed expansion without those damn ! ... let me know :p

Anamorphic - whenever possible stretch movie height by 33% to reduce border and improve image on widescreen TVs. If you have on of those, it's a must. Frame will have less borders, which means more bitrate for movie.
One time again I won't bother with this as I only play divx on computer screen ... you're on your own :p good luck !

I'm currently adding a greater user error control by overriding registry settings for progs each time it's possible :)
Already did it for DVDdecrypter and XviD.
I think it's a must have since I won't bother users with pages of "configure this that way ...".
It's also more user friendly.

[TaZ4hvn]

Well finally I post a first version:
http://forum.doom9.org/showthread.php?s=&threadid=53043

[bilu]

A new version will come soon.

I've redesigned the way R4R_ENC worked, putting everything into functions and using them with every batch variable needed as parameter.
This way the biggest part of the process is within AVS, which makes it much easier to debug and avoid a lot of batch parsing problems. :rolleyes:

The only thing missing now are two new resizing modes: Anamorphic and my version of TaZ's "PerfectCropResize". And the documentation, of course! :D Then I can add the rest of the features that TaZ is using on his tool, TaZit. ;)

The automatic resizing/compressibility decision is finished and this version is even able to generate logfiles! :D :D :D
It's prepared to deal with more than one resizer, but I'm only using NeutralBicubic because that's the way I like it :)

A little preview (30 min 4/3 PAL episode of Fawlty Towers):

For a range 60%-75% compressibility.

14-05-2003 - 04:47 : Check for subtitles.
14-05-2003 - 04:47 : Setting plugins and movie variables.
14-05-2003 - 04:47 : Deciding number of CDs needed.
14-05-2003 - 04:49 : Nr. of CDs needed = 0.500000
14-05-2003 - 04:49 : Compressibility tests (one percent).
14-05-2003 - 04:49 : Testing at 150 thousand pixels.
14-05-2003 - 04:49 : Compressibility = 85.469116 percent.
14-05-2003 - 04:51 : Testing at 185 thousand pixels.
14-05-2003 - 04:51 : Compressibility = 72.779419 percent.
14-05-2003 - 04:51 : Compressibility test results (one percent) = 185 thousand pixels.
14-05-2003 - 04:51 : Compressibility tests (five percent).
14-05-2003 - 04:51 : Determining difference between one and five percent comp.tests ...
14-05-2003 - 05:03 : Difference between 1 and 5 percent comp.tests = 2.357756 percent.
14-05-2003 - 05:03 : Testing at 185 thousand pixels.
14-05-2003 - 05:03 : Compressibility = 71.063408 percent.
14-05-2003 - 05:03 : Compressibility test results (five percent) = 185 thousand pixels.
14-05-2003 - 05:03 : Encoding at 185 thousand pixels.


Stay tuned, I hope to finish everything this week. You won't be disappointed ;)

NOTE:About "Difference between 1 and 5 percent comp.tests = 2.357756 percent.", formula is abs(Comp5size - 5*Comp1size)/Comp5size, and not abs(Comp1test - Comp5test).

Best regards,
Bilu

[TaZ4hvn]

Well, looks like you're on the way to finalize your tool :) good job !
I'd like to know how you generate a logfile, since i'd like to add it to my tool.
I also implemented a new resize/filters decision algo, dunno how it compares to yours, could be instructive to compare them.
All my best wishes to your project :p

[bilu]

R4R_ENC 2.0 (it deserves the version number :) )

Completely redesigned, the scripts made by batch files now call functions and include all the parameters needed.Got easier to debug and write code.

Also finished my CD fitting algorithm. It's pretty flexible, have a look in the code :)

http://pagpessoais.iol.pt/bruno_henriques/r4r_enc2.zip


Best regards,
Bilu

[TaZ4hvn]

Well I get it right now man !
I'm pretty interrested to see what you did on the resizing decision part to compare to mine :p

[DaveEL]

avs2avi 1.29 out should work with xvid correctly now.

http://daveel.leffe.dnsalias.com/avs2avi-129.zip

DaveEL

[bilu]

Everybody please update to the last AVS2AVI version, it has a new switch that is not important to this tool but also fixes a problem with decoding, since 1.26 only decoded with FFDshow.

R4R_ENC 2.0a should be released in a few hours, it fixes a lot of bugs from the 2.0 version, has corrected Anamorphic and Unamorphic (undoes Anamorphic) processes, and as flags to use subs just if you want (no errors in SWITCH like in pre-2.0 versions) and a flag for borders cropping ( you can use Anamorphic or Unamorphic and then crop borders if you want ).


Best regards,
Bilu

[bilu]

Here it is, and so far seems to work really good :)

Please post feedback about the needs for documentation, and bugs that may arise.

2 repeated warnings:

1) Use latest AVS2AVI;

2) My automatic deinterlacing function is generic and not perfect.
It's based on framerates and MPEG-2 flags that are not allways encoded correctly, or encoded at all.

Hardware telecined movies are one of this examples: when framerate= 29.97 fps I choose between Decimate(mode=1) and (mode=3) based on 3:2 pattern counting made by ParseD2V.

An hard-telecined has VIDEO flags in the D2V, so the duplicated fields generated by the telecine process will be replaced by blended frames, which is not as good as the correct Inverse Telecine process.
I won't say it is bad (yet) because I haven't tested this side effect.


It should work correctly over PAL movies and correctly flagged NTSC video and film streams.


Grab it at
http://pagpessoais.iol.pt/bruno_henriques/r4r_enc2a.zip

[EDIT] Grab it now, corrected ZIP file. And don't forget to run first the run1st.reg file!

[EDIT2] I won't make a standalone tool, that's TaZ project. :)
Mainly because I enjoy having a job queue.

I would like add VobSub Cutter to my project, but I would need to be able to get duration of each movie split and make time calculations to use with VobSub Cutter on command line. This would enable automated subtitle ripping for DirectVobSub playing! :D

Won't use KEnforcer and chapter info because TaZ has already tried and it conflicts with b-frames.


Best regards,
Bilu

[TaZ4hvn]

2) My automatic deinterlacing function is generic and not perfect.
On the few test I just did it works far better than the previous one !
I'm waiting for you to find the best solution here so that I can steal it to you :D

I won't make a standalone tool, that's TaZ project. Mainly because I enjoy having a job queue.
To be true, queue is on my mind too :p

Won't use KEnforcer and chapter info because TaZ has already tried and it conflicts with b-frames.
At the moment, pal ... anyhow I provide a flag to disable KFenforcer to use Bframes. But I really don't think Bframes are that much usefull for 'normal' bitrates.

Also did you give a try to my new resizing/filtering algo ?
Caus' I think we both suffer of the same problem concerning comptest, i.e. that artificial keyframes are inserted at each snap start leading to slightly oversized comptest.
I did some 95% comptest and comp. was still under the real one, I'd like to find a CLI tool able to discard this artificials Iframes in calculation but didn't find one yet :(
I can post links to related threads I read about this if you're interested ...

Regards.

[jonny]

Oversized frames are not only at the start of each snip (but at the end too if you use bframes... not discarding this frames can lead in an error up to 20%-25%).
With DivX things are easy (the bframes implementation is simple).
For XviD+bframes i use a strange method, i actually discard 3 frames at the start and 3 frames at the end of each snip, this is the Enc's "extended method"... and seems to reduce the error, but in some case the error is still too high... 7%-8% ... consider that in a normal case you obtain an error that range from 1% to 3% (soon i'll start experimenting a new method that should make miracles ^^).

Anyway parsing the log file generated by VDubMod or AVS2AVI it's essential for making the comp.test work correctly.

Do you need a little exe that parse this log and output the average frame size (discarding the correct frames)?
(drop me a PM if this can be enough)

[bilu]

Do you need a little exe that parse this log and output the average frame size (discarding the correct frames)?
(drop me a PM if this can be enough)


I did :)


On the few test I just did it works far better than the previous one !
I'm waiting for you to find the best solution here so that I can steal it to you


The previous version was really bad because it halved the vertical resolution. This one seems OK for everything except incorrectly flagged NTSC DVDs. I know that the side effect on this DVDs will be worst than doing a correct IVTC process, but I haven't tested yet to know how bad it is. Still there are no alternatives at the moment... except manual encoding with "eye-analisis" :(


Best regards,
Bilu

[TaZ4hvn]

Still there are no alternatives at the moment... except manual encoding with "eye-analisis"
Well that's what I'd understood, so I guess I just have to try to implement it :)

[bilu]

My CD fitting algo somewhat explained
===============================

Nr.CD Decision
============

fpsize=(1% comp.test size / 1024)
audiosize=audio size / 1024
CDsplit=(CDsize - margin)

NrCD_1p = (100*fpsize + audiosize)/(CDsplit * 1024)

NrCDmin_diff = abs(frac(NrCD_1p * MinComp))

NrCDs= (NrCDmin_diff < tolerance)? floor(NrCD_1p * MinComp):ceil(NrCD_1p * MaxComp)

For example, if NrCD_1p*MinComp=1.2 and NrCD_1p*MaxComp=1.4, it aims at 1CD. But if NrCD_1p*MinComp=1.3 and NrCD_1p*MaxComp=1.4, it aims at 2 CDs.

Tolerance default is 0.20, it means that if (NrCD_1p * MinComp) < 1.2 CDs it will fit in 1 CD by reducing resolution.
MinComp = minimum compressibility, MaxComp = maximum compressibility

NrCDs= ((fpsize + audiosize + margin) * MinComp < ((CDsplit*1024*(1+tolerance))/2))? 0.5 : NrCDs

Checks if it would fit in half CD.

DesiredSize=floor(NrCDs*(CDsplit*1024)- audiosize)

Now that you have a desired size, let's check if 1st pass is closer to MaxComp or to MinComp, and then define a "boost" to the fixed step based on the difference between 1st pass compressibility and MaxComp or MinComp, which one is closer.

Comptest1= float(DesiredSize)/fpsize
Compdiff1_max= abs(Comptest1 - MaxComp)/MaxComp
Compdiff1_min= abs(Comptest1 - MinComp)/MinComp

resdiff1=(Compdiff1_min < Compdiff1_max)? ceil(step*(1+Compdiff1_min)):ceil(step*(1+Compdiff1_max))

1% Comp.tests
=============
Uses the 1% comp.test calculated on Nr.CD decision on the first time, does new 1% comp.tests after looping (if needed).

If comp.test result < MinComp, decreases resolution by a resdiff1 step, else if comp.test result > MaxComp and resolution < MaxRes it increases resolution by the same resdiff1 step.

If comp.test result is within comp. range proceed to 5% comp. test(define step).


5% Comp.test (define step)
==========================
Calculates difference between 5% and 1% comp.test results and "boosts" the fixed step. It's not based in the calculated step from Nr.CD decision.


5% Comp.test
============
Does a 5% Comp. test, and if results are lower than MinComp decreases resolution by the new step and loops, else go for encoding. If Comp. test is bigger than 100% ( on small clips, since we aim at half CD ) does only first pass.

NOTE: Steps and resolutions are defined in Kpix/frame = (width * height).

Best regards,
Bilu

[bilu]

New version with bugfixes, same attachment name (r4r_enc2a.zip).

Replaced

NrCDs= (NrCDmin_diff < tolerance)? round(NrCD_1p * MinComp):round(NrCD_1p * MaxComp)

with


NrCDs= (NrCDmin_diff < tolerance)? floor(NrCD_1p * MinComp):ceil(NrCD_1p * MaxComp)


If NrCD_1p * MinComp = 1.3 and NrCD_1p * MaxComp = 1.4, it was aiming at 1 CD. Corrected.

Also added the 2nd pass skip for compressibilities over 100%, mentioned on the post above.


Bilu

[bilu]

Originally posted by TaZ4hvn
anyhow I provide a flag to disable KFenforcer to use Bframes. But I really don't think Bframes are that much usefull for 'normal' bitrates.


He's the boss, and says it all ;)

http://forum.doom9.org/showthread.php?s=&postid=316512#post316512


Bilu

[TaZ4hvn]

Well I can't tell who between Acaila and Syskin is right.
They are certainly both by far more into Bframes than me, and most af all I don't want to debate on which XviD features are best: that's not my point.
So that if anybody is wanting to use Bframes with my tool he just has to disable KFE, simple as that! Choice is up to the user, I don't force anyone not to use Bframes :D
To me chaptering jump accuracy is more important than Bframes and ... as I'm the first user of TaZit ...
And I'm still confident Zulu will sort problem with Bframes one day.
So I think debate on Bframes and/or KFE in our tools should be over, after all each of us is free to use tools he likes, isn't it ?

[bilu]

You're right, that's the attitude :)

I'll use this reply to post about another issue: about automatic subtitle splitting to use with DirectVobsub.

I'll have to make tests around this:

OGMInfo: http://cyrius.bunkus.org/tools.html

Usage: OGMInfo -p -l file.log movie.ogm

It gets duration info for both audio and video. It had a difference of 30 ms between them in a small clip (5 min) I tested.

for /f "useback skip=1 tokens=3" %%I in (`findstr "Duration" file.log`) do echo %%I

retrieves the audio duration info. I have no way to skip from the end of the file. But maybe audio duration will do fine. And then I need to create an Avisynth function to make time sums to use in Vobsub Cutter command line like this:

rundll32 vobsub.dll,Cutter <src> <dst> <countfromstart> <hh1:mm1:ss1:ms1> [<hh2:mm2:ss2:ms2>]


Let's hope it works :)

Bilu

[MinosK]

The command line of vobsub Cutter is:
rundll32 vobsub.dll,Cutter <src> <dst> <countfromstart> <hh1:mm1:ss1.ms1> [<hh2:mm2:ss2.ms2>]

(not rundll32 vobsub.dll,Cutter <src> <dst> <countfromstart> <hh1:mm1:ss1:ms1> [<hh2:mm2:ss2:ms2>])

Thanks for your great program..... Great job!!!

[bilu]

Good news for DVobsub users: it seems to work! :D

I've tried splitting a chapter (a 5 min first-pass clip) into three 15 MB parts, use OGMInfo to get the duration (total about 32ms deviation between audio and video durations), and make calculations based on the audio duration.

Clip was encoded with permanent subtitles and then splitted. Then I grabbed the original sub file and changed offset and colors, so I could see it over the permanent subtitles.

Then I grabbed the subtitles file and used Vobsub Cutter's GUI to cut into three subs.

When playing the three splitted clips all subtitles were correctly synchronized with the permanent subtitles, which means timings were correct. :) :) :)

The size of each split won't affect the delay, but the number of splits will. I'll try a larger clip ( this first pass was only 45 MB) divided into 10 clips ;)

If it works they can be splitted automatically, but another problem would be to deal with its size on size prediction: how could you take (subs filesize / nr CDs) into calculations of the number of CDs, if you haven't decided the number of splits yet? :confused:


But this is getting good ... ;)


Bilu

[bilu]

Originally posted by MinosK
The command line of vobsub Cutter is:
rundll32 vobsub.dll,Cutter <src> <dst> <countfromstart> <hh1:mm1:ss1.ms1> [<hh2:mm2:ss2.ms2>]

(not rundll32 vobsub.dll,Cutter <src> <dst> <countfromstart> <hh1:mm1:ss1:ms1> [<hh2:mm2:ss2:ms2>])

Thanks for your great program..... Great job!!!

Thanks MinosK :)

But I can't see the difference??? :confused:

Bilu

[Lobb0]

Originally posted by bilu
Thanks MinosK :)

But I can't see the difference??? :confused:

Bilu

Bilu,

hi amigo, here the difference:

<hh1:mm1:ss1:ms1> [<hh2:mm2:ss2:ms2>] wrong way

<hh1:mm1:ss1.ms1> [<hh2:mm2:ss2.ms2>] right way

Great job dude.
Best regards from Brasil.

[bilu]

That's a really small difference :D

Thanks a lot :)

Bilu

[TaZ4hvn]

About autoIVTC/deinterlacing:
I noticed that even if it doesn't harm PAL non interlaced it slow down the whole process by about 25%.
The idea I had is performing two comptests, one with autoIVTC on the other with it off and then compare the two clips, if difference is below a given threshold then turn ivtc off.
What do you think of this ?

[bilu]

Well,

It may work, given that the interlaced artifacts greatly increase the filesize. If you get a much lower filesize with deinterlacing, then it's the way to go.

We'll have to test it, but it seems a very good idea :)


Bilu

[TaZ4hvn]

Actually I was thinking at doing an even smarter process:
compare the two clips within avisynth doing Subtract and then checking luma of the difference, after we forced the same fps for the two clips of course :p

[EDIT] Also setting post=false in telecide doesn't seem to harm quality too much (if at all) and reduces slowdown from 25% to 10% ...

[bilu]

New deinterlace function, faster when not doing IVTC:


function Deinterlace(clip c,int D2V_Video,int D2V_Film) {
c= (D2V_Video > D2V_Film)? FieldDeinterlace(c,dthreshold=0): Decimate(Telecide(c,guide=1,gthresh=30),mode=3,threshold=1.0)
return c
}


I've been testing all day over panning scenes telecined movie (correctly flagged), but processing like it was an hard-telecined one, just to know the side effects of Decimate(mode=1). Results were very jumpy, and after trying alternatives (remember I was trying to maintain 29.97 fps frameratio, like on a mostly video hybrid clip) I found the best results with FieldDeinterlace(dthreshold=0).

Also got some jumpy parts with Telecide() over a PAL series (Fawlty Towers) on some frames and liked it better with FieldDeinterlace(dthreshold=0).

The side effects on both clips were a motion-blur effect when decombing is needed (won't affect progressive PAL), that are more acceptable than interlaced artifacts and, IMHO, to jumpy clips.

But I also think motion-blur should suck on anime :o

Deinterlace function is not updated in the ZIP file, I want to work in some more things before updating :)

Best regards,
Bilu

[bilu]

This is a very nice reading for all Decomb users. Recommended.

http://shelob.mordor.net/dgraft/journal/journal.html


Bilu

[bilu]

Hi guys,

This post is just to say that I've finally achieved an automatic splitting process for external subtitles to use with DirectVobSub.

Tested over a small clip divided in three with permanent subtitles and external subtitles flipped vertically on DVobSub so I could see both.

And they were perfectly synchronized in the three clips. :cool:


I must test it first inside R4R_ENC together with the new compressibility method based on a tool provided by jonny.

This tool has a "normal" and "extended" switch just like his program Enc, "extended" is for XVID with b-frames. I've already adapted my tool to automatically check if you're using b-frames or not. ;)


Compressibility tests will now take in account the Vobsub file size (in the same way it does with the audio size) whenever switches subs=1 and permsubs=0 in the settings.ini file.

Everything I mentioned is related to my next version to be soon released.



I cannot promise yet a nice manual, but I sure will try to do it with this tool:

http://forum.doom9.org/showthread.php?threadid=54079


NOTE: I'm not using the new generation Decomb filter yet, but the prior YV12 version.


TODO:
- Pack everything needed with R4R_ENC with a nice manual :)

- Support other containers (AVI,MP4 and maybe MKV):
Not much complicated if I have access to command-line tools for the muxing/splitting and for getting duration of each clip so I can integrate with the external subs splitting feature.
And start looking at container's overheads.

And to add command-line RAR support. It's a pity Vobsub uses RAR because there are no free compressors, just decompressors for this format :rolleyes:

Will have a flag on settings.ini so that we compress the subs first and have the compressed size in account when making the nrCDs/compressibility/resizing decisions. Overhead over splitted RAR files is not significant.


Stay tuned ;)

Bilu

[bilu]

R4R_ENC 2.0b
============

Grab it at

http://pagpessoais.iol.pt/bruno_henriques/r4r_enc2b.zip


New features:

- Uses jonny's compressibility method. Automatically checks if you're using b-frames in the registry to select between "normal" and "extended" modes.

- External subtitle splitting to use with DirectVobSub. It grabs each splitted clip duration with OGMInfo, calculates cut points with an AVS script and then use VobSub Cutter from the command-line to cut the subs file. Perfectly synchronized, I must say :)


TODO:
- Support other containers (AVI,MKV,MP4)
- Containers overhead
- RAR support for external subtitles
- A decent manual


I'm getting very happy to see how mature it is coming :)
It deserves a decent manual as a sign of maturity ;)


I need to look at newer Decomb versions than 4.1b3, had no time yet.



Best regards,
Bilu