R4R_ENC - Batch tool to use XVID,AVS2AVI and OGGMuxer inside Robot4Rip

[bilu]

Originally posted by TaZ4hvn
I did an encode wich ended 2cd with only 500MB on the second (so quant2
only is used) because the 1cd compressibility was 0.57 versus bound set to
0.60 !!
Nobody wants this, it's clear that sizing a bit less and using a less
sharpening filter would easily end with a 0.65 compressibility on 1cd.
So I'm working on it by now.

My concept (will try to code it later)

 
Parameters:
-MinRes  = Minimal resolution 
-MinComp = Minimum compressibility 
-GoodRes = Good resolution (applying Lanczos after this one)
-MaxComp = Maximum compressibility (just for resizing decisions)
-MaxRes  = Maximum resolution (we have to think about the players too )
-PrefRsz = Preferred resizer (NeutralBicubic by default,you can change) 

-AudioSize = Clip duration in seconds * (AudioBitrate/8)
-Margin = defined by the user in KBytes

(assuming 20 > MaxComp - MinComp > 10)


NewRes=MinRes;
PrefRsz=NeutralBicubic; 
Comp1= (1% Comp.test with NewRes + PrefRsz); 
Comp5= (5% Comp.test with NewRes + PrefRsz);
step=2  (means 2000 pixels in ResizeARP, about 44 width per 44 height)

1) Comp1; 

NrCD_1p= (100*Comp1)/(CDsize - Margin - Audiosize)
NRCDmax_diff=abs(NrCD_1p*MaxComp - round(NrCD_1p*MaxComp))
NRCDmin_diff=abs(NrCD_1p*MinComp - round(NrCD_1p*MinComp))

if NRCDmax_diff < NRCDmin_diff then NRCDs= round(NrCD_1p*MaxComp)
else NRCDs= round(NrCD_1p*MinComp)

2) DesiredSize  = NrCDs * (CDsize - Margin) - AudioSize
   Comptest1    = (100 * Comp1)/DesiredSize
   Comptest5    = (100 * Comp5)/DesiredSize
 
3) if Comptest1 > MaxComp then NewRes = GoodRes; Comp1 
 
4) if Comptest1 < MinComp then set PrefRsz = NeutralBicubic; NewRes = NewRes - step ; Comp1; goto 4)
   else if Comptest1 > MaxComp then 
	if PrefRsz <> Lanczos set PrefRsz = Lanczos ; Comp1; goto 4)
   	else if (MaxRes > NewRes + step ) then NewRes = NewRes + step ; Comp1; goto 4)

5) nrdiff=0;Comp5; Compdiff= abs(Comp5 - (5 * Comp1))/Comp5; resdiff=ceil(step * (1+Compdiff)); 

6) nrdiff=nrdiff + 1;
   if Comp5 < MinComp then NewRes=NewRes - nrdiff*resdiff; Comp5; goto 6)
   else ENCODE

Haven't given much tought to weights yet. First let's make this one work


Bilu

[TaZ4hvn]

Well i didn't understand strategy very well ... I will read it again
Anyway I will try to implement something this night I think, will tell ya.

[bilu]

Great news: I found a reliable way to deal with NTSC hybrid movies!

As referenced in http://www.doom9.org/ivtc-tut.htm , there is a tool called ParseD2V http://shelob.mordor.net/dgraft/tools/ParseD2V.zip that can be used like this:

ParseD2V Movie.d2v >log (with a space between ParseD2V and Movie.d2v and >log, where "Movie" is replaced by whatever name you're using)

"Then you can inspect the log file and you'll see frame numbers. If it says "Pattern break" next to the number then it is outside of 3:2 pattern, otherwise it is inside."

The output is like this:

0 -- 2 [3:2 pattern break]
1 -- 2 [3:2 pattern break]
2 -- 3
3 -- 0
4 -- 1
6 -- 2
7 -- 3
8 -- 0
9 -- 1
11 -- 2
12 -- 3
13 -- 0
14 -- 1
16 -- 2
17 -- 3

Since every time we get "[3:2 pattern break]" means it is Video and not Film, I found a way to count and check if a potentially hybrid NTSC movie is mostly Video or mostly Film, and then making a safe choice between Decimate(mode=1) and Decimate(mode=3)!!!

Made an example batch file for counting Video and Film Events.

====== Example batch file. Usage: batch movie.d2v ============
@echo off
set D2VFILE= %1
parsed2v %D2VFILE% > %D2VFILE%.LOG

for /f "usebackq delims=: tokens=3" %%I in (`find /c "[3:2 pattern break]" %D2VFILE%.LOG`) do set Video=%%I
for /f "usebackq delims=: tokens=3" %%I in (`find /c /v "[3:2 pattern break]" %D2VFILE%.LOG`) do set Film=%%I

echo Video=%Video%
echo Film=%Film%

====================================================

Will integrate into code later.

Bilu

[bilu]

Resize/Compressibility Algorithm Explained :

Parameters:

-MinRes  = Minimal resolution 
-MinComp = Minimum compressibility 
-GoodRes = Good resolution (applying Lanczos after this one)
-MaxComp = Maximum compressibility (just for resizing decisions)
-MaxRes  = Maximum resolution (we have to think about the players too )
-PrefRsz = Preferred resizer (NeutralBicubic by default,you can change) 

-AudioSize = OGG file in KBytes
-Margin = defined by the user in KBytes

(assuming 20 > MaxComp - MinComp > 10)

NewRes=MinRes;
PrefRsz=NeutralBicubic; 
Comp1= (1% Comp.test with NewRes + PrefRsz); 
Comp5= (5% Comp.test with NewRes + PrefRsz);
step=2  (means 2000 pixels in ResizeARP, about 44 width per 44 height)

If any doubts in the parameters part, please ask.

1) Comp1; 

NrCD_1p= (100*Comp1 + Audiosize)/(CDsize - Margin)
NRCDmax_diff=abs(NrCD_1p*MaxComp - round(NrCD_1p*MaxComp))
NRCDmin_diff=abs(NrCD_1p*MinComp - round(NrCD_1p*MinComp))

if NRCDmax_diff < NRCDmin_diff then NRCDs= round(NrCD_1p*MaxComp)
else NRCDs= round(NrCD_1p*MinComp)

2) DesiredSize  = NrCDs * (CDsize - Margin) - AudioSize
   Comptest1    = (100 * Comp1)/DesiredSize
   Comptest5    = (100 * Comp5)/DesiredSize
 
3) if Comptest1 > MaxComp then NewRes = GoodRes; Comp1 
 
4) if Comptest1 < MinComp then set PrefRsz = NeutralBicubic; NewRes = NewRes - step ; Comp1; goto 4)
   else if Comptest1 > MaxComp then 
	if PrefRsz <> Lanczos set PrefRsz = Lanczos ; Comp1; goto 4)
   	else if (MaxRes > NewRes ) then NewRes = NewRes + step ; Comp1; goto 4)

5) nrdiff=0;Comp5; Compdiff= abs(Comp5 - (5 * Comp1))/Comp5; resdiff=ceil(step * (1+Compdiff)); 

6) nrdiff=nrdiff + 1;
   if Comp5 < MinComp then NewRes=NewRes - nrdiff*resdiff; Comp5; goto 6)
   else ENCODE

In the first part we make a 1% 1-pass encode with MinResolution and
NeutralBicubic and then calculate the number of CDs needed for a
1st pass, in float, based on defined CD size, a defined free space
CD (margin) and audio track size.

Then calculate number of CDs for maximum and minimum compressibility.
The NrCD_diff part: since we can get different number of CDs from Max
and Min compressibility results in float ( example: 1.35 for Min and
1.60 for Max )it checks which result is closer to a number of CDs ( in
this case 0.35 for Min and 0.40 for Max ).

Result would be one CD . Obviously resolution would decrease under
Minimum until achieving Min compressibility and then a 5% comp.test.
If less than minimum on the 5% comp.test it would reduce even more
the resolution and then do the 5% test again, then ENCODE.

If result had been 2 CDs on this example the 1% comp.test would
probably have exceded the Max compressibility.We would then change
from Min resolution to Good resolution, and then a new 1% comp.test.
If compressibility on this comp.test was lower than minimum then
decrease resolution by a step of 2000 pixels (about 44 width per 44
height, if square - just to get the idea) , do a new comp.test and
evaluate again. Resizer forced to Neutral is explained on next
paragraph.

If compressibility was larger than maximum and resizer was Neutral,
change to Lanczos, do a new comp.test and evaluate again ( that's why
if after changing to Lanczos I force Neutral if compressibility
becomes lower than minimum). If still larger than maximum then
increase resolution by 2000 pixels.

BUG HERE: if (comp.test Lanczos - comp.test Neutral ) > (MaxComp -
MinComp) it will loop, because if Lanczos gets comp lower than
minimum,and Neutral gets comp higher than Maximum (that was the reason
why we chose Lanczos) , and given that after forcing
Neutral I'll reduce resolution, then I'll exceed MaxComp even more. And then
Lanczos again because it was Neutral before: Eventually stop with
result being Lanczos at a very low resolution, but within comp.range

Correcting the bug:

3) if Comptest1 > MaxComp then NewRes = GoodRes; Comp1;Lanczos=1
 
4) if Comptest1 < MinComp then         
	if Lanczos=1 set Lanczos=0; PrefRsz = NeutralBicubic; Comp1; goto 4)        
	else NewRes = NewRes - step ; Comp1; goto 4)

   else if Comptest1 > MaxComp then 
	if PrefRsz <> Lanczos and Lanczos=1 set PrefRsz = Lanczos ; Comp1; goto 4)
   	else if (MaxRes > NewRes + step ) then NewRes = NewRes + step ; Comp1; goto 4)

Added a flag to be able to use Lanczos on step 3). If it gets
compressibility lower than minimum then it will stop trying to use
Lanczos,and then do a new comp.test and reduce or increase resolution as
needed.
But if Lanczos comp.test exceeds compressibility it will increase
resolution until getting on comp. range. (MaxRes > NewRes ) is needed
to avoid growing resolution larger than permitted.

About 5) and 6) : Compdiff is a percentage, the difference between 1%
and 5% comp.test results.Resdiff is step amplified my Compdiff, if
there is a signifficant difference.If compressibility is lower than
minimum then resolution is decreased by a Resdiff step until it fits
into the Min-Max comp range, then ENCODE.

TODO: - maybe replace the fixed resizing step by one larger when
difference between Actual compressibility and Minimum or Maximum is
big, something like what's used on 5) and 6).

-Use weights at least on my NrCDdiff decisions: we shouldn't have gone
for 1 CD with MinResolution and NeutralBicubic on this example on a
1.35 result, maybe just for something smaller than 1.10 or 1.15

Waiting feedback

Best regards,

Bilu

[bilu]

Now it make sense for both TODO issues

1) Comp1; 

NrCD_1p= (100*Comp1 + Audiosize)/(CDsize - Margin)
NRCDmax_diff=abs(NrCD_1p*MaxComp - round(NrCD_1p*MaxComp))
NRCDmin_diff=abs(NrCD_1p*MinComp - round(NrCD_1p*MinComp))

if NRCDmin_diff < 0.15 then NRCDs= round(NrCD_1p*MinComp)
else NRCDs= round(NrCD_1p*MaxComp)

2) DesiredSize  = NrCDs * (CDsize - Margin) - AudioSize
   Comptest1    = (100 * Comp1)/DesiredSize
   Comptest5    = (100 * Comp5)/DesiredSize
 
3) Lanczos=1; Compdiff1= abs(Comptest1 - (MaxComp or MinComp))/(MaxComp or MinComp); 
   resdiff1=ceil(step * (1+Compdiff1)); if Comptest1 > MaxComp then NewRes = GoodRes; Comp1

4) if Comptest1 < MinComp then         
	if Lanczos=1 set Lanczos=0; PrefRsz = NeutralBicubic; Comp1; goto 4)        
	else NewRes = NewRes - resdiff1 ; Comp1; goto 4)

   else if Comptest1 > MaxComp then 
	if PrefRsz <> Lanczos and Lanczos=1 set PrefRsz = Lanczos ; Comp1; goto 4)
   	else if (MaxRes > NewRes) then NewRes = NewRes + resdiff1 ; Comp1; goto 4)

5) nrdiff5=0;Comp5; Compdiff5= abs(Comp5 - (5 * Comp1))/Comp5; resdiff5=ceil(step * (1+Compdiff5)); 

6) nrdiff5=nrdiff5 + 1;
   if Comp5 < MinComp then NewRes=NewRes - nrdiff5*resdiff5; Comp5; goto 6)
   else ENCODE

if NRCDmin_diff < 0.15 then NRCDs= round(NrCD_1p*MinComp)
else NRCDs= round(NrCD_1p*MaxComp)

On former example, does 1 CD if MinComp results are less than 1.15, regardless of MaxComp result.


Compdiff1= abs(Comptest1 - (MaxComp or MinComp))/(MaxComp or MinComp);
resdiff1=ceil(step * (1+Compdiff1))

Increases step based on difference between comp.test value and Min ou Max compressibility.
Will speed resizing until getting into comp. range

NOTE: This is not code, just algorithm design!

Bilu

[bilu]

Project is not dead, me and TaZ4hvn have been working on it

TaZ made a version that supports aspect ratio correction based on Stream_Information.txt generated by DVD Decryptor and other features he can describe better because I hadn't time to check it yet.

I made a complete rewrite of the current functionalities and inserted ParseD2V, now IVTC should not be a problem unless under extreme cases.
It even supports hybrid streams like Star Trek TNG and Babylon 5, because it chooses based on statistics the correct procedure between Decimate(mode=1) and Decimate(mode=3).

I've already coded the CD number decision today, but haven't tested it. Still missing is the resize code.

After testing I'll release this version, but there are plans for the future, namely MP4 muxing (easily doable) and AutoAnamorphic for those with widescreen output. I'm also looking for a way to use Vobsub Cutter command line to split and stay in synch, for those who want subs through DirectVobsub.

Last but not least, after everything is done we'll try to make the batch file compatible with older versions of Windows.

Hope to bring news soon,
Bilu

[TaZ4hvn]

Project is not dead, me and TaZ4hvn have been working on it

Far from it ! I'm working on it hours a day ... if I didn't post recently is because I'm so close to a first release I can't find time to write

TaZ made a version that supports aspect ratio correction based on Stream_Information.txt generated by DVD Decryptor and other features he can describe better because I hadn't time to check it yet.

Well I added so much stuff since the version I sent you I can't even remenber ... as I will post very soon I won't comment by now.

It even supports hybrid streams like Star Trek TNG and Babylon 5, because it chooses based on statistics the correct procedure between Decimate(mode=1) and Decimate(mode=3).

Well good to ear you achieved it to work, I didn't find on this at the moment, I still hope we can merge later in a complete achieved tool.

I've already coded the CD number decision today, but haven't tested it. Still missing is the resize code.

Why don't you use my AviSynth function, it's good enough ??

Last but not least, after everything is done we'll try to make the batch file compatible with older versions of Windows.

Looks like you ahd headaches with delayed expansion too

[bilu]

Originally posted by TaZ4hvn
Why don't you use my AviSynth function, it's good enough ??

I was talking about resizing/compressibility function. In the meantime I've finished it and will test it for the first time

About resizing functions, I want to have 3 different methods:

1) Crop All - all bitrate goes to movie, none to borders. For those with players that change AR.

2) "PerfectCropResize" - your function . It gets AR info from Stream_Information.txt file created by DVD Decryptor, and corrects image based on a PAR table. Uses borders. (I'll try to change it later to add borders at the end of processing, to make script faster and easier to compress)

3) Anamorphic - whenever possible stretch movie height by 33% to reduce border and improve image on widescreen TVs. If you have on of those, it's a must. Frame will have less borders, which means more bitrate for movie.

Looks like you ahd headaches with delayed expansion too

If you're speaking about CMD environment variable delayed expansion, I'm not using it.
What I really meant was to increase the universe of potential users of our tools!


Bilu

[TaZ4hvn]

If you're speaking about CMD environment variable delayed expansion, I'm not using it.

Damn you're stronger than me ... I'de like to avoid 'em but didn't find a way yet ... if you find a way to achieve delayed expansion without those damn ! ... let me know

Anamorphic - whenever possible stretch movie height by 33% to reduce border and improve image on widescreen TVs. If you have on of those, it's a must. Frame will have less borders, which means more bitrate for movie.

One time again I won't bother with this as I only play divx on computer screen ... you're on your own good luck !

I'm currently adding a greater user error control by overriding registry settings for progs each time it's possible
Already did it for DVDdecrypter and XviD.
I think it's a must have since I won't bother users with pages of "configure this that way ...".
It's also more user friendly.

[TaZ4hvn]

Well finally I post a first version:
http://forum.doom9.org/showthread.php?s=&threadid=53043

[bilu]

A new version will come soon.

I've redesigned the way R4R_ENC worked, putting everything into functions and using them with every batch variable needed as parameter.
This way the biggest part of the process is within AVS, which makes it much easier to debug and avoid a lot of batch parsing problems.

The only thing missing now are two new resizing modes: Anamorphic and my version of TaZ's "PerfectCropResize". And the documentation, of course! Then I can add the rest of the features that TaZ is using on his tool, TaZit.

The automatic resizing/compressibility decision is finished and this version is even able to generate logfiles!
It's prepared to deal with more than one resizer, but I'm only using NeutralBicubic because that's the way I like it

A little preview (30 min 4/3 PAL episode of Fawlty Towers):

For a range 60%-75% compressibility.

14-05-2003 - 04:47 : Check for subtitles.
14-05-2003 - 04:47 : Setting plugins and movie variables.
14-05-2003 - 04:47 : Deciding number of CDs needed.
14-05-2003 - 04:49 : Nr. of CDs needed = 0.500000
14-05-2003 - 04:49 : Compressibility tests (one percent).
14-05-2003 - 04:49 : Testing at 150 thousand pixels.
14-05-2003 - 04:49 : Compressibility = 85.469116 percent.
14-05-2003 - 04:51 : Testing at 185 thousand pixels.
14-05-2003 - 04:51 : Compressibility = 72.779419 percent.
14-05-2003 - 04:51 : Compressibility test results (one percent) = 185 thousand pixels.
14-05-2003 - 04:51 : Compressibility tests (five percent).
14-05-2003 - 04:51 : Determining difference between one and five percent comp.tests ...
14-05-2003 - 05:03 : Difference between 1 and 5 percent comp.tests = 2.357756 percent.
14-05-2003 - 05:03 : Testing at 185 thousand pixels.
14-05-2003 - 05:03 : Compressibility = 71.063408 percent.
14-05-2003 - 05:03 : Compressibility test results (five percent) = 185 thousand pixels.
14-05-2003 - 05:03 : Encoding at 185 thousand pixels.


Stay tuned, I hope to finish everything this week. You won't be disappointed

NOTE:About "Difference between 1 and 5 percent comp.tests = 2.357756 percent.", formula is abs(Comp5size - 5*Comp1size)/Comp5size, and not abs(Comp1test - Comp5test).

Best regards,
Bilu

[TaZ4hvn]

Well, looks like you're on the way to finalize your tool good job !
I'd like to know how you generate a logfile, since i'd like to add it to my tool.
I also implemented a new resize/filters decision algo, dunno how it compares to yours, could be instructive to compare them.
All my best wishes to your project

[bilu]

R4R_ENC 2.0 (it deserves the version number )

Completely redesigned, the scripts made by batch files now call functions and include all the parameters needed.Got easier to debug and write code.

Also finished my CD fitting algorithm. It's pretty flexible, have a look in the code

http://pagpessoais.iol.pt/bruno_henriques/r4r_enc2.zip


Best regards,
Bilu

[TaZ4hvn]

Well I get it right now man !
I'm pretty interrested to see what you did on the resizing decision part to compare to mine

[DaveEL]

avs2avi 1.29 out should work with xvid correctly now.

http://daveel.leffe.dnsalias.com/avs2avi-129.zip

DaveEL

[bilu]

Everybody please update to the last AVS2AVI version, it has a new switch that is not important to this tool but also fixes a problem with decoding, since 1.26 only decoded with FFDshow.

R4R_ENC 2.0a should be released in a few hours, it fixes a lot of bugs from the 2.0 version, has corrected Anamorphic and Unamorphic (undoes Anamorphic) processes, and as flags to use subs just if you want (no errors in SWITCH like in pre-2.0 versions) and a flag for borders cropping ( you can use Anamorphic or Unamorphic and then crop borders if you want ).


Best regards,
Bilu

[bilu]

Here it is, and so far seems to work really good

Please post feedback about the needs for documentation, and bugs that may arise.

2 repeated warnings:

1) Use latest AVS2AVI;

2) My automatic deinterlacing function is generic and not perfect.
It's based on framerates and MPEG-2 flags that are not allways encoded correctly, or encoded at all.

Hardware telecined movies are one of this examples: when framerate= 29.97 fps I choose between Decimate(mode=1) and (mode=3) based on 3:2 pattern counting made by ParseD2V.

An hard-telecined has VIDEO flags in the D2V, so the duplicated fields generated by the telecine process will be replaced by blended frames, which is not as good as the correct Inverse Telecine process.
I won't say it is bad (yet) because I haven't tested this side effect.


It should work correctly over PAL movies and correctly flagged NTSC video and film streams.


Grab it at
http://pagpessoais.iol.pt/bruno_henriques/r4r_enc2a.zip

[EDIT] Grab it now, corrected ZIP file. And don't forget to run first the run1st.reg file!

[EDIT2] I won't make a standalone tool, that's TaZ project.
Mainly because I enjoy having a job queue.

I would like add VobSub Cutter to my project, but I would need to be able to get duration of each movie split and make time calculations to use with VobSub Cutter on command line. This would enable automated subtitle ripping for DirectVobSub playing!

Won't use KEnforcer and chapter info because TaZ has already tried and it conflicts with b-frames.


Best regards,
Bilu

[TaZ4hvn]

2) My automatic deinterlacing function is generic and not perfect.

On the few test I just did it works far better than the previous one !
I'm waiting for you to find the best solution here so that I can steal it to you

I won't make a standalone tool, that's TaZ project. Mainly because I enjoy having a job queue.

To be true, queue is on my mind too

Won't use KEnforcer and chapter info because TaZ has already tried and it conflicts with b-frames.

At the moment, pal ... anyhow I provide a flag to disable KFenforcer to use Bframes. But I really don't think Bframes are that much usefull for 'normal' bitrates.

Also did you give a try to my new resizing/filtering algo ?
Caus' I think we both suffer of the same problem concerning comptest, i.e. that artificial keyframes are inserted at each snap start leading to slightly oversized comptest.
I did some 95% comptest and comp. was still under the real one, I'd like to find a CLI tool able to discard this artificials Iframes in calculation but didn't find one yet
I can post links to related threads I read about this if you're interested ...

Regards.

[jonny]

Oversized frames are not only at the start of each snip (but at the end too if you use bframes... not discarding this frames can lead in an error up to 20%-25%).
With DivX things are easy (the bframes implementation is simple).
For XviD+bframes i use a strange method, i actually discard 3 frames at the start and 3 frames at the end of each snip, this is the Enc's "extended method"... and seems to reduce the error, but in some case the error is still too high... 7%-8% ... consider that in a normal case you obtain an error that range from 1% to 3% (soon i'll start experimenting a new method that should make miracles ^^).

Anyway parsing the log file generated by VDubMod or AVS2AVI it's essential for making the comp.test work correctly.

Do you need a little exe that parse this log and output the average frame size (discarding the correct frames)?
(drop me a PM if this can be enough)

[bilu]

Do you need a little exe that parse this log and output the average frame size (discarding the correct frames)?
(drop me a PM if this can be enough)

I did

On the few test I just did it works far better than the previous one !
I'm waiting for you to find the best solution here so that I can steal it to you

The previous version was really bad because it halved the vertical resolution. This one seems OK for everything except incorrectly flagged NTSC DVDs. I know that the side effect on this DVDs will be worst than doing a correct IVTC process, but I haven't tested yet to know how bad it is. Still there are no alternatives at the moment... except manual encoding with "eye-analisis"


Best regards,
Bilu