Dynamic cropping?

[creaothceann]

I noticed that on an anime DVD, different amounts of black border were added per scene...

frame 03648
frame 04529
frame 05292
frame 09629
frame 09630
frame 40219
frame 40338
(see left border)

I can of course crop only the minimum amount per episode and live with the remaining black borders - going through the scenes and manually adjusting the cropping would be too much work. But is there an automatic filter that could do that?

[Overdrive80]

For example:

#crop&resize (0,0,-0,-0)

	#Opening
	D0=Trim(0,2607).Crop(4,0,-2,-0).Spline64Resize(720,480)

	#Resumen
	D1=trim(2608,4933).Crop(2,0,-2,-0).Spline64Resize(720,480)

	#Titulo
	D2=trim(4934,5137).Crop(2,0,-2,-0).Spline64Resize(720,480)

	#Capitulo - 1 parte
	D3=trim(5138,18199).Crop(2,0,-2,-0).Spline64Resize(720,480)
	
	#Cortinillas
	D4=trim(18200,18483).Crop(2,0,-2,-0).Spline64Resize(720,480)

	#Capitulo - 2 parte
	D5=trim(18484,32407).Crop(2,0,-2,-0).Spline64Resize(720,480)

	#Ending
	D6=trim(32408,34687).Crop(2,0,-2,-0).Spline64Resize(720,480)

	#Avance
	D7=trim(34688,0).Crop(4,0,-2,-0).Spline64Resize(720,480)
	
	UnalignedSplice(D0,D1,D2,D3,D4,D5,D6,D7)

[creaothceann]

Yeah... but doing that for ~585 minutes of material?

Originally Posted by creaothceann

[...] going through the scenes and manually adjusting the cropping would be too much work.

[Overdrive80]

Not exists function as you want for this reason I answered with that code. Bye.

[StainlessS]

If you can supply the trims only, then perhaps this might help:

Function CropResize(clip c,int "W",Int "H",int "Samples",float "Ignore",Float "Thresh", int "Matrix",bool "DEBUG", \
        int "CropMode", String "Resizer") {
    c
    Frames=FrameCount
    W=Default(W,width)                                  # Defaults to original clip width
    H=Default(H,height)                                 # Defaults to original clip height
    Samples=(!Defined(Samples)) ? 24 : (Samples > Frames) ? Frames : (Samples < 1) ? 1 : Samples    # See RT_QueryBorderCrop
    Ignore=Float(Default(Ignore,0.2))                   # See RT_QueryBorderCrop
    Thresh=Float(Default(Thresh,-32.0))                 # See RT_QueryBorderCrop
    Matrix = Default(Matrix,(width <= 720) ? 2 : 3)     # RGB Only  # See RT_QueryBorderCrop
    DEBUG=Default(DEBUG,False)                          # See RT_QueryBorderCrop
    CropMode=Default(CropMode,2)                        # Default = CropMore, See RT_QueryBorderCrop
    Resizer=Default(Resizer,"Spline64Resize(_W_,_H_)")  # Resizer to use, _W_ and _H_ replaced with W and H args
    cmmin=(IsRGB()?0:1)                                 # CropMode=0(CropExact) Only valid for RGB.
    Assert(CropMode>=cmmin && CropMode<=3,"CropResize: Invalid CropMode")
    QBCTrailer=Select(CROPMODE,"=","L=","M=","P=")
    QBC=RT_QueryBorderCrop(samples=Samples,thresh=Thresh,debug=DEBUG,ignore=Ignore,laced=False,matrix=Matrix)
    QBC=RT_TxtGetLine(QBC,CROPMODE)                     # Extract REQUIRED mode, CropLess, CropMore etc
    QBC=RT_StrReplace(QBC,QBCTrailer,"=")               # Change all to style without a trailer eg QBCropXM to QBCropX.
    Eval(QBC)                                           # Set QBCropX,Y,W,H variables
    Crop(QBCropX,QBCropY,QBCropW,QBCropH)
    FindStr="_W_" + Chr(10) + "_H_" + Chr(10)
    RepStr = "W"  + Chr(10) +  "H"  + Chr(10)
    RszCmd=RT_StrReplaceMulti(Resizer,FindStr,RepStr)   # Replace _W_ and _H_ with W and H
    Eval(RszCmd)
    return Last
}

Test Code:

Avisource("D:\test.avi")

W=720 H=576

A=Trim(0000,0999).Addborders(8 ,4 ,8 ,12).CropResize(W,H)
B=Trim(1000,1999).Addborders(12,4 ,4 ,12).CropResize(W,H)
C=Trim(2000,2999).Addborders(4 ,4 ,12,12).CropResize(W,H)
D=Trim(3000,3999).Addborders(16,0 ,0 ,16).CropResize(W,H)
E=Trim(4000,4999).Addborders(4 ,8 ,12,8 ).CropResize(W,H)

A++B++C++D++E

Addborders just to test out. Not had much testing at all, just done it now.

Suggest supply Laced=false as arg (makes default Planar HMOD in RT_QueryBorderCrop 2[Progressive] instead of 4).
EDIT: Just hack the default for Laced to false, easiest.

Needs RT_Stats, suggest v1.13, just upped.

EDITED: Made W and H optional defaulting to supplied clip c w/h.
(Can omit W,H above as test addborders all add up to 16, of course return w,h will be 16 greater than original source clip)
Removed Laced arg, as resizing, it should NOT be interlaced, thats another script, crop HMOD will default 2 for planar.
Most of the defaults in the function are there mainly for documentation, and would be defaulted to same values in
RT_QueryBorderCrop().
EDIT Of course Addborders() should not be used in real app.

[zerowalker]

It should be possible to detect the black borders and remove them.
The problem is that if it does it dynamically, it won´t work, as the resolution will differ.

But if you want to resize it should work.

For example

If blackborder = 12 pixels on left, remove them.
size = 708x480.
Resize to 720x480.

Something like that, it removes the black borders, them resize it to the correct resolution.
Though that will cause distortions to the aspect ratio intended.

[StainlessS]

The simplest solution is of course to just crop the maximum, why sweat for 6 pixels which are sometimes there and sometimes not.

[lisztfr9]

I thought that your video may jerk horizontally, - unless you pad it. Btw, remember "border control" in VD (and Avisynth)

Edit :

http://www.geocities.com/siwalters_uk/bdrcntrl.html

[StainlessS]

Thank you Mr Liszfr9, never actually tried BorderControl on either VD or AVS.

Looking up BC, I also found FillMargins (http://forum.doom9.org/showthread.php?t=50132)

which may or may not be better suited to the job.

[creaothceann]

StainlessS: Thank you, that's very helpful.

I guess a fully automated filter would look for scene changes and detect the borders after each one, so I'll search for that (unless there's a generally recommended function).

[StainlessS]

I seem to recall that ffmpeg has some kind of border detector (perhaps x264 also), take a peek.
(I really dont use anything more complicated than MeGUI).
Yes, ffmpeg does some frame by frame cropping, try google.

Maybe post in "MPEG-4 Encoder GUIs" forum, better for such help available there.

EDIT: ffmpeg crop:
https://ffmpeg.org/ffmpeg-filters.html#crop
Not now sure if it can produce required cropping, best ask in above forum.

[smok3]

Generaly speaking, what i would do;

1.
a. scene cut everything to clips
b. write something ffmpeg related that would do a border detection for each new clip
c. process everything, crop & scale
d. compare with originals just to see whats happening with different amounts of scaling < this may also turn out bad.
e. put everything back ...

or

2. Do a max crop + scale on enire clip.

decisions, decisions ...

[lisztfr9]

Does your video jerk ? That's the main point, if it does, use DepanStabilize this will dynamically center your video, if it doesn't, don't dynamic crop it because it will jerk even more.

[StainlessS]

creaothceann,

Can you give the below script a try out on your clip (Needs RT_Stats v1.13),
(only outputs list of croppings, sister func could do cropping based on output file)

Avisource("D:\test.avi").KillAudio()

TRIMSZ=1000     # Size of trims for BorderTorture
CTHRESH=0.666   # YCorrelation Threshold for JERK detection Alert
                # BorderTorture will probably create ALL Alert trims (border change NOT at scene change, I presume).

Function BorderTorture(clip c,String "fn",int "sz") {
    c
    myName="BorderTorture:"
    fn=Default(fn,"BorderTorture.txt")  RT_FileDelete(fn)
    sz=Default(sz,100)
    WW=Width HH=Height Frames=FrameCount
    nTrims=(Frames+sz-1)/sz     c2=0 oldcx=0 oldcy=0 oldcw=0 oldch=0
    wr_s="# BorderTorture: ("+String(WW)+"x"+String(HH)+") Creating output clip "+String(WW+8)+"x"+String(HH+8)+" Scene Start End X Y W H"
    RT_TxtWriteFile(wr_s,fn,append=true)    RT_Debug(myName,wr_s,false)
    GSCript("""
        For(n=0,nTrims-1) {
            b=true
            while(b || (cx==oldcx && cy==oldcy && cw==oldcw && ch==oldch)) {    # Make sure some change at every trim
                cx=Rand(4+1)*2  cy=Rand(4+1)*2  cw=Rand(4+1)*2  ch=Rand(4+1)*2
                b=false
            }
            e = ((n+1)*sz-1>=Frames) ? Frames-1 : (n+1)*sz-1
            wr_s = \
                RT_StrPad(String(n+1)+")",5)                                + " " + \
                RT_StrPad(String(n*sz),7)                                   + " " + \
                RT_StrPad(String(e),7)                                      + " " + \
                RT_StrPad(String(cx+4),4)                                   + " " + \
                RT_StrPad(String(cy+4),4)                                   + " " + \
                RT_StrPad(String(WW-cx-cw)+"("+String(-(cw+4))+")",11)  + " " + \
                RT_StrPad(String(HH-cy-ch)+"("+String(-(ch+4))+")",11)
            RT_Debug(myName,wr_s,false)
            RT_TxtWriteFile(wr_s,fn,append=true)
            t=Trim(n*sz,-sz).Crop(cx,cy,-cw,-ch).AddBorders(cx+4,cy+4,cw+4,ch+4)
            c2 = (c2.isClip) ? c2++t : t
            old_cx=cx oldcy=cy oldcw=cw oldch=ch
        }
    """)
    return c2
}

# Below Artificial Test Only, comment out
 BorderTorture(sz=TRIMSZ)
# End test

Function BorderDetect(clip c,String "fn",float "CThresh") {
    c
    myName="BorderDetect:"
    fn=Default(fn,"BorderDetect.txt")   RT_FileDelete(fn)
    CThresh=Float(Default(CThresh,0.666))
    WW=width    HH=height       Frames=FrameCount
    Samples = 256
    Samples=(Samples > Frames) ? Frames : Samples
    Eval(RT_QueryBorderCrop(samples=Samples,Prefix=""))                             # Local Vars X,Y,W,H
    Assert(X>0 || Y>0 || W<WW || H<HH,myName+" No border found")
    Assert(CThresh>=0.0 && CThresh<=1.0,myName+" CThresh 0.0 -> 1.0 ("+String(CThresh,"%6.2f")+")")
    RT_Debug(myName,"Width="+String(WW),"Height="+String(HH))
    RT_Debug(myName,"Common Border: ",String(X),String(Y),String(W),String(H))
    scene = 0   nAlert = 0
    GSCript("""
        if(Frames >= 250 && Int(Frames*0.85)>Samples) {         # Try not to use artificial black credits
            FrmStart    = Int(Frames * 0.05 + 0.5)              # Skip 5% BLACK intro
            FrmEnd      = Int(Frames * 0.90 + 0.5) - 1          # Skip 10% BLACK End Credits
        } else {
            FrmStart    = 0
            FrmEnd      = Frames-1
        }
        BRDMX = -1  # Find max luma pixel value in common border scanline nearest to image
        for(Samp = 1,Samples) {
            n = int(Samp * (Float(FrmEnd - FrmStart) / (Samples+1)) + 0.5) + FrmStart
            if(Y>0)     {tmp=RT_YPlaneMax(n,x=X,y=Y-1,w=W,h=1,threshold=0.0) BRDMX=Max(BRDMX,tmp)}
            if(X>0)     {tmp=RT_YPlaneMax(n,x=X-1,y=Y,w=1,h=H,threshold=0.0) BRDMX=Max(BRDMX,tmp)}
            if(Y+H<HH)  {tmp=RT_YPlaneMax(n,x=X,y=Y+H,w=W,h=1,threshold=0.0) BRDMX=Max(BRDMX,tmp)}
            if(X+W<WW)  {tmp=RT_YPlaneMax(n,x=X+W,y=Y,w=1,h=H,threshold=0.0) BRDMX=Max(BRDMX,tmp)}
        }
        RT_Debug(myName,"Luma Max in Common Border=",String(BRDMX))
        CUR_X=X CUR_Y=Y CUR_W=W CUR_H=H     # in case below fail (black frame)
        if(RT_YInRangeLocate(n=0,x=X,y=Y,w=W,h=H,Baffle=2,Thresh=0.0,Lo=BRDMX+1,Hi=255,Prefix="BDRL_")) {
            CUR_X=BDRL_X    CUR_Y=BDRL_Y    CUR_W=BDRL_W    CUR_H=BDRL_H    # Init current border
        }
        pe= -1  out=false   alert=false # Init
        wr_s="# BorderDetect: Scene Start End X Y W H Corr" RT_TxtWriteFile(wr_s,fn,append=true)    RT_Debug(myName,wr_s,false)
        For(n=0,Frames) {
            if(n<Frames) {
                b = RT_YInRangeLocate(n=n,x=X,y=Y,w=W,h=H,Baffle=2,Thresh=0.0,Lo=BRDMX+1,Hi=255,Prefix="BDRL_",debug=False)
                if(b) { # we ignore black frames, keep in current crop trim
#                   RT_Debug(myName,"Got ",String(BDRL_X),String(BDRL_Y),String(BDRL_W),String(BDRL_H),false)
#                   RT_Debug(myName,"Cur ",String(CUR_X),String(CUR_Y),String(CUR_W),String(CUR_H),false)
                    if(CUR_X!=BDRL_X || CUR_Y!=BDRL_Y || CUR_W!=BDRL_W || CUR_H!=BDRL_H) { # Cropping Changed
#                       RT_Debug(myName,"Changed",false)
                        XT=Max(CUR_X,BDRL_X)    WT=Min(CUR_X+CUR_W,BDRL_X+BDRL_W)-XT    # Find interesction of old/new images
                        YT=Max(CUR_Y,BDRL_Y)    HT=Min(CUR_Y+CUR_H,BDRL_Y+BDRL_H)-YT
                        ycorr=RT_LumaCorrelation(Last,n=n,n2=n-1,x=XT,y=YT,w=WT,h=HT)
                        if(ycorr > CThresh) {
                            Alert=true
                        }
                        Out=true
                    }
                }
            } else {
                out=True
                ycorr=0.0
            }
            if(out) {
                scene = scene + 1
                wr_s= \
                    RT_StrPad(String(Scene)+")",5)                              + " " + \
                    RT_StrPad(String(pe+1),7)                                   + " " + \
                    RT_StrPad(String(n-1),7)                                    + " " + \
                    RT_StrPad(String(CUR_X),4)                                  + " " + \
                    RT_StrPad(String(CUR_Y),4)                                  + " " + \
                    RT_StrPad(String(CUR_W)+"("+String(CUR_X+CUR_W-WW)+")",11)  + " " + \
                    RT_StrPad(String(CUR_H)+"("+String(CUR_Y+CUR_H-HH)+")",11)  + " " + \
                    RT_StrPad(String(ycorr,"%6.4f"),8)                      + " " + \
                    (alert?"# ALERT: Check Following Trim" : "")
                RT_Debug(myName,wr_s,false)             
                RT_TxtWriteFile(wr_s,fn,append=true)
                pe = n-1
                CUR_X=BDRL_X    CUR_Y=BDRL_Y    CUR_W=BDRL_W    CUR_H=BDRL_H # New Current Cropping
                nAlert = (alert) ? nAlert+1 : nAlert
                out=false   alert = false
            }
        }
    """)
    RT_Debug(myName,"Scene="+String(Scene),"Alert="+String(nAlert),false)
    return nAlert
}

BorderDetect(cthresh=CTHRESH)
return Last

Output on my Artificial test here:

00000004    16:37:17    BorderTorture: # BorderTorture: (640x400) Creating output clip 648x408 Scene Start End X Y W H  
00000005    16:37:17    BorderTorture: 1)    0       999     6    8    630(-12)    396(-4)      
00000006    16:37:17    BorderTorture: 2)    1000    1999    12   12   626(-10)    386(-10)     
00000007    16:37:17    BorderTorture: 3)    2000    2999    8    12   636(-4)     392(-4)      
00000008    16:37:17    BorderTorture: 4)    3000    3999    6    8    636(-6)     394(-6)      
00000009    16:37:17    BorderTorture: 5)    4000    4999    4    8    636(-8)     394(-6)      
00000010    16:37:17    BorderTorture: 6)    5000    5822    6    12   634(-8)     386(-10)     
00000011    16:37:20    RT_Debug: BorderDetect: Width=648 Height=408    
00000012    16:37:20    RT_Debug: BorderDetect: Common Border: 4 8 640 396  
00000013    16:37:23    RT_Debug: BorderDetect: Luma Max in Common Border= 16   
00000014    16:37:23    BorderDetect: # BorderDetect: Scene Start End X Y W H Corr  
00000015    16:37:28    BorderDetect: 1)    0       999     6    8    630(-12)    396(-4)     0.9765   # ALERT: Check Following Trim    
00000016    16:37:33    BorderDetect: 2)    1000    1999    12   12   626(-10)    386(-10)    0.9102   # ALERT: Check Following Trim    
00000017    16:37:38    BorderDetect: 3)    2000    2999    8    12   636(-4)     392(-4)     0.9441   # ALERT: Check Following Trim    
00000018    16:37:43    BorderDetect: 4)    3000    3999    6    8    636(-6)     394(-6)     0.9509   # ALERT: Check Following Trim    
00000019    16:37:48    BorderDetect: 5)    4000    4999    4    8    636(-8)     394(-6)     0.9705   # ALERT: Check Following Trim    
00000020    16:37:52    BorderDetect: 6)    5000    5822    6    12   634(-8)     386(-10)    0.0000    
00000021    16:37:52    BorderDetect: Scene=6 Alert=5

Still quite a way to go till complete, just want to know if I'm wasting my time.
Also, does your clip exhibit jerk as warned by lisztfr9 ?(Borders change within a continuous scene, I presume)

[StainlessS]

Sorry, bit of a phasing problem, detections were correct, output was wrong (off by 1).
Let that be a lesson to you wrong doers, beer and pubs can obfuscate your intentions and befuddle your noggins.

Corrected code in blue. (Wish the pubs didn't shut sooo early, it's still dark outside!)

[StainlessS]

Post #14 updated, with Jerk Alert (No connection Whatsoever with TheFluff ).

[creaothceann]

I will try that, but at the moment I'm having stability problems and not much free time to work on them (that's why I was planning this huge automated encoding session for now!).
My computer seems to just power down for some reason and I can't figure out why. (Memtest finds no error...) Right now I'm installing Windows in VirtualBox; try bringing that down, Avisynth...

Regarding lisztfr9's question: no, the cropping doesn't fluctuate within a scene, it's just as if the producer assigned different interns for each scene and they didn't agree what the exact values for the borders should be... I'll try to upload a sample later.

[StainlessS]

power down for some reason

Check your Event Viewer logs, In Windows Explorer (My Computer) and right click on
"My Computer" and click "Manage", then see logs in "Event Viewer". There are easier ways to
get there but I think that should work whatever your system/setup.

Also possible that you have malware.

[creaothceann]

Sample:
VTS_02_1.demuxed.m2v

I applied the above code to it, here are the results:
BorderDetect.txt
BorderTorture.txt

Originally Posted by StainlessS

Check your Event Viewer logs, In Windows Explorer (My Computer) and right click on "My Computer" and click "Manage", then see logs in "Event Viewer". There are easier ways to get there but I think that should work whatever your system/setup.
Also possible that you have malware.

Event Viewer didn't show any obvious source of error... and my virus scanner is up-to-date. I'll continue to investigate this.

[StainlessS]

Thanks, downloading m2v, but having severe problems with my connection, getting disconnected every few minutes.
The Torture test was really for testing the detector with a good clip, and really for anyone else wanting to try it out.
I'll be knocking up a sister func for the detector, maybe a day or two.

FreedownloadManager is telling me 4hrs 15 min, download time, going to supermarket, suspect will
be disconnected before I reach the end of the street. Damn.