Automatic Moving Crop (4:3 -> 16:9)

[FranceBB]

Hi everyone,
I have to deal with an old 4:3 DVD and I would like to crop it into 16:9 (please, don't kill me for this) and restore it.

The source is interlaced and with a lot of crap such as raimbow/dot crawl, a lot of static grain and other stuff, but I can get rid of everything.

Anyway, I'm here just to ask you about this crop because I remember that, time ago, when we used to watch things on our 4:3 TVs, sometimes the pan & scan function was used in order to do a moving cropping.

Silly example:

Amorphic source
http://www.animemusicvideos.org/guides/avtech/images/amamorphic-orig.jpg

16:9
http://www.animemusicvideos.org/guides/avtech/images/anamorphic-169.jpg

4:3 with PAN & SCAN
http://www.animemusicvideos.org/guides/avtech/images/anamorphic-pan.jpg

So, I was looking for a filter that is able to perform a PAN & SCAN automatically; "Animate" is a good one, but I have to specify the starting and ending frames.
Is there a way to make this process automatic?
If not, is there an editing software to use as replacement to avisynth? (so I can create a lossless with that software and then filter everything with avisynth and x264).

Thank you ^_^

[raffriff42]

KenBurnsEffect
http://forum.doom9.org/showthread.php?t=135776

[manolito]

Hi FranceBB,

do I understand you right: You want to convert a 4:3 source to 16:9 without pillar boxes (black borders left and right)? And you know that you will crop off information from the top and the bottom of your source?

If so then it is much simpler than using the Ken Burns Effect. Here is an old AviSynth script which I used a lot in the good old 4:3 days:

4-3to16-9.avs
# Requires stickboy's jdl-util.avsi and jdl-interlace.avsi
# Encoder Aspect Ratio must be set to 16:9 Anamorphic

# Interlaced = true
DestinationIsPAL = (DestinationHeight % 288 == 0) || (DestinationHeight % 216 == 0)
DestinationIsPAL ? Crop(0,72,-0,-72) : Crop(0,60,-0,-60)
Interlaced==true ? (DestinationIsPAL ? JDL_InterlacedResize(DestinationWidth,576,"Spline36Resize") : JDL_InterlacedResize(DestinationWidth,480,"Spline36Resize")) : (DestinationIsPAL ? Spline36Resize(DestinationWidth,576) : Spline36Resize(DestinationWidth,480))


For interlaced sources it requires Stickboy's interlaced tools. It will handle PAL and NTSC, and remember that the encoder must be set to 16:9 anamorphic. And of course the crop and resize values are for DVD only...


Cheers
manolito

[FranceBB]

@manolito... yep, I want to crop, but "moving" the "camera" (the crop) in order to avoid cutting important information.

@raffriff42... I'll try KenBurnsEffect tomorrow. Thank you.

[raffriff42]

I can't say I completely understand KenBurnsEffect, as I've only used it for the odd zoom. It's very powerful. But to be honest, your task would be a lot easier in Premiere or Vegas.

[manolito]

@manolito... yep, I want to crop, but "moving" the "camera" (the crop) in order to avoid cutting important information

Alright, but there is no way to automate this task. How can a software tool know where the important information is located?

I remember when the 16:9 format was already established, but most content was still broadcast at 4:3. At that time the 4:3 productions made sure that they were 16:9 safe. Which meant that no important information was in the top or bottom area which would eventually be matted or cropped.


Cheers
manolito

[FranceBB]

@manolito... I thought there was some built-in facial recognition or something like that. Anyway, some scenes of the DVD are in 16:9, which means that there are black borders up and down.
Another problem is made by the credits that are made at the very top or at the very bottom, so I need a precise crop etc.
It would be helpful if the software is able to recognize at least the black borders and the credits in order to avoid cutting stuff like the title etc.
As to the rest, I can set the crop values myself, it's not really a big deal since there shouldn't be a lot of "dangerous" scenes ^_^

[manono]

I thought there was some built-in facial recognition or something like that.
When the silly pan-and-scan versions of films are created, the scanner requires very heavy human intervention. There is no automatic way to do what you want.

https://en.wikipedia.org/wiki/Pan_and_scan

[hello_hello]

I've panned and scanned 4:3 to 16:9 a few times, and to do it properly it can be a lot of work, especially using Avisynth.
If you crop a little less from the top than the bottom you might get lucky and it'll be acceptable (often the focus tends to be mostly centred a little above the middle of the frame), but if you really want to do it properly, there'll probably be a lot of adjusting.

I usually crop to 16:9 dimensions and resize to square pixels. Occasionally, there might be a single shot where the focus starts at the top of the frame and finishes at the bottom etc, in which case you might need to find a way to zoom or pan down, but for live action stuff I found 99% of the time changing the cropping on a scene change is usually all that's needed to keep the important stuff in the 16:9 frame. Like this (that particular script was concert footage where almost every scene change needed a cropping adjustment, so it was time consuming):

trim(0, 3367).crop(16, 2, -16, -10).Spline36Resize(576,432).AddBorders(96, 0, 96, 0)\
++ trim(3368, 3487).crop(12, 74, -12, -74).Spline36Resize(768,432)\
++ trim(3488, 3973).crop(12, 104, -12, -44).Spline36Resize(768,432)\
++ trim(3974, 4223).crop(12, 14, -12, -134).Spline36Resize(768,432)\
++ trim(4224, 4353).crop(12, 104, -12, -44).Spline36Resize(768,432)\

The 4:3 videos I've panned and scanned so far had distinct opening and closing sequences containing the credits, but they couldn't be cropped without cropping the text, so rather than stretch to 16:9, I left the credits as 4:3 and added pillarboxing (as per the above example). A sudden switch to 16:9 when the episode starts adds a bit of effect. ;)

There might be an easier way to add the panning and scanning to scripts (I just kept copying and pasting and changing the frame numbers and cropping as I went), but I can't imagine there'd be a substitute for doing it manually. One downside might be due to the video qiality. 4:3 DVD isn't always that great, and by cropping so it'll fill a 16:9 screen you're effectively zooming in, so it might look like the quality has dropped.

[raffriff42]

It's always gonna be complicated trying to do this in Avisynth; usually it's better to use some good editing software. But sometimes it isn't better, so here is a demo of animated "pan & scan."

https://www.dropbox.com/s/yffey775ukwvir8/Automatic%20Moving%20Crop%20pan-n-scan-01-CTR.jpg?raw=1
centered

https://www.dropbox.com/s/7915u510gzpowbn/Automatic%20Moving%20Crop%20pan-n-scan-02-PLT.jpg?raw=1
pan left

https://www.dropbox.com/s/biemt7kusfx8pru/Automatic%20Moving%20Crop%20pan-n-scan-03-PRT.jpg?raw=1
pan right

#Last==assumed > 4:3 aspect ratio

## optional: show original frame numbers
ShowFrameNumber(x=Width/2, y=Height/2, scroll=false)

PLT = -1.0 ## pan full left
PRT = 1.0 ## pan full right
CTR = 0.0 ## center

## cropping to 4:3
hgt = Height
wid = Round(Float(hgt) * 4.0 / 3.0)
wid = wid - (wid % 2)

info = true ## set to true to show debugging info
curfrm = 0

## start center, pan left
R = Trim(0, 329).Animate(
\ 300, [* first frame for motion *]
\ 329, [* last frame for motion *]
\ "PanScan",
\ CTR, wid, hgt, info, [* pan <= first frame *]
\ PLT, wid, hgt, info) [* pan > last frame *]
\ .clip_id("clip A", info)
curfrm = curfrm+330

## start left, pan right
R = R++Trim(curfrm, -120).Animate(
\ 60, [* first frame for motion *]
\ 120, [* last frame for motion *]
\ "PanScan",
\ PLT, wid, hgt, info, [* pan <= first frame *]
\ PRT, wid, hgt, info) [* pan > last frame *]
\ .clip_id("clip B", info)
curfrm = curfrm+120

## start right, pan center
R = R++Trim(curfrm, -240).Animate(
\ 60, [* first frame for motion *]
\ 90, [* last frame for motion *]
\ "PanScan",
\ PRT, wid, hgt, info, [* pan <= first frame *]
\ CTR, wid, hgt, info) [* pan > last frame *]
\ .clip_id("clip C", info)
curfrm = curfrm+240

## cut to pan left
R = R++Trim(curfrm, -120)
\ .PanScan(PLT, wid, hgt, info)
\ .clip_id("clip D", info)
curfrm = curfrm+120

## cut to pan 3/4 right
R = R++Trim(curfrm, -60)
\ .PanScan(PRT*0.75, wid, hgt, info)
\ .clip_id("clip E", info)
curfrm = curfrm+60

## cut to pan center
R = R++Trim(curfrm, 0)
\ .PanScan(CTR, wid, hgt, info)
\ .clip_id("clip F", info)

return R

###############################
function clip_id(clip C, string s, bool info)
{
return (info)
\ ? C.Subtitle(s, align=7, size=C.Height/24)
\ : C
}

###############################
function PanScan(clip C, float pan, int wid, int hgt, bool info)
{
C = C.UUCrop2(wid, hgt, pan, 0.0)

return (info)
\ ? C.Subtitle(
\ "pan="+String(pan, "%0.2f"),
\ x=C.Width/2, y=0, size=C.Height/24)
\ : C
}

##################################
### crop to size with smooth animated pan & tilt
##
## @ pan -1=left, 0=center, +1=right; [default 0]
## @ tilt -1=up, 0=center, +1=down; [default 0]
##
function UUCrop2(clip C, int wid, int hgt, float "pan", float "tilt")
{
bdrWid = (wid - C.Width)
bdrHgt = (hgt - C.Height)

xctr = Min(Max(-1.0, Default(pan, 0.0)), 1.0)
yctr = Min(Max(-1.0, Default(tilt, 0.0)), 1.0)

bdrLt = Abs(Float(bdrWid) / 2.0)
bdrLt = bdrLt + bdrLt * xctr
bdrLt = Sign(bdrWid) * bdrLt

bdrTp = Abs(Float(bdrHgt) / 2.0)
bdrTp = bdrTp + bdrTp * yctr
bdrTp = Sign(bdrHgt) * bdrTp

(bdrLt<=0 && bdrTp<=0) ? Nop
\ : Assert(false, "UUCrop2: clip too small for target size")

return C.Spline36Resize(wid, hgt, -bdrLt, -bdrTp, wid, hgt)
}

[FranceBB]

Thank you all guys, really.
I'm actually doing it in two different ways: the proper one first and a light cut plus panorama tool as the second one.

I'm going to compare 'em this afternoon and I'll decide how I will proceed.
Thank you everyone, really. ^_^

[jmac698]

I did the inverse and used some tools to find the pan box relative to the 2.35:1 version of the film. It moved around a lot. It's really a director's choice, with some thought on artistic decisions. If there's two talking heads, do you show one, both but parts of faces, mostly one, or quick cut between them?

[burattino]

It's always gonna be complicated trying to do this in Avisynth; usually it's better to use some good editing software. But sometimes it isn't better, so here is a demo of animated "pan & scan..."

Hello, I know this thread is quite old, but I'm desperately looking for tilt only function in 4:3 -> 16:9 conversion.

I have an old video. It was converted to FHD with sidebars.
Frame size property without bars is: 1480x1080
I was doing fine with trim and crop technique until frame #9848

In there, one of two people got up to make a toast
The second man remains sited. It's all filmed with
extreme close-up, but at the same time they are both visible in the 4:3 frame
But for 16:9 transfer i need to tilt smoothly 'camera' up if only i could...




test.avs
Input.avi
Import("pan_&_scan.avsi")


a=trim(9042, 9847).crop(220, 188, -220, -60)#.UUCrop2(1480, 832, 0, -1)
b=trim(9848, 11042).crop(220, 60, -220, -188)#.UUCrop2(1480, 832, 0, -1)
#b=trim(9848, 11042).crop(220, 0, -220, 0).UUCrop2(1480, 832, 0, -1)
c=trim(11043, 15042).crop(220, 188, -220, -60)#.UUCrop2(1480, 832, 0, -1)

return (a+b+c)



raffriff42,
I was trying UUCrop2 function from your "pan_&_scan demo"
but could not make it to work. Maybe you can help.
Thank you