Weird error with crop()?

[eXtremeDevil]

I'm cropping a video with a lot of black borders, and this is my script:

DGDecode_MPEG2Source("r.d2v")
ColorMatrix(interlaced=true, threads=0)
tfm(order=1,pp=7,clip2=nnedi3()).tdecimate()
crop(28,12,-24,-6)
DAA()
Tweak(hue=0.0, sat=1.0, cont=1.0)
HQDN3D(ls=20, cs=0.0, lt=0.0, ct=0.0) #Denoise
LSFMOD(defaults="slow", preblur="OFF") # Sharpening + Prebluring
SmoothD (quant=2, zero_weight=6) #Deblocking
VMToon(strength=130, luma_cap=146, sharpen=false, threshold=4, thinning=100, ssw=3,ssh=3) # Linedarkening
awarpsharp2() # Sharpen
MAA() #Antialiasing
gradfun3() # Debanding


So right now when I try to load it in AvsP the error window appears and give me an infinite error lines.

If I remove the cropping, the video shows just fine. Also if I put the cropping at the end, but I've been told that you're not supposed to do that.

Also, if I leave the cropping there and quit SmoothD and DAA(), the video shows fine too. If I remove only SmoothD a green line appears at the bottom.

Are there any problems with DAA and Smooth? What's happening and how can I fix it?

Thanks.

[Atak_Snajpera]

do you really have to use so many filters? sometimes less is more. deblocking is reduntant after strong denosing.

[gyth]

It might be an alignment issue, try:
crop(28,12,-24,-8, align=true)

[eXtremeDevil]

Tried that, nothing changed :(

Now I can read some errors:

Traceback (most recent call last):
File "AvsP.pyo", line 6291, in OnSliderReleased
File "AvsP.pyo", line 8925, in ShowVideoFrame
File "AvsP.pyo", line 9467, in PaintAVIFrame
File "pyavs.pyo", line 322, in DrawFrame
File "pyavs.pyo", line 301, in _GetFrame
File "avisynth.pyo", line 277, in GetFrame
WindowsError: exception: access violation reading 0x0AA27000
Traceback (most recent call last):
File "AvsP.pyo", line 7123, in OnPaintVideoWindow
File "AvsP.pyo", line 9467, in PaintAVIFrame
File "pyavs.pyo", line 322, in DrawFrame
File "pyavs.pyo", line 301, in _GetFrame
File "avisynth.pyo", line 277, in GetFrame
WindowsError: exception: access violation reading 0x0AA27000

?

[eXtremeDevil]

OK, I missed that 8 there. Why that number? It seems to work now...

EDIT

It only works partially. I don't have any errors now and the green border has gone but now I have some weird shadows that are actually the same video but distorted (is that a word)... I can fix that thing by changing the 28 value to 24, but then I get a black border on the left...

I've noticed, the only thing that has changed is the 6 for the 8, the align value doesn't make a difference, if I put 6 again I get errors and the green line... I don't want to crop more than necesary, and I don't want black borders eigther... what can I do?

[manono]

...what can I do?
You could resize after the crop.

[gyth]

With interlaced YV12 your height crops have to be mod 4.
I think you are starting with interlaced YV12.
You could probably convert to non-interlaced before the crop, but I'm not sure how to do that.

Some of your filters are expecting mod 8 or mod 16 widths.

Are there any problems with DAA and Smooth? What's happening and how can I fix it?
Easiest way to fix it is to just comment out the lines.
It is highly likely you don't need (or want) all those filters.

[eXtremeDevil]

I've try and those filters on and off and they actually make some changes to the video that I like. I've try resizing but the green border still exists. And anyway, I don't want to resize here, but in x264 options with --sar.

Anything else I can try?

EDIT

And why wouldn't be good to put crop at the end, or after SmoothD?

[Guest]

And why wouldn't be good to put crop at the end, or after SmoothD? It's OK to do that. Generally, you crop early for performance reasons; the following filters have smaller frames to work with.

[Didée]

Well, in case of black borders it's also about the transition image-->border. Technically it is detail in the picture, and may affect spatial filters.

Without questioning the filterchain, what about keeping the image mod16 for the processing filters, after cropping?

DGDecode_MPEG2Source("r.d2v")
ColorMatrix(interlaced=true, threads=0)
tfm(order=1,pp=7,clip2=nnedi3()).tdecimate()
crop(28,12,-24,-6,true)
ox=width() oy=height()
xx=16-(ox%16) yy=16-(oy%16)
pointresize(ox+xx,oy+yy, 0,0,ox+xx,oy+yy)
DAA()
Tweak(hue=0.0, sat=1.0, cont=1.0)
HQDN3D(ls=20, cs=0.0, lt=0.0, ct=0.0) #Denoise
LSFMOD(defaults="slow", preblur="OFF") # Sharpening + Prebluring
SmoothD (quant=2, zero_weight=6) #Deblocking
VMToon(strength=130, luma_cap=146, sharpen=false, threshold=4, thinning=100, ssw=3,ssh=3) # Linedarkening
awarpsharp2() # Sharpen
MAA() #Antialiasing
gradfun3() # Debanding
crop(0,0,-xx,-yy)

Anyway: in the given position in the filterchain, SmoothD does not really act as a "deblock" filter, but more generally as a "DCT smoother". Block alignment has already been gone (crop), DAA & HQDN3D have smoothed, LSFmod has sharpened and resampled(!) ... if there have been any "blocks", they have been significantly mutated by then.

I assume that SmoothD generally can deal with such a situation, but it probably needs to be configured differently. In particular the settings about edge-weighting need to be deactivated, because of the missing block alignment. (I don't know whether those settings are activated or deactivated by default.)

[eXtremeDevil]

Didn't work. And what if I put SmoothD before the cropping, like this?

DGDecode_MPEG2Source("r.d2v")
ColorMatrix(interlaced=true, threads=0)
tfm(order=1,pp=7,clip2=nnedi3()).tdecimate()
SmoothD (quant=2, zero_weight=6) #Deblocking
crop(26,14,-24,-6)
DAA()
Tweak(hue=0.0, sat=1.0, cont=1.0)
HQDN3D(ls=20, cs=0.0, lt=0.0, ct=0.0) #Denoise
LSFMOD(defaults="slow", preblur="OFF") # Sharpening + Prebluring
VMToon(strength=130, luma_cap=146, sharpen=false, threshold=4, thinning=100, ssw=3,ssh=3) # Linedarkening
awarpsharp2() # Sharpen
MAA() #Antialiasing
gradfun3() # Debanding

What would be the difference?

[Didée]

Didn't work.
That's because I made a wrong calculation. Insted of using width%16, it needs 16-(width%16).
Script corrected, try again.

[eXtremeDevil]

Corrected where? The blue parts? Isn't the same? I has the same (bad) result...

You hadn't edited it yet xD It works!! Wow! I'll try to encode and see what happens.

Also, just for some tests for myself, if I wanted to resize, where would I have to put it?

[Didée]

I had a look into VMToon. The script is flawed. When "sharpen=false", the script returns the clip "darkend", which in turn is created with "U=1,V=1".

In your script, use

VMToon([settings]).MergeChroma(last)

[eXtremeDevil]

Sorry, I didn't understand that, english is not my first language, could you explain to me easily? xD That is for what? The resize thing? Or no reason?

Thanks for all :)

[Didée]

I said, instead of

VMToon([settings])

you should use

VMToon([settings]).MergeChroma(last)

[eXtremeDevil]

OK, done, what difference should I notice here?

And what about the resize? Where should I put it if I want to do it?

[Didée]

I'm not sure what you are seeing to start with, so what should I tell about the difference.

I was under the impression you are getting distorted chroma?
___

The resizer should come at the end. After MAA, before gradfun.

Alternatively, if everything else fails, you can resize first, immediately after the croping.

[eXtremeDevil]

I'll try later the resize thing, as I'm busy now.

I'm not seeing any distortions, what I said about something distorted was a secondary effect of the cropping + SmoothD.

So, that thing at the end of VMToon, it is because it improves something no matter what, or is it because you thought my display was wrong? If it is the second option, I guess I can remove it, right?

EDIT:

I tried the resize thing putting it in the two places Didée suggested. The AvsP preview is perfectly fine, but in the encode (x264) a green line appears at the top. This only happens if I use resize. What could it be?

EDIT 2:

I've just figured out two things:

First, if I put the resize (spline36resize(720,406)) after After MAA and before gradfun the resolution of the encode changes (probably because of the mod16 thing I have on my script for my filters. So I guess I'll put it inmediately after the cropping.

Second, the green border on the encode is only visible if I play the mkv with BSPlayer, if I use MPC HC or PowerDVD the encode is just fine. Why is that?

This is my final script. Please tell me if there's something out of order:

DGDecode_MPEG2Source("r.d2v")
ColorMatrix(interlaced=true, threads=0)
tfm(order=1,pp=7,clip2=nnedi3()).tdecimate()
crop(26,14,-24,-6,true)
spline36resize(720,406)
ox=width() oy=height()
xx=16-(ox%16) yy=16-(oy%16)
pointresize(ox+xx,oy+yy, 0,0,ox+xx,oy+yy)
DAA()
Tweak(hue=0.0, sat=1.0, cont=1.0)
HQDN3D(ls=20, cs=0.0, lt=0.0, ct=0.0) #Denoise
LSFMOD(defaults="slow", preblur="OFF") # Sharpening + Prebluring
SmoothD (quant=2, zero_weight=6) #Deblocking
VMToon(strength=130, luma_cap=146, sharpen=false, threshold=4, thinning=100, ssw=3,ssh=3).MergeChroma(last) # Linedarkening
awarpsharp2() # Sharpen
MAA() #Antialiasing
gradfun3() # Debanding
crop(0,0,-xx,-yy)