Settings to encode and clean an xvid video

[danielkun]

I want to encode a fraps footage that was originally encoded to xvid at a high bitrate, it's 1280x1024 and it's about 950MB for 13 minutes of video, I think it's a bit overkill so I reencoded it with x264, at 1024x768 using CRF 26, and the size results 250MB, which isn't bad but I still see some artifacts that were on the original video, like some smearing, a bit ringing and a bit blockiness on smooth colors or gradients.

Also the video has lots of text in the chat window, I tried M4G smooth video and the text becomes blurry, so I'm using just the flat matrix and lanczos resize.

I guess I should clean all that up, and I'd like to know which good avisynth scripts or plugins could help, specially considering xvid weaknesses, so I can have better compresibility and I specially want the text to remain sharp.

I'm using everything that can reduce the bitrate at a reasonable time: trellis 1, subme 6, b frames 3, referece frames 3, analize all, rdo, bime, b-prediction, b pyramid and some other usual settings. I guess those settings are right but if anything can be improved, I'm open to suggestions. :)

Thanks in advance.

[Terranigma]

Where did you obtain this clip?

[danielkun]

It's a fraps video from a friend, he made it from his playing in Lineage 2, he wanted good quality so he encoded at a high bitrate, but the size is too big imo. I downloaded it from filefront and I want to reduce its size so there's an option for those who don't have the bandwidth (or the space).

note: I updated my post with my x264 settings.

[Terranigma]

OK, I just wanted to know the origin of the clip before I offered any insight. :)

For downsizing you should use a Spline resizer such as Spline16/36Resize. Lanczos is really meant for upsizing and it's known to cause ringing and even a bit of smearing as well.

For denoising while maintaining sort of a sharp picture, you can use mvtools's, mvdegrain. It's rather slow though, but does what you want.

If your encode's coming out blocky, try using adaptive quantization with x264 and also alter the aq-sensitivity if needed; but if there's blocking in the source already, then use the deblocking filter, Deblock_QED()
(Note: adaptive quantization is a patch and is not in the original version of x264. If you'd like to use this, then use Cef's (http://mirror05.x264.nl/Cef/) custom builds)

function Deblock_QED ( clip clp, int "quant1", int "quant2",
\ int "aOff1", int "bOff1", int "aOff2", int "bOff2", int "uv" )
{
quant1 = default( quant1, 20 )
quant2 = default( quant2, 40 )

aOff1 = default( aOff1, quant1/2 ) # I've no clue if these are clever values or not!
bOff1 = default( bOff1, quant1/4 ) # So:
aOff2 = default( aOff2, quant1/4 ) # Also try all these 4 values @ 0 (zero),
bOff2 = default( bOff2, quant1/8 ) # and quant1=30, quant2=40~45 instead.
uv = default( uv, 1 ) # u=3 -> use proposed method for chroma deblocking
# u=2 -> no chroma deblocking at all (fastest method)
ox = clp.width() # u=1|-1 -> directly use chroma debl. from the normal|strong deblock()
oy = clp.height()

# With avisynth scripting, there is no information available about the position of the currently
# processed pixel ... there simply is no such thing like an "actual" processed pixel.
# So first I've to build up a grid covering the transitions between all 8x8 blocks,
# and make some LUTmania with it later. Yes, this is cumbersome.

block = blankclip(clp,width=6*4,height=6*4,color=$000000).addborders(4,4,4,4,color=$FFFFFF)
block = stackhorizontal( block,block,block,block)
block = stackvertical( block,block,block,block) .pointresize(32,32) .binarize(upper=false)
block = stackhorizontal( block,block,block,block,block,block,block,block)
block = stackvertical( block,block,block,block,block,block)
block = stackhorizontal( block,block,block)
block = stackvertical( block,block,block)
block = block .crop(0,0,ox,oy)
block = (uv!=3) ? block
\ : YtoUV(block.crop(0,0,ox/2,oy/2),block.crop(0,0,ox/2,oy/2),block)
block = block.trim(1,1) .loop(framecount(clp))


# create normal deblocking (for block borders) and strong deblocking (for block interiour)
normal = clp.deblock(quant=quant1,aOffset=aOff1,bOffset=bOff1)
strong = clp.deblock(quant=quant2,aOffset=aOff2,bOffset=bOff2)

# build difference maps of both
normalD = yv12lutxy(clp,normal,"x y - 128 +","x y - 128 +","x y - 128 +",U=uv,V=uv)
strongD = yv12lutxy(clp,strong,"x y - 128 +","x y - 128 +","x y - 128 +",U=uv,V=uv)

# separate border values of the difference maps, and set the interiours to '128'
strongD2 = yv12lutxy(StrongD,block,"y 255 = x 128 ?","y 255 = x 128 ?","y 255 = x 128 ?",U=uv,V=uv)
normalD2 = yv12lutxy(normalD,block,"y 255 = x 128 ?","y 255 = x 128 ?","y 255 = x 128 ?",U=uv,V=uv)

# interpolate the border values over the whole block: DCTFilter can do it. (Kiss to Tom Barry!)
# (Note: this is not fully accurate, but a reasonable approximation.)
strongD3 = strongD2.yv12lut("x 128 - 1.01 * 128 +","x 128 - 1.01 * 128 +","x 128 - 1.01 * 128 +",U=uv,V=uv).dctfilter(1.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0)# .yv12lut("x 128 - 2 / 128 +")

# apply compensation from "normal" deblocking to the borders of the full-block-compensations calculated
# from "strong" deblocking ...
strongD4 = yv12lutxy(strongD3,normalD2,"y 128 = x y ?","y 128 = x y ?","y 128 = x y ?",U=uv,V=uv)

# ... and apply it.
deblocked= yv12lutxy(clp,strongD4,"x y 128 - -","x y 128 - -","x y 128 - -",U=uv,V=uv)

# simple decisions how to treat chroma
deblocked = (uv<0) ? deblocked.mergechroma(strong) : uv<2 ? deblocked.mergechroma(normal) : deblocked

deblocked
return( last )
}


Since you're using trellis, then you should also use no-dct decimate as well (If it's not specified already), so add the following 3 commands to your commandline:
--no-dct-decimate --aq-strength 0.3 --aq-sensitivity 5.0


Other than that, everything looks fine. :)

[danielkun]

Thanks for the suggestions, I found them very helpful :)

Actually I've tried spline36, but I did it when I used the smooth matrix (with unsatisfactory results), so I guess I'll give it another go, this time without the matrix.

I'll try mvdegrain, I only used it when it was inside M4G painfully slow denoise and deblock function, my PC is very slow for this stuff (OE Athlon 64 3000+ single core), so that function is not an option. I might try only mvdegrain though, I think there is a mvdegrain2 function, is it any different?

I don't have problems with the encode blockiness, so AQ isn't needed. I've played with it before though and I've used much higher numbers, it's strength up to 1.1 and sensitivity 10, but I'll try your numbers on a future encode. I guess 1.1 strength is a bit too much :D
The source has some little blockiness, so I'll use Deblock_QED_MT2 as you said, I've seen didee's function, but I've no idea on how to tweak the parameters, I just use the default values :)

I didn't know that trellis must be used with no-decimate, is there a reason for this? And is it ok to use fast skip with this kind of video?

Actually my top priority is to have the chat text as readable as possible, with the smallest size. So I want to improve the compressibility while keeping sharpness, I didn't have any troubles with high motion, and I actually like the encode I did, but I'm sure it can be smaller while keeping a reasonable quality.

:thanks: a lot for the tips, I'll give em a try when I reach home

[Terranigma]

Actually I've tried spline36, but I did it when I used the smooth matrix (with unsatisfactory results), so I guess I'll give it another go, this time without the matrix.

Could you elaborate on what you mean by unsatisfactory result? The only thing I can think of, is the resizer not being as sharp as lanczos; but when it comes to retaining the most amount of detail with little to no ringing, then spline's the way to go.


I'll try mvdegrain, I only used it when it was inside M4G painfully slow denoise and deblock function, my PC is very slow for this stuff (OE Athlon 64 3000+ single core), so that function is not an option. I might try only mvdegrain though, I think there is a mvdegrain2 function, is it any different?
Yes, it does another backward/forward compensated search; it's not inherently better though. When using search=3, I can't spot a difference between degrain1/2. If mvdegrain's not an option like you say, then try ttempsmooth, FluxSmooth, or vaguedenoiser. I would also recommend fft3dfilter/FFT3dGPU but they're slow as well with fft3dgpu being faster since it relies on your gfx card.


I don't have problems with the encode blockiness, so AQ isn't needed. I've played with it before though and I've used much higher numbers, it's strength up to 1.1 and sensitivity 10, but I'll try your numbers on a future encode. I guess 1.1 strength is a bit too much :D

Yes, that's a rather high value. All that'd wound up doing is hurting your encodes rather than helping.


The source has some little blockiness, so I'll use Deblock_QED_MT2 as you said, I've seen didee's function, but I've no idea on how to tweak the parameters, I just use the default values :)
The Default is fine according to the author. :)


I didn't know that trellis must be used with no-decimate, is there a reason for this?


Well according to LoRd_MuldeR, it's a conflict of interest :p
DCT Decimate: If this setting is checked, then "DCT Decimation" will be disabled. This feature allows x264 to discard "unnecessary" DCT blocks. Those DCT blocks won't be written to the bitstream, which saves some discspace. Of course there will be a subtle quality loss at the same time, but usually the effect is negligible. Since DCT decimation leads to significantly smaller file in Quantizer Modes (CQ or CRF) it's recommended to always keep this setting enabled. Nevertheless, you should disabled DCT Decimation when Trellis quantization is used, since DCT Decimation might conflict with Trellis otherwise!


And is it ok to use fast skip with this kind of video?

That's actually up to you to ultimately decide. :)


Actually my top priority is to have the chat text as readable as possible, with the smallest size. So I want to improve the compressibility while keeping sharpness, I didn't have any troubles with high motion, and I actually like the encode I did, but I'm sure it can be smaller while keeping a reasonable quality.

:thanks: a lot for the tips, I'll give em a try when I reach home
Well you'll net the smallest size with the HQ-Insane Profile which can be found in megui, but the processing speed will drop significantly.

[danielkun]

Thanks a lot for the help, I really appreciate it, I find all that info valuable and it's helping me improve my encoding. :thanks: I know little about preprocessing and that's a major part, having knowledge about it, makes a great encoder, so I guess I'll crawl the avisynth sections of the forum :)

About the unsatisfactory results I mentioned, those were due mostly to the smooth matrix, cause it made the text a bit too blurry, that's why I said I would try the spline36resize without the matrix.

About AQ, now that I remember I used those AQ settings to encode a video that was very noisy and those were the suggested settings in another thread in order to keep most of the grain (including subme 1, no trellis, bime and intra, inter deadzones).It worked like a charm. I'll play later with the AQ settings on another video that isn't noisy.