Can't Compress this nicely

[ArabianSwan]

Hi.. I seek help with this source before I discard it for good.
This anime source is very nice and clean but after compression, it gets lots of noise, weird color dots that looks like rainbow.

I used rainbow filter(s), I tried denoise filters.. all that I know but nothing seems to help with this source. Not even high bitrate would help reduce the problem.

original source:
http://www.swan.seedhost.com/original.png

compressed
http://www.swan.seedhost.com/compressed.png

You can see how the compressed is so soft.. well I tried to kill the noise but got only artfacts >_< which I hate to do!

So any suggestions... !

[neuron2]

Your source looks great, why are you applying all the heavy duty filters?

Post your script.

[ArabianSwan]

well I tried to encode it without any filters but still got ALOT of noise and rainbow, so I used these filters:
--------------------------------------
SSIQ(diameter=11,strength=150,interlaced=false)
#
ConvertToYUY2()
Pixiedust(limit=2)
SmoothHiQ(3,20,25,230,5)
ConvertToYV12()
-----------------------------------------

Those the only filters I used with this along with crop and resize and I got that pic result >_<

NOTE: the first clip is from source WITH this script from vdub. Second pic is from the avi file after compression the same script with DivX.

[dbzgundam]

Your major problem seems to be edge enhancement if anything...

[Didée]

Suppose those halos are killing any codec, together with the general sharpness. "High bitrate" isn't enough for that, you'd need "extreme bitrate" for a clean reproduction. The artefakts you're getting in your encoding are called "mosquito noise" or "gibbs artefacing", and are caused by the very high frequencies of all those very sharp edges. That's a problem so inherent to DCT based codecs like fish in the sea. In last instance, it is simply a problem of precision and rounding.

In this thread (http://forum.doom9.org/showthread.php?s=&threadid=74003&highlight=BlindDeHalo) you'll find examples about how to treat that problem. Bottom line is, you'll either have to reduce the apparent sharpness in order to make it manageable for the codec, or have to use insane bitrate indeed. Thats how the world of lossy compression is like, welcome to reality :)

More things that can help to improve on the problem are: custom matrices (preferably one that keeps high frequencies), and usage of quarterpel motion estimation (supposely. Note that i myself have close to zero experience with animated content. More specific hints should come from animee experienced people.)


In any case, it makes no sense to apply filters *before* encoding, in order to fight artefacts that are only there *after* encoding.


edit -oh, and especially PixieDust is a filter that should not be used on such a source. (Said by one of the biggest nerds of PixieDust. Hear that! :D )

[ArabianSwan]

not only that... this is the best shot I can get... but if you see the avi file it is even worse specially when you stretch it to full screen.

I've done many encodes but never faced anything like this one really >_<

[Dali Lama]

Hi,

What's your encoding settings? If you encode the source losslessly does it look good? If so, then you need to adjust encoding parameters.

The bluriness looks like low quality encoding followed by a post-processing deblocker/deringer.

@Didee: Dust works fine on animated sources.

-Dali

[ArabianSwan]

I use the latest DivX... first pass quality base looks perfect.
But when I do the multipass - Nth passes, then I get this problem.

[Kagura]

Drop both SS filters. For a source as clean as this one, minor dehaloing followed by light denoising will be fine. Dust can also be used without problems if followed by a light blindpp to combat dust blocking.

Light denoising = something like [b]con3d[\b] on hq preset or [b]fluxsmooth[\b]
Dehalo/dering (same diff) = Didée's [b]blinddehalo[\b] or a masked [b]unfilter[\b]

Find the rainbow sections and, if they're not too bad, [b]rainbowkiller[\b] them.

[ArabianSwan]

yup right now I'm encoding a small clip using BlindDeHalo2 with Pixiedust only..

I will see how this will turn out ^_^ thanx

[ArabianSwan]

Ok... still no luck with BlindDeHalo2.. and I am almost giving up *I hate this ~_~*

So I'm attaching this small VOB (http://66.90.101.23/gainer.vob) about 52 MB if you please try and see what is the problem with this baka source >_< or is it only me getting old @_@

[Chainmax]

It's quite unusual that the source is clean and that the compressed file is the one that displays the results, filtered or not. I once had a problem when watching files and someone told me to use a different YV12 decoder which solved my issue. You can download a new YV12 decoder here (http://forum.doom9.org/showthread.php?s=&threadid=56972&highlight=helix). Anyway,where did you download the avisynth version of SSHiQ? From here (http://yatta.mellbin.org/misc/SSIQ.rar)? Have you experienced problems with it?
In any case, first of all I recommend you to use TIVTC with TFM(mode=2) as that is the best IVTCing short of using YATTA in my opinion. Other than that, I can't examine the screenshots because my monitor is about to go kaput and is way too dark at 100% brightness and contrast, but depending on how much noise the source has you might want to use hqdn3d (http://students.washington.edu/lorenm/src/avisynth/hqdn3d/)(0,0,3,3) or (0,0,2,2), then use the VDub version of SSHiQ (http://www.neuron2.net/hiq/smoothhiq.html) and use the preview window to adjust the settings so that most (or all) of the haloing is removed, then step through the video and see if there is still haloing and rainbow. If there is still some haloing, try BlindDeHalo2 with light settings and for residual rainbowing you might wanna try Bifrost (http://yatta.mellbin.org/misc/Bifrost.rar). Also, for residual noise either use stronger settings for SSHiQ or use PixieDust with light settings, or maybe Deen...why is this source unfit for Pixie, Didée?

[neuron2]

I don't understand your problem at all based on the clip you posted. It's hybrid but no big deal. Just do IVTC on it and encode it. It doesn't need any other filtering.

I used this script:

loadplugin("dgdecode.dll")
loadplugin("decomb.dll")
MPEG2Source("E:\tmp\Arab\gainer.d2v",cpu=0)
telecide(order=1,post=0)
decimate(5)

...then did a 2-pass XviD encode. No noticable softening or artifacts were observed. I don't understand what your problem is.

[ArabianSwan]

Problem is not on the source.. and I used all those you mentioned.. I've tested on this vob like 10 different scripts with no luck.

Problem is how the file looks like after compression.. we all know that after compression you get some noise that is why we use those filters but not like those I get from this one... and don't get better even with high bitrate which usually solve the compression noise :(


neuron2: what was the bitrate you used in xvid?
I can upload a small clip of my xvid encode to see what I mean by bad noise.

Ok this is the XVID (http://66.90.101.23/xvid.avi) file with 2 passes.

[dude051]

That anime, gainer, won't compress great at all. I've worked with the source, and its a great source. But thats the problem. Alot of anime sources today are nice and clean, which XviD actually sees this and compresses more than it should. This can be solved by two ways in my opinion. Either add noise and force the codec to quantisize correctly by adding more bits into the scenes... or create a custom matrix (or use one that retains more detail). Either way, detail is your goal... filtering more will cause the problem to get worse. If anything, you want to use no filters and if you must use light filtering in something like TemporalCleaner(2,2). I've messed with this source with a friend and the add noise approach doesn't work. So your best bet is to go for a new matrix (using XviD mind you :devil: ). Good luck encoding!

[Mug Funky]

you could always go with MPEG-2 :)

i'd just do a simple blur (maybe unfilter(33,33)) and encode as is. maybe deblock the input if it needs it (depends on the bitrate of the DVD source, but looking at that still there, it should all be fine).

if you must use Divx, make sure preprocessing (and postprocessing) are turned off - this source should encode fine without being touched at all, and should encode perfect with a very slight softening.

what bitrate are you aiming for?

[edit]
dude051: what do you mean about xvid overquantizing? i've found it'll overquantize if there's too much noise in a source (a problem that's probably not going to be looked at any time soon, as 99% of xvid encodes are from clean or denoised sources). if you mean blocking or banding, then i agree this could be a problem (had this with a super-clean source at q2, with regular h.263 matrix).. i don't understand why mpeg-4 ASP doesn't allow a higher DC precision, as i'm sure this would eliminate the blocks on gentle gradients (which clean anime is basically full of).

btw, what anime is it? i haven't seen that girl in anything...

[khmann]

you could try a different/flatter quant. matrix that preserves more detail in the high frequencies. my (mpeg) experience is that more accurate results are obtained at a given bitrate by smoothing the source than letting the encoder's matrix/"q" do it.

[dude051]

@Mud
i've found that XviD loves to over quantisize on clean sources because of the lack of detail (in Anime mind you).

i've found it'll overquantize if there's too much noise in a source (a problem that's probably not going to be looked at any time soon, as 99% of xvid encodes are from clean or denoised sources)

I dont really see that issue as over quantizing as much as just a bit starvation issue that forces XviD to not be able to compress it well. The over quantizing im talking about is one that XviD sees a lack of detail and therefore thinks it can compress more due to it's nature. This is why i suggested adding noise to force xvid to encode the detail that is there correctly (mainly flat even areas where DCT blocks occur), or his best bet is using a matrix that will retain more detai. Your right though, there isnt any need for filtering, the anime is Overman King Gainer and its got a great source. And as khmann said, using the matrix as a denoiser is usually a bad idea, esp. in these cases.