Encoding Anime Transformers. How I solved my restoration problem to get stuck in Xvid

[Pentajet]

I am trying to encode a low quality Italian version of the US cartoon "The Transformers". The video I started with was really really awful: interlaced, uncropped, full of aliasing everywhere, full of blocks and color banding everywhere. After several researches and a lot of hassle I managed to restore the video to an acceptable form. Here is how in case anyone is having the problems I mentioned above with similar cartoons.

First I used three ffdshow filters to get rid of color banding, macroblocks and the temporal noise caused by these two. The filters (turn on only in "decoder" not in "encoder") are:

DeBand (VERY IMPORTANT): set this to 3.75. This improves the video A LOT although it creates slight ghosting on some scenes.

MPlayer Temporal Noise Reducer. I played a little with the values and the best I found were 46,196,293 (TURN OFF ALL OTHER FILTERS THAT ARE SET TO ON BY DEFAULT)

Postprocessing Filter: Just used SPP Fast Deblocking.

I then used the command DirectShowSource("mysource.avi") to load the filtered video in Virtual Dub.

To get rid of the aliasing I used SangNom (with an anti-aliasing factor of 90) coupled with TlSophote. Before applying these last two filters I resized to twice the actual size and then saved.

Result: slightly blurry but otherwise perfect picture.

And you will ask: why is this stupid newbie posting this here?? Because although if I save to uncompressed YV12 format the resulting video is nearly flawless (at least compared to the source that I was working with) as soon as I try to compress it EVEN WITH A BITRATE of 8000 (yes 8000) the blocks and banding come back again. I even tried to encode it to Mpeg2 with TMPGEnc and still the blocks come back! What can I do? Does anyone have a suggestion as to how I can encode the video and keep the original quality that I achieved after filtering? Blockbuster has improved the situation a bit although the noise isn't very pretty and still it's not the quality I am looking for. I tried HuffyUV and no luck even with that.

[Suchy]

I had similiar situation. I filterd video, but after encoding blocks come back, even when I use high bitrate and low quants.

In my situations were blocks in lumination.

Try blend them in luma.

P.S.
Did You use CQM?

[Pentajet]

Thanks for your reply but I am a newbie to this although I am trying to learn. Can you please explain whats the difference between blocks in lumination and ordinary blocks? How do I blend them in luma? What's CQM? If you mean a constant quantization matrix I tried the one supplied by TMPGEnc when I encoded to Mpeg2 - the result was better than Xvid but blocks were still there.

[xyloy]

You'll get no blocks(and no ringing) with an H.264/AVC codec, such as x264. ;)

http://x264.nl/ to download x264 and MeGUI(or StaxRip, wich is more simple for some people).

Beware! .AVI is not designed for AVC! Better use .MKV or .MP4

See also the MPEG-4 AVC forum stickies.

[Pentajet]

I tried x264 but the blocks were still there. I didn't tweak it too much. I will try to have another go at it.

[xyloy]

One of the basic function of the H.264/AVC standard, wich x264 is based upon, is an advanced in-loop deblocking filter. ^^

The strange thing it's that it should be activated without need to tweak it.

I suggest you try this(it's a one-pass quality encode with x264 command line encoder) as a test:
X:\Program Files\x264\x264.exe --crf 18 --ref 8 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 7 --trellis 2 --analyse all --8x8dct --me umh --progress --no-dct-decimate --no-psnr --output "x:\movie.mp4" "x:\movie.avs"

Paths have to be modified in order to work. ;)

[Suchy]

I don't think that is caused by to low bitrate or too "lame" encoder. But who knows. :P

P.S.
CQM - Cutom Quant Matrix

[sysKin]

Are you sure ffdshow doesn't disable deblocking when CPU usage is too high?

[Pentajet]

Thanks for your suggestions and help which is much appreciatted.

Xyloy: I can't make it work - it tells me that I am missing files in my .NET framework. I am trying to fix it with little success so far.

I am 100% that everything is working fine because when saving to Uncompressed YV12 or RGB24 there isn't the slightest trace of a block.

Thre is one thing that works though. Turing off b-frames in Xvid, using Blockbuster in AVISynth and setting the quantizer to 1 with H263 matrix (I tried several custom matrices and none worked - Six-Of-Nine was a little better than the rest). Unfortunately this results in a 15600Kbps!!! and 3.4Gb per episode.

[xyloy]

It's MeGUI wich needs Microsoft .NET Framework 2.0 to be installed(Google it to download it), not x264.exe (unless I've miss something)

[DarkZell666]

stupid question : what resolution are you encoding to ?
if it's anything > than 640*480 blocking doesn't surprise me much :)

What do you call low quants ? same thing : anything > 2 will necessarily block (2 sometimes blocks but only with B-frames on, and in low-contrast areas)

I tried HuffyUV and no luck even with that. >> wierd, you must be doing something wrong because it's near-lossless (it simply throws away half the chroma information, that's all)

and no, x264.exe doesn't need .NET xD

[Pentajet]

I do have the NET Framework inslalled already but it doesn't work. It gives me an error that it's not installed (although it really is). I tried downloading an earlier version and it still didn;t work.

Zell I can't see what I'm doing wrong. As I said I am a newbie - just learning the basics. If I save the restored video to an Uncompressed format the picture is perfect - no blocks, no bands, nothing. If I use a codec the blocks and banding return. What you're saying about HuffyUV gave me an idea. I tried the ghost noise reducer in TMPEnc and it gave me surprisingly good results. The blocks are almost all gone. The file I got is 500Mb = Acceptable although not very happy with it - it isn't perfect and I have seen people getting better results with files as small as 250Mb. Also the TMPEnc filter improved the picture overall (it was somewhat unaturally dark before). I used the MPEG matrix (the others gave results as bad as Xvid). Looks like I will have to re-encode it as a DVD again but I will also keep trying the x264 solution although between MPEG2 and X264 I would prefer MPEG2.

[foxyshadis]

Banding is pretty common in YV12 with CG animation, it's why you need debanding on playback. Introducing a tiny bit of real noise or using an extremely high-bitrate matrix is the only way to get it past the codec, though; noise is probably more consistent.

SPP fast deblocking always looks terrible to me, full of blocking, but if it works for you I guess that's what's important.

What is decoding your output files? If it's ffdshow, have you tried it with all filters turned off vs pp turned on?

[Elic]

Pentajet
Hmm... Can you please put few pictures (source, filtered, and after XviD decompression) and/or small fragment of original video and same fragment encoded by XviD?

PS. Sorry but my English is not so sharp and for me, much simpler is to take look at video than to ask you proper questions :(

[Pentajet]

SPP Deblocking has done a great job for me - without it not only I wouldn't be able to encode but I wouldn't even be able to restore the original video. There were two single scenes however where SPP Deblocking totally blurred out the image in two sections full of red and blue macroblocks. I decided to encode those two scenes separately with mplayer deblocker in ffdshow and they turned out great too but I have to cut and join the video like this. I know it's not a great encoding practice but at least it looks good.

I am not using any filters on playback. My goal is to make the video look at its best without any need for post-processing filters.

I am no longer using CG Animation. I found out that the best matrix for this job is the MPEG matrix (at least among the ones I tried).

I am still working on the video. When I get have the final version I will post a link to a small specimen (that isn't against the board rules, I hope??)

I am still experiencing some problems:

1. Interlacing artfacts (visible only on the monitor not on TV) although I deinterlaced the video. No big deal for me - I don;t watch my DVD's and avi's on my Monitor.

2. Some visible noise (ringing) in some grey and purple areas (this is not the result of the encoding - both source and restored video are like this). Not a big deal if not for the fact that the main charcter in the series (in his truck form - these are transforming robots) is half grey!

3. The video is visibly blurry - definitely not sharp at all. But remember this is a US 80's toon not an anime where faceless characters, white guys who suddenly become purple, 12m charcters who beome 3m tall all of a suden etc, can be found every couple of scenes so the blur is not that noticeable. If only I could make mftoon work on my PC it would be great though.

I would like to thank everybody for their input as you have been very helpful on this board.

[foxyshadis]

Actually, vmtoon that comes with masktools 2 is a newer incarnation of mftoon, and should work as long as you have masktools2, warpsharp, and avisynth 2.5.6. You could use something like vaguedenoiser or peachsmooother if you want to get out the last of the ringing without hurting details, but they're both terribly slow.

[raeltheimperialaerosolkid]

Penta, try a single pass XviD at Q2 or Q3 constant. The video should be nearly lossless but without the oversize in Mbyte you got with the Q1 encode (that should be avoided anyway).

[DarkZell666]

ya know, if the blocks come back with HuffyUV I tried HuffyUV and no luck even with that ...

Now I'm thinking of something : those ffdshow filters you're using ... are you actually switching them on/off at the right moments ?

What you say the banding/blocks come back ... is it exactly as it was, as in "just as if it hadn't been cleaned" ? or is it rather "not as bad as if I hadn't cleaned" ?

posting screenshots is the best thing to do indeed, help us help you =)

[GodofaGap]

Very simple test to see if everything is working properly.

1. Save (a part) as yv12 (which works).
2. Load the uncompressed clip in VirtualDub and compress (e.g. XviD q2).
3. See if blocking comes back.

Since huffyuv is lossless there is no reason why it shouldn't work.

[Pentajet]

AN UPDATE:

I managed to get rid of some (not all ringing) by tweaking the controls on SPP deblocking. I managed to compress the video in xvid by tweaking brightness/contrast with almost no blocks or color banding in the resulting video (3000kbps/s = good to me)

Foxy: I will definitely try vftoon. Thankls for telling me. I didn't know.

Raedl...: Anything lower than 1 will not work unless I adjust the brightness/contrast.

DarkSel666: I am sure I am switching them on and off at the right moments. The blockiness/banding is different from the source - I mean it's the same kind of problem but not identical to the source.

GodOfAGap: That's exacly what I did to test it. The banding was a result of the encoding.

NOW, when everything was fine and the first eight episodes were nearly ready for burning a new problem came up. I get this episode where the robots find a big plane buried in ice and the sky in the episode is all full of banding AGAIN!! The other colors in the episode are fine - the banding is just in this particular color (deep "icy" blue - that was first shown in this episode only). I tried to post a picture as you suggested. As you can see the sky is made up of two kinds of blue and there is an abrupt change from one to the other.

THIS IS NOT A PROBLEM OF THE ENCODING THIS TIME - It's a problem in the filtered (as well as the unfiltered) source itself.

What I would like to do is modify one of the blue colors to make it resemble the other one as closely as possible. But I don't know how to do this. I've been searching this forum and the web for all day trying to find a filter that can change one single color in the source without affecting the others. I think that's the only possible solution since all other filters have failed for this episode and the problem concerns one color only.

This is the link to the bmp in which the problem is most evident:

http://www.verzend.be/v/5004120/astrum3.bmp.html

Here is a link to the scene where the problem is most apparent (9MB):

http://www.verzend.be/v/1975862/testnew.avi.html

[DarkZell666]

why is it I can't see ANY problem in the bmp or even in the video sample ? Am I blind or what ? oO I can't see ANY noticeable artifact what so ever ...

One last question from me : is your desktop set to 16bits colors ? I remember anime looking wierd-colored in 16bits display, but I hadn't seen that for quite a long time.
Prefer 32bits if possible :)

The only problem I can see in the sky is temporal noise, which you can get rid of with hqdn3D() for example. (And switching my screen to 16bits colors made the noise look even worse ...)
You'll maybe save 1000kbps and have a much cleaner picture imho :)

[Pentajet]

On the monitor it isn't very noticeable. I use the TV to watch them and if you set the contrast high enough the blocking/posterization/banding/temporal noise or whatever it's called is very visible on the blue sky. It's true that it's only in the deep blue sky and all other colors are fine (at least after all the filtering I did). It's not some big problem that's distorting the image in some abvious way but, for me, it's quite noticeable and after getting rid of the problem on all other colors I would like to clear the deep blue too. I will try the filter you suggested immediately.

[DarkZell666]

I'm surprised your TV enhances those problems, is it HD and/or LCD ?
a "normal" TV would be kind and "hide" that sort of thing ... (at least mine does :))

[Pentajet]

I tried the filter and unfortunately it didn't work. Thanks for the tip 'though. The only thing that works is changing the contrast and brightness. Unfortunately this fixes the blue colored sky but distorts the rest of the image. Anybody knows of a way to change brightness and contrast just for the one color I am having problems with without affecting the others??

DarkZel it's a normal TV. The problem is quite visible on it. My monitor is not "the best in town". It's an old 1998 monitor.

[Didée]

Banding is a general problem of digitally coded content, and the human HVS does not fit well to digital coding. The only way to hide the inherent digital banding from human HVS is noise, respectively "dithering".

Search for the avisynth filter "gradfun2db", or use a recent ffdshow build which incorporates this filter as "DeBand" option.

Another *important* point you must be aware of:

- do NOT touch contrast/brightness settings
- NEVER EVER use any sort of "gamma correction"

, at least not if you're sensitive to "banding" artefacts. Any of these adjustments will definetly enhance the inherent digital banding. Also check your driver settings regarding TV-out. It might be some inappropriate setting in that area.

In respect to banding, the only "safe" form of histogram adjustment are "offset" settings that push the complete range as a whole, keeping the range gradient 100% linear.

[Pentajet]

"In respect to banding, the only "safe" form of histogram adjustment are "offset" settings that push the complete range as a whole, keeping the range gradient 100% linear"

I don't know what this means are what are you refering to. If you could please explain what you mean and how to do this I would be grateful.

I tested grandfun3d and it didn't work. DeBand has made a great job in adjusting all other colors (in fact my first two posts were a lenghty discussion of the effects of DeBand on the video and the fact that re-encoding to xvid ereased the effect - a problem that I actually resolved by adjusting the brightness and contrast. If I didn't do that the video would still look like garbage) - the only problem is the deep blue. Further adjusting Brightness/contrast/hue/saturation has solved the problem with blue but yellow and grey don't look right - there is no banding in them - they just look unnatural with the adjustment.

If you could please explain or point me to somewhere that explains what histogram adjustments and gradients are I would be greatful as I am stil new to this.

[Didée]

Yes, you did mention that you used DeBand - probably I was in the afterglow of the uber-nerve-wracking worldcup game Germany-Argentinia. (We won after penalty shootout, phew.) Bear with me. :D

However: as a pre-processor before encoding, DeBand/gradfun2db is almost useless. It's the characteristics of DCT based codecs to destroy the improvements this filter may have made. (Note that this is not a "bug", either - it's more or less a necessary consequence of DCT'ed frequency coding.)
DeBand/gradfun2db is meant to be applied during playback, to hide the artificial boundaries that are created during encoding. And that's the only way to use it efficiently. What you see in any preview window lastly is irrelevant. Relevant is what you see after encoding.
And from what you told us, you got the ultimate proof indeed: you got the source from "looking ugly" to "looking good" ... *before* encoding. But *after* encoding, your words, everything "came back again" ...
So, try it *that* way: during playback.


To the histogram question - the effect is really very simple, but somewhat tricky to put in (few) words ...

My try #1: look at this:

original smooth gradient:
1 2 3 4 5 6 7 8

more contrast (or: more "gain")
1 2 4 5 6 8 9 10
^ ^
brightening by gamma adjustment:
1 4 6 7 8 8 9 9
^ ^ ^ ^

Try #2:

Take a sheet of checkered paper. Draw a 45° line, by using one cross per checker. Fine, no problem. Then draw a 20° line, a 75° line, and a circle. Ah-ha: big problem!

Bottomline: a digital coded picture never is really "smooth", because of the digital "quantum". The smallest stepsize is +/- 1, obviously.
Now, a "smooth gradient" corresponds to the 45° line you drawed on the checkered paper. If you enhance contrast of the output, this is similar to e.g. a 55° line you might try to draw - this line can not be smooth, there will be "jumps" in the line, because the checkering does not allow intermediate values.
Gamma correction is similar, and often even worse: gamma correction will produce relatively big "steps" in the dark parts of a frame, so that dark scenes very easily can get an ugly look.

That's what I meant by "offset" and "keeping linearity": if banding and/or blocking is such a big problem to you, then you must not adjust the levels by making the output curve more steepy, or even curv-ey like gamma does. Only off-setting the 45° line upwards or downwards, but keeping it at being 45°, is safe from producing more of these artefacts.

[Pentajet]

I understand Didee and thanks. I will try to be more careful when adjusting the contrast but I can't just not do it at all. It's the only way I found to actually reduce the banding in the sky. From what you said (and by examining a big zoom of the picture) it might be more probable that what's really happening is that the increase in contrast is producing more artifacts but it seems that on a larger scale these artifacts are actually hiding the more visible banding in scenes like the one I posted so as long as the video looks better the artifacts are welcome.

Ans I was watching the world cup too. Your next match is with us and you'll see we'll send you home! (well you're already home but anyway..........)