Anime Blockiness

[Son Gokou]

Greetings, I've been ripping some Dragonball Z anime from a DVD that I own recently in an attempt to find reliable settings that I can stick with in future anime encodes.

Settings I've been using are:

Xvid (latest build I think) with Virtualdub to encode, I use a virtual dub resize filter, 512 x 384 billinear so as to keep the 4/3 aspect ratio, 1 pass CBR with 1500 bitrate.

Here is a sample of my results:

http://www.webspawner.com/users/dragon142/samplescreen.jpg

You will see this particular blockiness, that only gets worse when I play it full screen.

I'd prefer to solve this without using smartsmooth or any other heavy smoothing filter, because I'd like to preserve as much of the quality of this DVD source as possible. Any help appreciated.

Edit: I'd like to ask a question also, why is it that I can use say, a 900 bitrate on a film type source such as "The Matrix" or "Gladiator" and I will get outstanding results with Xvid or DivX, but I can use an enormous bitrate like 1500 on an anime and it looks like utter crap (to be frank :P) I mean my first instinct would be that anime is easier, large blocks of color and all. Thanks again.

[sysKin]

Originally posted by Son Gokou
Settings I've been using are:

Xvid (latest build I think) Stable build (0.9) or unstable alpha?
with Virtualdub to encode, I use a virtual dub resize filter, 512 x 384 billinear so as to keep the 4/3 aspect ratio, 1 pass CBR with 1500 bitrate.
B-frames? Qpel? In general I should also ask you about motion precision and lumimasking, but _this time_ I'm not interested in those ;)
Here is a sample of my results:

http://www.webspawner.com/users/dragon142/samplescreen.jpg

You will see this particular blockiness, that only gets worse when I play it full screen.
First thing that comes to my mind - do what you can to remove black bars on the edges of your video. They have big effect on bitrate/quality. It doesn't matter if they are big or not, it just matters if they are there.
If changing aspect ratio a bit is unaccaptable, at least get rid of one row at one side, and add black pixels to the other side.
I'd just crop the black bars before resizeing - noone will notice ;)

Edit: I'd like to ask a question also, why is it that I can use say, a 900 bitrate on a film type source such as "The Matrix" or "Gladiator" and I will get outstanding results with Xvid or DivX, but I can use an enormous bitrate like 1500 on an anime and it looks like utter crap (to be frank :P) I mean my first instinct would be that anime is easier, large blocks of color and all. Thanks again. [/B]
MPEG4 is not bad for animation, but not very good either. Also, XviD (as well as any other mpeg4 encoder I know) is not bad for animation, but not very good either.
As you can see, all big blocks (at least on the picture) are in color, not in brightness. This is because codec ignores color until the very end - in motion estimation, scenechange detection, mode decision etc etc color is just ignored. This is much faster and gives good results with 'normal' movies, and with high quality animes as well (like "GITS" etc), but fails when encoding 'simple' animation...

Postprocesssing should remove any blocks from your video - did you try it?

Also, I recommend using 2-pass mode whenever you can - it should increase quality of high-motion scenes (and the picture you show us - that's high motion right?)

Well I have to finnish now, maybe I'll write more later ;)

Radek

[RadicalEd]

It's been said before but any gain in compressibility due to large blocks of color is overcome by the sharpness of almost every object in anime. Most edges are pointed and abrupt in luminace and chrominace making the quantization and DCT a lot more difficult. Those compression techniques were designed with natural images in mind, which usually have smooth, gradual changes in luminace/chrominace. That's why in general smoothing ups compressibility while sharpening lessens it.

[Hajime]

Hi Son Gokou

First for all, you're using CBR compression, this is not a good idea if you plan to get good quality with a reasonable bitrate, try again using 2 pass encoding. Iago wrote a very good guide for 2 pass encoding, check it: http://forum.doom9.org/showthread.php?s=&threadid=36915

[Son Gokou]

Stable build (0.9) or unstable alpha?

You know I'm not actually sure, I'm new to Xvid (you could probably tell) but here is the url of what I used.

http://www.roeder.goe.net/cgi-bin/fetch?file=XviD-04102002-1.exe

B-frames? Qpel? In general I should also ask you about motion precision and lumimasking, but _this time_ I'm not interested in those

I set the motion precision to ultra high, with lumimasking, I altered no other settings except the bitrate. Using a much higher bitrate like 2500 seems to fix all the blockiness which kind of concerns me :P I'd hate to think 400 mb an episode is the only way to solve it.

Anyway thanks for all the advice, I'd heard from someone somewhere that real media 9 was good for encoding South Park episodes, I have to wonder if it would work for this as well?

[RadicalEd]

Real9 is good for animated content but XviD is in no way inferior. Both are about equal although Real requires no tweaking of stats, so if you don't mind using the RealMedia format (although by this time next month I guess that won't matter anymore :) ) you might be better off that way. The results vary. For instance the Hellsing intro looked much better with XviD but the Love Hina intro looked better using Real9. Theres no real answer except to say that they're very similar.

[Teegedeck]

Originally posted by Son Gokou

I set the motion precision to ultra high, with lumimasking, I altered no other settings except the bitrate.

Now I think we got our delinquent: Lumi-masking is a big no-no for anime - the lumi-masking idea doesn't work for animated content because you immediately notice blocks in parts of the picture that got compressed stronger (while you might not realize it in a 'natural' movie). Don't use lumi-masking on anime and you'll be fine, I think.

[ChronoReverse]

Incidentally, you might want to try the bicubic resize filter setting rather than bilinear. Which is better is debatable, but I personally think bicubic looks better. If you want the best looking results, it'll be a good idea to use 2-pass encoding.

Which quantitization are you using btw?

[Prettz]

Originally posted by Son Gokou
I'd hate to think 400 mb an episode is the only way to solve it.

No you should never ever end up with more than 200MB. And you should try and stay between, say, 120MB to 160MB.
If you end up having to make the final result over 200MB for 1 tv episode, you did something wrong, like with the bitrate curve.

[JohnMK]

There's an order of magnitude less detail in anime vs. films, so I really don't think it's as simple as recommending bicubic. Just theoretically, there should be nothing wrong at all with bilinear resize for animation, and in fact its form of anti-aliasing/smoothing things out will really help more than hurt, for animation at least. Of course I'm no expert at all on these things, I'm just expressing my rational thoughts. I've never actually verified.

[Angrychair]

I do a lot of anime encoding... So I think I can give you some tips.

First off, if you don't want to use a smoother, I suggest you at least use a color stabilizer. Even a good source usually has lots of motion you don't notice because it either doesn't show up on the tv, or doesn't register in your mind. I suggest a light blending using a good 2d filter, and a good temporal color stabilizer such as flaxen's vhs (good options, tough to figure out) or avery lee's dynamic noise reducer (think that's what it's called) which has a MUCH simpler interface (1 slider) at the expense of tweakability.

Next, be sure to set your curve aggression to the highest setting, and use the largest quantizer range (2-32 or whatever) with high agression in bitrate towards motion. Since anime tends to have one of 3 things in each scene. Either, A: little or no motion, B: even panning motions, or C: Crazy huge amounts of motion with firey backgrounds and huge explosions.

Understanding this is your bread and butter. If you're using the color stabilizer and smoother, like I suggest, a lot of your scenes will have little to NO motion at all in them, saving tons of precious bitrate, that will be allocated to the higher motion scenes, stopping your 'blockiness'.

Hope this helps. (oh yeah, do like the other guy said, always encode in 2 passes)

[Marc FD]

>good temporal color stabilizer such as flaxen's vhs (good options, tough
> to figure out)

for me it was so slooow as hell, and always needed about 1 second to stabilize chroma after a scene change.

>or avery lee's dynamic noise reducer (think that's what it's called)
>which has a MUCH simpler interface (1 slider) at the expense of
>tweakability.

dunno if this only works for chroma.
what about CNR ? it's a specialized chroma stabilizer and it did wonders for me with anime VHS caps. it's why i've avisynth-adapted it.
I'm gonna add scene-change detection because it's one of the pb of the power of this filter.

>Next, be sure to set your curve aggression to the highest setting, and
> use the largest quantizer range (2-32 or whatever) with high
>agression in bitrate towards motion. Since anime tends to have one of
>3 things in each scene. Either, A: little or no motion, B: even
>panning motions, or C: Crazy huge amounts of motion with firey
>backgrounds and huge explosions.

i don't think you need to use high aggression. slow scenes will look ugly then. the best is to keep default settings. Koepi, foxer, Iago have tested and/or coded many things, and it works like a charm with defaults.

>Understanding this is your bread and butter. If you're using the color
> stabilizer and smoother, like I suggest, a lot of your scenes will
>have little to NO motion at all in them, saving tons of precious
>bitrate, that will be allocated to the higher motion scenes, stopping
>your 'blockiness'.

hmmm. bframes are exellent for gaining compression increase in slow scenes. and far less destructive than VBR tweaking. i always use it for anime encodes. (and i'm a anime-only encoder)

>Hope this helps. (oh yeah, do like the other guy said, always encode
>in 2 passes)

of course ^^

@SysKin

When do you think hinted-2pass could work with bframes ? speed-up and quality increase in 2pass encoding would be great.

[Angrychair]

Originally posted by Marc FD
for me it was so slooow as hell, and always needed about 1 second to stabilize chroma after a scene change.

dunno if this only works for chroma.
what about CNR ? it's a specialized chroma stabilizer and it did wonders for me with anime VHS caps. it's why i've avisynth-adapted it.
I'm gonna add scene-change detection because it's one of the pb of the power of this filter.

i don't think you need to use high aggression. slow scenes will look ugly then. the best is to keep default settings. Koepi, foxer, Iago have tested and/or coded many things, and it works like a charm with defaults.

hmmm. bframes are exellent for gaining compression increase in slow scenes. and far less destructive than VBR tweaking. i always use it for anime encodes. (and i'm a anime-only encoder)

of course ^^

@SysKin

When do you think hinted-2pass could work with bframes ? speed-up and quality increase in 2pass encoding would be great.

Complaints about speed are really pretty irrelevant when we're having a discussion about quality...

Secondly, I'm giving good advice, if you have a better virtualdub color stabilizer on your mind, by all means, don't hold back, we're trying to help out son goku here.

Thirdly, you may not think high curve agression is warranted, but in my digisubbing group the xvid users do definitely use high curve agression.

Anyway, I rarely use xvid myself (as it still sucks in my opinion) but for those that insist on using it, my advice is sound, as it comes from those that do a LOT of anime encoding.

Oh yeah, I'm pretty much strictly an anime encoder, too. I prefer divx5 to xvid, though.

[Marc FD]

>Complaints about speed are really pretty irrelevant when we're having a
>discussion about quality...

sorry, i though he was using 1-pass for speed purpose. it seemed logical.

>Secondly, I'm giving good advice, if you have a better virtualdub color
>stabilizer on your mind, by all means, don't hold back, we're trying to
>help out son goku here.

why VirtualDub ? i don't understand ? i though avisynth provides faster AND better filtering ? i've missed something ?

EDIT : and Chroma Noise Reducer is a VirtualDub filter, AFAIK.

>Thirdly, you may not think high curve agression is warranted, but in
>my digisubbing group the xvid users do definitely use high curve
>agression.

fansubbers teams have their own techniques. i've encoder friends in such groups, and i know it's not what i would call a "reference". every XviD anime encode i saw on internet was crap. maybe i'm just unlucky.

>Anyway, I rarely use xvid myself (as it still sucks in my opinion)

i agree with you. XviD sucks with anime. but i founded it's 10x easier to use than DivX, and i gives me better results, so i don't care.

RV9 is definitvely the best choice for quality, low bitrate and animation. the only problem is, it's a proprietary format.

> but
> for those that insist on using it, my advice is sound, as it comes
> from those that do a LOT of anime encoding.

doing a LOT don't always mean doing GOOD, does it ?
many many anime encoders who are doing a LOT of anime encoding are still using SBC. is it really that better ??

>Oh yeah, I'm pretty much strictly an anime encoder, too.
>I prefer divx5 to xvid, though.

If it gives you better results.
divx5 is simply too complex for me. and i don't know what it really does.
i've noticed too that it's seems to work good with anime for many people.

[sysKin]

Hi again,
I decided to check myself what are the gains and losses of using colour information for motion estimation. You know what? It's not quite the way I thought it was.
Encoding is, of course, slower. But the loss is only 5..7%, so I think quality maniacs might find it quite irrelevant.
For live movies, quality gain is very small indeed - 0.05dB is not something one can notice.
However, in case of colourful animation, the gain in terms of PSNR is at least 0.20dB - which is _something_. More importantly, big colour blocks are gone, improving visual quality. I'm not good at assessing visual quality, though.

What do you think?

The code is ready, I'm waiting for a blessing from main developers before I commit it.

@Son Gokou - thanks, I'd never try this without your thread ;)

Radek 'sysKin' Czyz

[ChronoReverse]

Sound really cool! Maybe there can be a xvid option that will allow choosing different ME? Or is this "hard-coded" when you build a binary? /me must go and download the xvid source one day

[sysKin]

Originally posted by ChronoReverse
Sound really cool! Maybe there can be a xvid option that will allow choosing different ME? Or is this "hard-coded" when you build a binary? /me must go and download the xvid source one day I'll just add a binary flag to xvid core.
I'm not going to change VfW library's interface - I think it's Koepi's job ;) But I guess it should either be added to motion precision 6, or as an additional tick box named similar to "increased colour precision"/"animation mode"/"maniacs please click here"/etc.

I'm still waiting for comments, but expect it today (/me is very optimistic).

If you can't wait, contact me at this IRC network/channel you see below.

sysKin

[trbarry]

First thing that comes to my mind - do what you can to remove black bars on the edges of your video. They have big effect on bitrate/quality. It doesn't matter if they are big or not, it just matters if they are there.

I didn't know this was that much of a big deal, though I always try to remove the black bars. But if it was really that important it would probably be fairly easy to write a filter that finds the transition between the black bars and the picture and vertically filters the heck out of only the few surrounding lines. Don't know if that would help compression or not but if the extreme top and bottom lines of a picture were very soft maybe nobody would notice.

- Tom

[Koepi]

Originally posted by sysKin
I'll just add a binary flag to xvid core.
I'm not going to change VfW library's interface - I think it's Koepi's job ;) But I guess it should either be added to motion precision 6, or as an additional tick box named similar to "increased colour precision"/"animation mode"/"maniacs please click here"/etc.
sysKin

I gave it a different name ;)

XviD-04112002-1:
- Fresh CVS checkout. GMC improvements.
- New mod. HQ quant type reimplemented.
- Added checkbox for chroma motion estimation (slow but better PSNR).

Regards,
Koepi

[ChronoReverse]

The reason you take out the black bars is because unless the transition lies exactly at the boundary between the blocks that xvid divides the image into, it's gonna be hard to compress (ie xvid doesn't like stark transitions) This will decrease compressibility and reduce the number of bits available for the image. Therefore you cut the useless black bars.

[RadicalEd]

I can't back the accusations that XviD sucks with anime
anyway, the Helix DNA is now opensource so pretty soon it should be able to more or less use Real9 as you would any other codec :]