Staxrip Debanding gradfun2db Problem

[DragooNick]

Hey,
so first of all I'm a total newby to encoding, but I'm trying to learn as much as possible.
Now I got a pretty weird problem, I got an Anime TV TS Source, so I used yadif to deinterlace and got stuck with a color banding,
I tried to use gradfun2dbfor the debanding and as long as I am in Staxrip using preview I can see what the filter does, but as soon as I encode,
the video looks as if the filter was never used at all :/

The avs file looks pretty much as this:
Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()
Yadif()
gradfun2db(1.2)
Trim(2348,2478)

Does anyone know where the problem could be?


Edit: http://jumbofiles.com/j66by3y3jmix <--- 34mb ts file with the problematic part of the stream

[sneaker_ger]

The debanded picture is very hard to encode (that's why you see banding in the first place, because broadcast encoding sucks). You can try to increase bitrate or use 10 bit x264 encoding, and use ordered dithering (not sure what gradfun2db uses by default).
See these threads:
http://forum.doom9.org/showthread.php?t=153589
http://forum.doom9.org/showthread.php?t=161411
http://forum.doom9.org/showthread.php?t=164352

Most current Anime series probably need IVTC (http://forum.doom9.org/showthread.php?t=163887), not deinterlacing, btw.

[Sapo84]

What kind of anime are you trying to encode?
Going 60fps sounds silly, nowadays Anime TS are usually 24000/1001 with constant patterns, that means that 9 times out of 10 the correct way to process them is
- Source (DGDecode already outputs YV12 video)
- Telecide
- Decimate

(There are a couple of exception to this rule, for example precures usually have 30000/1001 sections, sometimes mixed with 24000/1001 ones)

Using yadif will probably only make the video jerky and bloat the bitrate, and you should probably use that to increase the quality of the encoded video, so that the banding doesn't appear in the encoded video.

Anyway, gradfun2db(1.2) may not eliminate all the banding, in that case you should check that the picture is perfectly banding-free before encoding. Also try not to use stronger setting for the whole video, section filtering is time consuming but safer, because gradfun2db kills details in a very visible way (and adds a grainy looks you may not want in the picture).

Speaking of anime I usually go with gradfun3 (16 bit output) and then encode with 10bit x264, if you really want to eliminate banding imho there aren't really comparable solutions, 8bit x264 just doesn't cut it.
http://screenshotcomparison.com/comparison/107046
Just to link a worst case example.

If you're stuck to 8bit I'd say there are 2 solutions, bloating the filesize with lower crf/higher psy/higher fgo/higher AQ and hope that solves the problem, or using zones (a zone for every section with visible banding).
Depends really on the time available.

Doing a perfect encode from a TS Source is pretty difficult, I'd say a novice should start from something else (DVDs are probably the best learning source) and stay away from then until he really understands what he is really doing.
That or

DGDecode_Mpeg2Source()
TFM()
Tdecimate(mode=1)
Spline36Resize()
*denoise or sharpen if needen (usually not)*
gradfun2db() / Gradfun3()

encode with x264 10bit @crf18

And more or less it's what 90% of the people are doing (at least in Europe and America, japanese tend to abuse warpsharp and unsharpmask,).

[DragooNick]

Most current Anime series probably need IVTC (http://forum.doom9.org/showthread.php?t=163887), not deinterlacing, btw.

If you can tell me the difference? ^^" It's an interlaced source so why not deinterlacing? Or what exactly does IVTC different?

Edit: Sry I dont even know how to handle a ".avsi" file >_<" I'm not exactly sure whether I handle dll's right :S

What kind of anime are you trying to encode?
Going 60fps sounds silly, nowadays Anime TS are usually 24000/1001 with constant patterns, that means that 9 times out of 10 the correct way to process them is
- Source (DGDecode already outputs YV12 video)
- Telecide
- Decimate

(There are a couple of exception to this rule, for example precures usually have 30000/1001 sections, sometimes mixed with 24000/1001 ones)
Sry no Idea what your talking about ^^" but i think I saw something like 30000/1001 while encoding...


Using yadif will probably only make the video jerky and bloat the bitrate, and you should probably use that to increase the quality of the encoded video, so that the banding doesn't appear in the encoded video.

Anyway, gradfun2db(1.2) may not eliminate all the banding, in that case you should check that the picture is perfectly banding-free before encoding. Also try not to use stronger setting for the whole video, section filtering is time consuming but safer, because gradfun2db kills details in a very visible way (and adds a grainy looks you may not want in the picture).

Speaking of anime I usually go with gradfun3 (16 bit output) and then encode with 10bit x264, if you really want to eliminate banding imho there aren't really comparable solutions, 8bit x264 just doesn't cut it.
http://screenshotcomparison.com/comparison/107046
Just to link a worst case example.

If you're stuck to 8bit I'd say there are 2 solutions, bloating the filesize with lower crf/higher psy/higher fgo/higher AQ and hope that solves the problem, or using zones (a zone for every section with visible banding).
Depends really on the time available.

Doing a perfect encode from a TS Source is pretty difficult, I'd say a novice should start from something else (DVDs are probably the best learning source) and stay away from then until he really understands what he is really doing.
That or

DGDecode_Mpeg2Source()
TFM() <--- what filter is this?
Tdecimate(mode=1) <--- what filter is this?
Spline36Resize() <--- what is the difference to LanczosResize? ^^
*denoise or sharpen if needen (usually not)*
gradfun2db() / Gradfun3() <--- where do I get the dll for gradfun3?

encode with x264 10bit @crf18

And more or less it's what 90% of the people are doing (at least in Europe and America, japanese tend to abuse warpsharp and unsharpmask,).

http://screenshotcomparison.com/comparison/116842
http://screenshotcomparison.com/comparison/116841
http://screenshotcomparison.com/comparison/116840

Judging my own screenshots, I'm pretty pleased with the encode I did ^^"

http://screenshotcomparison.com/comparison/117346
here lies the problem, which non of you understood

as you can see this is the preview screen of staxrip,
as I enable the gradfun2db(1.2) filter, I CAN SEE THE DIFFERENCE
But after the encode the Picture looks as if the filter was never used at all, I dont know if I did mess up the adding of the filter,
or where the problem could be that it shows me the effect in preview but not in the encode? :S

I'm thankful for your tries to help but I really dont understand much of your words, I know I could use 10bit but this is a test I got to do in 8bit.

[sneaker_ger]

here lies the problem, which non of you understood.

I understood very well.

[DragooNick]

I understood very well.

I'm sorry but from your answer it didnt seem so ^^"
Since you took the debanding as a problem and not my problem
with the malfunction or missuse of the programm :S

I edited something in the post on top ;)

[sneaker_ger]

The problem is that the noise/dither from the debanding process is very hard to encode for 8 bit x264. After encoding with low bitrate the banding returns. That's why I told you to (1) use ordered dithering (either gradfun3() or flash3kyuu_deband(dither_algo=2)) (2) increase bitrate (a lot, maybe locally via zones) (3) use 10 bit x264 (which you can't/don't want for your test). (4) tune your x264 settings
You might have to do (1), (2) and (4) - not just one of them. Banding is really hard to get rid of.
I provided links to gradfun3 (third post in the first thread), flash3kyuu_deband and x264 recommendations in my first post. In all of them this issue is discussed to some extend.

[DragooNick]

The problem is that the noise/dither from the debanding process is very hard to encode for 8 bit x264. After encoding with low bitrate the banding returns. That's why I told you to (1) use ordered dithering (either gradfun3() or flash3kyuu_deband(dither_algo=2)) (2) increase bitrate (a lot, maybe locally via zones) (3) use 10 bit x264 (which you can't/don't want for your test). (4) tune your x264 settings
You might have to do (1), (2) and (4) - not just one of them. Banding is really hard to get rid of.
I provided links to gradfun3 (third post in the first thread), flash3kyuu_deband and x264 recommendations in my first post. In all of them this issue is discussed to some extend.

Ah ok then I just missunderstood you ^^" Just got confused that the preview showed some success >_<"

Ya the problem I got now is that for (1) I didnt find the dll for gradfun3() and even worse I dont even nearly know how to use flash3kyuu,
I read the txt but most of it is soo far from my knowlegde what it's used for that I feel totaly beaten :/
Or could I use that "flash3kyuu_deband(dither_algo=2)" frase you posted, after I put the dll into the plugin folder?
(2) sry neither know how that is done, nor what or how zones are used

[sneaker_ger]

gradfun3() should be part of the dither package you can download in the thread I mentioned.

Or could I use that "flash3kyuu_deband(dither_algo=2)" frase you posted, after I put the dll into the plugin folder?

Yes, it's that simple. It's an alternative to gradfun2db(), don't use both.

(2) sry neither know how that is done, nor what or how zones are used

Zones are not a must, and would be quite tedious anyways. To increase bitrate, lower CRF.

[DragooNick]

gradfun3() should be part of the dither package you can download in the thread I mentioned.

Sadly there is no gradfun3.dll in that pack :/


Yes, it's that simple. It's an alternative to gradfun2db(), don't use both.
Sure I will try that now!


Zones are not a must, and would be quite tedious anyways. To increase bitrate, lower CRF.
hmm okey ^^"

Thanks for your help and patience with me :]

[sneaker_ger]

Sadly there is no gradfun3.dll in that pack :/

The plugin file names don't have to be the same as the in-script function names.

[DragooNick]

The plugin file names don't have to be the same as the in-script function names.

hmm ok I will try it then ^^"

for the test with flash3kyuu_deband, I got the same problem,
but then did as you said turn up the bitrate and got a result,
so now I 100% got what you ment with it comes back ... lol

sadly I dont think this will help me since I had to go up from crf 18 to 14 and this will be too much for the final filesize :/ Or will the compressability go up too? Probably not hmm? ^^
I already did the 1. encode in slower, so there is just 1 step slower left *g*

Edit: as I try to use gradfun3() it tells me that there is no such function >_<"
I think my method of adding plugins just plainly suck ... ^^" Or I dont know what exactly I do wrong at all :/

[sneaker_ger]

sadly I dont think this will help me since I had to go up from crf 18 to 14 and this will be too much for the final filesize :/ Or will the compressability go up too? Probably not hmm? ^^
I already did the 1. encode in slower, so there is just 1 step slower left *g*

See the third thread I linked for x264 tunings.

Edit: as I try to use gradfun3() it tells me that there is no such function >_<"

IIRC you have to copy all .dll (including those that are in sub-files) and .avsi files to the plug-in folder, except for libfftw3f-3.dll, which has to go into the windows\SysWOW64 (for 64-bit windows) or windows\system32 (for 32-bit windows). Not 100% sure. Ask in that thread if you have problems using gradfun3().

[cretindesalpes]

Edit: as I try to use gradfun3() it tells me that there is no such function >_<"
Unpack all the .dll and .avsi files from the Dither package into your Avisynth\plugin folder. You'll probably need external dependencies like masktools or removegrain, too. Sorry if I sound harsh, but RTFM is usually a good starting point when you cannot make things work at the first try.

[DragooNick]

See the third thread I linked for x264 tunings.

IIRC you have to copy all .dll (including those that are in sub-files) and .avsi files to the plug-in folder, except for libfftw3f-3.dll, which has to go into the windows\SysWOW64 (for 64-bit windows) or windows\system32 (for 32-bit windows). Not 100% sure. Ask in that thread if you have problems using gradfun3().
I will look into it some more.

Then I were not really wrong with what I did ;)

Unpack all the .dll and .avsi files from the Dither package into your Avisynth\plugin folder. You'll probably need external dependencies like masktools or removegrain, too. Sorry if I sound harsh, but RTFM is usually a good starting point when you cannot make things work at the first try.

... :/ I just thought that I could have screwd on that part, as both of your explanations proved that I were doing the things right >_<"
Ya and I also got masktools already. Sadly most of the manuals arent written for total beginners ^^" And already require a lot of knowledge :S
Ok as I try to use gradfun3() I get that message now

Script error: mt_expand does not have a named argument "chroma"
(mt_xxpand_multi.avsi, line 50)
(dither.avsi, line 1223)
(dither.avsi, line 109)
(...\Test\last exile ginyoku no fam 7 temp files\last exile ginyoku no fam 7 - final lower DEBANDING.avs, line 11)

probably because I left the gradfun3() <--- empty?

[cretindesalpes]

Script error: mt_expand does not have a named argument "chroma"
:D

Really, you should at least open the manual, it's not that difficult. One of the first things you'll see, highlighted in flashy yellow, is the minimum version requirement for Masktools…

[DragooNick]

:D

Really, you should at least open the manual, it's not that difficult. One of the first things you'll see, highlighted in flashy yellow, is the minimum version requirement for Masktools…

believe me or not I DID FUCKIN READ THAT O.o But somehow I dled masktools-v2.0a9.zip and thought this was more actual than masktools-v2.0a48.zip,
but please dont ask me why XD Since I actually face palmed me pretty hard as I checked that again ... thanks ^^"

Okey I got it to work but same as with gradfun2db I can see the great difference in the preview but not in the encode,
so I still have to turn up the bitrate or adjust the settings like discussed here? http://forum.doom9.org/showthread.php?t=164352&page=2 (but there can occur other problems though ^^")
Or what would be your tip for my problem? :] @cretindesalpes?

[mp3dom]

Gradfun2db, Gradfun3, flash3kyuu... all have the same goal of attenuate/fix the color-banding problem. They work differently but the end result is pretty much comparable under certain aspects. You have fixed the banding problem at 'source' level so the filters are ok. You can set the filters to use an ordered dithering (Gradfun3/flash3kyuu only) which is better for the encoder but worst for the eyes but the weak part of the ring is that you want to stay at low bitrate. When you have low bitrate, the dithering is the first thing that an encoder throw away because dither means very little/subtle noise on very similar color gradation. On low rates, it's preferable to keep real details rather than subtle grain. You can change settings on x264 to try to keep dithering more, but since the bitrate is "fixed", the more bitrate you use to keep dithering, the less bitrate you'll have for complex parts.
As someone already told you, the only way you have to keep dithering AND details is to increase the bitrate or use a 10bit x264 version.

[DragooNick]

The question is DO I have to keep the dithering to get the banding fixed? And why are they so useless at crf18? sneaker_ger said because that it returns because of low bitrate,
still dont get it why the preview option of staxrip shows me a good result and it just returns then if I actualy encode it >_<"

http://jumbofiles.com/j66by3y3jmix <-- I've cut out the problem part of the ts (just 34mb) if anyone wants to look at it or mind telling me a good way to encode this in 8bit with a good quality and not too huge filesize ^^"

[mp3dom]

Dithering is what you need to preserve to avoid banding. If you don't keep the dithering (added noise) you return back to the main problem. Filters are made to fix problems at source/master level, not encoding level. The final (encoded) result is poor because the added noise gets 'removed' at compression stage due to the low bitrate (H.264/AVC is ->lossy<- at least with that profile/level so less bitrate means more compression and lower quality). To keep dithering you need huge amount of bitrate (if you're encoding at 8bit). Just as a reference note, even bluray can suffer from banding and they are encoded up to 40 Mbps. You can't encode a 720p video at an average bitrate of 1 Mbps and pretends to have it indistinguishable from the source. Raise the bitrate and you'll see dithering preserved... or simply switch to a 10bit profile.

[DragooNick]

Hmm ok I do think I got the difference now :/
Whats your opinion on that http://screenshotcomparison.com/comparison/117411 now?

[AzraelNewtype]

If you can tell me the difference? ^^" It's an interlaced source so why not deinterlacing? Or what exactly does IVTC different?


To be perfectly honest, the main problem you posted the thread about aside, if you don't know the answer to this you probably shouldn't be encoding at all, with or without dithered grain added.