Need help with settings for anime encoding...

[Makaveli84]

I'm ripping anime episodes from a retail DVD (SD quality/res obviously) and encoding them with x264. The source is not quite to my liking, and has some problems that I'm trying to address with my encoding settings. Specifically, I'm looking to flatten out the picture to some degree (i.e retain less detail sharpness and noise) while emphasizing lines/edges and making them sharper and more pronounced, but with keeping those lines/edges as clean as possible (no "zigzagging" for me please!). I'm not sure how to deviate from the animation tuning and some other general settings, so I'm open to some suggestions. Here's what I've got so far:

- subme=7 (I usually set this to 9 for max details, but maybe a 7 value better suits my needs in this case).

- merange=16 (I usually set this to 24 or even 32 but since I'm looking to get a smoother picture, maybe I shouldn't go beyond 16).

- psy-rd=0.2 (Again, lowering it from the default 0.4 for less details).

- aq-strenth=??? (0.6, more, less? What yields sharper and more pronounced edges while keeping them clean and noise free?)

- deblock=1:1 (I'm reluctant to go higher since I'm applying an indiscriminate 'blend' deinterlacing filter to start with).


Any thoughts? Thanks...

[AMED]

It seems odd that you are trying to clean\repair the image by tweaking x264 settings.

Wouldn't an Avisynth plugin be more geared to do what your asking? Probably best to post in the Avisynth section with a small sample of the video your trying to fix.

[Makaveli84]

Yes, well, thank you AMED, but that's not very helpful. It doesn't matter that there are "better" ways to do it. I'm looking for help/advice on how to tweak x264 settings for such a scenario in general. I tend to agree with you regarding Avisynth but that's beyond the point. Let's just assume that it's an academic/hypothetical question. :)

[foxyshadis]

People tend to not waste their time with hypothetical questions, especially when numerous better options exist. If you're using ffmpeg, you should be using a video filter, either smartblur or hqdn3d (or both). Both can easily give that look, while x264 cannot. It's odd that you're using psy-rd or aq at all, though, since those specifically keep more noise in flat areas.

[Makaveli84]

People tend to not waste their time [...]I wouldn't want to waste anyone's time. My question was simply about which x264 settings/values affect some aspects of the output. Regardless of why I'm asking, people who have this information can easily answer with things like "lower psy-rd" or "lower aq" or "no psy-rd" or "no aq", etc...


especially when numerous better options exist.I have compelling reasons that make me use a minimalist approach for this task.


If you're using ffmpeg [...]I am.


you should be using a video filter, either smartblur or hqdn3d (or both). Both can easily give that look, while x264 cannot.Let me explain: I don't do much ripping and this is my first time dealing with an interlaced source. After reading about it and testing out various deinterlacing filters (yadif, w3fdif and others), the one that worked best for my source was a blending filter (everything else causes wavy/dancing background). This deinterlacing filter also blurs out the picture to a certain degree, so I'm reluctant to apply more blurring. I'm actually quite set with blend filter + low (or no) psy-rd/aq. Only problem is that edges and lines are also losing their sharpness. Now having previously read other doom9 threads dealing with x264 settings for anime encoding, I was under the impression that it was possible to fine tune some settings to achieve some contrast between line/edge sharpness and flat surface blurring. Maybe I got it wrong when I thought that certain x264 settings can enhance lines and edges?


It's odd [...]That's my 2nd "odd" in this thread. Can I get one more? Anyone? :rolleyes:


that you're using psy-rd or aq at all, though, since those specifically keep more noise in flat areas.Thank you. I'm glad I kept reading. That's the kind of advice that this oddball (:rolleyes:) was looking for (particularly about turning off aq altogether, will try it out).

[poisondeathray]

Let me explain: I don't do much ripping and this is my first time dealing with an interlaced source. After reading about it and testing out various deinterlacing filters (yadif, w3fdif and others), the one that worked best for my source was a blending filter (everything else causes wavy/dancing background). This deinterlacing filter also blurs out the picture to a certain degree, so I'm reluctant to apply more blurring. I'm actually quite set with blend filter + low (or no) psy-rd/aq. Only problem is that edges and lines are also losing their sharpness. Now having previously read other doom9 threads dealing with x264 settings for anime encoding, I was under the impression that it was possible to fine tune some settings to achieve some contrast between line/edge sharpness and flat surface blurring. Maybe I got it wrong when I thought that certain x264 settings can enhance lines and edges?



To reiterate - you're probably doing this the wrong way.

99.999999999999999999999% certainty an anime DVD won't have interlaced content. You probably want to IVTC, not deinterlace . Blend deinterlacing almost worst thing you can do. You're going to end up with jerky and blurry playback. x264 settings are the least of your worries

ffmpeg can IVTC now with -vf fieldmatch and decimate filters - but it still has problems compared to the avisynth solutions for some reason

[Makaveli84]

To reiterate - you're probably doing this the wrong way.

99.999999999999999999999% certainty an anime DVD won't have interlaced content. You probably want to IVTC, not deinterlace . Blend deinterlacing almost worst thing you can do. You're going to end up with jerky and blurry playback. x264 settings are the least of your worries

ffmpeg can IVTC now with -vf fieldmatch and decimate filters - but it still has problems compared to the avisynth solutions for some reason

IVTC = Inverse Telecine? My research had initially led me to "-vf fieldmatch and decimate" but I hit a wall reading this (https://www.ffmpeg.org/ffmpeg-filters.html#fieldmatch). This is surely not the right forum for this, but since we're at it, can you provide an example or sample ffmpeg command for the fieldmatch + decimate filters?

[poisondeathray]

Clarify is this NTSC DVD ? or PAL DVD ? IVTC doesn't apply to "PAL" region DVD's


Currently, what works for IVTC in ffmpeg (most of the time) is



-vf "fps=30000/1001,fieldmatch,yadif=deint=interlaced,decimate"



Yadif is only used on the residual combed frames after field matching . It still "barfs" on some samples and DVD's, but so far it's better than the old -vf pullup filter (the old method of IVTC in ffmpeg / mencoder / mplayer)

[Makaveli84]

Clarify is this NTSC DVD ? or PAL DVD ? IVTC doesn't apply to "PAL" region DVD's
Sorry, thought I had mentioned it in my first post. It's NTSC, 29.97 fps, 720x480, 4:3 AR.



Currently, what works for IVTC in ffmpeg (most of the time) is



-vf "fps=30000/1001,fieldmatch,yadif=deint=interlaced,decimate"



Yadif is only used on the residual combed frames after field matching . It still "barfs" on some samples and DVD's, but so far it's better than the old -vf pullup filter (the old method of IVTC in ffmpeg / mencoder / mplayer)

After my last post, I dug deeper and I'm glad to see that I got it partially right. I had just ran a sample encode test using:
-vf "fieldmatch,yadif=deint=interlaced,decimate"

Should I explicitly add the fps part? Why is it needed?

[poisondeathray]

fps isn't required, but sometimes ffmpeg gets the source framerate slightly off , thus final file has slightly off frame rate - you're safer with it than without it

[foxyshadis]

Let me explain: I don't do much ripping and this is my first time dealing with an interlaced source. After reading about it and testing out various deinterlacing filters (yadif, w3fdif and others), the one that worked best for my source was a blending filter (everything else causes wavy/dancing background). This deinterlacing filter also blurs out the picture to a certain degree, so I'm reluctant to apply more blurring. I'm actually quite set with blend filter + low (or no) psy-rd/aq. Only problem is that edges and lines are also losing their sharpness. Now having previously read other doom9 threads dealing with x264 settings for anime encoding, I was under the impression that it was possible to fine tune some settings to achieve some contrast between line/edge sharpness and flat surface blurring. Maybe I got it wrong when I thought that certain x264 settings can enhance lines and edges?

It doesn't matter what you're afraid of if you didn't even try. You asked for smoother interiors leaving sharp lines, that's exactly what smartblur and hqdn3d do; even if the lines aren't perfectly sharp, it still does a good job of leaving them alone. hqdn3d can quickly improve mildly noisy video, with the trade-off that dark scenes can get a bit muddier, so try it with defaults as a test. If there's a ton of detail, like Ghibli or Gonzo, you'll probably get an "oil painting" effect, but you'd get something even more uneven if you tried to tune x264 that way. x264 will give you detail pumping, with good frames and muddy frames mixed, if you tried to strip away too much detail with it.

The only way to make lines sharper with x264 is to raise me and subme. Note that you may need to trade a bit of aq and psy-rd if you make it too smooth and don't use the 10-bit encoding, or you can end up with bad banding.

[Makaveli84]

Clarify is this NTSC DVD ? or PAL DVD ? IVTC doesn't apply to "PAL" region DVD's


Currently, what works for IVTC in ffmpeg (most of the time) is



-vf "fps=30000/1001,fieldmatch,yadif=deint=interlaced,decimate"



Yadif is only used on the residual combed frames after field matching . It still "barfs" on some samples and DVD's, but so far it's better than the old -vf pullup filter (the old method of IVTC in ffmpeg / mencoder / mplayer)

I'm having some issues with this filter chain, and I can't seem to find a suitable ffmpeg sub-forum on Doom9, so I'll continue posting here even if this is not about x264 anymore.
Problem is that "fieldmatch,yadif=deint=interlaced,decimate" leaves a few combed frames scattered here and there that are very noticeable. Some of these combed frames are getting smoothed down with "smartblur,hqdn3d", but some still remain even after blurring. Apparently, those frames are not marked as interlaced so yadif is not taking them into account. Any ideas?

[Makaveli84]

It doesn't matter what you're afraid of if you didn't even try. You asked for smoother interiors leaving sharp lines, that's exactly what smartblur and hqdn3d do;Again, you missed an important aspect which is that I was using the "linearblend" deinterlacing filter. So you're right, but only partially so. I actually did try, and linearblend deinterlacing + smartblur and/or hqdn3d looked horrible. Applying "smartblur,hqdn3d" only became an option after poisondeathray pointed out that I should be using IVTC rather than blend deinterlacing.


even if the lines aren't perfectly sharp, it still does a good job of leaving them alone. hqdn3d can quickly improve mildly noisy video, with the trade-off that dark scenes can get a bit muddier, so try it with defaults as a test.After numerous trials, I think I'm good with "smartblur=ls=0.75:lt=15,hqdn3d". This does a good enough job and allows me to follow with an "unsharp" sharpening filter that enhances lines and edges while hardly reintroducing any of the noise that was previously cleansed out. I would've preferred a dedicated edge sharpener + anti-aliasing, but it doesn't look like ffmpeg has any such filters.


The only way to make lines sharper with x264 is to raise me and subme. Note that you may need to trade a bit of aq and psy-rd if you make it too smooth and don't use the 10-bit encoding, or you can end up with bad banding.What would you suggest? I'll be trying out various combinations, but a good starting point will definitely make my life easier...

[poisondeathray]

I'm having some issues with this filter chain, and I can't seem to find a suitable ffmpeg sub-forum on Doom9, so I'll continue posting here even if this is not about x264 anymore.
Problem is that "fieldmatch,yadif=deint=interlaced,decimate" leaves a few combed frames scattered here and there that are very noticeable. Some of these combed frames are getting smoothed down with "smartblur,hqdn3d", but some still remain even after blurring. Apparently, those frames are not marked as interlaced so yadif is not taking them into account. Any ideas?

Try reducing the combing threshold, or MI

In ffmpeg, they are cthresh, combpel

https://www.ffmpeg.org/ffmpeg-filters.html#toc-fieldmatch

[fvisagie]

I can't seem to find a suitable ffmpeg sub-forum on Doom9

You can also try the ffmpeg users' mailing list via ffmpeg.org.

[Asmodian]

I can't seem to find a suitable ffmpeg sub-forum on Doom9, so I'll continue posting here even if this is not about x264 anymore.

You can also try the ffmpeg users' mailing list via ffmpeg.org.

This sub-forum is probably ok as well. It is for H.264 not x264 exclusively. ffmpeg is a good H.264 encoder. :)

[huhn]

This sub-forum is probably ok as well. It is for H.264 not x264 exclusively. ffmpeg is a good H.264 encoder. :)

the h264 encoder in ffmpeg is x264.
https://trac.ffmpeg.org/wiki/Encode/H.264

[Asmodian]

Right, so asking how to use ffmpeg in our H.264 encoding forum is reasonable. :)