Possible to reach visually transparent encodes of anime with 35-40% compression?

[N'Cha]

Hi

I have a lot of anime blu-ray, and using discs is a pain in the as* (personal opinion) + I like to just pick my favorite scenes (fighting anime) sometimes and watch them. For movies (live action I mean, not anime) BR, I just remux them.
But for my anime, I quickly realized that it would take a ton of storage (like tens of Tb) if I were to just remux them with no encoding, because I have anime series that are tens or hundreds of episodes (bleach, DBZ etc. for example).
Obviously I want to keep a very good image and the lossless audio.

My question is quite simple:
Despite the fact that encoding would result in losing, on paper, quality, is it though possible to achieve a visually transparent quality of my anime BR encodes (I guess ther's maybe a limit of what our eyes can perceive and the quality loss they can accept? Especially for less demanding "images" like anime (as opposed to live action movies)... all that compressing the files for like 35-40%? Encoding time is not really a concern (though i don't want to spend 4 hours encoding a single episode, 1 or 1h30 for a 22 min episode is my limit), as I have 2 PC and one of them have a 5950X and 64gb ram, and the other has a 7950X and 96gb of ram).
Oh and it would be to watch on a 85" screen...so i'm not interested in doing mini encodes and alike.
If not, what's the best type of quality I can hope knowing what I just said? Like 90% of the visual quality? (which is acceptable too)

Thanks in advance

[rwill]

Just try it. Do a CRF 18 preset slower encode of your average content and check if you are Ok with the quality, speed and rate.

Regarding your 85" screen, screen size is not relevant, view distance to screen size is relevant. If you are sitting really close you will start to see things even with a BR.

[N'Cha]

i was hoping answers beyond the "try it" (or else I wouldn't have created a post, I know I could try it. And what works with an anime and a specific episode, might not give me the same satisfaction with another episode or anime). My question is more geared toward "is it possible to do visually transparent while compressing roughly 30-40%.
About my view distance I like too be immersed so i'm like 3 meters (roughly 9.8 feet) from the screen (small living room anyway).

[RanmaCanada]

There is an whole topic about it in the forums. Yes it's old, but not much has changed since then sadly. But to get that low, you will probably have to move to AV1, as it compresses anime really well. I have no experience with VVC.

But honestly, build a nice media server and then put all your content on it in raw format. You'll be happier, and will use less money building it then you would in regards to electricity costs and time to encode the files.

[microchip8]

There is no silver bullet. It is nearly impossible to recommend settings for 35-40% compression as there are too many variables across different content. The best you can do is trial and error of various settings and find a "sweet spot" that is good enough but not perfect or spot on. You'd still have content that "doesn't fit", though.

[benwaggoner]

35-40% for animation content from Blu-ray is pretty straightforward, as Blu-ray's H.264 1 sec GOP and 3 B-frame max are pretty limiting compared to what long-GOP HEVC can do.
Anime can benefit from some different settings, as there is a lot of spatial redundancy and some extreme temporal redundancy.

Try something like:
--preset slower --tskip --tskip-fast --tu-inter-depth 4 --tu-intra-depth 4 --limit-tu --nr-inter 150 --selective-sao 2

With the big flat areas and sharp edges, the combination of tskip and min 4x4 TU size can squeeze a lot of bits out. And some temporal deadzoning as denoising can help remove some unintended temporal noise

[N'Cha]

Thank i'm gonna try these settings as you confirm that 30-40% is reachable for a transparent encode :D
Will try with CRF 16, 15 and 14

[N'Cha]

Bframes 12 is enough for anime content or could I really benefit from bframes 16?

Was thinking either --bframes 12 --rc-lookahead 100 or --bframes 16 --rc-lookahead 120

And not sure if I should let the encoder set its level automatically (probably gonna end up being 5.0 if I let it on auto) or if I should set it to 4.1 knowing I aim at very high quality encode

And should i set --no-open-gop or --open-gop for anime? (it's to be palyed from my HTPC or my shield or plex, not sure for the latter)

What do you guys think?

Thanks

[excellentswordfight]

Bframes 12 is enough for anime content or could I really benefit from bframes 16?

Was thinking either --bframes 12 --rc-lookahead 100 or --bframes 16 --rc-lookahead 120

And not sure if I should let the encoder set its level automatically (probably gonna end up being 5.0 if I let it on auto) or if I should set it to 4.1 knowing I aim at very high quality encode

And should i set --no-open-gop or --open-gop for anime? (it's to be palyed from my HTPC or my shield or plex, not sure for the latter)

What do you guys think?

Thanks
For anime I would say use atleast 8 bframes, but as I havnt used much above that I cant tell how much anime gains with x265 for that. But as for lookahead there is no relationship to your number of bframes, you can use up to your keyint, but at some point you will get very little extra benefits from it. I usually use 2-4s.

The reason why you would like to set level manually is cause that will set the proper vbv limits for that level, without specifing it will be unrestrictive. This can cause some playback issues on some devices or if you are doing in-home streaming on wifi etc (and this is why I always use it).

The smaller the gop sice the bigger is the benefit of open-gop, so if you are going for a say 10s keyint, I dont think it will make a big differnce eitherway. But i have never seen any playback issues of using open-gop with hevc so I always keep it on (unless I encode for fixed gop streaming cases) as it is benifitial for compression.

This is what I use for 1080p24 bluray anime titles, should be a decent starting point:

--preset slow --profile main10 --level-idc 41 --keyint 240 --min-keyint 24 --rc-lookahead 96 --bframes 8 --psy-rd 1 --tskip --aq-mode 3 --selective-sao 2

if you want it to be "visually lossless", you can disable sao and increase psy-rd, but then you might not reach your compression goals. So i would play a bit with different crf, psy-rd and selective-sao values until you find what you are looking for.

[N'Cha]

For anime I would say use atleast 8 bframes, but as I havnt used much above that I cant tell how much anime gains with x265 for that. But as for lookahead there is no relationship to your number of bframes, you can use up to your keyint, but at some point you will get very little extra benefits from it. I usually use 2-4s.

The reason why you would like to set level manually is cause that will set the proper vbv limits for that level, without specifing it will be unrestrictive. This can cause some playback issues on some devices or if you are doing in-home streaming on wifi etc (and this is why I always use it).

The smaller the gop sice the bigger is the benefit of open-gop, so if you are going for a say 10s keyint, I dont think it will make a big differnce eitherway. But i have never seen any playback issues of using open-gop with hevc so I always keep it on (unless I encode for fixed gop streaming cases) as it is benifitial for compression.

This is what I use for 1080p24 bluray anime titles, should be a decent starting point:

--preset slow --profile main10 --level-idc 41 --keyint 240 --min-keyint 24 --rc-lookahead 96 --bframes 8 --psy-rd 1 --tskip --aq-mode 3 --selective-sao 2

if you want it to be "visually lossless", you can disable sao and increase psy-rd, but then you might not reach your compression goals. So i would play a bit with different crf, psy-rd and selective-sao values until you find what you are looking for.Wouldn't it be better to set level 5.1 main tier due to higher bitrate supported? because with 4.1 if not going for high tier, there's not enough bitrate, but if going for 4.1 high tier, which has enough bitrate, is there risks for it to be less compatible than 5.1 main tier nowadays? Well as far as it can be read by MPC-HC, nvidia shield and maybe plex, on potentially a tablet (for the "in plane" times) i'm good with any as far as it can support 40Mb/s (so 4.0, 4.1 or 5.0 high tier or 5.1 main tier) but maybe some encoding parameters are only available/applicable with certain encoders levels and/or tier?

[excellentswordfight]

Wouldn't it be better to set level 5.1 main tier due to higher bitrate supported? because with 4.1 if not going for high tier, there's not enough bitrate, but if going for 4.1 high tier, which has enough bitrate, is there risks for it to be less compatible than 5.1 main tier nowadays?
In this case x265 will default to high tier, so that is 50Mbps for 4.1 so that is more than enough if you wanna bring bluray titles <10Mbps, and is actually higher than 5.1 main tier.

But you are right, you can absolutly do that instead as its so overwhelmingly common for hevc decoderc now days to support 5.1. But I still use the practice of always go for the lowest level as long as the framesize and max bitrate matches my target.

[N'Cha]

I think I'm gonna set pick 4.1 main (for 20Mb/s max) knowing that 20Mb/s is the most I want for my encodes)

[N'Cha]

Problem: any profile i set/try (4.1, 5.0, 5.1) with main tier (even tried with high tier), it says "x265 [error]: frame rate is out of range for specified level" only if i remove the level i set to leave it on auto, it works...

I also get "deprecated pixel format used" for using --pix_fmt yuv420p10le

EDIT: Strangely, if I remove --map 0 (which I need as i want to pass all audio and subtitles) it now works... but ineed to pass all streams with --map 0 so not a solution

[excellentswordfight]

Problem: any profile i set/try (4.1, 5.0, 5.1) with main tier (even tried with high tier), it says "x265 [error]: frame rate is out of range for specified level" only if i remove the level i set to leave it on auto, it works...

I also get "deprecated pixel format used" for using --pix_fmt yuv420p10le

EDIT: Strangely, if I remove --map 0 (which I need as i want to pass all audio and subtitles) it now works... but ineed to pass all streams with --map 0 so not a solution
What does your workflow looks like? what encoder-tools are you using? Cause if you are piping from ffmpeg to x265 why would you pass anything other than your main video track? You cant process the other stuff anyway with x265, ir are you doing everthing in ffmpeg directly?

If piping input from ffmpeg to x265 this works just fine for 1080p24 8bit:

"ffmpeg.exe" -i "input" -an -f yuv4mpegpipe -strict -1 - | "x265.exe" --y4m --profile main10 --level-idc 41 ...

You dont actually need to convert it to 10bit before feeding x265, if you set profile main10 x265 will handle the conversion internally.

[N'Cha]

I found the problem! There is an image embedded in the MKV (i kept the "attachment picture, i thought it was used to display automatically a thumbnail) on stream #0:5. That's why when I remove -map 0 it worked

How can I copy all the stream 0 (the video, all audios and subs) but remove only the attachement image? by default I used -map 0

I tested with mkvtoolnix removing the iamge, and now I get no error and the encoding starts, but a ffmpeg command would be more practical


Stream #0:5: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 640x360, 90k tbr, 90k tbn (attached pic)
Metadata:
filename : cover.jpg
mimetype : image/jpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> hevc (libx265))
Stream #0:1 -> #0:1 (copy)
Stream #0:2 -> #0:2 (copy)
Stream #0:3 -> #0:3 (copy)
Stream #0:4 -> #0:4 (copy)
Stream #0:5 -> #0:5 (mjpeg (native) -> hevc (libx265))
Press [q] to stop, [?] for help
[swscaler @ 0000020394192700] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000203943c0b80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000203942b5800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000203943c6a80] deprecated pixel format used, make sure you did set range correctly
x265 [info]: HEVC encoder version 3.5+96-9c9ab68fc
x265 [info]: build info [Windows][GCC 12.2.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [error]: frame rate is out of range for specified level
[libx265 @ 0000020392748800] Cannot open libx265 encoder.
[vost#0:5/libx265 @ 0000020393090cc0] Error initializing output stream: Error while opening encoder for output stream #0:5 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

[excellentswordfight]

I found the problem! There is an image embedded in the MKV (i kept the "attachment picture, i thought it was used to display automatically a thumbnail) on stream #0:5. That's why when I remove -map 0 it worked

How can I copy all the stream 0 (the video, all audios and subs) but remove only the attachement image? by default I used -map 0

I tested with mkvtoolnix removing the iamge, and now I get no error and the encoding starts, but a ffmpeg command would be more practical


Stream #0:5: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 640x360, 90k tbr, 90k tbn (attached pic)
Metadata:
filename : cover.jpg
mimetype : image/jpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> hevc (libx265))
Stream #0:1 -> #0:1 (copy)
Stream #0:2 -> #0:2 (copy)
Stream #0:3 -> #0:3 (copy)
Stream #0:4 -> #0:4 (copy)
Stream #0:5 -> #0:5 (mjpeg (native) -> hevc (libx265))
Press [q] to stop, [?] for help
[swscaler @ 0000020394192700] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000203943c0b80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000203942b5800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000203943c6a80] deprecated pixel format used, make sure you did set range correctly
x265 [info]: HEVC encoder version 3.5+96-9c9ab68fc
x265 [info]: build info [Windows][GCC 12.2.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [error]: frame rate is out of range for specified level
[libx265 @ 0000020392748800] Cannot open libx265 encoder.
[vost#0:5/libx265 @ 0000020393090cc0] Error initializing output stream: Error while opening encoder for output stream #0:5 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!
In this case -map 0 -map -0:5 (the "-" before 0:5 is exluding the specified stream). Otherwise you can use -map 0:0 -map 0:1 etc to select the streams you wanna keep.

[N'Cha]

In this case -map 0 -map -0:5 (the "-" before 0:5 is excluding the specified stream)I also tried "-map 0 -map -0:t" As "t" is for attachment I just found out...but it doesn't work, it doesn'tp ass the verification but indeed with -0:5 it works. Thank you

What the purpose of attachement? I kept the attachment on all remuxes I did, i thought it was for displaying the image of the episode or movie on plex, kodi etc. Whoops

[excellentswordfight]

I also tried "-map 0 -map -0:t" As "t" is for attachment I just found out...but it doesn't work, it doesn'tp ass the verification but indeed with -0:5 it works. Thank you

What the purpose of attachement? I kept the attachment on all remuxes I did, i thought it was for displaying the image of the episode or movie on plex, kodi etc. Whoops
yeah the issue you are encountering here is cause its mjpeg so it gets identified as a videotrack by ffmpeg.

If your target is plex, I woudlnt care for any type of image attachments etc, as plex has that feature built in and will populate covers etc automatically. Not sure how great it is with anime, but in my experience it does a great job with that in general.

[N'Cha]

well my target is my PC (it can read anything) but i also plan to buy a nvidia shield in a few months and to maybe use plex (not sure, as it would be hard to leave MPC-HC + Madvr)

Thanks for thelp

p.s: I picket "level 5.0" (main) eventually, for the 25Mb/s and I guess nvidia shield can mange that without problem + i tested on my 3 years phone and it works + on USB on my 5 years old TV it works too, so I guess nowadays, it must be rare the devices that don't read HEVC and/or high than 4.1 level

[microchip8]

well my target is my PC (it can read anything) but i also plan to buy a nvidia shield in a few months and to maybe use plex (not sure, as it would be hard to leave MPC-HC + Madvr)

Thanks for thelp

p.s: I picket "level 5.0" (main) eventually, for the 25Mb/s and I guess nvidia shield can mange that without problem + i tested on my 3 years phone and it works + on USB on my 5 years old TV it works too, so I guess nowadays, it must be rare the devices that don't read HEVC and/or high than 4.1 level

You should leave the Levels decision to the encoder. It's "smart" enough to pick one based on settings and input.

I have great experience with Kodi on the NV Shield. It can basically play anything, including HDR10. I haven't tried DoVi and HDR10+, though.

[Boulder]

well my target is my PC (it can read anything) but i also plan to buy a nvidia shield in a few months and to maybe use plex (not sure, as it would be hard to leave MPC-HC + Madvr)

If you are looking to use the thing only for Kodi, I can recommend a Homatics Box R 4K Plus. With a recent firmware and CoreELEC 20.3, all the UHD thingies (HDR10(+), DoVi) should work out of the box just fine.

[benwaggoner]

Wouldn't it be better to set level 5.1 main tier due to higher bitrate supported? because with 4.1 if not going for high tier, there's not enough bitrate, but if going for 4.1 high tier, which has enough bitrate, is there risks for it to be less compatible than 5.1 main tier nowadays? Well as far as it can be read by MPC-HC, nvidia shield and maybe plex, on potentially a tablet (for the "in plane" times) i'm good with any as far as it can support 40Mb/s (so 4.0, 4.1 or 5.0 high tier or 5.1 main tier) but maybe some encoding parameters are only available/applicable with certain encoders levels and/or tier?
For anime content, bitrate doesn't need to go that high anyway, so I doubt there would be visible benefit going beyond 4.1.

The HEVC profile/level max bitrates are pretty reasonable overall. Level 5.0's 25 Mbps isn't quite high enough for 4K with a lot of fine film grain, but the others are fine by me.

[N'Cha]

On a test CR 14 gives me a bitrate of 18,5 Mb/s compared to the like 38MB/s of the original file (and the file goes from 6,79Gb to 3,26Gb despite the mix of CRF14, presets slower and the command line you told me were important for anime (tskip, tu-inter 4 etc.) with 56minutes of encoding per 24min episode, so very satisfied with the compression of 45% despite very reserving settings!). With CRF 14.5 I get like 17,2Mb/s and with CRF 15 I get 16 Mb/s . I was concerned that both crf 15 and 14.5 was not enough for anime, compared to that very high bitrate of the original episode (even if it's more efficient). Isn't high bitrate important for at least stuff like banding? I thought that the lower the bitrate is, the more chance there is to have bitrate (on the original files I mean)

[N'Cha]

If you are looking to use the thing only for Kodi, I can recommend a Homatics Box R 4K Plus. With a recent firmware and CoreELEC 20.3, all the UHD thingies (HDR10(+), DoVi) should work out of the box just fine.I'll check this product thanks

[benwaggoner]

On a test CR 14 gives me a bitrate of 18,5 Mb/s compared to the like 38MB/s of the original file (and the file goes from 6,79Gb to 3,26Gb despite the mix of CRF14, presets slower and the command line you told me were important for anime (tskip, tu-inter 4 etc.) with 56minutes of encoding per 24min episode, so very satisfied with the compression of 45% despite very reserving settings!). With CRF 14.5 I get like 17,2Mb/s and with CRF 15 I get 16 Mb/s . I was concerned that both crf 15 and 14.5 was not enough for anime, compared to that very high bitrate of the original episode (even if it's more efficient). Isn't high bitrate important for at least stuff like banding? I thought that the lower the bitrate is, the more chance there is to have bitrate (on the original files I mean)
CRF of 14 is probably overkill, yeah, with the other anime-oriented features in use. CRF 16 is where I'd start, and you can pick the highest value that remains visually transparent for you. And encoding in 10-bit can help reduce banding without any bitrate increase.

The source frame rate isn't really material here. H.264 can encode anime quite efficiently as well, so the source may be pretty close to lossless. If you do a --csv-log-level 2, you'll get per-frame quality metrics. There's likely a frame QP value beyond which you start to get visible degradation, so making sure you aren't getting QPs above that threshold much on "interesting" (non-black, non-static) frames is a better way to find your optimum CRF value. And CRF roughly maps to QP.

To save some more bits, increasing the --pbratio and perhaps --ipratio can help. Lots of anime b-frames are pretty much just duplicate frames except for perhaps noise. --frame-dup should be great for anime if you're using a post 3.5 build, but I've not been able to get it to do much in my testing.