Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th April 2021, 11:57   #1  |  Link
Mitra
Registered User
 
Join Date: Mar 2021
Posts: 6
What are the best settings/recommendations to decrease Generation loss?

What are the best settings/recommendations to decrease Generation loss?


Please have a look at this page.

Last edited by Mitra; 13th April 2021 at 14:31. Reason: the title is incomplete
Mitra is offline   Reply With Quote
Old 8th April 2021, 12:39   #2  |  Link
butterw2
Registered User
 
Join Date: Jun 2020
Posts: 212
Go lossless.

You should host + embed your image rather than wait for approval.
butterw2 is offline   Reply With Quote
Old 8th April 2021, 16:33   #3  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Metropolitan City of Milan, Italy
Posts: 1,767
I think that lossless encoding is still way too large for serious use other than in professional sector.
It doesn't matter if you're using Lagarith, HuffYUV, UtVideo, FFV1, MagicYUV or even H.264/H.265 lossless, it will still require lots of bits to encode and far more than what we can handle right now.
Heck, even in professional environments, although the source is lossless (generally a series of .tiff) it's then encoded as a very high bitrate mezzanine file and stored as lossy, so... it's just a matter of "how lossy" it's gonna be.
Now, on to your particular example.
You posted a link to Wikipedia that shows this:




This is a JPEG file encoded as lossy at constant bitrate with a very high quantization. What this shows is that every time you re-encode something, the quality will be affected as something is gonna be lost. The example, however, shows a pretty extreme case scenario. What happens in that case is that the image is analyzed by the DCT (Discrete Cosine Transform) to get values out of it. What the encoder is gonna do then is to assign a quantization pass (a number that goes from 0 to 99) based on the bitrate it has to decide which frequencies are more important and which ones are less important. The closer they are to 0 the more they are important and the closer they are to 99 the less they are important according to the encoder to the human eye. Therefore, if a quantization pass of, let's say, 20, is selected, everything AFTER 20 is gonna be cut off and everything before 20 is gonna be preserved. All nice and dandy, right? No! Here is the problem: what happens is a quantization error, what it means is that once the values have been quantized and the encode happened, the resulting image has "artifacts" (of course, since we have cut off something) but those artifacts are made of high frequencies. What does it mean? Well, it means that, after the first encode, when the encoder is shown the already encoded image as source and proceeds to encode it a second time it will still perform the same operations as before, HOWEVER what the DCT (Discrete Cosine Transform) shows this time is something peculiar: those artifacts introduced by the former encoding appear as high frequencies values which the encoder thinks they're part of the image and therefore it will try to preserve them. How? Well, to preserve something you have to assign bits to it, but the bitrate is constant, therefore it will assign more bits to those high frequency artifacts by taking them from other less important values (which are instead really part of the image). So what happens then? Well, you guessed it, those parts of the image create other artifacts and the resulting image is worse and this cycle repeats over and over again 'till all you're left with is a meaningless series of uncorrelated high frequency artifacts.

So... are we doomed? Will all our dearest family memories fade away in time in a lossy nonsense due to quantization artifacts?

The answer is: No.

The example above is very extreme and it's something that is NOT gonna happen in real life for some very simple reason:

1) Encoders are getting better and better not to provide those high frequency spikes as artifacts (take x262 vs x265 at the same bitrate for instance, if you analyze the spectrum it will NOT show anything remotely as bad).

2) CRF can help tackling the issue a lot. It's been a while since encoders were limited to constant quantization or bitrate and the invention of CRF tackles this. Sure, it will still be lossy and sure, you'll still lose something, but CRF will essentially be like telling the encoder: "Hey, I want this quality everywhere, feel free to shoot the bitrate high as long as you don't hit the maxrate/buffersize", which greatly improved things over constant quantization encodes from 2002 or so.



I gotta say, though, that sure, from a mathematical point of view, every time you re-encode a file and this encode is not lossless you're gonna lose something, however it's just a matter of "what" and "how much" you're gonna lose. Metrics like PSNR, SSIM and (to some user-and-not-broadcast-related extent VMAF) can help you "check" how much you're losing but it's very unlikely to end up with something like the very last picture at the very bottom.


Now, on to the XAVC part, since you seem to be so much interested in that.
The peculiarity of XAVC is that, although it's lossy (so you're losing something) each and every frame is treated as Intra. What this means is that there aren't any motion estimation errors occurred by the motion compensation of the encoder and you don't have GOPs like IPBB etc 'cause nothing is predicted and every frame is "Intra" so "I". Does it mean that you won't lose anything and you can re-encode every time you want in XAVC Intra Class 50 - 100 - 200 - 300 etc? The answer is: No! You will still lose something! Besides, XAVC is constant bitrate (unless selected otherwise but it's not fully supported by Sony officially anyway) so there will be some errors. So... Sony's encoder is not a "Holy Grail" that will solve everything nor is their XAVC specification which is based on H.264.

If you're willing to try and you know how to use SSIM/PSNR, you can try to encode 10 frames in XAVC Intra Class 300 with the following x264 settings 50'000 times and then check with the metrics how it much you've lost. If they say anything other than "infinity" then it means that it was lossy and something has been lost and I bet you're gonna find out that something has indeed been lost:


AVS Script:

Code:
ColorBars(3840, 2160, pixel_type="YV16")
ChangeFPS(50)

trim(0, 10)

ConvertBits(16)
ConverttoDoubleWidth()
BAT:

Code:
avs4x264mod.exe "XAVC.avs" --x264-binary "x264-10b.exe" --input-depth 16 --preset medium --profile high422 --keyint 1 --bitrate 500000 --vbv-maxrate 500000 --vbv-bufsize 500000  
--no-cabac --slices 8 --overscan show --colormatrix bt709 --range tv --log-level info --thread-input --transfer bt709 --colorprim bt709 --videoformat component --nal-hrd cbr 
--output-csp i422 --output "raw_video.h264"

pause

and write a loop to repeat it 50'000 times. xD

Quote:
Originally Posted by Mitra View Post
What are the best settings/recommendations to decrease Generation Loss?
1) Avoid useless re-encodes: for instance, if you have to apply several filters or color grade etc using different tools, use a lossless intermediate like v210 or if you're using frameservers like Avisynth or VapourSynth or FFMpeg or VirtualDub etc FFV1, UTVideo, HuffYUV, Lagarith etc are your friends so that you can re-encode to lossy only 1 time instead of serveral times.

2) Avoid mixing codecs!! This is VERY important and something that people often don't understand. Using different codecs which have different implementations and / or different transforms can lead to a lot of degradation. For instance, if your source is in, let's say, H.265 and you re-encode to VP9 and then back to H.265 you're gonna lose way more than just re-encoding it 2 times in H.265. This is because VP9 like VP8 etc are based on a very different implementation and treat as important some of the things that are not important for H.265 and vice-versa, so, please, avoid mixing codecs whenever possible.

3) I would definitely recommend using CRF whenever you can instead of other ancient methods like constant quantization. There are still cases in which constant bitrate is required like in the professional sector when you only have a certain amount of bandwidth on the satellite and you cannot afford overshoots, however, if the encodes are for personal use, please use CRF, 'cause those "overshoots" are gonna preserve quality (or at least try to preserve it).

4) Learn from the community and use the right settings for the right file. There are tons of examples and tweaks here on Doom9, especially for x264/x265 and there's never a "write once use every time" thing as contents may have different peculiarities. For instance, Sport Events are nothing like Movies which are nothing like Anime etc. There are lots and lots of psychovisual optimizations that can be tweaked in those encoders and you just have to find the right settings (some people even use --zones to encode different parts of movies with different settings!).


That's just my opinion, but I hope it helps.

Cheers,
Frank.

Last edited by FranceBB; 8th April 2021 at 16:44.
FranceBB is offline   Reply With Quote
Old 8th April 2021, 20:10   #4  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,954
Lossy workflows aren't used that much when doing multigenerational processing in professional video workflows, certainly not in feature film production.

When working with an i-frame only format, most video tools will pass through unchanged frames. So an assembly edit won't involve much reencoding. A color pass will. But generally there aren't many passes that are changing individual pixel levels; lots of filters are stacked together losslessly in RAM inside a tool. The mezzanine file has everything locked in, and generally the only processing that happens after that would be downscaling for adaptive streaming.

It occurs to me that a codec-aware tool could pass through quantization and other parameters intelligently. For example, a lot of colorist work may only impact the DC coefficients, so those could be modified without having to change quantization. Or if it does, it could be done in an intelligent way that doesn't introduce new quantization errors. And unimpacted macroblocks could also be functionally passed through with some bit twiddling. Of course CABAC would need to be recalculated and all that, but doing higher quality multigenerational processing with lossy formats could be done better than it is today.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.