a better way to design a codec?

[deadrats]

lately i've been wondering if all major encoder authors such as xvid, divx, x264, main concept, apple, wmv, vp8, ateme et all haven't been going about creating codecs the wrong way all along.

why is it that all the above codecs try to improve quality of the encode largely via deeper motion vector analysis, larger motion search ranges, quarter pixel support, "tricks" with b frames (adaptive b frames, pyramid b frames), many have some sort of psychovisual feature and motion compensation features and the end result seems to be lots of complex code, lots of options (which result in post after post on message boards across the net on "what's the best setting for quality" and for the most part it seems that most users still aren't satisfied since they look for ways to improve their video further with filters via various means such as avisynth.

so that begs the obvious question: instead of going to the hassle of coding all the various advanced features of modern h264 and vc-1 codecs and trying to optimize code for speed and quality, why not just create a basic reference encoder that confirms to the most basic specs of the standard but build in enhancement filters into the codec?

for instance divx comes with a built in deinterlace filter (it sucks, needs improvement) a built in denoise filter (not bad at all), all h264 encoders have an inloop deblocking filter, x264 also has a denoise filter built in, xvid has a chroma enhancing filter built in, why not go all out and build in a color correction filter, a good deinterlacing filter, a good high quality denoising filter (maybe as good as neat video's), a warp sharpening filter and don't worry about deeper motion analysis, b and p frame tricks, motion compensation like gmc and mb-tree or psychovisual enhancements or algorithms for dealing with dct left over coefficients, like trellis.

it seems to me that this would be a simpler approach and would result in higher quality videos, if the sharpening, deblocking, denoising, color correction and deinterlacing were all done in loop.

what do you guys think?

[Selur]

problems I see:
a. from my point of view: a encoder is ment to compress uncompressed material an not do filtering (unless it is part of the compression scheme like inloop filtering); -> the encoder should assume that the input he gets is perfect and it's aim should be to preserve as much of the data he is getting fed. It should not try to remove stuff on it's own, that stuff might be important!
b. automatic filtering of material without user interaction is not really an option. The encoder has no real idea of what quality is, so how should it know when to filter and when not on it's own? So if user interaction is needed to specify filtering, a gui is probably also needed.

and for the most part it seems that most users still aren't satisfied since they look for ways to improve their video further with filters via various means such as avisynth.
normally you filter stuff with avisynth because:
a. someone screwed up before you (e.g. didn't properly master the material)
b. you aim for some artistic effect
c. you are not happy with the way the material is presented, e.g. you don't like that it's interlaced, you don't like the colors,...
- filtering should be done before the encoding and not by the encoder
- if the codec specification would include XY high quality filters you might not only have to pay for the patents that cover the encoding but also the filtering routines

it seems to me that this would be a simpler approach and would result in higher quality videos, if the sharpening, deblocking, denoising, color correction and deinterlacing were all done in loop.
If that would be true shouldn't it be possible to use MPEG-1 + some fancy filtering and end up with material that could rival MPEG-4 AVC&Co ? (I mean MPEG-1 incorporates the simple approach that is behind the MPEG codecs,..)

what do you guys think?
I hope this doesn't happen,...

Cu Selur

[Dark Shikari]

The x264 internal denoising filter is a relatively useless toy that few people use, and the "inloop deblocking filter" is not meant to "fix" the source, it's part of the compression process.

"Enhancement" filters don't work in-loop; an inloop sharpener, for example, would make every frame even sharper than the previous frame, since the effect would accumulate.

[mandarinka]

You know, I have trouble believing you are meaning what you just wrote in the entry post.
You have been posting here for at least a year IIRC, so I have a feeling as if you are likely to be aware how your "most basic specs" thing makes no sense.
And the way you specifically mention f.e. warp sharping... I really don't want to be rude, but your post sounds 100 % like a silly prank. Sorry.

[deadrats]

You know, I have trouble believing you are meaning what you just wrote in the entry post.
You have been posting here for at least a year IIRC, so I have a feeling as if you are likely to be aware how your "most basic specs" thing makes no sense.
And the way you specifically mention f.e. warp sharping... I really don't want to be rude, but your post sounds 100 % like a silly prank. Sorry.

i meant every word of it; lately i've been working with a bunch of greek comedy dvd's, as well as dvd's of the old "21 jump street" tv show (probably the worst professional authored hollywood dvd's ever created) and i've been experimenting with various codecs and various settings to see how i could get the best quality encode at roughly 60-65% the bit rate of the original.

what i found is surprising, no matter how high i crank the settings, how much i massage the various settings, for any encoder, the resulting output was always crappy. x264 gave the best results but i had to crank up he settings to absurd levels, like mb-tree turned up to it's strongest setting, sub-me=11, motion search at 64+tesa, 16 reference frames+16 b-frames, all partitions, weighted p=2, weighted b=1, psy-rod and psy-trellis and so on. the problem is that even with an i7 3770k+8gigs ram even on SD material it brought the system to it's knees and even then the encode wasn't that good.

then i decided to try some good color correction filters, a good denoiser and a good sharpener and after some test encodes, i found the proper settings for the filters and i found that even with x264's settings turned all the way down (like diamond+16, full pixel, no partitions, no weightp or b, no trellis, no psychovisual, etc) i still got much better quality encodes than no filters but all the encoder settings maxed out.

thus it seemed to me like a logical conclusion that maybe all codecs should be designed from the ground up with filters built in; i thought that the same way you can filter a frame using an external filter and then pass it to the encoder that the code for both could be merged and that the processing could be done internally.

it seems kind of counter intuitive to me that something like an inloop sharpener would make every frame sharper than the next, why doesn't that happen when you filter using external filters, why doesn't it happen with the built in denoiser, why doesn't it happen with the inloop deblocker?

ah well, just a thought...

[LoRd_MuldeR]

Well, the idea of an "in loop" filter is that it is applied on the frame before that frame is used as a reference by other frames. So some of the "sharpening" effect would propagate into those frames that use the filtered frame as reference. But then those "second generations" frames will be filtered too an become even sharper. This increased sharpness is then propagated into the "third generation". This goes on and on. Thus it accumulates.

Also: Why should video encoders (or video compression formats) be designed to "repair" defects in the source? The video encoder has to assume that the source is "perfect" quality and, apart from Psy optimizations, its goal is to retain the source as closely as possible! If there are any specific "defects" in the source, they have to be fixed before the re-encoding - if possible at all. I think there are far too many different kinds of "defects" that might appear in a source. Also it's pure speculation in which way the "look" of the source should be "modified" in order to satisfy the viewer. Consequently the "swiss army knife" filter that will magically fix all arbitrary defects of a given video (without causing any bad side effects!) simple doesn't exist in the real world. Even worse: Assume there was a video compression format with various built-in "repair" filters and now you want to encode a good quality source. All those filters would not only be useless, they would even change the intended "style" of the source with high probability. Nobody would use such format for serious encodes! After all it's up to you to decide what "defects" exactly need to be fixed in a particular source (if any) and then pick the suitable pre-processing filter(s)...

[Asmodian]

it seems kind of counter intuitive to me that something like an inloop sharpener would make every frame sharper than the next, why doesn't that happen when you filter using external filters, why doesn't it happen with the built in denoiser, why doesn't it happen with the inloop deblocker?


It does happen with the inloop deblocker, that is why it is inloop.

I think you don't quite understand what is meant by "inloop" (neither do I) but this blog post (http://mrutyunjayahiremath.blogspot.com/2010/09/h264-deblocking-filter.html) might give you an idea. :)

Edit: LoRd_MuldeR's post is much easier to read and understand but this is the issue as that blog puts it:
"Due to coarse quantization at low bit rates and block-based transformation, motion compensation, typically results in visually noticeable discontinuities along the block boundaries, as in Figure 1 (a blocky image). If no further provision is made to deal with this, these artificial discontinuities may also diffuse into the interior of blocks by means of the motion-compensated prediction process."

I don't understand how your idea works. I really wouldn't want the same filters you ran on your terrible DVDs run on my nice BBC Life blurays. I wouldn't want them run on the output from my HD camcorder. I wouldn't even want them run on my decent DVDs. Automatic filtering would need to be amazingly (impossibly?) smart and not instead of all the techniques used now but in addition to them.

[deadrats]

Well, the idea of an "in loop" filter is that it is applied on the frame before that frame is used as a reference by other frames. So some of the "sharpening" effect would propagate into those frames that use the filtered frame as reference. But then those "second generations" frames will be filtered too an become even sharper. This increased sharpness is then propagated into the "third generation". This goes on and on. Thus it accumulates.

i understand the theory of why some might think that the above would happen but what you describe also applies to any preprocessing filtering that's done.

my understanding of the basic process is thus: decode frames, apply filter to frame, pass results to encoder, so even with external filters doesn't the filtering effect get applied to the frame before it's used as a reference by other frames?

i mean i understand what you and DS describe but how is that different from what happens with an external filter? isn't the process the same?

furthermore, even if your prediction of the end results is 100% accurate, my understanding of how reference frame work is that with a closed gop frames can't reference frames outside the gop they belong to, thus this theorized propagation of sharpness would end once a new gop started and could thus be limited by using small gop's, say gop=frame rate, no?

Also: Why should video encoders (or video compression formats) be designed to "repair" defects in the source? The video encoder has to assume that the source is "perfect" quality and, apart from Psy optimizations, its goal is to retain the source as closely as possible!

in theory maybe, in practice this isn't what actually happens, is it? if the goal was to have the output be as close to the source as possible then there wouldn't be any configuration settings at all, what the encoder authors would do, and i mean for all codecs, is analyze the source, calculate peak PSNR, average PSNR, peak SSIM and average SSIM, then use the resulting values to internally tune the encode to produce the output that is mathematically closest to the original at any given bit rate.

but that's not what any codec does, instead it allows the
end user to configure all sorts of parameters and in fact most of the better encoders, such as divx, x264, ateme (even the version found in nero recode) and xvid have 1 or more psychovisual enhancement settings that users are encouraged to use.

it seems to me that all the more advanced codecs attempt to improve the quality of the encode in comparison to the source, so instead of having to write lots of complicated code and testing it, then tweaking it and so on it may be a bit simpler to just code a few high quality filters and build them into a basic reference encoder and call it a day.

i would also assume such an encoder should be easier to multithread and my guess is that it would probably be faster than the current approach.

i'm also curious why none of the encoders use the PSNR/SSIM approach i mentioned above, if the goal is for the output to be as close as possible to the source then wouldn't it make more sense to not have any configuration settings at all, just let the encoder calculate the output based on the above metrics and call it a day?

edit: what would be the settings for configuring x264 for closest SSIM or PSNR between source and output? i know about the tune settings SSIM and PSNR but all that does is change 2 or 3 settings within the encoder, what about the rest of the settings?

also, what about the I/P and P/B ratio settings? instead of the defaults, wouldn't 1x for each result in the best quality? while i'm at it, here's another thing that's been bothering me for a long time, why is it that encoders don't use a gop sequence that matches the source, so that I frames stay I frames, P frames stay P frames and B frames stay B frames?

to me it seems logical that if the source has a gop that's 30 frames long with a gop arrangement of say BBBIBBBPBBBPBBBPBBBPBBBPBBBPBP then it's kind of counter productive to use a gop of say 250 with a sequence of BBBBIBBBBPBBBBPBBBBPBBBBPBBBBPBBBBPBBBBPBBBBPBBBBPBBBBPBBBBP, i don't see how one can expect to achieve maximum quality encodes when you change I frames to B frames, B frames to P frames and P frames to ! frames.

or maybe i'm missing something...

[schweinsz]

Deadrats just needs the tools that could detect if certain image to be encoded needs the sharpness, denoising, color correction, etc, and decide the degree of sharpness, denoising and color correction, etc, that the certain image needs, and at last conducts the sharpness, denoising and color correction, etc, operations on the certain image.
He/She just needs that the encoder could do these automatically and finely, free him/her to operate these manually.
To the best of my knowledge, there BE or will BE these algorithms/implementation.

[pandy]

what do you guys think?

Great idea, perhaps human visual perception emulator + cognitive human brain model functionality?

[LoRd_MuldeR]

i'm also curious why none of the encoders use the PSNR/SSIM approach i mentioned above, if the goal is for the output to be as close as possible to the source then wouldn't it make more sense to not have any configuration settings at all, just let the encoder calculate the output based on the above metrics and call it a day?

Actually PSNR and SSIM are used a lot in encoder development/tuning. But relying too much on them is not good! That's because there is no clear and universal definition of "difference" or "error" between the original source and the encoded video. PSNR simply calculates the average squared difference, separately/independently for each pixel. This is simple, but not how the human perception works. SSIM's sliding window approach seems to capture the usual compression artifacts better, but it's still not perfect. The perfect quality metric simply does NOT exist and probably will never exist! Therefore experience shows that tuning too much for PSNR (SSIM) will result in a "smooth" or "washed out" look. That's because the human eye does not only want the "per-pixel difference" to be as small as possible. It also wants the "complexity" to be similar to the original, even if that means a slightly bigger "per-pixel difference". This is exactly the idea that has been implemented in x264's Psy RDO and Psy Trellis features. And those "Psy" optimizations will obviously hurt PSNR and SSIM, because of the way how PSNR/SSIM measures the "error".

Also, even if we optimized only for PSNR or only for SSIM, there still would be many different "variants" with (virtually) identical PSNR/SSIM score but quite different look. It's not unambiguous at all!

Just look at those pictures that all have the very same PSNR value and yet look quite different:
http://img19.imageshack.us/img19/5602/psnrssim2.png
(note that here the "MSE" essentially is the same thing as PSNR and "MSSIM" is what we usually call just "SSIM")

edit: what would be the settings for configuring x264 for closest SSIM or PSNR between source and output? i know about the tune settings SSIM and PSNR but all that does is change 2 or 3 settings within the encoder, what about the rest of the settings?

As you already have figured out, there are "--tune" parameters to tune x264 for maximum SSIM or maximum PSNR. And, for obvious reasons, it only changes/disables those settings that hurt PSNR/SSIM. Also SSIM or PSNR are often used to compare different encoder settings, e.g. to compare the effect of different "--preset" options. Consequently "--tune ssim/psnr" doesn't mess more with your settings than it is necessary to allow meaningful SSIM/PSNR results! If you want to maximize overall SSIM or PSNR, you should combine "--tune ssim/psnr" with "--preset veryslow" (or "--preset placebo", if you have a lot of time to waste). But you don't want "--tune ssim/psnr" in your final encode, for the reasons explained above...

[burfadel]

The x264 internal denoising filter is a relatively useless toy that few people use.

I find that using around say, 400-500 can be used to 'improve' quality. It seems to remove 'low-level' noise that you can't see in the final encode, such that using CRF mode the file size is a bit smaller. You can then use a slightly lower CRF to compensate for the size, meaning for a given file size it is potentially fractionally better quality. I guess in two pass it would mean better bitrate distribution.

It isn't a massive different, it's just one of those 'little tweak' things that I find beneficial.

[akupenguin]

the encoder should assume that the input he gets is perfect and it's aim should be to preserve as much of the data he is getting fed.

I can think of one reason you might want to deviate from that model, though it's purely theoretical at this point: If you're more certain of some features of the input video than others, you might want the encoder to assume the features you are certain of are perfect, while letting it take whichever decision costs the fewest bits out of the sets you're uncertain of (or some tradeoff between those extremes). A denoise filter could in principle return a probability distribution over videos, rather than a point estimate thereof. This still wouldn't require filters to be merged with codecs, as long as you have a standard notation to represent that distribution.

[iwod]

Deadrats just needs the tools that could detect if certain image to be encoded needs the sharpness, denoising, color correction, etc, and decide the degree of sharpness, denoising and color correction, etc, that the certain image needs, and at last conducts the sharpness, denoising and color correction, etc, operations on the certain image.
He/She just needs that the encoder could do these automatically and finely, free him/her to operate these manually.
To the best of my knowledge, there BE or will BE these algorithms/implementation.

Actually deadrats asked the question that i had in my mind long ago ( when i was still doing lots of encoding ), the this proposed solution was something i have thought would exist someday and not yet been done.

A Great Sets of Filter for, Anime, Sports, Movies, TV etc and auto apply them to encode.

[paradoxical]

But why should that be included in the codec? What actual, verifiable benefit does it do there versus just applying the filtering to the frames before sending them to the encoder?

[jackoneill]

lately i've been wondering if all major encoder authors such as xvid, divx, x264, main concept, apple, wmv, vp8, ateme et all haven't been going about creating codecs the wrong way all along.

[...]

what do you guys think?

Oh, yes! All those people, who have the knowledge and skills to write such software, they're all doing it wrong!!! You alone saw the error of their ways.

[Selur]

what would be the settings for configuring x264 for closest SSIM or PSNR between source and output?
closes results should be gained with lossless encoding ;)

[deadrats]

But why should that be included in the codec? What actual, verifiable benefit does it do there versus just applying the filtering to the frames before sending them to the encoder?

speed - if the code for the filter was merged with the code for the encoder it would reduce latencies between having to decode the frame, filter it and then encode it (at least in theory).

smaller code size - again, less code, in theory should result in faster execution, also easier to maintain.

eliminate the need to test with dozens of different filters to find the best one.

higher quality, i'm presuming that were the code for filter and encoder merged the resulting quality would be higher, as the codec would be designed with the filter's capabilities in mind.

you guys really can see why it is reasonable to think that were someone to merge say the code for neat video (if it were open source) and x264 that this would result in superior encodes? or if the x264 developers built in a good quality deinterlacer or color correction filter, you guys don't think that it would result in higher quality encodes?

[deadrats]

Oh, yes! All those people, who have the knowledge and skills to write such software, they're all doing it wrong!!! You alone saw the error of their ways.

you statement is an attempt to end a discussion via application of a technique known as "appeal to authority", a tactic often used in debates to try and shut someone up, to wit "so and so says such and such and he/she knows more than you thus you must be wrong".

the reality is that even people with vast amounts of experience can overlook something and believe it or not even non codec authors have functioning brains.

and second, divx has had a built in denoise and deinterlace filter for a while now, as has been pointed out x264 also incorporates a built in denoiser (regardless of whether or not they think its any good), the h264 standard calls for a built in deblocker, so the idea of integrated filters is not that out there nor did i claim that i envisioned it.

what i wondered was why codec authors don't spend more time working on expanding the number of built in filters and/or improving their quality rather than the other methods they use to try and improve the quality of the final output.

all you have to do is look around this forum to see how many people use some filter as a standard part of their encoding methods, how many people use avisynth and filter every video they encode and it's my understanding that professional studios apply low pass filters to all their encodes when they are sending out the blu-ray masters to authoring houses.

to me it just makes sense to put it all into one package and call it a day.

[deadrats]

closes results should be gained with lossless encoding ;)

actually closest results would be to not re-encode at all, but sometimes you have to. besides, lossless results in huge file sizes...

oh wait, you were just kidding...

[Selur]

actually closest results would be to not re-encode at all, ...
If not re-encoding wields another result (SSIM AND PSNR wise) than lossless reencoding, the reencoding wouldn't be lossless. :D

what i wondered was why codec authors don't spend more time working on expanding the number of built in filters and/or improving their quality rather than the other methods they use to try and improve the quality of the final output.
Why should it be incorporated into a single codec? I mean this approach should be source and not encoder dependent.
Plus the general quality problem still exists (when to apply which filter how often).
Just looking at some basic filters (same is true for most filters) like denoiser and deinterlacing shows that applying a filter without being sure that it is needed can have devastating results.
Since filter and codec development both requires similar but also quite different expertises, it seems to be a good idea to leave the filter development to people that only do this and not burden codec developers with it.

Your idea might have it's merit, but it kind of assumes that one knows when and when not to apply which filter to get the perfect result.

Cu Selur

[paradoxical]

speed - if the code for the filter was merged with the code for the encoder it would reduce latencies between having to decode the frame, filter it and then encode it (at least in theory).

I need more than just "theory".

smaller code size - again, less code, in theory should result in faster execution, also easier to maintain.

This makes no sense. Your proposal ADDS code to the encoder. Meaning there is more to debug and more that can break.

eliminate the need to test with dozens of different filters to find the best one.

Why? There is no perfect filter out there. If there was, people wouldn't still be making new ones.

higher quality, i'm presuming that were the code for filter and encoder merged the resulting quality would be higher, as the codec would be designed with the filter's capabilities in mind.

Based on what exactly? You're heavy on assertions and light on any actual evidence.

you guys really can see why it is reasonable to think that were someone to merge say the code for neat video (if it were open source) and x264 that this would result in superior encodes? or if the x264 developers built in a good quality deinterlacer or color correction filter, you guys don't think that it would result in higher quality encodes?

No, I see nothing reasonable about that versus just having them be separate.

what i wondered was why codec authors don't spend more time working on expanding the number of built in filters and/or improving their quality rather than the other methods they use to try and improve the quality of the final output.

Because there is no benefit to doing so versus having them be external filters. If there was, they would have already done this. Your idea is not as novel as you think it is.

If you think your idea is great and everyone else is wrong, modify x264 or write your own encoder that works the way you claim is better and prove all the experts wrong. This would actually go farther than just assertions.

[iwod]

Deadrats just needs the tools that could detect if certain image to be encoded needs the sharpness, denoising, color correction, etc, and decide the degree of sharpness, denoising and color correction, etc, that the certain image needs, and at last conducts the sharpness, denoising and color correction, etc, operations on the certain image.
He/She just needs that the encoder could do these automatically and finely, free him/her to operate these manually.
To the best of my knowledge, there BE or will BE these algorithms/implementation.

Like i said i have had some early ideas on similar things when i was still doing lots of encoding. I dig out those notes, and it turns out it wasn't the encoder we need.
Like some have said the codec is suppose to produce the closest image possible to the source. To add filters infront of it would be acting exactly like Real Encoder and Decoder. It looks good in the low end but will never be top notch.

Doing Low Bitrate encoding requires lots of fiddling with filters. What i thought of ( and possibly the OP wanted ) was a Automatic way of choosing the best possible sets of filters, according to the encoding materials ( Sports? Anime? Movies? TV? ), before feeding this into encoder.

So that we could just seat back, throw in the source, enter a few parameters, and leave it to work.

Of course the Pre Filter should also take notes, or only tuned to work with specifics encoders. Since difference encoder has different characteristics.

So after all this shouldn't really be a Encoder problem.

[schweinsz]

Like i said i have had some early ideas on similar things when i was still doing lots of encoding. I dig out those notes, and it turns out it wasn't the encoder we need.
Like some have said the codec is suppose to produce the closest image possible to the source. To add filters infront of it would be acting exactly like Real Encoder and Decoder. It looks good in the low end but will never be top notch.

Doing Low Bitrate encoding requires lots of fiddling with filters. What i thought of ( and possibly the OP wanted ) was a Automatic way of choosing the best possible sets of filters, according to the encoding materials ( Sports? Anime? Movies? TV? ), before feeding this into encoder.

So that we could just seat back, throw in the source, enter a few parameters, and leave it to work.

Of course the Pre Filter should also take notes, or only tuned to work with specifics encoders. Since difference encoder has different characteristics.

So after all this shouldn't really be a Encoder problem.

So we need more intelligence. The pre-filter should be selected according to the input image quality, the bitrates, the resolution, the is-interlace and the target quality, etc.

[benwaggoner]

But why should that be included in the codec? What actual, verifiable benefit does it do there versus just applying the filtering to the frames before sending them to the encoder?
The big reason to include filtering inside the encoder is if filter settings would be adjusted by feedback from the encoding process.

For example, if the codec could adjust the strength of a denoising feature if the noise would otherwise introduce visible artifacts.

Or switching to a softer scaling mode if the image's detail is too high to avoid artifacts.

I've long been a proponent of moving bit-depth conversion into the codec so the codec can figure out the most efficient way to dither the image and to code gradients instead of being stuck with no way to tell dithering from image texture.

[Rumbah]

Apart from adding a lot of complexity to the decoder that could cost a lot of money and/or energy for consumer hardware I guess the main problem would be finding the "magic formula".

You would have to find the algorithm that tells you what is wrong with the video to automatically correct it with all the filters. That would be the perfect one click solution that would probably make you rich.

It kind of reminds me of the itanium where they hoped they could write a very intelligent compiler and therefore reduced the hardware complexity (e.g. static scheduling, no jump prediction, etc.). And it turned out that the performance was low as they couldn't write a compiler that delivered the perfect code that the architecture needed to really shine.

[paradoxical]

The big reason to include filtering inside the encoder is if filter settings would be adjusted by feedback from the encoding process.

For example, if the codec could adjust the strength of a denoising feature if the noise would otherwise introduce visible artifacts.

Or switching to a softer scaling mode if the image's detail is too high to avoid artifacts.

I've long been a proponent of moving bit-depth conversion into the codec so the codec can figure out the most efficient way to dither the image and to code gradients instead of being stuck with no way to tell dithering from image texture.

It might have those benefits and it might not. Either way this would significantly bloat the encoder and add all sorts of complexity. Which goes against his claim of:

smaller code size - again, less code, in theory should result in faster execution, also easier to maintain.


Also, I see no reason why the same logic that would be added to the encoder couldn't be part of the filter's logic. Either way, the coded needed to implement the logic to provide these theoretical benefits don't match up to the claims of simplifying the encoder.

[benwaggoner]

It might have those benefits and it might not. Either way this would significantly bloat the encoder and add all sorts of complexity. Which goes against his claim of:
Lots of encoder features act as "filters" already, with a feedback loop. Adaptive deadzone and in-loop deblocking have filter-like effects.

Also, I see no reason why the same logic that would be added to the encoder couldn't be part of the filter's logic. Either way, the coded needed to implement the logic to provide these theoretical benefits don't match up to the claims of simplifying the encoder.
It's more about having the encoder being able to feed back to the filter to better optimize the video for encoding. Lots of people manually iterate filter settings to get better encodes, and there are cases where the encoder itself could do a better job of knowing what the right parameters are.

But yes, it definitely would not reduce complexity!