Weird effect on interlaced 1080i HDTV AVC

[eXtremeDevil]

The title resumes the case very well hehe. Here are the frames (already deinterlaced) as the letters go upper:

http://www.mediafire.com/?2lt3ez3hs5pdug5

Info:

It's a HDTV capture from Spain, Europe, so it is PAL.
Is a DVB-S broadcast.
Is a 1080i AVC with MBAFF interlacing.
The show is SNL, which is american, NTSC, so maybe something went wrong in a possibly convertion to pal in order to broadcast here.

That's all the facts I can think about now. No matter which deinterlace filter I appy, the same effect always prevail.

Maybe is just the way the credits are done, but I post here just to make sure.

Thanks!

[Guest]

Post a sample of the unprocessed source.

[eXtremeDevil]

Here: http://www.mediafire.com/?3g0oztj9pzi2c33

[Guest]

It' strange. The scrolling credits exhibit field blending that does not appear on the video itself. I can't think of anything useful you can do to this.

[eXtremeDevil]

Could be because of the NTSC - PAL thing? Can you explain it again slower? English is not my navite language hehe.

[Guest]

When you use SeparateFields() and view the individual fields, then you see blends of multiple pictures but only for the credits, not the video behind them.

[eXtremeDevil]

So is it just the way it's ogirinally printed? Or a defect of something?

[Guest]

It's in the source material, yes. How it got that way is hard to say, but crappy standards conversions are legion.

[eXtremeDevil]

OK. Two last questions, any deinterlacing recomendation? Is it worth to make a 1080p version of the source?

[Guest]

As I said, I can't think of anything useful you can do to this.

Regarding your second question, it depends on what you consider worthwhile, i.e., what your goals are.

[eXtremeDevil]

My goal is to keep the video for myself trying to get the best compression and the best quality as possible. What I was actually asking is that if it is worth, in general, to make a 1080p out of a 1080i, or if it is always a waste of time.

With the recommentaion, I meant to ask if there is any special deinterlacing filter for MBAFF HDTV AVC sources, if not, I will just use one of the usuals.

[Guest]

Personally I would leave the original untouched and deinterlace in my player. But as they say, different strokes for different folks.

There is no special deinterlacing for MBAFF AVC. Use whatever pleases you.

[eXtremeDevil]

Thanks for all your help.

[Mole]

I would also keep it as interlaced.

You might as well let your player deinterlace it during playback. With time, processing power as well as deinterlacers will improve, so in a few years time there may even be a QTGMC like quality (or better) real time deinterlacers.

So why bother with processing it now when it may potentially improve with time.

[eXtremeDevil]

I know you guys have told me to keep the original interlaced file. And I probably will do that. But in the meantime, just for experimenting and gain knowledge, I'm trying to create an encode that will weight almost the same as the source (for example from a 2.5 GB ts make a 2.4 or 2.45 mkv). These are the results I am having, after investigate a lot and making at least about 20 different tests:

http://screenshotcomparison.com/comparison/160092

http://screenshotcomparison.com/comparison/160093

My question is, is that all the quality I'm going to have? Can I improve the encode even more to leave it almost exactly like the source?

Thanks!

[Mole]

Well, in order to give you any answers, you should at least tell what you did. What deinterlacer did you use, and did you deinterlace to 50p or 25p?

Also, the quality can only be as good as the source, so I don't really know what you expect with a question like "is that all the quality I'm going to have?"

[eXtremeDevil]

Doesn't matter the deinterlacer I used, I'm comparing the encode to the avs applied to the source, not the raw source. With the quality I mean, if you look to the encode, some parts are smoother than the original video. This is the avs:

DGsource("Video.dgi")
YADIFMod(edeint=nnEDI3())
crop(2,2,-2,0)
spline36resize(1920,1080)

And it is at 25.000 FPS.

Thanks.

[Mole]

Oh, you're talking about ENCODING.
Well, information of the codec and setting would be very useful.
What codec did you use? x264? With what program? MeGUI? What encoding settings did you use?
If you used x264, then encoding settings like
"cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x133 blah blah blah" would be really useful.

It is kind of difficult to give pointers without these infos man.

But in a nutshell, most of the time it's the bitrate which is responsible for drop of quality. If you want better quality, crank up the bitrate.

Also, unless there are some kind of limitations, I would recommend to deinterlace to 50p. If you deinterlace to 25p, you are basically throwing away half the data.

Also, if your goal is to keep it as a personal collection, there is really no point in deinterlacing it. Keeping it unaltered will of course always be better than a processed one since the source is the source.

The only reason people deinterlace is because they may be converting to some other type of medium/format which doesn't support interlaced video very well or at all, or they need to manipulate it or put effects on it.

[eXtremeDevil]

You're right, I forgot some things. Sorry.

I'm using MeGUI, x264, x264_64.exe being specific (MeGUI auto-chooses that one). After a lot of modifications now these are the settings:

program --level 4.1 --preset veryslow --crf 18.5 --deblock -1:0 --keyint 25 --min-keyint 13 --ref 9 --psy-rd 1.0:0.15 --no-dct-decimate --no-fast-pskip --output "output" "input"

Although some of them doesn't show on the command line, I think. I'm using crf, I tryed with const. bitrate but the encoding time was twice the crf one, so it wasn't useful to me. On my latests tests I'm having a 8200~ bitrate and the original ts has 8500~ bitrate so I'm not going to use more than that to make the encode look the same as the source.

If you need additional info please let me know.

[Mole]

This is strange. Different encoding modes shouldn't take any longer time, unless you chose 2-pass, which naturally will take about 1.5 times longer. (First pass doesn't take as long as 2nd pass)
What will take longer time are other settings such as -me and -subme.

[eXtremeDevil]

That's what I meant, with the 2pass. So, looking at my results, can I make even less smooth, almost like the source?

[Mole]

Yes, by increasing the crf/bitrate....

The lower the bitrate, the more smooth x264 will make the video.

Try it and you'll see.

[eXtremeDevil]

Yes, if I use 9000 or 9500 I get kind of what I want. But again, the original source is only 8500 so it is ilogical to have the same quality with a bitrate of 9500. I want like the same quality on the same bitrate.

[Mole]

No it is not. The problem is actually your "logic" that your new processed video for some reason should have the same bitrate as the source.

This new video has been deinterlaced, so you can't really compare it to the interlaced source anymore.

Also, you are comparing your encode with it's uncompressed source.
Didn't it perhaps occur to you how much better the uncompressed interlaced source might have looked like before it was compressed to it's size?
The drop in quality may have been comparable or even more so than your result.

[eXtremeDevil]

I'm comparing it with my avs, not the raw source. The avs has the filters already applied.

"Didn't it perhaps occur to you how much better the uncompressed interlaced source might have looked like before it was compressed to it's size?"

Sorry, I'm not following you, english is not my native language. What I can tell you is this: I'm comparing a frame from the source with the filters applied, and I don't understand why my encode has to weight more to have the same look. At least should weight the same.

[Mole]

But you have already applied filters to the interlaced source. Then you are comparing the compressed video with your uncompressed avs.
But you want a better result from the compression, comparing with the avs, but wanting the size of the original interlaced source.

Your logic of wanting the size to be the same as the interlaced source is just illogical.

English is also not my native language and I don't wanna bother with explaining it any more to you.

Maybe somebody else will care to have more motivation to explain the illogical nature of your comparisons.

[eXtremeDevil]

OK then.

[eXtremeDevil]

Wait, I think I get it. The source is interlaced, therefore it will be more compressed. So if I want the same video but progressive, the weight will have to increase to maintain the same quality, is that it?

[pbristow]

eXtremeDevil: It depends somewhat on the content of the video, but let's take a simple example:

I have a video 10 seconds long, at 25fps, in interlaced Pal format. So that's 250 frames, each with 720x576 pixels in them. But it's a "true video" source (not a Hollywood movie converted to Pal), so each of those frames has information about two different moments in time. It's more appropriate to think of it as 500 *fields*, each of which has 720x288 pixels.

Now I deinterlace the video. I have two options: I can deinterlace to the original frame rate (25fps), or to the field rate (50fps).

If I go to 25fps, I'm basically converting half of the original fields into full frames. The frames might have a *bit* more detail than the original fields, if I've used a clever enough deinterlacer... but the fact is that I've thrown away nearly half of the original data. I should be able to get a lower bitrate for the same level of detail as the original source... but it won't be the same *quality* as the source, because I've thrown away nearly half of the data that me eyes could have used to tell me what's going on in the scene!

If I instead go to 50ps, then I have taken 500 fields and turned all of them into full frames, perhaps filling in details on the extra lines using clues available from the adjacent fields (like my eyes do when I'm watch the interlaced version). I therefore have twice as many pixels to encode as I started with. Now, a lot of those extra pixels are going to be very similar to their neighbours, so I won't actually need twice the bitrate to endode them... But I will certainly need *more* bits than the original source, to encode this new version to the same level of detail.

As to quality: The 50fps version will ceratinly look smoother than the 25fps version, and should be at least as smooth as the original, interlaced version... But it would be misleading to say it could ever be the same or better *quality* than the original. It is better *in some ways*, but there is no escaping the fact that you have changed the original data, and in the process will probably have lost something that the human eye - or at least, *some* human eyes - would regard as "better quality".

In summary:
- "Level of detail" can be measured. Getting more of it tends to require more bitrate.
- "Quality", like beauty, is "in the eye of the beholder". :)

[eXtremeDevil]

OK, maybe I was using the wrong word. I'm converting a 1080i25 to 1080p25. And the result I get is, with the source at 8500 and the encode at 9600, the encode has lower detail, I mean, smoother. You can see that on my screenshots. And of course the encode is more heavy. Why is all that?

[pbristow]

Also: You are talking about lossy compression schemes.

If the original video was encoded using a *lossless* codec, and the deinterlaced video was compressed with *the same*, lossless codec, with exactly the same settings, then the resulting bitrate go up for the 50fps deinterlaced version, or down for the 25fps deinterlaced version.

However, with lossy compression schemes, this can change. The act of compression changes the details of the picture. These changes of detail can take the form either of blurring, or added artefacts (mosquito noise, distorted edges, blocking, etc.). When you re-compress a video that has been decoded from a lossy source, there is no guarantee that the codec wil make the same decision about how to represent each detail of the picture as the original one did - even if it is exactly the same codec). And if anything has changed about the picture in between - such as deinterlacing - then decisions will *deifnitely* be different. So the bitrate will be different. And if the differences in the picture are interpreted by the codec as "more detail", then it will need more bits to encode them.

You can test this: Just take your original interlaced source, and without deinterlacing or filtering or doing anything else, re-encode it using the same codec and the same settings. Two things will happen:
1. The re-encoded version will not look identical to the original version; It is less accurate, or less "authentic", than the original. This is one aspect of what people mean by "quality".
2. The bitrate will be different from the original. Whether it is higher or lower depends on what is happening in the pictures, and how they interract with the strengths and weaknesses of the codec in question.

[eXtremeDevil]

I will try. But remember, I'm comparing the encode detail with the details on the avs preview. Not the original video, but with the filters already applied. And I don't know the settings of the original video, it was a DVB-S capture.

[pbristow]

Ok, but you can do the same test with any video. Encode it once, using your choice of codec, and a known set of settings. Then take the file that comes out, and re-encode wit the same codec, same settings. There will be differences.

I'm also confused because of what you're saying about "after filtering". What filtering are you doing, and at what stage? Whatever it is, that will *also* have an effect on bitrate.

[eXtremeDevil]

I mean, I put the video on the avs, apply the filters, and take a screenshot. And that's the one I compare, and not a screenshot of the original file.

[pbristow]

I mean, I put the video on the avs, apply the filters, and take a screenshot. And that's the one I compare, and not a screenshot of the original file.


So... You're *visually* comparing the output of the avs file, with the output of the avs file after encoding?
Can we see that, then? I need a screen capture of the output from avs script, and a screen capure of *THE SAME FRAME* after encoding.

(All the talk of bitrates made me assume you were comparing two files, so that's what I menat when talk about "the original".)

But I think I know what I'm going to see: The inevitable results of lossy DCT-based compression. That means either loss of detail, i.e. blurring (which I think is what you mean by "smoothing"), or mosquito noise, or both.

Bear in mind that the raw data rate of HDTV video (in the US) is:
16 bits/pixel x (1920 x 1080) pixels/frame x 30 frames/second
= 995328000 bits/second, or nearly a Gigabit per second.

With x264 *lossless* compression, you might get that down to 2 or 3 Mbps, with no changes to the image data. To reduce teh bitrate any further requires lossy compression, which is what you're using. And that means a certain amount of detail will be lost.

[eXtremeDevil]

The comparisons are in one of my previous posts (hosted at screenshotcomparion.com). I understand all of what you're saying, but, in simple words, I actually can't get the same detail level with the same bitrate as the original, right?
I have a "AVC without loss" profile on megui from a friend, the result was a video with 111mbps, and the detail were exactly the same as the source avs filtered output.

[pbristow]

The comparisons are in one of my previous posts (hosted at screenshotcomparion.com). I understand all of what you're saying, but, in simple words, I actually can't get the same detail level with the same bitrate as the original, right?


You can't get the same level of detail, FULL STOP. Not with the codec and settings you're using. Unless you re-encode the source video, *completely unchanged in any way*, using a lossless codec, *ANY* change to the data is going to affect the image in some way, most likely in a loss of detail or an introduction of little speckles of "fake" detail (called "mosquito noise", because of the way they seem to flit around the edges of objects on the screen). Luckily, it's usually in small ways that aren't very noticable: That's why lossy codecs are so useful. But even if you used ten times the original bit rate, if the compression scheme is not *designed* to preserve every detail of the original image (i.e. is "lossless"), then it won't.



I have a "AVC without loss" profile on megui from a friend, the result was a video with 111mbps, and the detail were exactly the same as the source avs filtered output.

Yep. That's a lossless codec. Getting it down to 111Mbps is better than usual: There must be a lot of static elements in the original video, or else the original AVC encoding of it has already shaved off enough fine detail to fit what's left into so few bits.

Welcome to the messy and disappointing world of video encoding! :)

[eXtremeDevil]

Wait, I'm encoding in AVC, isn't that a loseless codec?

[pbristow]

Wait, I'm encoding in AVC, isn't that a loseless codec?

AVC has a lossless *mode* (which is the best lossless codec there is, I'd say), but that is not the mode that is used for broadcast, or for BluRay, etc.

Here's everything you need to know about H.264/AVC: http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC

[eXtremeDevil]

And is there a way to encode in the loseless mode or to change between the codec modes?

[pbristow]

You've already done it! :) In MeGui, remember?

[eXtremeDevil]

I just selected a profile, apparently loseless, which is why there was no difference in the encode. Now I'm using other profile, with the same codec, AVC, and there is detail lost. It is because of the crf, for example, right? I mean, now I'm telling the codec to compress and on the other profile there was no compression settings, is that right?

[pbristow]

That's right (well, nearly). :)

CRF (Constant Rate-Factor) is *one* way deciding when details should be thrown away, and when they should be kept, in order to bring the average bitrate down. If a particular sequence has a lot of fine details or a lot of fast motion, then it uses extra bits to encode that section, and then uses *fewer* bits later on when hopefully they won't be missed, to bring the average rate back down to target.

CQ (Constant Quantizer) is an older way of deciding things, which is usaully not as good (it throws away more details at times when people might notice).

The oldest and simplest way of doing it is CR (Constant Rate) encoding, which just keeps throwing away as much detail as necessary to keep the bitrate constant all the time.

The lossless mode is actually the simplest, as it avoids making these decisions altogther: It just uses as many bits as necessary to encode every last scrap of detail. Hence: Very Big Files! :)

[eXtremeDevil]

So, in conclusion, every time you encode with a lossy codec, you lose 'detail', and even if you use a higher bitrate than the source video the result will be a large file with some detail loss, just because that is the way the things are, right? :P

[pbristow]

Yep. :)

[eXtremeDevil]

And what do you think of my encode? Is it good?

[StainlessS]

and even if you use a higher bitrate than the source video the result will be a large file with some detail loss, just because that is the way the things are, right? :P

Yep

What if really high (excessive) fixed bitrate used when resultant file will have padding added on every frame,
would that also be lossy (mpg1/mpg2/mp4) ?

(suspect answer is still yes but unsure)

[pbristow]

StainlesS: In practice, I don't think that situation would actually crop up. If you set a huge bit rate, and the codec manages to encode everything it needs to in fewer bits than that, I don't know of any codec that would actually pad it out with extra bits.

However, the question of whether it's truly *lossless* in that situation depends on the specific codec you're using. Some codecs are designed to do their initial processing stages - the motion detection and matching, for example - in a completely lossless way (this is the case with x264, for example); Others were designed with the assumption that DCT compression was always going to be applied after the motion stage anyway, so there was no point worrying about whether the first stage was perfectly lossless. For example, I know that DivX (at least up as far as version 6.8) was never completely lossless, no matter how much space you allowed it for all its DCT coefficients. (I frequently run comparisons of material using AviSynth's "Compare()" function, as well as a visual check using "Subtract().Levels(96,1,160,0,255)".)

XviD would always stay closer to the original source than DivX, but still with quite a few errors of one LSB... (Not usually noticable on general video, but disappointing when you're trying to find the perfect codec for storing "mezzanine" material that needs to be re-processed later, or reference clips for test purposes). I *suspect* that was because, even though DCT-based compression wasn't being applied (because there was plenty of bitrate available), the DCT stage itself was still being performed during encoding and had to be reversed on decoding, and the number of bits used in that computation wasn't quite enough to completely prevent rounding errors. I can't confirm that, though, as I haven't had the time/patience/brain-space to delve into the standards and figure out exactly what's going on. (Basically, once I realised how good x264 was there was no point wasting time studying the other two any further! :) )

With x264, I don't know if the lossless mode even perfoms a DCT at all. It would makes sense not to, as it would save redundant processing; but if the decoding part of the H264/AVC standard had already been specified to *require* a DCT stage before anyone had the idea of adding lossless encoding to the spec., then maybe it does...? However, *if* it does, then it seems to do it with perfect precision: I've never seen even so much as single LSB error in anything I've encoded with x264's lossless mode, except when I accidentally set the colourspace option wrong. :) (Colourspace conversions always introduce errors in the LSB). Its also achieves better compression than any of the other lossless codecs I've tried, even ArithYUV.

The first thing I do with all my Fraps video-game captures these days is convert them to x264(lossless). They take up a tenth of the disk space that way, with no loss of integrity. :)

But I digress...

[pbristow]

And what do you think of my encode? Is it good?

That's an unanswerable question. Good compared to what? :)

It's like the question "which is the best (whatever)...?", which is specifically banned by the rules here. The answer depends entirely upon context, and on what the user is trying to achieve, as well as on subjective factors that, when people try to discuss them, usually cause more arguments than enlightenment. :\

Try to figure out what *you* mean by the word "good", and then you can ask a specific question about that factor (or those factors - there may be several!).
Then, if can understand the question, I (and others) can try to answer it.

[eXtremeDevil]

Compared to the source, looking at my screenshots, I'm trying to make the encode to be as heavy as the original source :P