SD TV show was upscaled to HD; how should I encode it for best quality/size

[Wowfunhappy]

So I have a TV show that was originally filmed in 576p, and then upscaled to 1080p for the BluRay release. I want to rip this BluRay so that I can add it to my massive digital archive, but I'm not sure how to encode it.

I am far more interested in getting the best quality per gigabyte than I am in absolute quality. To this end, I figure that retaining the BluRay's original 1080p resolution would not be particularly efficient. So what should I do? Should I downscale the video back down to 576p? Should I downscale it to a lower resolution, such as 480p, to account for artifacts that may have been added when the video was upscaled? And does it matter what scaler I use?

Thank you!

[foxyshadis]

If it came from DVD it might have compression artifacts, but that's unlikely; one of the only shows I know of done that way is Last Exile. Most likely it'll be remastered from the original video, which will be much higher quality than DVD and won't show any artifacts. There aren't really any other common upscaling artifacts, unless it was done very slipshod, but over-sharpening may have been done afterwards which would require its own fixes. (Army of Darkness is a great example, and that was from film!)

I would encode a short segment of both the original and resized down (960x540) at super high quality, play both fullscreen, and see if you can tell the difference. If not, why bother? Then you can save at least half the bitrate no matter what quality you prefer.

[Wowfunhappy]

If it came from DVD it might have compression artifacts, but that's unlikely; one of the only shows I know of done that way is Last Exile. Most likely it'll be remastered from the original video, which will be much higher quality than DVD and won't show any artifacts. There aren't really any other common upscaling artifacts, unless it was done very slipshod, but over-sharpening may have been done afterwards which would require its own fixes. (Army of Darkness is a great example, and that was from film!)

I would encode a short segment of both the original and resized down (960x540) at super high quality, play both fullscreen, and see if you can tell the difference. If not, why bother? Then you can save at least half the bitrate no matter what quality you prefer.

Thank you. Sorry for the late reply, I've had finals...

I can sometimes tell the difference, depending on the scene. But it's not always the original BluRay that looks favorable: sometimes the scaled-down one actually looks better. For example, there's a couple of shots where you can see aliasing along the edges of diagonal objects, and it usually looks WORSE in the original, upscaled version. It's really strange.

Oh, and for whatever it's worth: these are BluRays of the earlier seasons of the new Doctor Who.

Any particular reason you recommended scaling to 960x540 as opposed to 720x576?

[foxyshadis]

Just because 960x540 is exactly half of 1920x1080; if it's 4:3 then 720x540 instead, sure, but the new Who is 16:9. As long as you keep the aspect ratio, it doesn't really matter what size you use; don't bother messing with anamorphic DVD resolutions, it'll drive you crazy.

It sucks that there isn't one good answer that'll apply generally, but at least you tested and know not to waste your time and space on a full-resolution, unless you really want to.

[SeeMoreDigital]

Out of interest.... What is this TV show?

[benwaggoner]

I can sometimes tell the difference, depending on the scene. But it's not always the original BluRay that looks favorable: sometimes the scaled-down one actually looks better. For example, there's a couple of shots where you can see aliasing along the edges of diagonal objects, and it usually looks WORSE in the original, upscaled version. It's really strange.
Alas, not really that strange at all. The pre Season 5 New Who all had interlaced credits. If a not particularly careful software upscaling was done, it might have detected an interlaced source and so you're getting some deinterlacing aliasing as well as scaling aliasing.

And upscaling in general can introduce aliasing issues as well. That's still almost a 2x scale on the vertical, and a ~2.7x scale on the horizontal.

Now, a good upsampling should generally look better than a TV's own upsampling, with scene-by-scene settings tweaks and ideally rerendered credits and other simple elements.

[movmasty]

So I have a TV show that was originally filmed in 576p, and then upscaled to 1080p for the BluRay release. I want to rip this BluRay so that I can add it to my massive digital archive, but I'm not sure how to encode it.

I am far more interested in getting the best quality per gigabyte than I am in absolute quality. To this end, I figure that retaining the BluRay's original 1080p resolution would not be particularly efficient. So what should I do? Should I downscale the video back down to 576p? Should I downscale it to a lower resolution, such as 480p, to account for artifacts that may have been added when the video was upscaled? And does it matter what scaler I use?
Thank you!
Also scaling to the alleged res of 576 will get rid of artifacts,
but i would follow fox and do a round reduce by half to 960x540

About the scaler, i have a special tip, do a bilinear vertical reduction to 540,
and then horizontal with lanczos

Bilinear works better with simple ratios, and very very better when applied in only one direction,
but in this way increase the compressibility while almost as sharp as bicubic.

Besides applying the resize two times in one direction is also slightly faster.

[Ghitulescu]

So I have a TV show that was originally filmed in 576p...

That's a contradiction in terms.
If it was originating from a TV camera it should have been interlaced.
If it comes from film (by scanning etc.) it may be progressive, as film is progressive, but being broadcast it had to be again interlaced. Some are only fake interlaced (ie the two halves make a true frame), but remain interlaced.

I would keep it as it was broadcast, the scaling characteristics of the new chipsets are impressive and beat many software algorithms.

If you need (the better word would be "want") however to reencode it, you can select a portion of it, in particular with a lot of movements or changes (flashlights, aggressive zooming, grass, trees, etc.) and see what bitrate would be appropriate.

There is no mathematical rule, I am afraid.

[Wowfunhappy]

Hello again! Foxyshadis's suggestions generally worked well for the Season 3 and 4 BluRays I had, so thanks.

I recently got my hands on a Seaon 1 BluRay as well—however, for whatever reason, this one is interlaced (so, the BluRay itself is 1920 by 1080i, as opposed to the others which were upscaled to 1080p). Does this change what I should do?

Why the heck would they have upscaled it but left it interlaced, anyway?

@movmasty, I'll be sure to try the scaling filters suggested this time around, assuming they still apply to an interlaced source.


Edit:
Out of interest.... What is this TV show?
As I said above, New Doctor Who. Seasons (series?) 1-4 were recorded in SD.

[movmasty]

@movmasty, I'll be sure to try the scaling filters suggested this time around, assuming they still apply to an interlaced source.
.
You have to deinterlace before,
resizing by 1/2 you could aslo discard a field,
but will get higher quality applying the smart deinterlace filter

[Wowfunhappy]

What deinterlace filter should I use? Most people on Google seem to use Yadif, but there's a bunch of different options for it and I can't figure out what they mean.

Also, do I still want to scale with bilinear vertical and horizontal lanczos?

Sorry if these are noonish questions, there's a lot to take in!

Edit: The other thing that's confusing me is the S1 BluRay seems to be 30 (29.970) fps, whereas the show was supposedly filmed at 25 fps. I'm not personally noticing any judder, but...

[movmasty]

Use smart deinterlace, that only deinterlaces moving areas
yes, bilinear vertical and horizontal lanczos
About FPS watch advancing one by one, if dont notice telecine artifacts
was resampled by merging, creating intermediate frames..

[Wowfunhappy]

Okay, thanks.

I'm getting there... last problem. It looks like every sixth frame (or, occasionally, every seventh frame) is an exact duplicate of the previous one, except with a slightly different color balance. Any way to get rid of that?

[ndjamena]

??

What episodes are you looking at?

My Doctor Who s1-4 BDs are encoded as 24/1.001 fps progressive.

The Tenth doctor Specials are a bit of a mess though. The only problem I can see with the actual episode upscales is that whoever encoded them didn't remove the ITU padding on the sides. Are yours different to mine?

[benwaggoner]

I recently got my hands on a Seaon 1 BluRay as well—however, for whatever reason, this one is interlaced (so, the BluRay itself is 1920 by 1080i, as opposed to the others which were upscaled to 1080p). Does this change what I should do?

Why the heck would they have upscaled it but left it interlaced, anyway?
The credits were originally rendered interlaced. Most frames are probably progressive, though, so using a good deinterlacer that passes through progressive frames well would be helpful. Hopefully the progressive frames (most of the show) are actually flagged as progressive in the bitstream.

[Wowfunhappy]

??

What episodes are you looking at?

My Doctor Who s1-4 BDs are encoded as 24/1.001 fps progressive.

The Tenth doctor Specials are a bit of a mess though. The only problem I can see with the actual episode upscales is that whoever encoded them didn't remove the ITU padding on the sides. Are yours different to mine?

I think so?

I currently have the Season 1, 3, 4, and 4 Specials Blurays. I imported 3, 4, and Specials from Amazon.co.uk—Season 1 I got off of eBay from a US seller.

Contrary to what I said before, most episodes on the S1 Blu-Ray are 24 fps progressive. The two problematic episodes are:

-Season 1 Episode 1 (Hence my last post; I hadn't checked the other eps)
-The Next Doctor Special

These two episodes are 30fps interlaced, wherein every sixth frame (or, occasionally, the seventh frame) is a duplicate of the previous one with slightly different color. I used smart deinterlace to deal with the first problem, but I'm still not sure what to do with the duplicate frames. I'd like to get rid of them if at all possible.

(And yes, there's ITU padding on the sides, which I cropped out)

[ndjamena]

http://horman.net/doctorwho/specials.php

I've think I've re-encoded The Specials twice already to fix the frame rate conversion.

It would be nice if they release proper PAL versions somewhere down the road. Honestly, they really ought to redo 1-4 just to get rid of that padding. I can't believe the BBC would allow something like that to get through. What they've produced doesn't qualify to their own broadcast standards for SD to HD upscaling.

[movmasty]

One solution could be to encode at 15 fps,
or find a program able to remove duplicates

[vivan]

These two episodes are 30fps interlaced, wherein every sixth frame (or, occasionally, the seventh frame) is a duplicate of the previous one with slightly different color. I used smart deinterlace to deal with the first problem, but I'm still not sure what to do with the duplicate frames. I'd like to get rid of them if at all possible.That's called telecine and used for compatibility: BD doesn't support 25 fps, so it uses "59.94i" (29.97 interlaced) mode.

Use TFM (http://avisynth.nl/index.php/TIVTC) to get progressive frames and then use TDecimate to remove dublicates.

One solution could be to encode at 15 fpsWhat a great advice!

[movmasty]

That's called telecine and used for compatibility: BD doesn't support 25 fps, so it uses "59.94i" (29.97 interlaced) mode.

Use TFM (http://avisynth.nl/index.php/TIVTC) to get progressive frames and then use TDecimate to remove dublicates.

What a great advice!
No, that is called duplicate frames,
telecine is all another thing, works with fields, mixing fields from different frames,
Wowfunhappy just sees duplicate frames, telecine gives artifacts into the frames,
telecine cant be resolved with deinterlace,

Decimate is not good to remove dupes, you cant be sure that removes a dupe and not a good frame,
to remove dupes you have to work with the duplicate flag,
unless the dupe was added before of filtering/encoding,
then the dupe will have slight differences and flagged as normal frame
if this is the case, the only way to remove them is manually or decimate by 2.

[huhn]

That's called telecine and used for compatibility: BD doesn't support 25 fps, so it uses "59.94i" (29.97 interlaced) mode.

blu ray supports 50i so it supports PsF too.

[sneaker_ger]

BluRay does indeed support 25 Hz / 50 Hz. The problem is that not all players in the US support it so they do such bad conversions for the American market.

[Wowfunhappy]

http://horman.net/doctorwho/specials.php

I've think I've re-encoded The Specials twice already to fix the frame rate conversion.

Is there any chance you could upload a copy of that AviSynth script you made? Would like to try it.

Although, you mention that "The Next Doctor"'s frame pattern changes a couple of times... will that put the script permanently out of sync with the rest of the episode, or will just those parts be a bit screwed up?

[hello_hello]

Is there an official pulldown pattern to convert PAL to NTSC? I asked Wikipedia, but they don't list a 25 to 29.940 pulldown pattern.

I just noticed the link in post #17. If I'm understanding it correctly, for every seven frames there's two repeated fields. So.....

TFM().TDecimate(cycle=7, cycleR=1)

That should take you back to 25.6886fps. You could slow that down to 25fps, but I wouldn't bother as then you have to re-encode the audio to match.

I assume the following is how Srestore was being used to fix the frames with bad chroma, assuming they're not deleted using the above process. You'd want to use tdecimate or Srestore, not both.

You could try this instead of tfm and tdecimate.

yadif(mode=1)
srestore(25) #although I wonder if you should use 25.6886fps again.

I guess 25fps is sped up to 25.6886fps before the pulldown is applied. Unless there's an extra repeated field every so often but I'm probably not clever enough to work out how often. It wouldn't surprise me if it was sped up, although contrary to the info in the link, I thought it was generally slowed down to 24.975fps before telecine is applied, so you need to remove one frame in six.

TFM().TDecimate(cycle=6, cycleR=1)

That'll give you 24.975fps. Although maybe it's neither. Maybe it's telecined from 25fps and there's an occasional extra repeated field. It'd probably be infrequent enough not to care, but you could try this:

TFM().TDecimate(mode=7, rate=25)

[movmasty]

Hurrà for the High Quality Blu Rays!!!!!!!!!!!

[vivan]

I just noticed the link in post #17. If I'm understanding it correctly, for every seven frames there's two repeated fields. So.....every sixth frame (or, occasionally, the seventh frame) is a duplicateIt should be 6.03 (1200 / 199).
So, yeah, the correct answer is (without using 1200 frames cycle)TFM().TDecimate(mode=7, rate=25)

[hello_hello]

Now that's settled can I vent a little?

WTF is Moffat trying to do to Doctor Who? It started off with promise back in 2005, but it's slowly degenerated into nonsensical crap. The only thing that could save it now is a Star Trek style, "go back in time and re-write history" reboot, where the whole story starts off from scratch and Moffat doesn't exist. It's exceeded the John Nathan-Turner, Classic Who levels of embarrassment, for different reasons, but who'd have thought that was possible?

Sorry.... vent over. I have to release a little pressure when I have the opportunity or one day I'll be watching a Doctor Who episode and my head will explode.

[hello_hello]

My Doctor Who s1-4 BDs are encoded as 24/1.001 fps progressive.

I don't have any Doctor Who I can compare, but I checked an episode of Downton Abbey. The British shows at 23.976fps seem to be PAL that's been slowed down. The audio pitch was obviously different. Here's a question though.....

Would British TV shows be "filmed" at 24fps and sped up to 25fps, or "filmed" at 25fps and slowed down for international Bluray. I'm just wondering if technically most 25fps British TV shows run at the wrong speed or if in fact it's the 23.976fps versions that have been speed altered. Anyone know?

[Sparktank]

Would British TV shows be "filmed" at 24fps and sped up to 25fps, or "filmed" at 25fps and slowed down for international Bluray. I'm just wondering if technically most 25fps British TV shows run at the wrong speed or if in fact it's the 23.976fps versions that have been speed altered. Anyone know?

I think it depends on the creator?

For BBC's Sherlock series, the promotion photos on the days of their shooting, the markers clearly showed 25fps.

I got series 1 & 2 of Sherlock on Bluray for Canada. They're at my mom's right now but I think I remuxed an episode to an external somewhere because I was curious about the international distribution as well.

IIRC (from memory), the Canadian bluray (should be the same for the US BD) is that it's 50i / MBAFF.
We can watch it straight from blruay and it's the same as it is on Netflix (CAN) so I don't think they did a pitch shift for that series.

I also got BBC's Merlin season 3-5 on BD (the only HD seasons/"series") but actually never really looked at it.
They played fine straight from disc, and appeared the same on Netflix (CAN).

Actually, I'm going to try get them back to have an in-depth look.

i've been quite curious about this and so has a friend from UK.
He asked me how BBC shows turn out on CAN Netflix, but there's no real way for me to know since Netflix doesn't have any sort of stats or allowe DirectShow.
And I don't have a capture card of any sort to accurately record a sample.

[hello_hello]

Thinking about it, I think at least some of it must be natively 25fps. The Doctor Who Bluray being discussed here, for example.
itunes cater to old Apple TV devices and apparently they have a limit of 720p at 24fps, so 25fps video tends to be anamorphic at 960x720 and 16:9. If British shows were natively 24fps, it doesn't seem likely they'd speed it up only to have to make it anamorphic.

I was surprised at the audio pitch difference for the Downton Abbey episode I looked at. I wonder why they don't pitch correct it. I know when it comes to DVDs the audio pitch usually isn't fixed, but surely it wouldn't hurt the quality in a noticeable way.

If you discover anything interesting when checking out your video, please let us know. Personally I think using telecine to go from 25fps to 29.940fps is preferable to frame blending. Especially if you tend to re-encode as it should be quite reversible.

[nevcairiel]

BBC films natively at 25fps, including Doctor Who and what have you. International Blu-ray releases are not really a good indicator as they are often "internationalized" for non-EU markets where 25fps support isnt mandatory for $reasons.

[Sparktank]

Just got my blurays and dvd's to my place last night.
It was not fun to sort everything out.

Looking at Sherlock series 1.
Distributed in the USA and Canada by Warner Home Video Inc.
Also says "Distributed by BBC Worldwide Americas".

It's actually 1080i60.

On a 200mb sample, I indexed that with DGdecNV and it shows:
Starting from last IDR frame to EOF:

IDR 193853196
1669:FRM 7 0 1 1
1670:FRM 6 2 1 1
1671:FRM 6 4 1 1
1672:FRM 5 6 1 1
1673:FRM 6 8 1 1
1674:FRM 6 10 1 1
1675:FRM 5 12 1 1
1676:FRM 6 14 1 1
1677:FRM 6 16 1 1
1678:FRM 5 18 1 1
1679:FRM 6 20 1 1
1680:FRM 6 22 1 1
1681:FRM 5 24 1 1

SIZ 1920 x 1088
FPS 30000 / 1001
CODED 1682
PLAYBACK 1682
0.00% FILM
ORDER 1

I remember now that I watch the first few minutes of it.
It looks like it's poorly interpolated to 29.97fps.

Here's the 200mb sample. It actually ended at 186mb.
EDIT: https://mega.nz/#!mooEQYzQ!B3CDhGfw-F1ereNKzN7fxuD5P3ijiAgHOKeE9jrreTw

eac3to sample log:
eac3to v3.31
command line: eac3to "V:\BDMV\PLAYLIST\00001.mpls" 2: 1080i60.h264 -200mb
------------------------------------------------------------------------------
M2TS, 1 video track, 2 audio tracks, 1 subtitle track, 1:28:11, 60i /1.001
1: Chapters, 8 chapters
2: h264/AVC, 1080i60 /1.001 (16:9)
3: AC3, English, 5.1 channels, 448kbps, 48kHz, dialnorm: -27dB
4: AC3, English, 2.0 channels, 192kbps, 48kHz, dialnorm: -27dB
5: Subtitle (PGS), English
[v02] Extracting video track number 2...
[v02] Creating file "1080i60.h264"...
Video track 2 contains 3364 fields.
eac3to processing took 1 second.
Done.

eac3to check log on whole episode:
eac3to v3.31
command line: eac3to "V:\BDMV\PLAYLIST\00001.mpls" -check
------------------------------------------------------------------------------
M2TS, 1 video track, 2 audio tracks, 1 subtitle track, 1:28:11, 60i /1.001
1: Chapters, 8 chapters
2: h264/AVC, 1080i60 /1.001 (16:9)
3: AC3, English, 5.1 channels, 448kbps, 48kHz, dialnorm: -27dB
4: AC3, English, 2.0 channels, 192kbps, 48kHz, dialnorm: -27dB
5: Subtitle (PGS), English
[a03] Extracting audio track number 3...
[s05] Extracting subtitle track number 5...
[a04] Extracting audio track number 4...
[v02] Extracting video track number 2...
Video track 2 contains 317120 fields.
Subtitle track 5 contains 1380 captions.
eac3to processing took 22 minutes, 13 seconds.
Done.


I imagine Merlin is 1080i60, too.
I guess after some time, you just get used to it.

Maybe I should start ordering from the UK for these shows.
No wonder they were dirt cheap at Best Buy.

EDIT: Looking at Canadian bluray disc of BBC Merlin, series 5, disc 1, episode 1.
It's actually 23.976.
I guess they converted down? I tried to look up info on the filming, as seasons 4&5 were shot in HD so they were the first to get put on bluray (later a bluray boxset would include seasons 1-3 (SD) in HD). Some fancy talk about the cameras and how they work.
But, tbh, I have no idea if any pitch altering was done to the series for BBC America distribution.
Haven't looked at the DVD's yet. Still unpacking everything and trying to get them all in one place.

If anyone wants to look at a sample, here's 153MB of the first episode of season 4.
used eac3to to convert first stream to FLAC (it had two dtshdma streams but were the same stream; there's supposed to be commentaries on some episodes so I don't know if that's how the discs were made (dupe audio where there is no commentary) or if it's something about DVDFab Passkey) and then remuxed with MKVtoolnix, trimming on an IDR frame.
Anyone with UK discs can compare and tell if any pitching done?

For a series shot in HD for season 4&5, it looks noisy/sharpened in quite a few scenes.

57 seconds
https://mega.nz/#!z5pmmSyC!2D6M5GsfKzpZoszFHHVw4zBFSaVKTFR0pnFpAu5-aEw

eac3to log:
eac3to v3.31
command line: eac3to "V:\BDMV\PLAYLIST\00071.mpls" 2: Merlin.h264 3: Merlin_1.flac 4: Merlin_2.flac 5: Merlin.sup
------------------------------------------------------------------------------
M2TS, 1 video track, 2 audio tracks, 1 subtitle track, 0:45:16, 24p /1.001
1: Chapters, 7 chapters
2: h264/AVC, 1080p24 /1.001 (16:9)
3: DTS Master Audio, English, 2.0 channels, 24 bits, 48kHz
(core: DTS, 2.0 channels, 1509kbps, 48kHz)
4: DTS Master Audio, English, 2.0 channels, 24 bits, 48kHz
(core: DTS, 2.0 channels, 1509kbps, 48kHz)
5: Subtitle (PGS), English
[a03] Extracting audio track number 3...
[v02] Extracting video track number 2...
[a04] Extracting audio track number 4...
[a03] Decoding with libDcaDec DTS Decoder...
[s05] Extracting subtitle track number 5...
[a04] Decoding with libDcaDec DTS Decoder...
[a03] Encoding FLAC with libFlac...
[a04] Encoding FLAC with libFlac...
[v02] Creating file "Merlin.h264"...
[a03] Creating file "Merlin_1.flac"...
[a04] Creating file "Merlin_2.flac"...
[s05] Creating file "Merlin.sup"...
[a03] The original audio track has a constant bit depth of 24 bits.
[a04] The original audio track has a constant bit depth of 24 bits.
Video track 2 contains 65112 frames.
Subtitle track 5 contains 474 captions.
eac3to processing took 12 minutes, 27 seconds.
Done.
a03 and a04 turned out to be identical.