Restoring "Remember that Night"

[Boulder]

I'm trying to restore David Gilmour's Remember that Night, but I've run into some issues. SRestore with different frate values produces choppiness at some frames, TFM cleans things up but bobbing the output shows that some fields are not OK.

Here's a link to a sample:
https://drive.google.com/file/d/0BzeF_1syecQwNjllTkI3WTVxRFk/view?usp=sharing

Can anyone verify whether TFM is the way to go and that the source is just butchered and cannot be restored properly?

[kuchikirukia]

I'd be interested to know how they managed that. It looks like it's blended with something with a mismatched frame rate, since it seems to go in and out of sync.

But anyway,
QTGMC()
... doesn't look all that objectionable.

[johnmeyer]

The first few fields are reversed (BFF instead of TFF). After that, looking at individual fields, there are some frames where there is no temporal difference between fields. Then there are many frames where one of the fields is a blend, and the other is a good field.

I suspect that the source of this video is not the DVD, but instead is a capture from YouTube or some other source. I have seen this sort of thing from such captures. Suggestion: start with the DVD, if possible.

[Boulder]

That clip's from the original Blu-ray..

At first I thought it was just deinterlaced telecined stuff but FixBlendIVTC couldn't do a thing about it. The video is quite dark throughout the whole concert so it's difficult to get a good sample to analyse. Here's a different one, don't know if it's any better or if it just shows the same symptoms: https://drive.google.com/file/d/0BzeF_1syecQwUWNmQTViaXA3NXc/view?usp=sharing

[johnmeyer]

Wow, that's on the Blu-Ray? I don't own a Blu-Ray so I don't know how ripping is done. Can you tell, by playing the Blu-Ray disc, whether this problem exists on the disc? If it is, a transfer like that is incompetent.

I collect all manner of old concerts, and while this one is not old (most of my stuff is from the 60s), Pink Floyd is sort of old, so I put the DVD at the top of my Netflix queue, and I'll check it out. When I get it, if I find something that may be of use, I'll pass it along.

[Boulder]

Ripping BDs is basically the same as with DVDs, you even have the same problems with bad transfers as you can see here :D

There's a slightly unnatural look to the video when you watch it on a big screen. Fortunately the music's good so you don't notice it unless you start looking for it. Sometimes these ones shot in Europe (this one's from the UK) are shot in 25fps and you can SRestore it without any issues but unfortunately that's not the case here.

[johnmeyer]

Sometimes these ones shot in Europe (this one's from the UK) are shot in 25fps and you can SRestore it without any issues but unfortunately that's not the case here.Hmmm ... I wonder if there is a PAL version, or does such a thing not exist for Blu-Ray? The idea would be to get a disc that is as close to the "native" format in which it was shot.

[johnmeyer]

In answer to my own question, it does appear that a PAL version does exist. Whether it has the same problems is a question I cannot answer.

Remember That Night (UK-PAL) (http://www.amazon.co.uk/Remember-That-Night-Blu-Ray-DVD/dp/B000WZ8Y24/ref=sr_1_1?ie=UTF8&qid=1440648723&sr=8-1&keywords=gilmour+night)

[creaothceann]

Only one way to find out. ;)

[kuchikirukia]

50i blended to be 60i?
Anyway OP, where are you getting that this is telecined?

[Boulder]

I saw one part in which this pattern of three cleanish, two slightly ghosted ones appeared. However, I'm not exactly sure if it was bobbed at that point because I've gone back and forth with this one.

[kuchikirukia]

I'd say it's 60fps. There's a point in the cycle where two fields are close, but I'm not sure it's actually synced to a short cycle. It probably steps out, since I do notice some motion in some.

[Boulder]

I ended up QTGMCing the video and encoding at 59.94 fps. It's not perfect but very much watchable.

From what I've read, the concert was shot with multiple different cameras of varying quality - which you can also tell by the fact that some parts look much worse than others. Maybe the cameras were also shooting with different parameters and then the author had to make this weird hybrid mess out of the material to get at least something out :D

[johnmeyer]

Netflix delivered the NTSC DVD today and I looked at it field-by-field.

It has exactly the same problems as the Blu-Ray footage you posted

So, the next step is to see if I can find the PAL version. I'm pretty sure the only way to get good video is to rent or buy the PAL version.

Whoever was responsible for the PAL to NTSC conversion clearly didn't come to class the day they taught "standards conversions."

[feisty2]

Whoever was responsible for the PAL to NTSC conversion clearly didn't come to class the day they taught "standards conversions."

PAL --> NTSC
you just need to:
1. Deinterlace (if interlaced)
2. Resize to 720x480
3. pulldown (go directly from 25 to 29.97 or slow it down a little bit to 24fps then apply a standard 3:2 stuff)

doubt if there is anyone in this world gotta "go to some class first" then figure out how to do this simple 3-step work...seriously?

[johnmeyer]

PAL --> NTSC
you just need to:
1. Deinterlace (if interlaced)
2. Resize to 720x480
3. pulldown (go directly from 25 to 29.97 or slow it down a little bit to 24fps then apply a standard 3:2 stuff)

doubt if there is anyone in this world gotta "go to some class first" then figure out how to do this simple 3-step work...seriously?Well, maybe so, but did you download and look at the OP's video? It was from the Blu-Ray, and I just confirmed that the NTSC DVD is screwed up in exactly the same way. So, whoever was responsible for this conversion actually did need to go to class.

Also, the original was PAL interlaced ("50i"), NOT progressive, so standards conversion gets a little more complicated than what you outlined, and the trick of slowing down to 24 fps and then setting the 3:2 pulldown flag isn't an option.

[feisty2]

Also, the original was PAL interlaced ("50i"), NOT progressive, so standards conversion gets a little more complicated than what you outlined.

more complicated, how?
it will be 50p after step1, then you throw half of the frames away, then it's 25fps, then you just do the pulldown


and the trick of slowing down to 24 fps and then setting the 3:2 pulldown flag isn't an option.

why? you speed movies up to PAL, why slowing PAL down to a movie isn't an option?

[johnmeyer]

more complicated, how?
it will be 50p after step1, then you throw half of the frames away, then it's 25fps, then you just do the pulldown.Does that produce smooth video? Maybe it is just the way you are describing it, but it doesn't sound quite right.

why? you speed movies up to PAL, why slowing PAL down to a movie isn't an option?Once again, you are describing a technique used for progressive material. AFIK, you can't set the DVD pulldown flag for interlaced material.

Here's a good starting point for doing interlaced PAL to interlaced NTSC conversions. I think the code is from the AVISynth documentation:

PAL to NTSC Interlaced (old Neuron code) (http://forum.doom9.org/showthread.php?p=1064777#post1064777)

You can also use QTGMC for these conversions. Nick Hope has posted some very nice scripts that can provide some amazing results although, like every script that uses motions estimation, you need to check the results carefully to make sure it doesn't create any noticeable artifacts.

[feisty2]

I was just stating the obvious, you wanna get advanced, still pretty simple
1. Deinterlace (50p now)
2. Motion flow (svp kinda stuff) to 300fps
3. Select one frame out of every 5 frames (60p now)
4. Throw half of the fields away (60i, NTSC now)

[johnmeyer]

I was just stating the obvious, you wanna get advanced, still pretty simple
1. Deinterlace (50p now)
2. Motion flow (svp kinda stuff) to 300fps
3. Select one frame out of every 5 frames (60p now)
4. Throw half of the fields away (60i, NTSC now)Yup, that's it.

[feisty2]

But seriously... No one gives a damn about PAL or NTSC now like, really
PAL or NTSC, it's just a video file to a computer! Nothing more.

[johnmeyer]

Back to the OP's problem ...

I'm working on getting the PAL version of this concert. I hope to have something to report in a few days. My expectations are that the video will be perfect.
==========================

As for the odd comment about PAL and NTSC ...

PAL and NTSC most definitely DO matter, even for those who only play video on a computer. They matter because they are still the only hard and fast video standards that ever existed and, if you deliver video to clients, as I do, they provide a way -- with 100% certainty -- that I can ensure that my client can play the results of my work. Every television set and every DVD player ever made plays one of these formats.

I can guarantee it.

By contrast, I can put video on a thumb drive and give it to someone, and I have zero percent assurance that he or she will be able to plug it into a TV set and have it work. Maybe it will, and maybe it won't. I have no way of knowing. Even on their computer, it may or may not work if they don't have the latest codec pack, or if their computer is too slow to play a high bitrate HD video stream.

And, if they have a hand-held Android/iPhone/etc., there are so many variations on what each generation will play that I either have to deliver video at 640x480 at no more than 1.5 mbps (and even then it may not work on older devices), or I have to make sure the clients all have the latest generation phones or tablets.

So, if you deliver video for a living, PAL and NTSC are not only still relevant, they are vital.

I am just about to do two more weddings, and while I will try to deliver an HD version on a thumb drive, my main deliverable will be an NTSC DVD. That media cannot be erased, is likely to last forever (based on my own aging tests, as well as major commercial longevity tests); and will be easily playable for decades to come (I can still easily play CDs, which were introduced over thirty years ago in 1984).

[Boulder]

Thanks John, I appreciate your efforts. I just don't understand why they didn't use the PAL version for the whole world as I'd assume that any Blu-ray capable device nowadays is able to playback both?

This is one concert I always seem to return to, I've watched it many, many times on a lazy Saturday evening. In fact, in three weeks I'll be seeing Mr. Gilmour perform at the very same place, only nine years later :D

[sneaker_ger]

AFAIK 50Hz is not mandatory for BluRay world-wide.

[Boulder]

OK, then I'm mistaken. I checked at Wikipedia, which says that 1920x1080i50 is a supported format so I thought it would apply to any case.

[johnmeyer]

OK, I got the PAL DVD, and it turns out to be 25p, not 50i. Here's a short clip, cut directly from the VOB using Womble's excellent MPEG Video Wizard cutter:

Gilmour (PAL).mpg (https://www.mediafire.com/?iek1k3h13z80al6)

Knowing that it started out life as 25p may, perhaps, help in restoring the 60i version, although my recommendation is to get the PAL DVD, if this video is important to you.

[johnmeyer]

Here's the MediaInfo (except for the subtitle streams) printout for the first VOB file:

General
Complete name : E:\VTS_01_1.VOB
Format : MPEG-PS
File size : 1 024 MiB
Duration : 24mn 28s
Overall bit rate : 5 847 Kbps

Video
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings, BVOP : Yes
Format settings, Matrix : Default
Duration : 24mn 28s
Bit rate mode : Variable
Bit rate : 4 835 Kbps
Width : 720 pixels
Height : 576 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 fps
Standard : PAL
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.466
Stream size : 846 MiB (83%)

Audio #1
ID : 189 (0xBD)-128 (0x80)
Format : AC-3
Format/Info : Audio Coding 3
Mode extension : CM (complete main)
Muxing mode : DVD-Video
Duration : 24mn 28s
Bit rate mode : Constant
Bit rate : 448 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Compression mode : Lossy
Delay relative to video : -80ms
Stream size : 78.4 MiB (8%)

Audio #2
ID : 189 (0xBD)-129 (0x81)
Format : AC-3
Format/Info : Audio Coding 3
Mode extension : CM (complete main)
Muxing mode : DVD-Video
Duration : 24mn 28s
Bit rate mode : Constant
Bit rate : 448 Kbps
Channel(s) : 6 channels
Channel positions : Front: L C R, Side: L R, LFE
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Compression mode : Lossy
Delay relative to video : -80ms
Stream size : 78.4 MiB (8%)

[johnmeyer]

One last thing.

If your German is pretty good, here is a description of the technical details of how the performance was recorded:

David Gilmour – Remember That Night – Live at the Royal Albert Hall (https://klangspektrum.wordpress.com/2012/03/07/david-gilmour-remember-that-night-live-at-the-royal-albert-hall/)

I haven't used German for fifty years, so I can't quite tell whether it was filmed in 24p or 25p, and then converted (sped up or slowed down), but it was definitely filmed using progressive, not interlaced.

[Boulder]

"They took advantage of the system Alchemist Ph.C-HD from Snell & Wilcox for HD cross-conversion from 25P to 59.94"

GAH :(

I've already tried using SRestore to get back to 25p but it doesn't work all the way. Of course, it's possible that the material is too difficult because it's very dark throughout the video and you would need many different thresholds according to the scene in question.

[johnmeyer]

I wonder if it was Snell or Wilcox who screwed up?

Yes, exposure can really screw up algorithms that are trying to track motion. I've had some success (not 100% of course) creating the vectors for MVTools (or SRestore) using an autogained version of the video. This evens out the exposure and provides some detail (sometimes) in the really dark areas. I then feed these vectors to MVTools2, but have it actually operate on the original, unaltered video.

Of course the simplest thing is to spend $15 and get the PAL DVD or Blu-Ray from Amazon UK or another source, now that we know for sure that there is no problem with those discs (as my short upload demonstrates). Trying to fix what you've got is always going to involve compromise.

When people bring me their vinyl records or 78 rpm disks for restoration, I always look up online to see if the same material is available on CD. If it is, I refuse to do the restoration because I can never fix something to sound (or look) as good as the original source material when transferred correctly. Occasionally I'll still do it, but only if there is substantial difference between the CD and the original vinyl record (e.g., most of the Beatles catalog).

[feisty2]

Yes, exposure can really screw up algorithms that are trying to track motion.

do it with frequency methods (like dct=1 for mvtools)
these kind of stuff (exposures, fades...) just, won't affect the image much, frequency-wise.

[Boulder]

Just a quick thought: would it possibly help if I inverted the colours of the analysis clip for SRestore? I'm only able to test it on Sunday.