Horizontal noise bars - what are they called, and how to remove?

[johnmeyer]

I have been asked to fix this video, but have not yet received the DVD. It is of a 1945 American football game. I don't have access to the original film, but I was told that the original film to video transfer was done professionally. Whatever created these horizontal bars happened after that time.

I have been told that they are static (i.e., they do not crawl up or down the screen as the video is playing).

Question 1: does anyone know what this interference pattern might be called (so I can Google it)?

Question 2: does anyone have some general ideas of what they would use to fix it? If they truly are static, then I might just be able to get away using a mask of some sort and then adjust the exposure and color saturation. If they move, I'll have to devise another approach.

Question 3: Does anyone know what might have caused this?

I ask the last question because the person who purchased this disc may be able to find a source that is closer to the original transfer if we can pinpoint what caused the problem, and by knowing that, figure out how to go around the person involved.

Thanks!

Here's a link to an un-edited clip (MPEG-2 format from DVD):

Noise Bar Clip (https://www.mediafire.com/file/q1dahfsntbq14o4/1945_Cards_v_Lions_Clip.mpg/file)

.

[FranceBB]

Uhm... I've seen those in Analog BetaCAM as well, but I thought (and I might be completely wrong) they were introduced by the live analog transmission method occurred at the time.
The reason why I'm saying this is that in some very old oxide tapes I've captured there were live feeds which have been recorded at the time and they had this kind of issue.
What got my attention, though, is that you're having this issue on something which isn't supposed to have been aired... ever... so I don't know how they got there.
Could it be that it was somehow transferred from the original 1945 reel to Oxide BetaCAM and then from Oxide Analog BetaCAM to DVD and that step introduced them?
I'm just guessing 'cause back in the days there were no SDI cables and there might have been something introduced by some kind of VTR, but I'm really just speculating...
If someone knows, please let me know, 'cause I'm very interested in finding out why they're there.
As to how I fixed it, I'm afraid I didn't... Well I wasn't "superficial" enough to let it the way it was, but I don't remember what I used... I remember I've got some poor results when I was attempting to fix it "digitally", though.
What I found out at the time was that the Thompson VTRs were much more prone to show this than the Sony ones as there was some analog integrated circuitry that was attempting to fix this with some very good results in some scenes and some pretty poor ones in some others in the official Sony ones. I've never found out the root cause, though, so I'm interested to see if someones knows.

[GMJCZP]

Well, while a better alternative appears, and seeing the problem, I remembered the birth of vinverse2, I don't know if making any modification to the Didée script can do something:

Here (https://forum.doom9.org/showthread.php?p=1584186#post1584186)

[johnmeyer]

Well, while a better alternative appears, and seeing the problem, I remembered the birth of vinverse2, I don't know if making any modification to the Didée script can do somethingDidée posted some amazing scripts, back when he was active, so when the video arrives, I'll definitely look at the script and see if I can get ideas.

Could it be that it was somehow transferred from the original 1945 reel to Oxide BetaCAM and then from Oxide Analog BetaCAM to DVD and that step introduced them?This is a very real possibility. The person I am working with has ties to NFL Films: he provides them with content, and they send him some of their content in return. He claims that a previous disc from them had similar problems. NFL Films does professional scans of the film and then, back when this was done (analog days) transferred the results to broadcast video formats.

[Sharc]

Question 1: does anyone know what this interference pattern might be called (so I can Google it)?

Question 3: Does anyone know what might have caused this?

My guessing:
Q1: Mains interference (power frequency + harmonics) during analog signal transfer

Q3: Poor wire screening and/or groundloops between devices

Edit:
https://en.wikipedia.org/wiki/Mains_hum
https://en.wikipedia.org/wiki/Ground_loop_(electricity)

In video systems
In analog video, mains hum can be seen as hum bars, (bands of slightly different brightness) scrolling vertically up the screen. Broadcast television frame rates are chosen to match the line frequency, to minimize the disturbance these bars cause to the picture. A hum bar can be caused by a ground loop in cables carrying analog video signals,[5] poor power supply smoothing, or magnetic interference with the cathode ray tube.

[Boulder]

Q2: Chances are that some of vcmohan's plugins might help here. I think he has several which tackle (pun intended) this kind of area.

[kedautinh12]

Here mod for vinverse:
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Vinverse_avsi.avsi

[johnmeyer]

Thanks for the link to Didée's vinverse. GMJCZP linked to something Didée wrote that apparently was the predecessor to that function. I'll definitely try it out.

I've been Googling "hum bars" ever since someone else said it might be some manifestation of AC hum. Most of the visuals I've found for that, as well as my own experience trying to remove hum bars with a "humbucker" (balun to interrupt ground loops) is that AC hum bars tend to be much, much broader (wider, top to bottom, with only a few horizontal lines over the entire frame). This looks more like a beat between two frequencies, like you sometimes get with OTA adjacent channel interference

[GMJCZP]

If this is related to broadcasting interference, and if vinverse2 for some reason does not work, john, review this information, especially pay attention to the contribution of tritical. I uploaded the first two pages of the post because I couldn't see the images today. Remember, only use this Plan B if you are not satisfied with vinverse2, good luck.

Kurish post (http://www.filedropper.com/whatisthisartifactingcalled)

Post original today: https://forum.doom9.org/showthread.php?p=1404025#post1404025

[Emulgator]

Do we see a complete frame or a cropped one ?
Somehow I am missing upper/lower border artifacts...

Without a sample no clues for now, just tossing thoughts for association:
79 horizontal bars with square-modulated luminance...40 bars dark, 39 bright.
(Seems to be Y only to me, lowering Y alone would saturate colors)

A shift upwards a bit after half width.
Analog PAL has such behaviour that picture content of first line starts a tiny bit past half width.
But with US footage this should be out of the equation.

Still: any resizing using integers ?
(DLP menu preview had that, leading to 15/16 stuff, missing/doubled lines and so on.

Just a shot in the dark.

[johnmeyer]

The video was shipped to me today, so I'll have it on Friday or Saturday.

[johnmeyer]

Here's an unedited clip from the DVD:

Noise Bar Clip (https://www.mediafire.com/file/q1dahfsntbq14o4/1945_Cards_v_Lions_Clip.mpg/file)

I found that the last two minutes of the DVD is a blank screen (the tape had run out) which included the noise bars. It provides a hint as to what caused the problem. I note that the bars are not horizontal and that they jitter up/down slightly. I've attached a snapshot of the bars.

The 3:2 pulldown must first be removed before doing anything else.

I'll post my results, after I try a few of the suggestions.

[FranceBB]

I found that the last two minutes of the DVD is a blank screen (the tape had run out) which included the noise bars.


So it was a tape... My hypothesis turned out to be correct.


Here's an unedited clip from the DVD


Woah! This very much looks like what I had to deal with with those tapes, so I really bet it was an oxide analog BetaCam, 'cause I haven't seen those things in the Digital BetaCAM.




I note that the bars are not horizontal and that they jitter up/down slightly.


Yeah, if you were the one who was in charge of capturing it, your best allied would have been this monster: Sony DVW-A500P

https://i.imgur.com/8xcJiJ8.png

one of the boards you see on the left hand side is a logic with a series of analog filter and the one close to it is for analog to digital conversion and it's done in a really good way once the VTR has been referenced with an SD 720x480 30i limited TV Range Colorbars via SDI as it would be able to not only attenuate that issue but also provide good levels.
Unfortunately, you don't have access to the original tape and in the file they sent you the "damage" has already been done 'cause whatever has been used to play it and capture it:

A) Had a really poor analog circuitry and TBC
B) Wasn't referenced

The problem with "A" is common in the sense that I've seen it being "bad" in many professional VTRs like the ones from Thompson or in the J-30 and I consider anything but the official Sony ones to be not good enough. About the "B" that is pretty bad 'cause if a VTR isn't referenced, anything that could go wrong WILL go wrong. Sure, VTRs can generate their own reference signal and adjust automatically according to the input, but what they do is generally a pretty poor scene-by-scene based guess which is anything but good. Personally I've been capturing tapes with the model I posted above (referenced with a PAL SD colorbars signal via SDI 'cause I live in PAL Land) and a BlackMagic Decklink configuration via SDI saving everything in v210 4:2:2 planar lossless as Derek (Blue_MiSfit) suggested (Emulgator also helped me when I had to remote it) and I've now captured around 300 tapes (which have been Remastered and sent to our LTO6 archive). Unfortunately there's nothing you can do about the capturing method and you've received a lossy file as well, so it's gonna be tough to remove those things digitally but I believe you will find a way ('cause the power of Avisynth is endless). Still, it saddens me that you have to do an extra work and spend a lot of time when your job would have been easier if whoever was in charge of capturing it did "the right thing"... :(


Keep us posted John and best of luck.

[vcmohan]

From the image posted in the first post of this thread, I think as already mentioned by some one, the bars are due to power line frequency interference. This can be easily removed using my plugin manyPlus ( also libfftw3f-3.dll) in the path. F1Quiver can be used as long as the direction of noise bars do not change much in the frame. Need to do a turnLeft() prior and later TurnRight.
Alternative is to use F2Quiver. In place of F1Quiver you can also try Fan function. Same process of turnleft, Fan, TurnRight may work.

[johnmeyer]

So it was a tape... My hypothesis turned out to be correct.I think I misled you when I said "the tape ran out." I meant to say, "the film ran out." This was, as I said in my OP, a transfer from a 1945 16mm color film. What I just received was a DVD (hence the MPEG-2 file which was losslessly cut from the rip).

However, you may very well be correct because, like with so much video, it has probably been through a tortured path, even though the source was, initially, from a very professional organization.

Of course the fact that pros dealt with this doesn't mean it was done correctly. At some point these bars got added, and whoever let that happen must have been legally blind to have not noticed, or they just didn't care.

... the bars are due to power line frequency interference. This can be easily removed using my plugin manyPlus ( also libfftw3f-3.dll) in the path. F1Quiver can be used as long as the direction of noise bars do not change much in the frame. Need to do a turnLeft() prior and later TurnRight.
Alternative is to use F2Quiver. In place of F1Quiver you can also try Fan function. Same process of turnleft, Fan, TurnRight may work.Thanks. I will give those a quick try. Defreq worked very well on the middle of the video, but left some residual bars at the top and bottom.

[FranceBB]

I think I misled you when I said "the tape ran out." I meant to say, "the film ran out."


Ah yeah, got it.


whoever let that happen must have been legally blind to have not noticed, or they just didn't care.


Yeah... I'm pretty sure they didn't care...
You would be surprised by how many people are in some professional organizations who just don't care and wanna get through the day to just get to lunch time and then home... It's a shame, really... :(

[Ghitulescu]

"Timeliness" is the keyword here, in such companies.
Nobody wants to wait, therefore they use hardware solutions which provide excellent results even without too much operators' involvement.

If one doesn't like the material, he'll let on Auto and goes for lunch.

[vcmohan]

Thanks. I will give those a quick try. Defreq worked very well on the middle of the video, but left some residual bars at the top and bottom.
Further I tested on your screenshot as the video had only a logo. Screen shot clips amplitudes so that sinusoids will become step functions, which can not be filtered out. These will have almost all harmonics. However I tested with following code using my manyPlus plugin.


imagereader("c:\images\BaseBall.jpg",end = 100)
turnleft()
crop(0,0,800,1060)

#F1q=F1QTest(0,100,25,10,30,10,40,100,60,100,70,20,80,20,100,100, custom = true)
#
F1q=F1Quiver(0,100,25,10,30,10,40,100,60,100,70,20,80,20,100,100, custom = true)
F1q=turnRight(F1q)

you can see lot of residual noise. One way to reduce is use
F1q=F1Quiver(0,100,25,0,30,0,40,100,60,100,70,0,80,0,100,100, custom = true, morph = false, uv = true)
But it will introduce too many artifacts. Normally such sharp filters should not be used, but you can try.

[Frank62]

There is a step in the lines in the middle of the picture, about two pixels up seen from left to right. Could it be useful to handle the two halves seperately?
(Just saw it, if this is senseless, just forget... I don't know much of these brillant frequency plugins).

[vcmohan]

Yes. There are several inclined line ups where more than 2 pixels shifts are seen. These will require possibly segmenting the frame into several parts and process separately. In 2D freq display appears to be too complicated to analyze. May be the original video will look better

[johnmeyer]

I finished the project. The attached photo shows the same frame as in my post #1, but with the improved version on the right.

I ended up using Defreq, and got some help with the settings from "jagabo" over in the Videohelp.com forum. Here is the script I used:

loadPlugin("C:\Program Files\AviSynth 2.5\plugins\defreq.dll")
AVISource("E:\fs.avi").KillAudio().converttoYV12(interlaced=true).AssumeTFF()
tfm()
source = tdecimate(cycle=10,cycleR=2)
TurnRight(source)
defreq(fx=16.6, fy=0.0, dx=2, dy=2, sharp=0, info=false, show=0)
defreq(fx=49.4, fy=0, dx=2, dy=1, sharp=0, info=false, show=0)
TurnLeft()
output = aWarpSharp(depth=5).Sharpen(0.3)
return output

#Alternate output, for debugging
#interleave (source,output)
#stackhorizontal (source,output)

I also did some dirt removal, stabilization, some additional sharpening, and color correction, the last one in order to reduce the reddish hue.

It's not perfect (you'll see some residual banding, if you look for it, but it is a LOT easier to watch.

[Frank62]

Impossible for each professional studio in the world to my knowledge. Can only be done with avisynth and always experimenting, developing, learning. Looks really great!

[StainlessS]

Nice job John/Jagabo, the client will likely be most happy. https://www.cosgan.de/images/smilie/sportlich/f030.gif

[johnmeyer]

Nice job John/Jagabo, the client will likely be most happy. https://www.cosgan.de/images/smilie/sportlich/f030.gifStainlessS, the color correction was done with GamMac.

[FranceBB]

Well done, John! Avisynth once again proved to be the best frameserver to tackle these things :)

[kedautinh12]

Can anyone compile defreq conformable with x64??

[real.finder]

Can anyone compile defreq conformable with x64??

there are one from 2007 https://github.com/realfinder/UniversalPluginsFolders/blob/master/plugins64/25/defreq.dll I think it's from http://members.optusnet.com.au/squid_80/defreq64.zip

it's work but it's better if someone did update it to new color formats and HBD

[kedautinh12]

Thanks

[GMJCZP]

Good job john! Thank You vcmohan!

[patul]

Impressive work John, out of curiosity, did you try to remove the residual banding? My impression is that residual is similar to residual combing, which I often see in YT videos and got me irritated & frustrated to remove that residual combing. Sadly those videos are from some official club football youtube channel, I guess they just don't care to properly edit/process their videos. This issue is typically found in some old footage, which I bet they were from interlaced sources.

Confession: I'll have to "blame" you guys for the change of my taste :D, because the more I follow these avisynth threads, the more obvious errors/mistreatment of those videos and I can't stand them anymore hence I have to do something just to be able to watch the damn video. Thank you guys for this.

[johnmeyer]

There was a LOT of residual banding, mostly at the top and the bottom. This was probably due to the bars not being the same distance across the frame. Also there was a discontinuity in the center of the frame. This caused a slight vertical band of residual "stuff," although as you can see from the frame capture, this is pretty subtle.

If this had been for a paying customer rather than for a friend who does volunteer work for NFL Films (he finds one-of-a-kind old game films) I would have done more work.

The only thing that I was a little disappointed with was the GamMac color correction. It is a very useful tool, but I have never figured out how to tune it. I think the result was just a tad too blue. I probably should have left it with the slightly warm (too red) color.

[GMJCZP]

If exists residual combing vinverse (not vinverse2) maybe is a good solution.

[StainlessS]

I was a little disappointed with was the GamMac color correction
First image here [Bot RHS] : https://forum.doom9.org/showthread.php?t=173695&highlight=Gammac
Has BluMul set at 0.95, you can see effect compare to the image above it.
[maybe a tad more reduction would be better for that image].
Theres also a lot of green [the default lockchan] in both images, and both are a bit blue.
[ I'm lousy at color correction, look great when I do it, next day looks awful :) ]

EDIT:
Some wildlife from around here john, [no cougars in this neck of the woods],
4 remaining out of clutch of 9 [EDIT: Gaggle of 9]
https://i.postimg.cc/3kj1ZjMk/Goslings.jpg (https://postimg.cc/3kj1ZjMk)