How, mathematically, downscaling works?

[luquinhas0021]

I talk about, here, turn an image into a polynomial function f(x,y) = z.
In upscaling task, all it has to do is replace into function the coordinates of the interpolated pixels.
What about downscaling task: is there something that can be done in the polynomial that is the image function?

[Katie Boundary]

http://entropymine.com/imageworsener/resample

^this explains things in simple English with lots of pretty pictures. It's a good place to start.

[LoRd_MuldeR]

I talk about, here, turn an image into a polynomial function f(x,y) = z.
In upscaling task, all it has to do is replace into function the coordinates of the interpolated pixels.
What about downscaling task: is there something that can be done in the polynomial that is the image function?

As always, the input image consists of sampling points, i.e. the image value (brightness or color) is only "known" at certain discrete points.

In between those sampling points the image value is not known, so a 2D interpolation (e.g. BiLinear or BiCubic) is used to approximate the image data in between the original sample points:
https://upload.wikimedia.org/wikipedia/commons/thumb/9/90/Comparison_of_1D_and_2D_interpolation.svg/1024px-Comparison_of_1D_and_2D_interpolation.svg.png

Finally, new sampling points are taken from the interpolated curve (function), which then form the output image. Again the output image will only contain discrete sampling points.

The difference between "upscaling" and "downscaling" is that the former creates an output image with more (denser) sampling points, while the latter creates an output image with less (sparser) sampling points - compared to the input image.

(BTW: Why does "downscaling" require interpolation at all? Why not simply discard a certain fraction of the sampling points from the original image? The reason is that, depending on the scaling factor, the remaining sampling points of the "downscaled" image may be located at a position where there is no sampling point in the original image! So the image value at the location of the "new" sampling point needs to be interpolated/approximated from the original sampling points)

[nevcairiel]

(BTW: Why does "downscaling" require interpolation at all? Why not simply discard a certain fraction of the sampling points from the original image? The reason is that, depending on the scaling factor, the remaining sampling points of the "downscaled" image may be located at a position where there is no sampling point in the original image! So the image value at the location of the "new" sampling point needs to be interpolated/approximated from the original sampling points)

Thats not the only reason for proper downscaling interpolation - even if you were to hit a point exactly at all times, you might still drop a lot of information of the original image without taking it into account at all. Hit one bright pixel in an otherwise mostly dark image, and suddenly a much larger area might be bright. Thats why downscaling typically averages areas to account for all information and not distort by random chance.

[luquinhas0021]

That's not the only reason for proper downscaling interpolation - even if you were to hit a point exactly at all times, you might still drop a lot of information of the original image without taking it into account at all. Hit one bright pixel in an otherwise mostly dark image, and suddenly a much larger area might be bright. That's why downscaling typically averages areas to account for all information and not distort by random chance.

There's, two, the aliasing problem: simply discard pixels, like Point Downscaling does, causes edge discontinuity. instead of, for example, SSIM Downscaler, that attempts to try to reconstruct the edge.

[Katie Boundary]

As always, the input image consists of sampling points, i.e. the image value (brightness or color) is only "known" at certain discrete points.

In between those sampling points the image value is not known, so a 2D interpolation (e.g. BiLinear or BiCubic) is used to approximate the image data in between the original sample points:

This discussion has already been had. The short version is that Mulder's explanation is wrong because it pretends that images are waveforms.

More information was provided by MP4 Guy here (https://forum.doom9.org/showpost.php?p=1711511&postcount=12)

[LoRd_MuldeR]

This discussion has already been had. The short version is that Mulder's explanation is wrong because it pretends that images are waveforms.

Nope!

Just like an audio signal is a continuous curve in 1D time-space (aka "Waveform"), an image is a continuous function in 2D space. And both are sampled for digital storage, i.e. the signal value is only captured/store at discrete (infinitely small) points. Only difference is that for an audio signals we store the signal value at discrete points in 1D time space, while for an image we store the image value at discrete points in 2D spatial space.

Neither are there any "stairsteps" in a digital/sampled audio signal, nor do the "pixels" in a digital/sampled image have any "area". Representing pixels (i.e. samples of a digital image) as solid-color squares that actually take up an area is nothing but a "workaround" which is used by image editors - because you wouldn't be able to see an infinitely small sample points. It is still not an accurate representation (not at all) of what the sampled image data actually is ;)

Watch this video, especially the "there are no stairsteps" chapter:
https://xiph.org/video/vid2.shtml


https://upload.wikimedia.org/wikipedia/commons/thumb/9/90/Comparison_of_1D_and_2D_interpolation.svg/1024px-Comparison_of_1D_and_2D_interpolation.svg.png

[Sharc]

Just a thought on the "stairsteps" (the temporal aspect maybe a bit off topic though):
In a movie each picture (or frame in digital video) is displayed for 1/24s = 41.7ms. During this time the picture does not change, means our eyes are fed with a sequence of pictures of 41.7ms duration each, which is per definition a "stairstep" function. The interpolation filter is the human apparatus (eyes, brain ....). What is this human filter like? Some kind of lowpass, I assume ..... (btw. certain animals have "interpolation filters" which are totally different from humans, that's why they don't go to the cinema, I presume).

[nevcairiel]

Neither are there any "stairsteps" in a digital/sampled audio signal, nor do the "pixels" in a digital/sampled image have any "area".

While I agree this is true for audio, for video its not as simple. Digital sensors in a camera aren't infinitely small dots that capture one infinitely small sample at given distances, the pixels in a sensor are an area of a certain size, so the pixel-area representation is closer to their actual sampled signal then an infinitely small dot in the center of the pixel.

[Sharc]

While I agree this is true for audio, ....
Being nit-picking it's not even fully true for audio, as real A/D converters and sampling devices have a non-zero aperture time. The effect is normally negligible though and one can assume "sharp" samples.

[wonkey_monkey]

This discussion has already been had. The short version is that Mulder's explanation is wrong because it pretends that images are waveforms.

More information was provided by MP4 Guy here (https://forum.doom9.org/showpost.php?p=1711511&postcount=12)

I don't see how MP4 Guy's post contradicts LoRd_MuldeR's perfectly well-informed explanation.

The interpolation filter is the human apparatus (eyes, brain ....). What is this human filter like?

I don't think there is any such temporal filtering. That's why movies seem to "judder" and have a "filmic" effect.

Being nit-picking it's not even fully true for audio, as real A/D converters and sampling devices have a non-zero aperture time. The effect is normally negligible though and one can assume "sharp" samples.

Isn't audio usually passed through a high-pass filter before sampling for exactly this reason? To avoid "sharp" samples?

[Katie Boundary]

While I agree this is true for audio, for video its not as simple. Digital sensors in a camera aren't infinitely small dots that capture one infinitely small sample at given distances, the pixels in a sensor are an area of a certain size, so the pixel-area representation is closer to their actual sampled signal then an infinitely small dot in the center of the pixel.

Yes, there's that. There's also the fact that all bets are off as soon as any resizing is done. If a 1920x1080 image is resized to 720x480 using Area Averaging, then suddenly each pixel represents the average color of a rectangular piece of the image. And if you're dealing with CGI/machinima, then you're at the mercy of whatever algorithm is used to render images...

In other words, sampling is just one of many models of reality, and presenting it as the only model is wrong.

[Sharc]

I don't think there is any such temporal filtering. That's why movies seem to "judder" and have a "filmic" effect.
I think there must exist some kind of "human interpolation filter" (temporal resolution limit) in order to perceive the individual pictures as a more-or-less fluent motion. 24fps is a cost driven lower limit (cost of the film material = storage cost) which is for most people bearable without causing nausea. That does however not mean that 24 pictures per second is fast enough to represent fast changing natural processes undistorted. An example is the backward turning wheels caused by the too low picture sampling rate producing aliasing.

Isn't audio usually passed through a high-pass filter before sampling for exactly this reason? To avoid "sharp" samples?
Hmmm..., don't you mean low-pass filter, means remove all frequencies which are above half the sampling frequency in order to avoid aliasing (=folding back the high frequencies to the useful frequency range causing signal distortion)?

[StainlessS]

24fps is a cost driven lower limit (cost of the film material = storage cost) which is for most people bearable without causing nausea.

Do I not recall that 24 FPS projectors display each frame twice, taking it up to 48 FPS and within reasonable range of the more natural lower limit of about 50 Hz ? (otherwise flicker is apparent @ 24).

[hello_hello]

While I agree this is true for audio, for video its not as simple. Digital sensors in a camera aren't infinitely small dots that capture one infinitely small sample at given distances, the pixels in a sensor are an area of a certain size, so the pixel-area representation is closer to their actual sampled signal then an infinitely small dot in the center of the pixel.

Yeah but the "pixels" capture the three primary colours individually so you could argue a pixel is a combination of samples, and the final colour for each pixel is computed by using information from neighbouring pixels (in a way I don't understand) to increase the resolution, and ignoring the gaps between "light cavities".... each pixel could be seen as something the camera confabulated from it's sensor and doesn't really represent a particular "area", but rather it's a "sample" at a particular point in the image.
http://www.cambridgeincolour.com/tutorials/camera-sensors.htm

Maybe another way of looking at it.....

I don't know what Katie imagines a pixel could be other than a sample. That makes no sense. If you resize down, each new pixel might represent some sort of average of pixels from the original image, but that doesn't mean a new "sample" wasn't created.

[Sharc]

Do I not recall that 24 FPS projectors display each frame twice, taking it up to 48 FPS and within reasonable range of the more natural lower limit of about 50 Hz ? (otherwise flicker is apparent @ 24).
Possibly yes. However, I would assume that displaying the same picture twice does not improve the time resolution of the picture sequence as the same picture is displayed for 2*1/48 = 1/24 second. So I think there must probably be another reason for the 48fps projector playback, or some "psychovisual" or "anti-flicker" effect (?)
This is different for video when we bob (instaed of single rate deinterlace) a video. The time resolution of the single rate deinterlaced video is 30fps while the bobbed sequence produces 60 different pictures per second (for action or panning scenes).

[StainlessS]

So I think there must probably be another reason for the 48fps projector playback, or some "psychovisual" or "anti-flicker" effect (?)


Yes, anti-flicker. (same for PAL 50 fields/sec or NTSC 60 fields/sec, 25/30 Frames/sec would flicker).

[raffriff42]

Isn't audio usually passed through a high-pass filter before sampling for exactly this reason? To avoid "sharp" samples?Yes, if you mean low pass - and the same applies to video:
wikipedia/Anti-aliasing_filter/Optical_applications (https://en.wikipedia.org/wiki/Anti-aliasing_filter#Optical_applications)

[Midzuki]

Yes, anti-flicker. (same for PAL 50 fields/sec or NTSC 60 fields/sec, 25/30 Frames/sec would flicker).

30fps does flicker. In 2008 I created a short interlaced MPEG-2 clip which emulated a 29.97 f*s-per-second sequence, authored a DVD disc with it, and watched it on a 29-inch analog TV set.

What an annoying, disgusting, terrible experience,
is all I can say :D

[Katie Boundary]

That's funny. None of my old CRT televisions ever flickered. Maybe it was just your TV and had nothing to do with the frame rate.

[StainlessS]

25/30 Frames/sec would flicker

By that I meant 25 or 30 FRAMES/SEC (ie not 50i, 60i) as on a monitor.
(50i and 60i simulate the double frame exposure as in projector). (though some may still claim to see flicker)

[Midzuki]

That's funny. None of my old CRT televisions ever flickered. Maybe it was just your TV and had nothing to do with the frame rate.

If you never fed your old analog TV sets with a sequence made of half of the fields replaced with 240 (or 288) black horizontal lines, then you have no idea of what I was talking about :)

Also, the human peripheral vision is much more sensible to flickering than the central vision. My peripheral vision did perceive the flickering of ANY analog TV set. Fortunately, my peripheral vision did not get in the way when I sat right in front of the old heavy tubes :)

[wonkey_monkey]

Katie simply likes to be contrary. Dare to claim the sky is blue and you'll probably get a reply spelling out why red is the only obvious choice.

24fps is a cost driven lower limit (cost of the film material = storage cost)

That was the reason, but it hasn't been for a long time. 24fps is stuck with because it has become the convention. It sets a certain tone and gives a certain impression because of what viewers are used to.

Do I not recall that 24 FPS projectors display each frame twice, taking it up to 48 FPS and within reasonable range of the more natural lower limit of about 50 Hz ?

Wasn't that down to how the shutters worked in projectors? A shutter had to come down to block the projection while the film advanced to the next frame. Wasn't there some compromise with how quickly the shutter could be made to work and how quickly the film could move which means the shutter comes over the projection once without the film moving? So each frame kind of gets displayed twice?

Now I think about that I'm not sure it adds up...

[johnmeyer]

Yes, most film projectors use a shutter that not only closes during the time that film is pulled down, but then also closes two more times while the film frame is stationary in the projector gate. This increases the flicker rate from 24 fps to 72 fps. The reason for this is that early movie houses (during the silent era) found out that many in the audience got headaches from the 24 Hz. flicker. Even if you don't directly perceive it, the flicker can still wear you down.

24 fps was used for film both because of cost considerations (faster speed uses up VERY expensive film at a MUCH faster rate) and also because it was difficult and expensive to make cameras and projectors that could run much faster.

As for as whether people can perceive flicker on any device, that very much depends on both the person and the source material. A great example of this is the technology that was briefly really hot in the late 1990s. It was called DLP and used a Texas Instrument chip that consisted of millions of micro-mirrors on a pivot that reflected (or didn't reflect) light through a rotating RGB wheel. Each field of video was projected three times, once for each color. As a result, when movement was fast, the red, green, and blue components of each field were projected at slightly different points in time. Some people claimed they could see fringing along the edges of fast-moving objects and as a result hated DLP. Others said they saw nothing and were really happy with it.

So, I am not surprised that some people might see an artifact related to how an image is painted on the screen, while others will not be able to see it. We are all wired a little differently.

[StainlessS]

If you never fed your old analog TV sets with a sequence made of half of the fields replaced with 240 (or 288) black horizontal lines,)


Thanks for the clarification Midzuki, I did not originally understand what you meant.

John,
Thank you also for you comeback, I was hoping that you would comment. I had posted about double projection in some thread a few years ago but nobody followed up and just seemed to ignore my comment, was not sure if I had totally imagined it. (well I was nearly there with double projection).

EDIT: Cheers RaffRiff below.

[raffriff42]

So each frame kind of gets displayed twice?Three times (72 Hz) on the better projectors.
wikipedia/Movie_projector/Shutter (https://en.wikipedia.org/wiki/Movie_projector#Shutter)

[wonkey_monkey]

Oh, I think I remember now about the projector shutter, now you've described it. It has to be large enough to cover the whole frame, but to keep size down it's a small disc, so it's like 50% of the area is blacked out. It the projection rate is 24fps, then for 1/48th of a second there is black. If you speed it up and show each frame three times, it's still the same amount of black, but split into smaller chunks, with the frame only advancing on every third chunk of black. Another way to solve the problem (I assume) would to have a larger disc or bar spinning round, so the shutter is only in place for a fraction of the time, but I guess the other way was easier, or something.

I know I'm extremely sensitive to frame rates and flickers, thanks to all my work with video. When I first got a 100Hz TV (CRT) it was so solid compared to 50Hz. And I hated DLP for exactly the reasons described :)

[Sharc]

Interesting. The shutter activity explains the flicker (and ways to reduce it), but still our eyes and brain get fed with a sequence of discrete pictures separated by 1/24s which most humans perceive as a more or less fluid motion, probably thanks to some interpolation or inertia (kind of low pass interpolation filter) of the human seeing apparatus. Is there a model or consensus of this "filter"?

[StainlessS]

I Believe I saw some experiments on BBC Open University TV program about 20+ years ago where they showed 24FPS frames @ 24FPS [EDIT: or 25] and without any double projection, looked a lot like Charlie Chaplin movies IIRC. [EDIT: There was probably some TV jiggery pokery to allow you to visualize on PAL TV].

EDIT: OT. On another Open University prog, remember them showing effect of switching OFF error detection and correction
[EDIT: switching OFF highest 1 or 2 levels of error correction]
whilst playing a DVD disk, results were totally appalling garbage (well beyond any kind of secondary repair).
Without the many layers of correction (between about 5 and 9 I think), the DVD storage medium is not so very good.

[johnmeyer]

24 fps on a modern LCD is shown without any shutter, so most people reading this have probably experienced 24 fps sans flicker.

As for our brain being fed "a sequence of discrete pictures," that is pretty much how all movies and TV perform their magic.

However ...

Original television, on a CRT, was actually much closer to a continuous process. If you neglect, for a moment, the retrace and also the vertical blanking interval (VBI), the picture is actually created by an electron beam scanning across the phosphor-coated face of the tube. It is painting in exact synchronization with the TV camera and -- this is the important part -- as it scans, every atom excited on the phosphor comes from a later instant in time than the previous atom. Put another way, there was no storage and no concept of a "frame buffer" or any such thing back in the 1930s and 1940s when this stuff was created, and therefore every instant in time got recorded, transmitted, and then displayed in the exact moment the light hit that part of the camera's electron tube.

So, in the case of PAL or NTSC original television, it actually was not a discrete series of pictures, but a continuous process.

I've had to think about this quite a bit when trying to figure out how to recover good results from Kinescopes which capture that continuous scan onto "a sequence of discrete pictures."

P.S. I just saw StainlessS' OT postscript. I am not surprised by the results of turning off DVD error correction. I've burned many thousands of DVD discs and I buy them 200 at a time. Whenever I open a new package, I do extensive testing on the first burn using "DVD Speed." That software reports on the number of correctable and uncorrectable errors. Even a "good" burn (a quality metric the software provides) produces thousands of errors, almost all of which are corrected and therefore are never seen by the person viewing the video.

Error detection and correction applies to all storage, from RAM to hard drives; the problem is not unique to DVDs.

The first thing I learned in the electronics portion of my computer training (where we learned about NAND gates, etc.) was about error detection and correction, with the simplest scheme being the good old-fashioned parity bit (add all the bits in the byte and if the number of ones is an even number set the parity bit to one; if odd, keep the bit as a zero).

For those who are mathematically inclined, the technology of error correction is absolutely fascinating, with all sorts of clever ways of doing corrections without having to double the amount of storage (like done with a RAID). There are even algorithms for correcting more than one error from the same byte (or whatever the storage size might be).

[raffriff42]

Thanks for pointing that out, johnmeyer. It's good to be reminded that flicker (varying overall light output) is a different issue from the motion distorting effect of low temporal sample rate - which has been called strobing and wagon wheel effect.

...interesting aside about the way analog TV samples time continuously, in a way. So do rolling shutters in cameras.

[Sharc]

Interesting also to observe how many efforts have been put in the spatial domain while the temporal aspects remained more or less stagnant.

P.S. It's perhaps not quite true for TV, as most TV sets today provide true motion interpolation for Film material which can be enabled/disabled (Soap Opera Effect and interpolation artefacts)

[wonkey_monkey]

John, based on your description of the CRT process, wouldn't we see a lot more rolling shutter type problems with old material? I assume it didn't take the full 1/50th or 1/60th of a second to scan one field, but did it somewhat quicker?

Or maybe film was used more in the situations where it would have been apparent.

[johnmeyer]

I have wondered the same thing about rolling shutter. I am pretty sure my description of the process is correct because there most certainly was no circuit that could store an entire frame of video back then.

Instead, I find myself questioning the usual explanations for what creates the rolling shutter artifacts: are they wrong? I ask this because my good cameras don't create rolling shutter effects, and yet I think the actual sensor technology is similar from high to low end.

[Asmodian]

The scan was pretty fast, at least faster than 1/60th of a second per field. Does anyone know the percentage of the signal was V-blank and other non-image timing info and what percentage was image? It was that fast. ;)

Rolling shutter artifacts should be due to a mismatch between the camera's and projector's shutters?

edit: sorry, getting a bit off topic here. :o

[Sharc]

Has the rolling shutter effect possibly been masked (halved) by the double field rate of interlacing?

Edit:
Ooops! Asmodian was faster

Edit2:
@john: From what I have read the rolling shutter effect exist for CMOS sensors but not for CCD sensors. Do your good cameras have CCD sensors?

[Katie Boundary]

If you never fed your old analog TV sets with a sequence made of half of the fields replaced with 240 (or 288) black horizontal lines, then you have no idea of what I was talking about

Whoa there, that's something completely different. If the screen is alternating between visible fields and black fields at the refresh rate, then you're basically just taking a normal video stream and blacking-out either all of the top fields or all of the bottom fields, which will be ugly but won't have anything to do with flickering. On the other hand, if the black fields alternate between top and bottom, then you're actually only getting 15 visible frames per second.

[wonkey_monkey]

Katie Boundary vs Midzuki. Popcorn, anyone?

[johnmeyer]

@john: From what I have read the rolling shutter effect exist for CMOS sensors but not for CCD sensors. Do your good cameras have CCD sensors?My circa 2005 Sony FX1 (prosumer, relatively large) is the last camera I have which has a CCD sensor. My CX-700 (standard consumer hand-held camcorder) has CMOS, as do other cameras. The CX-700 has absolutely no rolling shutter artifact.

[hello_hello]

Is this the effect of rolling shutter?

A video of a CRT computer monitor (refreshing at 85Hz) taken with an Android phone.

rolling.mkv (http://www.filedropper.com/rolling) (3.5MB).

The monitor's refreshing at 85Hz as anything too much below that results in flicker for me.

[hello_hello]

I wouldn't mind understanding this one. How does the camera seem to fall into sync with the Plasma refresh rate (60Hz in this case) and what's creating the circular effect to the right? The phone was a high end model when I bought it, but it's four or five years old and they probably have better cameras in all but the cheapest phones these days.

Plasma.mkv (http://www.filedropper.com/plasma_1) (13.4MB)

[StainlessS]

1st one down to non synchronized CRT raster scan and camera capture of each frame, also as there are several dark bars
(CRT blanking) per frame, then looks like CRT refresh rate was several times greater than camera capture rate
[somewhere between ~28->42FPS, 2 full bars and just about starting a new 3rd bar].
(leastwise thats how it looks to me).
2nd one, no idea [did it fall out of sync after a some time ?].

EDIT: The 2nd one, the circular effect, is it a screen reflection (of something else in the room).
(not perhaps your head :) )

EDIT: To below post, LOL, always good for a giggle [I was hoping it was your head] https://www.cosgan.de/images/smilie/froehlich/a065.gif

[hello_hello]

(leastwise thats how it looks to me).
2nd one, no idea [did it fall out of sync after a some time ?].

It didn't fall out of sync although I didn't keep trying for long periods, but I could repeat it falling into sync over and over. Move the phone away from the TV, start the video app, move it back to the TV and it'd always fall in sync pretty quickly as it did in the video.

EDIT: The 2nd one, the circular effect, is it a screen reflection (of something else in the room).
(not perhaps your head :) )

Damn! As well as dummy of the week, I've given myself idiot of the week award too.
There's an exercise ball on top of a bookshelf at the back of the room, behind which is a lamp with a dim bulb. It was daylight when I took the video so I forgot I'd left it on.
Weird thing is, it stood out to me immediately in the video but I've some how managed to mentally block it out when watching the TV normally from the same position.
I thought it was some sort of artefact along the lines of the coloured lines that run across the screen when the camera syncs. Doh!

[Sharc]

My circa 2005 Sony FX1 (prosumer, relatively large) is the last camera I have which has a CCD sensor. My CX-700 (standard consumer hand-held camcorder) has CMOS, as do other cameras. The CX-700 has absolutely no rolling shutter artifact.
In 2013/14 (?) CMOS sensors with Global Shutter function which capture all pixels at once instead of one row at a time have been introduced. This should have eliminated the Rolling Shutter effect of earlier CMOS sensors.

Edit:
BUT from the CX700 manual, section "Troubleshooting", page 102:
The subjects passing by the screen very
fast appear crooked.
This is called the focal plane phenomenon.
This is not a malfunction. Because of the way
the image device (CMOS sensor) reads out
image signals, the subjects passing by the lens
rapidly might appear crooked depending on
the recording conditions.

[wonkey_monkey]

I think the first one must be at least partially down to rolling shutter, because the grey bars are horizontal to the camera, not to the CRT screen.

Second one, I guess it's just a coincidence that when moving from the window to the TV the camera drops to an integer division of the refresh rate. Or perhaps it has some anti-flicker code in there somewhere for fluorescent lighting which happens to work with TVs too.

[Midzuki]

Whoa there, that's something completely different. If the screen is alternating between visible fields and black fields at the refresh rate, then you're basically just taking a normal video stream and blacking-out either all of the top fields or all of the bottom fields, which will be ugly but won't have anything to do with flickering.

Yes, the trick DOES cause intense and undeniable flickering,
BUT it has nothing to do with flickering. :rolleyes: