Reducing bandwidth by downsizing - streaming - upsizing

[jmartinr]

Apparently it can be advantageous to downsize video before transmitting, upscaling afterwards. See this article (http://www.microsoft.com/whdc/archive/convrg1.mspx#EBAA) for an explanation.

I did some testing. Using at the same time a pixel aspect ratio of 4:3 I encoded an SD (progressive) video to 400 x 400. After that I display it at 1066 x 800. It looks good and at SD size it even looks great! Take a look here (http://members.home.nl/jmartinr/test.html) for the results.

You need a to have a recent version of Adobe Flash Player Installed to watch the video. The video is encoded with x264 at 600 kbps and the encoding is kept simple to enable watching it with QuickTime. I had to do some sharpening to enhance the look of the upsized video.

You need a somewhat higher video rate to get a good result compared to a normal small size video (it has to be sharp), but I think it's less then you would need for a SD sized encode.

What do you think?

[Sharktooth]

BS. ;)
when downscaling you drop details... when upscaling you cant restore the lost details... so the upscaling is not necessary at all unless you use a ubercomplex algorithm that can fake part of the missing details with complex interpolations... but that's far from being done in realtime... expecially in a flash player and it will always have a lower quality than encoding at original res...
however your video is blurry. a real 1066x800 video will look definatly much better.
dont listen M$... they cant even make a good video codec...

[HDT]

Both files are the same size...the point being???

[jmartinr]

@ Sharktooth a real 1066x800 video will look definitely much better Sure it does, but with what bitrate? It's about streaming a decent looking video with a low bitrate. But I must admit 1066x800 is a bit far fetched. ;)

@HDT It's the same video shown at different sizes on the website (if your screen is big enough). I think 768x576 looks good.

I also think it's interesting to use the pixel aspect ratio to shave off some bits. Hey, it's free! :cool:

[HDT]

Ah sorry, I see what your doing now. It is the same file via different transport/reconstruction

[Sharktooth]

jmartinr: it's still useless. if you watch both videos full-screen they look identical... it's the same video.
you wont get any quality boost by upsizing unless you use complex algos which are in most cases not possible to apply in real-time. it also depends on the source...
it's just like watching a photo with a magnifing glass. it wont have more details or more quality, you just see it bigger.
also, with modern video formats, the higher the res the more the encoder will be efficient...

[LoRd_MuldeR]

Forget about the "fake-upscale a previously downscaled stream" idea and better look at new streaming techniques like SVC:
http://en.wikipedia.org/wiki/Scalable_Video_Coding

BTW: In fact Youtube does exactly do that. If you watch the video in the default embedded Flash Player (none-fullscreen), you see a slightly upscaled video...

[HDT]

jmartinr: it's still useless. if you watch both videos full-screen they look identical... it's the same video.
you wont get any quality boost by upsizing unless you use complex algos which are in most cases not possible to apply in real-time. it also depends on the source...
it's just like watching a photo with a magnifing glass. it wont have more details or more quality, you just see it bigger.
also, with modern video formats, the higher the res the more the encoder will be efficient...
I think you've got the wrong pair of lenses on like I had to start with :)
This is not looking to get more detail, but simply to reconstruct modest dimensions to larger dimensions with good interpolation etc. and still have the same apparent sharpness and detail from the same viewing distance, as is done by hd disc players when playing a sd dvd which is upscaled from 576 to 1080 [e.g.].
The same is being done in software with windvd and nvidia's new 'cuda' engine, though I have not tried either of these as yet.
One has to be carefull though of how you go about assessing this on a digital panel were 1:1 mapping comes into the equation.

[Sharktooth]

videocards do that in hardware, not via software... and they actually have a LOT of processing power. that means they can use complex algos...
doing the same stuff via software in real time is not an option. so, no, i've got good lenses... ;)
same happens in HD players, they have dedicated hardware... and the "reconstruction" is sitll questionable... some of them, infact, look like crap with standard DVDs...
upsizing (or upscaling) is always bad since the algos introduce data that was not present on the source. they cant invent details that are not there... interpolation is not a magic wand...

[CruNcher]

Sharktooth you allways underestimate the Psy effects on our eyes and Brain sure such simple upscaling stuff like many hardware/software does today isn't efficient enough but the new Realtime Superresolution algorithms that gonna come out now in the Consumer World in form like of Toshibas DVD (XBE) Player and also now in ATIs Video Cards will give Blu-Ray a run for it's money the Psy effect is so heavy that you belive it is HD the sharpness is their some details that got lost due to the quantization are back also (also depends on the DVD source Quality the higher the less the effect of preserved details).
The most important thing Psy wise is the sharpness factor as this goes lost due to quantization (im also quiete sure that it will recover some of the lost grain layer that gets especialy lost on B-frames) how many details are in the Recording itself (for the objects you see) isn't of interest to our eyes @ first only when you do a direct comparision and even then in Motion it would be hard to spot. Im also sure with your arguments against it the Blu-Ray guys will start Marketing campaigns against this, tough imho the only way to conquer this are lower Blu-Ray prices so im very happy in both ways even if it fails in the end that means lower Blu-Ray price :)

About Didees stuff it is nice tough it has one major drawback being Pre-processing and not Post-processing (the Lossy Encoder will allways kill most of the effects (unless you except the extreme higher bitrate for it) the same like it kills the gradfun dither effect tough with Psy RD and VAQ (X264,Dark_Shikari) the thing looks different again and still needs to be Visualy tested)

[jmartinr]

@Sharktooth: I guess you're right. I did some more encodes with normal SD dimensions and a video bitrate of 500kbps. I've put them at the same page (http://members.home.nl/jmartinr/test.html) as the others.

You can take a look for yourself. But i think that the most ordinary and last video with normal dimensions looks best.

This thread is all about what's practically feasible with low bitrate (500 is not much). I have seen a lot of blurriness in low bitrate videos. This seemed a way to overcome this blurriness. But I guess I underestimated the power x264 nowadays has to get a decent video along with low bitrates.

This case is checked now. But might it be different in other cases?

[HDT]

videocards do that in hardware, not via software... and they actually have a LOT of processing power. that means they can use complex alogs...
doing the same stuff via software in real time is not an option. so, no, i've got good lenses... ;)
same happens in HD players, they have dedicated hardware... and the "reconstruction" is sitll questionable... some of them, infact, look like crap with standard DVDs...
upsizing (or upscaling) is always bad since the algos introduce data that was not present on the source. they cant invent details that are not there... interpolation is not a magic wand...
I understand what your saying about video cards here but it's not strictly true. Hardware accelerated yes, but still dependent on software.
Hd players on the other hand, whilst still software driven are no doubt more pure hardware, in which the quality of the output is largely fixed and independent of the software.

[Sharktooth]

i meant, that software is executed by the GPU, not the CPU... so if you upsize the video in your CPU executed software (like the flash plaeyer) you cant use ubercomplex upsizing algos and keep the realtime playback.

@jmartinr: unless some complex upsize algos are used then no... but for the reasons i tried to explain above, that is not an option unless you make your own software player and implement those algos in a way they can be executed on the GPU (using either CUDA, OpenCL, etc...).

[jmartinr]

I guess I overdid the resizing a bit in my first tries, so I did a new test with 512 x 512 encoded video shown at 768 x 576. I dropped the pre-sharpening, because it doesn't work out well and eats bitrate. Encoding at 512 x 512 means that there are about 1.5 pixels less to encode. Following the rule of thumb that four times more pixels need twice the bitrate, I guessed I could do an encode at 500 instead of 600 Kbps.

So that's what I did. Look here (http://members.home.nl/jmartinr/test.html) again for the result of the test. Please have a look and tell me what you think. Does the last encode look OK? Also if you compare it with the reference?

Basically this method is the same LoRd_MuldeR mentioned, used by Youtube: playing a slightly upscaled video in Flash Player. I just get a little bit more out of playing with the pixel aspect ratio.

Update: I've put an extra reference video in the test page, this one encoded at 500 Kbps to compare with the other 500 Kbps video

[smok3]

it tells me to get the latest flash, when i have ver 10 beta intalled.

[2Bdecided]

Apparently it can be advantageous to downsize video before transmitting, upscaling afterwards. See this article (http://www.microsoft.com/whdc/archive/convrg1.mspx#EBAA) for an explanation.I think that article, and what you're doing, and not as closely related as you might think. That guy is basically arguing that the camera sensor and the display should be higher resolution than the transmission channel. Otherwise you're limited to optical (or no) filtering on both, with a sub-optimal result. This is all true, but if what you have available is a 1920x1080 sensor and a 1920x1080 channel, you might as well use them. Of course 1440x1080 is popular for exactly the reason that you don't lose much at all.

What you're doing is subtly different: you're downsizing for transmission because, at a given bitrate, a higher resolution = more artefacts, and a lower resolution = more blurred. You need to trade off artefacts vs sharpness.

Your video isn't really a worst case for encoding - it looks OK full resolution - you might find a more "challenging" video benefits more from the approach.

If you are changing the resolution, then I don't think there's much advantage to using non-square pixels when the target is a PC display.

Cheers,
David.

[jmartinr]

@ smok3: I'll look into that. You can still download them if you want, just a few MB's per video.

@ 2Bdecided: Thanks for the reaction. I think you sum it up quite well. When I've a bit more time, I'll encode the video to 400 and 300 Kbps. Should make it more challenging.
If you are changing the resolution, then I don't think there's much advantage to using non-square pixels when the target is a PC display.
Now why's that? I've always read that the human vision system is less sensitive in the horizontal direction. So it shouldn't really matter that you encode for a PC display.

All this also depends on how far you are from your monitor. If you're on a comfortable distance not really looking at the details, like many people are, you wont notice the difference. But, if you're paying attention and look real close, you will.

[smok3]

yes i did, they 512x512 version looks as crappy as 768x576 one, you need some better source..., what was this, some mpeg2 camcorder?

[Didée]

I've always read that the human vision system is less sensitive in the horizontal direction.
... and if you put a thick book under the rear of your PC case, the PC will run faster, since it can work downhill then. :cool:

That claim about the human HVS has never been true.

The truth is: old-school CRT displays are more sharp in the vertical direction (distinct scanlines) than in the horizontal direction (continuous, bandwidth-limited signal).

Since most of us have been grown-up with CRT televisions, the brain has been trained to accept some blurriness in the horizontal direction. It's only about conditioning (we're all a dog of Pavlov), not about some obscure limitation of the human HVS.

[jmartinr]

@ smok3: It was progressive PAL DV shot with a Sony DCR-HC90E which can record progressive video (yes, I know that's not real good quality progressive, but it is still good for encoding compared to interlaced stuff).

It was indoors, so the lighting was not great. Still OK for me though as a test video, because those are my usual conditions. ;)

@ Didée: I'll do some more research.

[Sharktooth]

do whatever you want... as i said, upscaling during playback isnt giving you more quality.
also, as i said, the encoder is more efficient at higher resolutions, the real issue here is finding a correct resolution for a given bitrate and for a given content. no upsizing/upscaling will help... in no way...

[CruNcher]

jmartinr did you used the default settings of the cam how was the sharpening setting balanced out ?
to get perceptable away from the blurrienes you need to sharpen the source again but tough then you have the problem that most likely the Encoder is gonna kill this effort of yours, trying to apply Didees Seesaw or Limitedsharpen (or some of his new creations not up2date currently with whats new) could help HVS wise in combination with X264s Psy-RD/trellis :)

[jmartinr]

@CruNcher: Thanks for the tip. I just used the automatic settings of the camera. And for the encode I used fft3dgpu at default settings.

You're right that I could do some better sharpening but I wanted to keep the compares basic and simple. Might do some compares with better sharpening later.

[smok3]

maybe post the original clip with your desired bitrate?

[jmartinr]

@ smok3: it tells me to get the latest flash, when i have ver 10 beta installed. Bug should be fixed. I'm not trying to obtain a desired bitrate, I'm just playing around trying to find out how to get an OK picture with low bitrates.

Did some more testing at 400 and 300 Kbps with some sharpening. Results are on the bottom half of this page (http://members.home.nl/jmartinr/test.html) again.

When I had a first glance at the videos fullscreen they all looked a bit unsharp and the simple 512 x 512 encodes looked best. But looking on the website at 768 x 576 you can see the difference.

Concluding from these videos I would say that sharpening didn't help. And that the 512 x 512 encodes look OK but not sharp.

Which ones would you prefer to look at. The 512 x 512 or the 704 x 576?

[smok3]

generally speaking:

a. i don't like youtube at this curent state is unwatchable for me
b. i wan't to see 1:1 pixels, let me decide for fullscreen (or double size) if i wish so
c. figure out what your actual camera can produce resolution-wise.

[2Bdecided]

I think they all look surprisingly similar. There are fewer artefacts at 512x512 than 704x576 for 300kbps, but it's close. Note: all your 704 links look much sharper than your original 768 link - your pre-encode resize is suspect!

I don't know what screen resolution you're running, but here (1280x1024) I need an HD source for full screen to look sharp. All of these look really soft at full screen, as does most web video (except for Vimeo, but that stutters on my old machine!)

Cheers,
David.

[jmartinr]

@ 2Bdecided: all your 704 links look much sharper than your original 768 link Which original do you mean? They're all encodes. The references being done by cropping 16 pixels of a 720 video. So the references are slightly anamorphic too (12:11).

Checked the videos and saw that a fft3dgpu filter with default settings was missing in the LimitedSharpen ones (used that in all the encodes). Fixed that now, but it don't think it makes a difference.

For the results I think that one of your earlier comments sums it all up: trade off artefacts vs sharpnessAnd I guess this is not a very subtle way of doing this as opposed to filtering or using custom quant matrices.

[CruNcher]

http://www.nkbv.nl/blobs/wedstrijden/Test%20Embedded/Wereldrecord%20Speedklimmen%20WK07%20704x576%20LimitedSharpen%20400.mp4 <- subjectively imho this is a very good result could need maybe some fine tuneing now :) it seems to have brought back what the camera compression (and or settings lost)

[jmartinr]

Because CruNcher liked the LimitedSharpen :) I did some more testing. Nothing anamorphic this time, so I encoded at 704 x 528 and show that at the same resolution. Results are low on the same page (http://members.home.nl/jmartinr/test.html) again.

And well... I think it looks sharp.

[2Bdecided]

That's more like it, though now we have 1:1 pixel mapping from camera to screen, I think maybe your camera itself adds some aliasing.

This is being really picky though - it's a million percent better than lots of web video at comparable bitrates!

Cheers,
David.

[jmartinr]

That's more like it, though now we have 1:1 pixel mapping from camera to screen, I think maybe your camera itself adds some aliasing.

It's not exactly 1:1 because now it's vertically downsized. Because the source is anamorphic, that's the only thing you can do if you really want no upsizing at all. The source is progressive.

Aliasing... avoiding that might get complicated.

[2Bdecided]

You need a to have a recent version of Adobe Flash Player Installed to watch the video. The video is encoded with x264 at 600 kbps and the encoding is kept simple to enable watching it with QuickTime.I forgot to ask: what x264 parameters did you set to make it play properly in real time?

Cheers,
David.

[jmartinr]

I forgot to ask: what x264 parameters did you set to make it play properly in real time?
Just the QuickTime profile with turbo first pass copied from MeGUI's log. I use a batch encode. The relevant lines are:
"C:\Program Files\megui\tools\x264\x264.exe" "%%~fP" --pass 1 --keyint %keyint% --bitrate %bitrate% --bframes 1 --direct auto --subme 1 --partitions none --me dia --threads auto --thread-input --sar %sar% --progress --no-psnr --no-ssim --stats "%%~dpnP.x264.stats" --output NUL
"C:\Program Files\megui\tools\x264\x264.exe" "%%~fP" --pass 2 --keyint %keyint% --bitrate %bitrate% --ref 5 --bframes 1 --b-rdo --direct auto --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4 --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --sar %sar% --stats "%%~dpnP.x264.stats" --output "%%~dpnP.%x264cont%"

"C:\Program Files\MP4Muxer\mp4creator" -create="%%~dpnP.%x264cont%" -rate=25 -qtw=%width% -qth=%height% "%%~dpnP.%x264cont%.mp4" 2>&1|find /v "Error decoding sei message"

"C:\Program Files\megui\tools\mp4box\mp4box.exe" -add "%%~nP.%x264cont%.mp4"#1 -add "%%~nP.aac.mp4"#1 -new "%%~nP.mp4"


Mp4creator version: DVBPortal MP4 Multiplexer/Demultiplexer Version 0.9.2 Aug-09-2008

Mp4box version: the MeGUI one

The funny thing is, I mux twice. I first put the dimensions in with mp4creator and then mux that again with MP4Box.
MP4Box then takes over the matrix.

As you can see, aspect ratio is in the X264 stream and in the matrix. Not in the PASP Atom.
Since this thread ended with an encode with a 1:1 PAR, that doesn't seem so important anymore.

[2Bdecided]

Thank you - that's great.

Cheers,
David.