Super-resolution via Motion-Compensated Deblocking

[opieant]

A few months ago, a paper was published by some Berkeley folks that describes a deblocking method that produces video that upscales well. The title of the paper is "MOTION-COMPENSATED DEBLOCKING AND UPSCALING FOR VIEWING LOW-RES VIDEOS ON HIGH-RES DISPLAYS."

In the paper, there are some rather snazzy comparison images that show off the potential of their method. The authors make use of principles of super-resolution, spatial deblocking, and temporal (motion-compensated) deblocking. They use 9 frames for the motion-compensation, but it looks like more or less could be used as needed.

The "Spatial + MC POCS" method is supposed to be reasonably fast compared to other methods and yields impressive results, so I'm wondering if any of the developers here have had an opportunity to test it. I remember reading that the author of MVTools had tinkered with super-resolution but had set it aside. Would be very cool to see what this method could do for some of my own videos. :)

[2Bdecided]

I think this script from 2006(!), while not "the same", at least partly anticipates this approach...

http://forum.doom9.org/showthread.php?p=812569#post812569

Cheers,
David.

[opieant]

I think this script from 2006(!), while not "the same", at least partly anticipates this approach...

http://forum.doom9.org/showthread.php?p=812569#post812569

Cheers,
David.

Thanks for the tip. That looks reasonably nice, but it appears to be super-resolution without specific consideration of deblocking and MPEG/H.26x DCT. To a certain degree, super-resolution alone can do nice deblocking, but that depends on the quality of the source.

Some users of that script from 2006 were having major smearing problems, and the author scharfis_brain said that his script "merges everything", so that isn't ideal. Also, you were having problems with its performance (0.05fps), and I suspect that this new, not-yet-implemented method is probably quite a bit faster.

The new method described in the paper is supposed to be ideal for "global" motion, so it is known to work well when the entire frame is shifting (not just objects within the frame). At the very least, it would be interesting to see what an implementation of it would do differently on scenes with a stationary background and motion in the foreground. Hmm...

[2Bdecided]

It would be interesting to try.

The "classic" "real" super resolution algorithms and demos mostly work with a pan over a static shot. As-is, it seems they're fairly useless for "normal" video.

Cheers,
David.

[cantonesejim]

is there a link to a copy of the paper?

[Didée]

Google search (http://www.google.de/search?hl=de&q=%22MOTION-COMPENSATED+DEBLOCKING+AND+UPSCALING+FOR+VIEWING+LOW-RES+VIDEOS+ON+HIGH-RES+DISPLAYS%22&btnG=Google-Suche&meta=), 2nd result.

[Undead Sega]

has this method of Super-resolution being used in any way? ecause i must say judging from their proposal, that looks pretty impressive.

[Undead Sega]

ummm is it something i said?

anyone may i ask either way?

[opieant]

has this method of Super-resolution being used in any way? ecause i must say judging from their proposal, that looks pretty impressive.

Since we're not getting much feedback here, I would guess that only the researchers that came up with this method have really tried it.

Of course, if any AVISynth developers around here take interest in implementing this, I'm sure they won't hide it from us. :)

[Undead Sega]

im prety sure this is possible, and i like to think that using Motion Compensation for deblocking is a genius idea! :D and not to have paramters so that it can do the job automatically wheather its just light or strong blocking artifacts visible on the video. Seeing that example as well, really does look like it has almost recovered back to its original state :D

[Terka]

http://www.hindawi.com/GetArticle.aspx?doi=10.1155/2008/763254

[Undead Sega]

very interesting, however no mention of deblocking, but as i said interesting due to its different approach of Super-Resolution.

[opieant]

very interesting, however no mention of deblocking, but as i said interesting due to its different approach of Super-Resolution.

Agreed. The paper I referred to specifically addressed MPEG including deblocking and DCT. Many of the super-resolution algorithms I've seen described seem to expect a clean, low-resolution source, but most compressed video that really needs to be improved is low-res, at least somewhat blocky, very likely has noise, and is some kind of MPEG. Any method that can analyze or at least consider the coding of the video is likely to be more effective at recovering information because it's one step closer to the source than just comparing whole pictures from a decoder's output to one another without considering how the decoder formed each picture.

[Undead Sega]

how further agreeable can this be? hahaha, looking at the tools available in AviSynth and filters, Motion Compensation is readily used, stable and adjustable (and free :D)

i suppose it is only a matter of knowing how thier approach works and how can this be implented as a AviSynth script file, i suggest we ask the authors of these excellent filters if they can help :D

[opieant]

The issue with the existing motion-comp filters is that they only consider decoder output, although they do have the ability to consider block sizes to a certain degree which helps. They are somewhat generic, which can allow a fair or good amount of sub-pixel detail recovery, but not really great. Also, they're usually slow, so they are only practical at ~5fps if you don't mind waiting a very long time before you can watch a video. The performance isn't close to real-time, but to be really useful it would need to be (using multiple cores, assembly coding with instruction set optimizations, CUDA support, and any other stuff that is difficult but not impossible).

One example that can help emphasize why super-resolution plugins/filters should be working at the decoding level is the deblocking support built into DGDecode. If you use a D2V source file and the "cpu" parameter in any mode that includes deblocking (1 through 6), DGDecode will use quantization info from the MPEG data to control the amount of deblocking to be done without really over-processing or under-processing the picture. That's a pretty ideal situation for just deblocking (no super-res). If you use a non-D2V source and then call upon the deblocking features in DGDecode, it has no knowledge of the quantization values and all it can do is guess itself or have you specify a guess. The guessed value is going to be constant unless you apply different rules to lots of very small picture groups, and it will be non-ideal no matter what the guesses are.

[Undead Sega]

hahaha, wow that really is indepth explaination, almost of the rails for me to understand, hahaha naah just joking :D but that is an interesting theory about the deblocking using DGDecode, however there are many other sources that wont deprive from MPEG encodes.

but you are correct, the performance speed on those filters are indeed very slow to what u ahve mentioned, 5fps :( and there was a thread her not too long ago about implementing such filters to use CUDA or GPU.

I am thinking if probably contacting the people who pitched this idea and if thre may be anything experimental or a usable application to test this.

Either way, anyone, do you think this method can be done in AviSynth?

[Undead Sega]

Also, would this make a good tool for deblocking on different kind of compression videos?

[Undead Sega]

hey everyone, i know you lot maybe abit bored of this, but i got in contact with one of them, and he gratefully replied back to me :D

he mentioned that although he doesnt have the sourcecode, it is implemented in some of HP products (that he is not sure of) and either way, the development of this is part of HP, thus they would own the sourcecode and so on.

he also stated that they implemented the algorithm in C++ and a linux platform. (using some libraries to read , write and access video frames of course).

lastly to say, he said the algorithm is patented, and that he says specifically that if we follow the paper, you can impelment the steps yourself in software.

[opieant]

hey everyone, i know you lot maybe abit bored of this, but i got in contact with one of them, and he gratefully replied back to me :D

he mentioned that although he doesnt have the sourcecode, it is implemented in some of HP products (that he is not sure of) and either way, the development of this is part of HP, thus they would own the sourcecode and so on.

he also stated that they implemented the algorithm in C++ and a linux platform. (using some libraries to read , write and access video frames of course).

lastly to say, he said the algorithm is patented, and that he says specifically that if we follow the paper, you can impelment the steps yourself in software.

Very interesting. I hope HP is using this for something critically important like blowing up YouTube videos, and not wasting it on toys like medical equipment. ;)

[Undead Sega]

i mentioned AviSynth to him as well if he has heard of it, and through the reply, neither did he mention he heard of it or not, but simply says that it can be done if one followed the paper.

i suppose it is possible to be done through AviSynth. anyone up for the challenge or interested?

[Undead Sega]

anyone at all may i ask?

[Socio]

anyone at all may i ask?

I have been feeding videos via AVS scripts to a program called Video Enhancer so I can use its built in super resolution for resizing.

So if you’re looking for an immediate solution you can try it.

[Undead Sega]

it just dosnt work like that, because Video Enhancer doesnt work the same way as what the people has proposed in the paper, therefore, the adjustable Motion Compensation in AviSynth is practical, thats why i am asking if anyone might be interested.

anyone?

[Didée]

I have a feeling that you don't realize - not even vaguely - the complexity of that hypotetical plugin. It rather feels like - you have seen one(!) nice picture, now keep buggin' "I wanna wanna have that".

[Sagekilla]

As Didee said earlier, the closest thing we have to real super resolution is TempGaussMC. That's more or less the best SR you can possibly get right now. Otherwise, you also have MDegrain from MVTools which is motion compensated averaging. Normal super resolution for still images is composed of just averaging several images together, and MDegrain is about the best approximation you're gonna get.

[zee944]

As Didee said earlier, the closest thing we have to real super resolution is TempGaussMC. That's more or less the best SR you can possibly get right now. Otherwise, you also have MDegrain from MVTools which is motion compensated averaging. Normal super resolution for still images is composed of just averaging several images together, and MDegrain is about the best approximation you're gonna get.

Is there a specific thread where TempGaussMC's and MDegrain's capabilties are displayed? I can't find it. Also, can they be combined?

[Sagekilla]

TempGaussMC does use MDegrain. It uses degraining + motion compensation as part of it's deinterlacing scheme.

Link to thread: http://forum.doom9.org/showthread.php?p=1158000

[Terka]

http://zoi.utia.cas.cz/node/6
http://staff.utia.cas.cz/sroubekf/papers/TIP_07.pdf

[Undead Sega]

if i am correct, TempGaussMC is a deinterlacer?

also, the proposed method of Super Resolution includes deblocking and recovery like of blurred areas.

[Didée]

Yes, TGMC is a deinterlacer. It uses MC to turn half-height fields into full-height frames. Per definition, TGMC is a temporal superresolution filter.
_____

Regarding this thread's topic ...

- you can't "just implement the method". Unlike "normal" filters for Avisynth, you need to include a source filter that decodes the source. (For the children - imagine that all needed stuff had to be put into DGDecode, and you'd have mpeg2source() doing all the work.)

- the paper is about MJPEG compression. Each block is Intra coded. Mpeg-X is mostly Inter (delta) coded. This has very important implications (I won't bother to explain them to you - either you see, or you see not). Their claim that the method can be ported to other DCT-based codecs is highly simplified, if not even blue-eyed. With delta compression, the game is a quite different one.

- They also use a neural network for upsizing (did you even note?). We don't see any results of the single filter steps, we only see the "all together" result. Also we don't know what was the training data for the NN. If they show the results of the clip that was used to train the NN, then the result is not surprising.


That's just three points quickly written down. You could as well be presented with a very long list that explains why things are hundred times harder to do than you seemingly are imagining. But even this little post here takes more time than it's worth to invest...


In any case:

a) implementation of that method is nothing one would do in a few evening's free time. It is complex, demanding, and needs plenty of time.

b) It has been noted by now. Either someone will take the task, or not. Continuing with a weekly "Has anyone done it yet?!?" will not help, but is only annoying.


Mind you ... I would like to see an implementation of that method, too. Really. But you need to watch out from a realistic point, not from Cloud Seven above.

[Undead Sega]

hii, s sorry for the late reply, but i have read this since it was posted and all was taken account seriously. but like you and many others still very much would like to see this implemented, as this method has never really been done before. and from what is described it looks very promising. i know i am keep on going on and on about it, but i have recently contacted one of the authors who gave me more information, including that he doesnt have the time in his current sitution to rewrite the code as he doesnt have it.

i think this is very helpful and useful, but what he says is that, the code was written by fair amount of C code, they used the jpeg image library to read and write images, to access the bitstream and to encode etc, in addition they used ffmpeg package.

one has to access the bitstream and Qp values, and then use that for the algorrithm, which he mentions is fairly straightforward to implement just simple implementation from the equations in the paper (as he says). To add to that, he did say dealing with the bitstream was 'more painful'

i hope this helps for you lot, and might shine some more light as well. i will try to get in contact again, but he has not yet replied back as i think i asked abit too much in the responding email :D