Super-resolution via Motion-Compensated Deblocking

opieant · 19th February 2009, 07:30

A few months ago, a paper was published by some Berkeley folks that describes a deblocking method that produces video that upscales well. The title of the paper is "MOTION-COMPENSATED DEBLOCKING AND UPSCALING FOR VIEWING LOW-RES VIDEOS ON HIGH-RES DISPLAYS."

In the paper, there are some rather snazzy comparison images that show off the potential of their method. The authors make use of principles of super-resolution, spatial deblocking, and temporal (motion-compensated) deblocking. They use 9 frames for the motion-compensation, but it looks like more or less could be used as needed.

The "Spatial + MC POCS" method is supposed to be reasonably fast compared to other methods and yields impressive results, so I'm wondering if any of the developers here have had an opportunity to test it. I remember reading that the author of MVTools had tinkered with super-resolution but had set it aside. Would be very cool to see what this method could do for some of my own videos.

2Bdecided · 19th February 2009, 10:57

I think this script from 2006(!), while not "the same", at least partly anticipates this approach...

http://forum.doom9.org/showthread.ph...569#post812569

Cheers,
David.

opieant · 19th February 2009, 21:54

Quote:

Originally Posted by 2Bdecided

I think this script from 2006(!), while not "the same", at least partly anticipates this approach...

http://forum.doom9.org/showthread.ph...569#post812569

Cheers,
David.

Thanks for the tip. That looks reasonably nice, but it appears to be super-resolution without specific consideration of deblocking and MPEG/H.26x DCT. To a certain degree, super-resolution alone can do nice deblocking, but that depends on the quality of the source.

Some users of that script from 2006 were having major smearing problems, and the author scharfis_brain said that his script "merges everything", so that isn't ideal. Also, you were having problems with its performance (0.05fps), and I suspect that this new, not-yet-implemented method is probably quite a bit faster.

The new method described in the paper is supposed to be ideal for "global" motion, so it is known to work well when the entire frame is shifting (not just objects within the frame). At the very least, it would be interesting to see what an implementation of it would do differently on scenes with a stationary background and motion in the foreground. Hmm...

2Bdecided · 20th February 2009, 02:04

It would be interesting to try.

The "classic" "real" super resolution algorithms and demos mostly work with a pan over a static shot. As-is, it seems they're fairly useless for "normal" video.

Cheers,
David.

cantonesejim · 21st February 2009, 23:29

is there a link to a copy of the paper?

Didée · 21st February 2009, 23:48

Google search, 2nd result.

Undead Sega · 22nd February 2009, 21:08

has this method of Super-resolution being used in any way? ecause i must say judging from their proposal, that looks pretty impressive.

Undead Sega · 24th February 2009, 23:16

ummm is it something i said?

anyone may i ask either way?

opieant · 25th February 2009, 00:47

Quote:

Originally Posted by Undead Sega

has this method of Super-resolution being used in any way? ecause i must say judging from their proposal, that looks pretty impressive.

Since we're not getting much feedback here, I would guess that only the researchers that came up with this method have really tried it.

Of course, if any AVISynth developers around here take interest in implementing this, I'm sure they won't hide it from us.

Undead Sega · 27th February 2009, 11:51

im prety sure this is possible, and i like to think that using Motion Compensation for deblocking is a genius idea!

and not to have paramters so that it can do the job automatically wheather its just light or strong blocking artifacts visible on the video. Seeing that example as well, really does look like it has almost recovered back to its original state

Terka · 2nd March 2009, 15:23

http://www.hindawi.com/GetArticle.as...55/2008/763254

Undead Sega · 2nd March 2009, 18:13

very interesting, however no mention of deblocking, but as i said interesting due to its different approach of Super-Resolution.

opieant · 2nd March 2009, 19:47

Quote:

Originally Posted by Undead Sega

very interesting, however no mention of deblocking, but as i said interesting due to its different approach of Super-Resolution.

Agreed. The paper I referred to specifically addressed MPEG including deblocking and DCT. Many of the super-resolution algorithms I've seen described seem to expect a clean, low-resolution source, but most compressed video that really needs to be improved is low-res, at least somewhat blocky, very likely has noise, and is some kind of MPEG. Any method that can analyze or at least consider the coding of the video is likely to be more effective at recovering information because it's one step closer to the source than just comparing whole pictures from a decoder's output to one another without considering how the decoder formed each picture.

Undead Sega · 2nd March 2009, 20:45

how further agreeable can this be? hahaha, looking at the tools available in AviSynth and filters, Motion Compensation is readily used, stable and adjustable (and free

)

i suppose it is only a matter of knowing how thier approach works and how can this be implented as a AviSynth script file, i suggest we ask the authors of these excellent filters if they can help

opieant · 3rd March 2009, 00:46

The issue with the existing motion-comp filters is that they only consider decoder output, although they do have the ability to consider block sizes to a certain degree which helps. They are somewhat generic, which can allow a fair or good amount of sub-pixel detail recovery, but not really great. Also, they're usually slow, so they are only practical at ~5fps if you don't mind waiting a very long time before you can watch a video. The performance isn't close to real-time, but to be really useful it would need to be (using multiple cores, assembly coding with instruction set optimizations, CUDA support, and any other stuff that is difficult but not impossible).

One example that can help emphasize why super-resolution plugins/filters should be working at the decoding level is the deblocking support built into DGDecode. If you use a D2V source file and the "cpu" parameter in any mode that includes deblocking (1 through 6), DGDecode will use quantization info from the MPEG data to control the amount of deblocking to be done without really over-processing or under-processing the picture. That's a pretty ideal situation for just deblocking (no super-res). If you use a non-D2V source and then call upon the deblocking features in DGDecode, it has no knowledge of the quantization values and all it can do is guess itself or have you specify a guess. The guessed value is going to be constant unless you apply different rules to lots of very small picture groups, and it will be non-ideal no matter what the guesses are.

Undead Sega · 3rd March 2009, 21:53

hahaha, wow that really is indepth explaination, almost of the rails for me to understand, hahaha naah just joking

but that is an interesting theory about the deblocking using DGDecode, however there are many other sources that wont deprive from MPEG encodes.

but you are correct, the performance speed on those filters are indeed very slow to what u ahve mentioned, 5fps

and there was a thread her not too long ago about implementing such filters to use CUDA or GPU.

I am thinking if probably contacting the people who pitched this idea and if thre may be anything experimental or a usable application to test this.

Either way, anyone, do you think this method can be done in AviSynth?

Undead Sega · 8th March 2009, 00:01

Also, would this make a good tool for deblocking on different kind of compression videos?

Undead Sega · 9th March 2009, 15:15

hey everyone, i know you lot maybe abit bored of this, but i got in contact with one of them, and he gratefully replied back to me

he mentioned that although he doesnt have the sourcecode, it is implemented in some of HP products (that he is not sure of) and either way, the development of this is part of HP, thus they would own the sourcecode and so on.

he also stated that they implemented the algorithm in C++ and a linux platform. (using some libraries to read , write and access video frames of course).

lastly to say, he said the algorithm is patented, and that he says specifically that if we follow the paper, you can impelment the steps yourself in software.

opieant · 9th March 2009, 17:36

Quote:

Originally Posted by Undead Sega

hey everyone, i know you lot maybe abit bored of this, but i got in contact with one of them, and he gratefully replied back to me

he mentioned that although he doesnt have the sourcecode, it is implemented in some of HP products (that he is not sure of) and either way, the development of this is part of HP, thus they would own the sourcecode and so on.

he also stated that they implemented the algorithm in C++ and a linux platform. (using some libraries to read , write and access video frames of course).

lastly to say, he said the algorithm is patented, and that he says specifically that if we follow the paper, you can impelment the steps yourself in software.

Very interesting. I hope HP is using this for something critically important like blowing up YouTube videos, and not wasting it on toys like medical equipment.

Undead Sega · 9th March 2009, 18:29

i mentioned AviSynth to him as well if he has heard of it, and through the reply, neither did he mention he heard of it or not, but simply says that it can be done if one followed the paper.

i suppose it is possible to be done through AviSynth. anyone up for the challenge or interested?

19th February 2009, 07:30	#1 \| Link
opieant Registered User Join Date: May 2004 Posts: 27	Super-resolution via Motion-Compensated Deblocking A few months ago, a paper was published by some Berkeley folks that describes a deblocking method that produces video that upscales well. The title of the paper is "MOTION-COMPENSATED DEBLOCKING AND UPSCALING FOR VIEWING LOW-RES VIDEOS ON HIGH-RES DISPLAYS." In the paper, there are some rather snazzy comparison images that show off the potential of their method. The authors make use of principles of super-resolution, spatial deblocking, and temporal (motion-compensated) deblocking. They use 9 frames for the motion-compensation, but it looks like more or less could be used as needed. The "Spatial + MC POCS" method is supposed to be reasonably fast compared to other methods and yields impressive results, so I'm wondering if any of the developers here have had an opportunity to test it. I remember reading that the author of MVTools had tinkered with super-resolution but had set it aside. Would be very cool to see what this method could do for some of my own videos.

21st February 2009, 23:48	#6 \| Link
Didée Registered User Join Date: Apr 2002 Location: Germany Posts: 5,407	Google search, 2nd result. __________________ - We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)

19th February 2009, 10:57	#2 \| Link
2Bdecided Registered User Join Date: Dec 2002 Location: UK Posts: 1,673	I think this script from 2006(!), while not "the same", at least partly anticipates this approach... http://forum.doom9.org/showthread.ph...569#post812569 Cheers, David.

20th February 2009, 02:04	#4 \| Link
2Bdecided Registered User Join Date: Dec 2002 Location: UK Posts: 1,673	It would be interesting to try. The "classic" "real" super resolution algorithms and demos mostly work with a pan over a static shot. As-is, it seems they're fairly useless for "normal" video. Cheers, David.

21st February 2009, 23:29	#5 \| Link
cantonesejim Registered User Join Date: Jan 2007 Posts: 13	is there a link to a copy of the paper?

22nd February 2009, 21:08	#7 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	has this method of Super-resolution being used in any way? ecause i must say judging from their proposal, that looks pretty impressive.

24th February 2009, 23:16	#8 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	ummm is it something i said? anyone may i ask either way?

27th February 2009, 11:51	#10 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	im prety sure this is possible, and i like to think that using Motion Compensation for deblocking is a genius idea! and not to have paramters so that it can do the job automatically wheather its just light or strong blocking artifacts visible on the video. Seeing that example as well, really does look like it has almost recovered back to its original state

2nd March 2009, 15:23	#11 \| Link
Terka Registered User Join Date: Jan 2005 Location: cz Posts: 704	http://www.hindawi.com/GetArticle.as...55/2008/763254

2nd March 2009, 18:13	#12 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	very interesting, however no mention of deblocking, but as i said interesting due to its different approach of Super-Resolution.

2nd March 2009, 20:45	#14 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	how further agreeable can this be? hahaha, looking at the tools available in AviSynth and filters, Motion Compensation is readily used, stable and adjustable (and free ) i suppose it is only a matter of knowing how thier approach works and how can this be implented as a AviSynth script file, i suggest we ask the authors of these excellent filters if they can help

3rd March 2009, 00:46	#15 \| Link
opieant Registered User Join Date: May 2004 Posts: 27	The issue with the existing motion-comp filters is that they only consider decoder output, although they do have the ability to consider block sizes to a certain degree which helps. They are somewhat generic, which can allow a fair or good amount of sub-pixel detail recovery, but not really great. Also, they're usually slow, so they are only practical at ~5fps if you don't mind waiting a very long time before you can watch a video. The performance isn't close to real-time, but to be really useful it would need to be (using multiple cores, assembly coding with instruction set optimizations, CUDA support, and any other stuff that is difficult but not impossible). One example that can help emphasize why super-resolution plugins/filters should be working at the decoding level is the deblocking support built into DGDecode. If you use a D2V source file and the "cpu" parameter in any mode that includes deblocking (1 through 6), DGDecode will use quantization info from the MPEG data to control the amount of deblocking to be done without really over-processing or under-processing the picture. That's a pretty ideal situation for just deblocking (no super-res). If you use a non-D2V source and then call upon the deblocking features in DGDecode, it has no knowledge of the quantization values and all it can do is guess itself or have you specify a guess. The guessed value is going to be constant unless you apply different rules to lots of very small picture groups, and it will be non-ideal no matter what the guesses are.

3rd March 2009, 21:53	#16 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	hahaha, wow that really is indepth explaination, almost of the rails for me to understand, hahaha naah just joking but that is an interesting theory about the deblocking using DGDecode, however there are many other sources that wont deprive from MPEG encodes. but you are correct, the performance speed on those filters are indeed very slow to what u ahve mentioned, 5fps and there was a thread her not too long ago about implementing such filters to use CUDA or GPU. I am thinking if probably contacting the people who pitched this idea and if thre may be anything experimental or a usable application to test this. Either way, anyone, do you think this method can be done in AviSynth?

8th March 2009, 00:01	#17 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	Also, would this make a good tool for deblocking on different kind of compression videos?

9th March 2009, 15:15	#18 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	hey everyone, i know you lot maybe abit bored of this, but i got in contact with one of them, and he gratefully replied back to me he mentioned that although he doesnt have the sourcecode, it is implemented in some of HP products (that he is not sure of) and either way, the development of this is part of HP, thus they would own the sourcecode and so on. he also stated that they implemented the algorithm in C++ and a linux platform. (using some libraries to read , write and access video frames of course). lastly to say, he said the algorithm is patented, and that he says specifically that if we follow the paper, you can impelment the steps yourself in software.

9th March 2009, 18:29	#20 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	i mentioned AviSynth to him as well if he has heard of it, and through the reply, neither did he mention he heard of it or not, but simply says that it can be done if one followed the paper. i suppose it is possible to be done through AviSynth. anyone up for the challenge or interested?