Video Super Resolution for everyone

[DmitryPopov]

Super Resolution is a technique of increasing images or video resolution with superb quality. For each processed frame it uses information not only from this frame but also from other frames. If picture in your video doesn't change too fast, then information from several frames can be added to create a picture of a bigger size. No algorithms of ordinary image resize/resample can provide this quality just because there's not enough information in one frame.

Video Enhancer is a tool which uses Super Resolution technique to increase resolution of your video.

Also, it can use a hundred of VirtualDub filters to process your video: deinterlacing, denoising, deshaking, color correction etc.

Main page:
http://www.thedeemon.com/VideoEnhancer/

Compare Super Resolution to bicubic and Lanczos3:
http://www.thedeemon.com/VideoEnhancer/compare.html

Edit:
Version 1.0 didn't work with files containing mp3 audio stream. Now this is fixed, so don't be scared by error reports in the beginning of this thread.

[Guest]

Is this free software?

[DmitryPopov]

It's a shareware.
Registration will be accessible in a couple of days.

VDFilter (DirectShow wrapper for VirtualDub filters) is freeware (for non-commercial use).

[Didée]

Sweet application. Conversion is done in zero time, and the output file consists of 1 frame. :)
Tested both operation modes with both AVI and AVS input, on a Celeron 2.6, XP SP1, without dotNET.

[DmitryPopov]

What was the input file?
What codec used for output compression?
Any errors?

[check]

Also tested, same error.
o 640x480 AVI input decoded through ffdshow encoded to uncompressed 2048x1536,
o 640x368 AVS encoded to uncompressed 2048x1152
both of these gave the same error.

As a side note, a shortcut to the readme in the start menu would be nice.

[DmitryPopov]

Thanks! I'll check that out.
What kind of video decoded by ffdshow?

I've found a file that gives this kind of error. It has VBR audio stream, it can be the cause...

[check]

the video is xvid-mp3.vbr. When I click "go", it begins to draw the progress screen (I assume that's what it is), writes the first frame to disk and displays the success message. I'll take a video of it in action (or not :P). What a software launch hey? :(

[check]

Here is a video of the error in action: http://project357.com/ul/upload/output.wmv
As you can see, it instantly ends. The *interesting* part is that when I clicked start, it loads up a second ffdshow video filter for the processing and requests the first frame only.

I also just somehow got it to display the first frame of the encoding, but it still crashes at the same point, the GUI just updates correctly now. EDIT: hmm, seems that the GUI will update if the destination image is smaller than a certain level. 1280x960 will update the GUI to display the first frame, 2048x1536 will not.

EDIT: vbr mp3 audio is not the cause. cbr mp3 avi files also produce the same error.

[foxyshadis]

Since I work mostly with still images for my resizer, I hardly included any temporal information, but I suspect it wouldn't be too hard to cajole mvtools into providing some backup. I'm always in the market for more ways to make it the slowest filter around. =D

[Revgen]

Super Resolution is a technique of increasing images or video resolution with superb quality. For each processed frame it uses information not only from this frame but also from other frames. If picture in your video doesn't change too fast, then information from several frames can be added to create a picture of a bigger size. No algorithms of ordinary image resize/resample can provide this quality just because there's not enough information in one frame.

Video Enhancer is a tool which uses Super Resolution technique to increase resolution of your video.

Also, it can use a hundred of VirtualDub filters to process your video: deinterlacing, denoising, deshaking, color correction etc.

Main page:
http://www.thedeemon.com/VideoEnhancer/

Compare Super Resolution to bicubic and Lanczos3:
http://www.thedeemon.com/VideoEnhancer/compare.html


Not too bad. However I don't think it's good enough to pay for yet. EEDI2 does a pretty good job and Foxyshadis' EEDI2 resizer function works even better.

The only thing that would tempt me to buy this is if you provided good support for multithreading since I have a Dual-Core CPU. MVTools doesn't really work with multithreading that well so it would give your tool an advantage over any MVTools-EEDI2 based script.

[DmitryPopov]

Thank you all guys for valuable info!

The problem appears when an mp3 audio stream is present. DirectShow says sample has no time stamp. I'll fix this soon. Till that moment you can still work with videos without mp3 streams inside.

Multithreading support is the next thing planned to be implemented. My algorithm can be paralleled very good.

[MrTroy]

I have mixed feelings about this one.

In your Star Wars example:
- the archway looks really good
but
- there are all kinds of resizing artefacts around Yoda's head

[DmitryPopov]

Yes, but you can see his eyes. Does any other upsample method show them? Hardly.

Overall quality is going to be better soon, that was just a version 1.0, just the beginning.

BTW, can anyone post here a frame from the example resized with his favourite upsample method? Like EEDI2?

[*.mp4 guy]

ImageSource("m4x.jpg").Converttoyv12(matrix="pc.601").Pointresize(100, 60)


MultiSWAR(400, 240, warping=1, StepRatio=2, sharper=true, Sharpen=6, Smoothing=true, Sthresh=128, Sstrength=.8, EDetail=true, Esharp=.33, FDetail=1, Sthresh2=1.1)
image (http://img231.imageshack.us/img231/7664/multiswar8rj.png)

The reason that there is color bleading is becaue MultiSWAR only works in the yv12 colorspace, which in practical usage isn't an issue.

[Didée]

That's a spot-on confirmation of Dimitry's point, mp4guy. MultiSWAR shows straightened lines, faces blurred and warped-away into posterization. In a word, this shot looks pretty poor.
Temporal superresolution is not about straightening lines in the first place. It is about digging-out detail out of the swamp below the pixel "quantums". And this you will never achieve with spatial filtering.

---

Dimitry - haven't had much time for testing, up to now. The application/GUI definetly has its buggies that should be fixed, you probably know. For example, my above error report was false: the application actually does the conversion; however, directly after pressing "start", the message window "Done" comes up immediately (always). But the conversion in fact is running in the background, then ... one just must not confirm that "Done" message, in order to get a result.

However, bugs aside:

You know that in the past, mostly I haven't been all that gracious with MSU filters ...
... But this time, I give a definite commendation. This is very good and very promising work! Especially when considering it's only a first release.

Well done. Applause.

.
.
.

Downside is that it's proprietary ... no surprise I've a corresponding framework for Avisynth already, for quite some time ...
but getting *those* results is not *that* easy. Really not.

[*.mp4 guy]

I realize that the results on that frame aren't great, I only posted it becuae DmitryPopov asked for examples of other filters. That said It still performed better then any of the other spacial only filters. The detail restoration that DmitryPopov's filter managed was impressive, and obviously no spacial filter could manage those results because the detail is quite simply not in the source frame.

[Revgen]

Dimitry - haven't had much time for testing, up to now. The application/GUI definetly has its buggies that should be fixed, you probably know. For example, my above error report was false: the application actually does the conversion; however, directly after pressing "start", the message window "Done" comes up immediately (always). But the conversion in fact is running in the background, then ... one just must not confirm that "Done" message, in order to get a result.

I've only had this issue using .avs files. AVI files work fine. Try using FFDShow's MakeAVI's and it should work.

[Revgen]

@Dmitry

Does your filter work directly in YV12? Because whenver I load a YV12 video, your app lists "uncompressed RGB" as the first option. Also, when I load an avs file using FFDshow, FFDshow processes the video in "RGB Flipped" colorspace instead of YV12 like it would in WMP.

[Revgen]

Okay I did comparisons between Foxyshadis' EEDI2 resizer and VRS and EEDI2 looks a lot better to my eyes. I'll post pics later.

[DmitryPopov]

Thanks again for valuable info and testing!
I haven't got much experience with AviSynth, and I guess most of ordinal users haven't got it either.

Thanks for MultiSWAR example!

Now Super Resolution is done in RGB, no YV12 at this moment.
That's because all the VirtualDub filters work in RGB32 and you can make any chains of VD filters and Super Resolutions in Video Enhancer.
But if there's really a need for YV12 support, that can be done for sure.

[Sirber]

Most people here use avisynth. VirtualDub is going down as well as VFW.

[Revgen]

Most people here use avisynth. VirtualDub is going down as well as VFW.

Sirber, although I agree, I don't think we should be starting a VFW flame war just this minute.

[Chainmax]

scharfis_brain made a superresolution script for a clip of mine here (http://forum.doom9.org/showthread.php?t=109422). He also made a thread about it here (http://forum.doom9.org/showthread.php?t=81868).

[Revgen]

Okay here is the test I've done. I used a 720p basketball game that I capped awhile back and used PointResize to resize it down to 720x480. PointResize makes the clip look uglier than Lanczos so it should be a challenge each resizer. I used both SuperResolution as well as FoxyShadis' EEDI2 Resizer (http://forum.doom9.org/showthread.php?t=112249&highlight). Here is the package (http://www.filefactory.com/?02e555) I created that has all functions and plugins setup. Just import the ekg.avs into your script.

Here are PNG's (http://www.filefactory.com/?24da86). It has the original HD video, PointResize vid, SuperResolution, and EEDI2.

Here is the setting I use for EEDI2 Resizer.

ekg(grain=0,iter=1,bp=1,sharp=0)

I don't use grain or shapening since the clip is free of grain and is already quite sharp.

IMHO the EEDI2 Resizer more closely resembles the original clip although it does cause slight oversharping in some areas. Keep in mind that I've not tried any other material except sports material since it's the only HD material that I usually cap.

EDIT: EEDI2 Resize package updated.

[Revgen]

scharfis_brain made a superresolution script for a clip of mine here (http://forum.doom9.org/showthread.php?t=109422). He also made a thread about it here (http://forum.doom9.org/showthread.php?t=81868).

Yeah, I forgot about that one. It's slow since it uses MVtools and there is no way to speed it up with SetMTmode since MVtools doesn't work well with MultiThreading.

[foxyshadis]

I like the results with:

ImageSource("sw4x.jpg",end=0)
ox=width
oy=height
PointResize(width/4,height/4)
makemod(8,crop=true)
converttoyv12(matrix="PC.601")
LimitedSharpenFaster(SMode=4,LMode=3,Strength=80,special=true,wide=true,soft=25)
EKG(grain=10,iter=1,bp=1,sharp=2,wsharp=0,gthresh=.5)
EKG(grain=20,iter=1,bp=1,sharp=2,wsharp=.5,gthresh=.5)
spline36resize(ox,oy)
ConvertToRGB32(matrix="PC.601")

http://foxyshadis.slightlydark.com/random/yoda1.jpg (http://foxyshadis.slightlydark.com/random/yoda1.png)
My function's been updated quite a bit (including today!), though I didn't post about it, just updated the first post, so you might want to update the package. ^^;

Since I have no context I can't test the new motion-adaptive part, and it's a bit annoying to start from a jpeg, but that's the way it goes.

A single run with extra sharpening also looks good:

ImageSource("sw4x.jpg",end=0)
ox=width
oy=height
PointResize(width/4,height/4)
makemod(8,crop=true)
converttoyv12(matrix="PC.601")
LimitedSharpenFaster(SMode=4,LMode=3,Strength=80,special=true,wide=true,soft=25)
EKG(grain=20,iter=1,bp=1,sharp=2,wsharp=0,gthresh=.8)
spline36resize(ox,oy)
LimitedSharpenFaster(SMode=4,LMode=3,Strength=80,special=true,wide=true,soft=25)
ConvertToRGB32(matrix="PC.601")

http://foxyshadis.slightlydark.com/random/yoda2.jpg (http://foxyshadis.slightlydark.com/random/yoda2.png)

That grain is simply my taste, not a necessary part of the script. It's a bit hard to balance between the "flowy" look two iterations leads to and a detailed look. Motion might help with that, as you guys've shown, but I have yet to do more than tinker with it.

[Revgen]

My function's been updated quite a bit (including today!), though I didn't post about it, just updated the first post, so you might want to update the package. ^^;

You managed to update it 3 minutes after I uploaded the package.:D

I'll update it right now.

[Chainmax]

Yeah, I forgot about that one. It's slow since it uses MVtools and there is no way to speed it up with SetMTmode since MVtools doesn't work well with MultiThreading.

Some people prefer speed, some people prefer image quality. Both deserve to see methods that fit their specific needs. Also, you can't argue with such an amazing result as scharfis_brain got on the 4x resize in my thread :).

[Didée]

scharfis_brain made a superresolution script for a clip of mine here (http://forum.doom9.org/showthread.php?t=109422). He also made a thread about it here (http://forum.doom9.org/showthread.php?t=81868).

Yeah, I forgot about that one. It's slow since it uses MVtools and there is no way to speed it up with SetMTmode since MVtools doesn't work well with MultiThreading.

That script can be sped up considerably, with just a small change. (Mates, do set those darn idx vars...)
However, this script is a very raw proof-of-concept, and is fully unusable in practice. Tests with re-upsizing a 50% downsized movie source have shown it puts out a softer & less detailed result than a plain lanczos or spline resize, d'oh. And heaps of artefacts. It surely does show the potential on some certain selected frames, but that's just it. Perhaps it's good for such a maximum-digital-crap source like that of Chainmax. But surely not for your "average" source, whatever that might be.


I used a 720p basketball game ... and used PointResize to resize it down to 720x480. PointResize makes the clip look uglier than Lanczos so it should be a challenge each resizer
Invalid test, or at least extremely suboptimal. Period.
With this method, you play the ball directly into eedi2's arms, and tie VSR to a pole, to hinder it from participating at the game. (Too tired for an explanation right know ... if you know what you're doing, you know that's right.)

BTW, are you sure you posted the right screenshot for VSR? There is virtually no difference between that VSR screenshot, and a plain Lanczos-upsize of the pointsample-destroyed version...

Of course, EDI algo's do have a big strength in spatial interpolation. And spatial interpolations you need even in temporal supersampling. That was my point in the "proprietary" comment - the user can not interfere somewhere in the middle of the process, only use post-processing afterwards. But at that time, it's too late for applying EDI ... So you're left at the developper's mercy to have done the job as good as possible.

That's why I love Avisynth. Everything under user's control. :)

Source: zoomed by 2x -vs.- sharp TSR with Avisynth
http://img155.imageshack.us/img155/3205/baseframezoom2x7xo.th.png (http://img155.imageshack.us/my.php?image=baseframezoom2x7xo.png) http://img126.imageshack.us/img126/8049/baseframetsrupsize3sx.th.png (http://img126.imageshack.us/my.php?image=baseframetsrupsize3sx.png)


I like the results with ...
Foxyshadis, everything okay with you? Can we help, is there hope? :D
(Waaay too artificial for my taste ... in this case, I'd prefer the beautifully smooth bicubicresize'd version.)

---

What happened -- me standing up for MSU against the masses? The world is out of order. :D

[CruNcher]

Dmitry Nice any HDTV Manufacture gonna use that in HDTVs (Hardware implementation) for upscaleing SD input to HD resolutions maybe Samsung ;) hmm btw is it possible to get a list of products in wich your algorithms are used :P

[Revgen]

Invalid test, or at least extremely suboptimal. Period.
With this method, you play the ball directly into eedi2's arms, and tie VSR to a pole, to hinder it from participating at the game. (Too tired for an explanation right know ... if you know what you're doing, you know that's right.)

Oooh... I must be an EEDI2 conspiracist.:D

I'm interested in an explanation anytime you're ready. BTW is there another resizer you would like me to use like Lanczos?

BTW, are you sure you posted the right screenshot for VSR? There is virtually no difference between that VSR screenshot, and a plain Lanczos-upsize of the pointsample-destroyed version...

It is from VSR. The differences are extremely subtle compared to EEDI2 resizer. I can create some .ebmp files that aren't converted to RGB and send them if you're interested.

[foxyshadis]

EEDI2 was developed specifically to solve the "every other field missing" problem, which is what you're asking of it, whereas MSU was made to solve the much more common (for its problem set) bilinear/bicubic downsized video problem.

Sorry, Didée, I think a plain resize just looks as if there's a clouded glass window between the screen and me. Looking again, I like the second better, though; I don't think the problem of eedi2 smoothing everything out after a couple of iterations is really possible to workaround in script, so I think combining it with regular resize and a crazy sharpening (SeeSaw would be better than LS here, but LS needs very little tweaking ;)) will give a more pleasant result.

[Revgen]

Okay I used LanczosResize instead of PointResize and differences are not as big now. SuperResolution does a good job of increasing resolution while keeping details. EEDI2 tends to smooth details a little more. This can be fixed somewhat by using more sharpening. However, it still can't beat EEDI2 when it comes to getting rid of aliasing. If you look at the yellow frame supporting the backboard you can notice a lot more aliasing in the SuperResolution screenshot.

Keep in mind that most sources won't have the diagonal lines, arches, circles, and other tricky stuff that a basketball court would have. SuperResolution should do fine with most movies.

Here's the PNG screen shots (http://www.sendspace.com/file/yhoejc)

Quick Summary:

heavily aliased content=EEDI2

blury low-rez content=SuperResolution


EDIT: Link updated. Using Sendspace now.

[foxyshadis]

Your upload is bad. eedi and sr pngs fail crc after multiple downloads. (One of the reasons I dislike filefactory, if it isn't corrupt on upload it'll probably corrupt randomly on download.)

[Revgen]

PNG's have been uploaded to sendpace. Hopefully it works better.

[CruNcher]

@Didée
yeah the RTL upsize looks amazing too, when we will be able to test it ourselves ? :D

@foxyshadis
your 2nd one reminds me more of the MSU Cartoonizer but that's not bad we really need some fancy Video Effect Scripts for Avisynth :D

and btw lets do a Motion Compensated Logo Remover :D

[DmitryPopov]

Since I have no context I can't test the new motion-adaptive part, and it's a bit annoying to start from a jpeg, but that's the way it goes.


Just to remind, there's a video clip (starwars, 50 frames) used for testing:
http://www.thedeemon.com/VideoEnhancer/ve-test.avi (2 MB, msu-ls-codec)
If you need to compare temporal algos.

[*.mp4 guy]

Okay I used LanczosResize instead of PointResize and differences are not as big now. SuperResolution does a good job of increasing resolution while keeping details. EEDI2 tends to smooth details a little more. This can be fixed somewhat by using more sharpening. However, it still can't beat EEDI2 when it comes to getting rid of aliasing. If you look at the yellow frame supporting the backboard you can notice a lot more aliasing in the SuperResolution screenshot.

Keep in mind that most sources won't have the diagonal lines, arches, circles, and other tricky stuff that a basketball court would have. SuperResolution should do fine with most movies.

Here's the PNG screen shots (http://www.sendspace.com/file/yhoejc)

Quick Summary:

heavily aliased content=EEDI2

blury low-rez content=SuperResolution


EDIT: Link updated. Using Sendspace now.

MultiSWAR (http://www.yousendit.com/transfer.php?action=download&ufid=AF3144BA29AFC48C), for comparisons sake.

[Revgen]

MultiSWAR (http://www.yousendit.com/transfer.php?action=download&ufid=AF3144BA29AFC48C), for comparisons sake.

Hey not too bad. It's similar to EEDI2. There is, however, still aliasing on the yellow frame holding the backboard, but not nearly as bad as SuperResolution.

What settings were used?

[MfA]

Just to remind, there's a video clip (starwars, 50 frames) used for testing:
http://www.thedeemon.com/VideoEnhancer/ve-test.avi (2 MB, msu-ls-codec)
If you need to compare temporal algos.
It would be much better to have the high resolution version and a script to create the downsized version (my preferred resize would be 1-2-1 gaussian blur with 2x decimation in each dimension). That way you can also do objective testing.

[*.mp4 guy]

Hey not too bad. It's similar to EEDI2. There is, however, still aliasing on the yellow frame holding the backboard, but not nearly as bad as SuperResolution.

What settings were used?

I think I used something very close to this, I can't be sure however becuase I deleted the script.
MultiSWAR(1440, 960, warping=1, StepRatio=4, sharper=true, Sharpen=8, Smoothing=true, Sthresh=128, Sstrength=.8, EDetail=true, Esharp=.25, FDetail=1, Sthresh2=1.1)

If you wan't all of the aliasing removed MultiSWAR can do that to, you just loose some detail. These settings remove more or less all aliasing from that frame.
MultiSWAR(1440, 960, warping=1, StepRatio=4, sharper=true, Sharpen=8, Smoothing=true, Sthresh=128, Sstrength=1, EDetail=true, Esharp=.25, FDetail=1, Sthresh2=1.1)

[foxyshadis]

EEDI-based algorithms tend to turn details (ie, faces) into random squigglies, and lower-detail areas into very sharp curves. Whereas at least normal resizing methods look rather fuzzy (and/or oversharpened) across everything. I'll play with mine once I have some more free time.

[Dayvon]

Super Resolution is a technique of increasing images or video resolution with superb quality. For each processed frame it uses information not only from this frame but also from other frames. If picture in your video doesn't change too fast, then information from several frames can be added to create a picture of a bigger size. No algorithms of ordinary image resize/resample can provide this quality just because there's not enough information in one frame.

Video Enhancer is a tool which uses Super Resolution technique to increase resolution of your video.

Also, it can use a hundred of VirtualDub filters to process your video: deinterlacing, denoising, deshaking, color correction etc.

Main page:
http://www.thedeemon.com/VideoEnhancer/

Compare Super Resolution to bicubic and Lanczos3:
http://www.thedeemon.com/VideoEnhancer/compare.html

If this was AVISynth compatible and also added little to no slowdown.... I'd definitely look at picking it up. I like the results and have always been intrigued with upsizing.

BTW, It was said earlier that this filter is fast? Like TIVCT+TDeint fast?

[unmei]

with the version i downloaded today, it seems i have to press stop when the preview doesn't advance anymore. It doesn't look like it would complete otherwise (xvid avi input, uncompressed avi output).

[billou2000]

is there a support for interlaced content?

[DmitryPopov]

Current version of Super Resolution algorithm works with progressive video. In Video Enhancer you can use deinterlacing video filters before SR. Native support for interlaced content is planned for future versions.

[DmitryPopov]

Today version 1.1 released.
Now works with files containing mp3 audio stream. Previous version stopped processing on such files right at the start. Please check it out.
http://www.thedeemon.com/VideoEnhancer/

[Doobie]

I only now read this thread. It took so long becaue the title sounded so much like spam.

The last place I want to fudge details is in the encoding. There it would dramatically increases the size of the encoded file while providing only minimal quality improvement. Enlargment of the video should be done on playback.

[laserfan]

Is Video Enhancer completely self-contained, or are helper apps needed? I tried a couple of "other" filters in a chain, and their Preview buttons didn't seem to do anything i.e. I expected some sort of ability to see the impact of the filter as I adjust its settings?