Using TomsMoComp for Upscaling

[trbarry]

I recently introduced my new TomsMoComp filter in this thread (http://forum.doom9.org/showthread.php?threadid=28778) but mentioned there was another feature of it that I'd post more about later.

There is a mostly experimental easter egg feature that allows it to vertically double the size of your clip, in a way that seems to present a bit more apparent detail than just scaling it. It was a side effect of part of the deinterlace algorithm but I thought it was interesting so I made it a feature.

So what am I up to here, and why would I want to do that?

One possibility is to upconvert DVD's and make my own faux-HDTV clips. But I'll post later about that after some more research and more coding.

But for now I wanted to consider upsizing as a way of making and viewing low bit rate clips. For instance, making yet another 1 CD encoding of The Matrix.

Sometimes, either because it does not IVTC or deinterlace well, or because you just want to save bits it may be useful to keep only the even fields of a clip. That way avoids weave artifacts and on low bit rate encodings can be a fairly efficient way of storing video.

The idea is that if material is interlaced or poorly telecined then the odd lines have less reliability than the even ones because they are not samples from the same point in time. (or vice versa)

So if you can only save a limited number of bits maybe it is best to choose only the good ones, from the even lines.

To test this out I made an 2 pass Xvid 1 CD rip of The Matrix, encoded with the following AVS script:


LoadPlugin("D:\mpeg2decB\release\MPEG2DEC2.dll")
mpeg2source("E:\E_VCR\Matrix\MatrixC.d2v")
separatefields
selecteven
crop(8,31,704,176)


The encoding was very fast on my P4 because the script basically doesn't do much. But notice the strange 704x176 resolution. I encoded this at a video rate of only 591 kbps. This rate just happens to match Doom9's last 2002 MPEG-4 Codec Shoot-out. You can find his excellent write-up about this here (http://www.doom9.org/codecs.htm). I didn't otherwise stick to his resolution or parms but at that link you can find a lot of sample pics. I'll post some matching ones below.

But why encode at 704x176? It is not very practical to view yet, with little short squashed images. So to display the clip I had to use the following second AVS script, using the TomsMoComp fancy doubler. I could do that either in ZoomPlayer or Virtualdub.


LoadPlugin("D:\TomsMoComp_Avisynth\TomsMoComp.dll")
LoadPlugin("D:\AVISynth\SimpleResize.dll")
avisource("E:\E_VCR\Matrix\MatrixE2.avi")
TomsMoComp(1,-1,1)
SimpleResize(704,304) # not needed when using ZP hardware overlay scaling


This allowed me to view the clip with a 704x304 rez. The samples below are taken from vdub and mostly match some of the shots in the Doom9 shootout.

see:
www.trbarry.com/Matrix_Wires.jpg
www.trbarry.com/Matrix_Wires_NO_VF.jpg
www.trbarry.com/Matrix_Profile.jpg
www.trbarry.com/Matrix_Heads.jpg
www.trbarry.com/Matrix_Kick.jpg
www.trbarry.com/Matrix_Pistol.jpg
www.trbarry.com/Matrix_Fire.jpg
www.trbarry.com/Matrix_Wall.jpg

All of the pics above were made using the TomsMoComp vertical filter option, except the 2nd one. The VF option makes things just slightly softer but without that option the doubling algorithm can sometimes create artifacts on very thin diagonal lines. See the wire in the second pic.

Of course for higher bit rate encodings of quality material you wouldn't want to throw away half the info in a clip. But for one CD rips I thought maybe this was an interesting way to go about it. But I don't usually make 1 CD rips so maybe I'm overlooking something.

I'm probably also going add a horizontally expanding version of the same algoritm, to see how far I can push this. Maybe some of you might tell me not to waste my time but, if there was any real interest in this it could probably be added as a feature to ffdshow or the Xvid dshow display filters. If it had both H & V expansion than it might be a better way to view small old 320x240 clips on a bigger screen.

If there isn't much interest then I'll probably just enhance this to see if I can process DVD's into faux-HD upconverts like ShowTime makes.

Any comments?

- Tom

[poptones]

I don't have much enthusiasm for making encodes that only work with special software, but this filter is working great for making a progressive DVD from an interlaced source. I know I said that before, but I was pretty blown away. Richard and I had talked a bit about this and I looked into the Faroujda stuff but now I'm honestly not certain there's any need. I still think it would be cool to have it as a more specialized bob, 'tho. Just to make it easier to use.

Check this out:

http://personal.ayrix.net/~poptones/deinterlace/tomshirl.jpg
http://personal.ayrix.net/~poptones/deinterlace/jaggedshirl.jpg

One uses the old bob() method. The other uses your filter with the requisite split>crop>interleave to get rid of the bounce. The difference is pretty damn impressive, I think.

[poptones]

Addendum: how the hell did you do that? Would you mind describing the algorithm of the "-1" setting?

[trbarry]

I don't have much enthusiasm for making encodes that only work with special software,

poptones -

Me neither actually, though I could care less if things run on standalone players. But this was sort of a proof of concept thing, not even really what I set out to write.

But I do seem to have blinders on when it comes to encoding at more than 30 fps in Avisynth. The (jiggle free) TomsMoComp code aready runs fine at 60 fps in DScaler but somehow I assumed for the Avisynth version that I'd only need one frame for every 2 fields.

But I will add a 50/60 FPS flavor later for video source.

I just saw your cross post "addendum". Since I just got back from drinking dinner maybe it would be better if I explained the algorithm later. ;)

But look at the big block of comments in member WierdBob.inc, in the zip.

Cheers,

- Tom

[Defiler]

How exactly does this differ from merely firing up the clip in BSPlayer and forcing a particular aspect ratio?
Intriguing thread, I must say.

[trbarry]

How exactly does this differ from merely firing up the clip in BSPlayer and forcing a particular aspect ratio?

Normal scaling methods just smoothly stretch along the horizontal and vertical. TomsMoComp attempts to fill in pixesl that would preserve edges, even when diagonal.

- Tom

[Defiler]

Awesome.

[vlad59]

That's a pretty good idea Tom !! ;)

I'm going to make a test myself and check how much CPU is required to play the encoded script (even with hardware resize).

Instead of using
separatefields
selecteven

perhaps it should be interesting to use VerticalReduceby2 (it average a little the result and will increase the compressibility of the movie) or maybe the vertical filter of your TomsMoComp is enought (just an idea I'm going to check that).

[trbarry]

vlad -

I think VerticalReduceBy2 would work better if you had a noisy clib and wanted it softer, but it would only work if you were sure the clip was progressive or had already been well deinterlaced or IVTC's. Otherwise it might blend data from unmatched fields, causing ghosting.

But on clean source (most DVD or HDTV) I prefer to just separate them. I think I get the better compression by simply having to encode fewer pixels. But there's been limited testing so far and I probably won't do a lot until I can also expand horizontally. I've no idea how that will affect things.

- Tom

[vlad59]

@Tom

Yep, you're right. In france 95% of the Dvd are not interlaced and no IVTC problem so I forgot it ;) .
I'll check the CPU load at home tonight.

[vlad59]

I've done a little test and the results are good.

The CPU load was not too big : 20 or 30% more on my P3-550.

I was testing on an anime and during normal playback it was hard to see a big anormal thing. With virtualdub I found some little pb especially on the title : the name of the anime is in blue on a black backgroud and due to the vertical filter there was little blue waves on top of each character.

I also found a interesting point, you must crop all black bars (top, bottom, right and left), otherwise you'll see those mini waves on each side (very annoying).

That's all for my first report, it could be very interesting as soon as it added on a directshow filter because a light deblocking (only light to keep the edges) could be done before your filter.
I'll carry on with a normal film.

EDIT : my source is very noisy, it's maybe because of that

[trbarry]

Hi Vlad -

Are you talking about just vertically doubling, or deinterlacing? An early version of the filter seemed to have problems with overlayed static video text over film, but I thought I had that fixed.

And I guess I never tried it on black bars.

- Tom

[vlad59]

I'm just talking about vertical doubling (my source is not interlaced at all and pure PAL). But as I said my source is very very noisy so it must be an explanation (It's the less compressible DVD I have and I usually use TemporalSmoother (4 or 5) to compress it). It's also not a good masterised DVD so the text is not full static, it's slightly moving (like on old film).I'll try with a normal film.

For the black bars, it wasn't intentional : I just forgot to crop right an left and I've seen those litlle waves. Ooooops :rolleyes:

[Blight]

This seems like an interesting idea. I'd like to see it implemented into FFDShow, with both V and H resizing (for those pesky 320x240 clips that's still out there) and if possible, It'd be nice to throw a WarpSharp ontop of that to further bring out the edges on these low quality clips.

[trbarry]

WarpSharp is going to be someone else's job.

I've got the H doubling mostly done but so far it doesn't seem to look as nice as the vertical. It's probably just a bug but maybe there is a limit how far we can stretch things this way.

I think I'll also jump into the wayback machine here and respond again to poptones first comment at the top, that it is not nice to make clips that have to be played by a special process. That's true but probably I should have already pointed out that ANY of these clips are perfectly playable in any player like ZP that supports ANY kind of resizing. I just think they can maybe be played a bit better upsized with this new method.

But there is only a small difference.

Now back to figuring out why my horizontal stretching still looks like sh**.

-Tom

[Blight]

I'm not sure if this is how your filter works, but ... here's my view on how to do interesting value-adding scaling.

Run a motion search algo. between frames, find the direction of the motion in a frame, then find edges of object and transfer the data along the motion route.

I think this approach would yield nice quality, but ... WAY CPU intensive as motion search code takes time to do well. A possible solution in case such as FFDShow is to use the motion vectors already used in extracting the MPEG4 video data without having to find them yourself.

[vlad59]

I just checked with Excalibur (very good old movie) wich has a very good quality. And there is no little waves around the titles. So noise was the problem with my anime.*

EDIT : I'm still looking for a good test of your new Deinterlace filter. But all the interlaced DVD I have (mainly anime) are just NTSC Dvd slow down to 25fps si it's not a good test. still looking...

[trbarry]

Go rent some adult xxxx. ;)

- Tom

[int 21h]

I'm interested in the idea of a decent 352x240 scaler for computer, 90% of the time I watch VCDs and such on my standalone so it isn't an issue, but 10% of the time I watch VCDs and TV Series at work, and a higher quality process to watch these on the computer monitor would be great.

Also, it maybe a useful filter to use for just regular analog conversions to DVD video ;)

Can't wait for the ffdshow addition! :p

[trbarry]

I've never compiled or even downloaded the source for ffdshow but I've heard it has some Unix oriented quirks that make it harder for a VS6 programmer like me.

But I'll check with Milan after I get this stupid horizontal stretch working right.

- Tom

[vlad59]

Originally posted by trbarry
Go rent some adult xxxx. ;)

- Tom


Ehhhhhh, don't know :rolleyes: :rolleyes:
Sounds like a good idea for a good weekend ..... ;)

[trbarry]

Vlad -

I wasn't really making a suggestion about what you should do for the weekend ;) , only that porn DVD's are typically shot on video and you stated you had nothing to test/deinterlace.

One of the highest quality DVD's I now own is a title called "Whispering Hearts" that I recently ordered. It has the distinction of being the first and maybe only porn DVD shot with HDTV video cameras, though it's mastered as a regular video DVD. But I figured it would be a great but controversial subject for my experiments to upscale DVD's back to 720p HDTV using TomsMoComp. ;)

Later I'll probably post some (G-rated) pics.

- Tom

[avih]

lol, pls do ;)

[Defiler]

Originally posted by trbarry
One of the highest quality DVD's I now own is a title called "Whispering Hearts" that I recently ordered.
How is it? :D

[vlad59]

Originally posted by trbarry
Vlad -

I wasn't really making a suggestion about what you should do for the weekend ;) , only that porn DVD's are typically shot on video and you stated you had nothing to test/deinterlace.


I already know that porn DVD are mainly interlaced. You just gave me a good excuse to join the the useful and the pleasant (I use BabelFish to translate a french expression : "Joindre l'utile à l'agreable", don't know if it mean something in english).

But unfortunately ;) , I found a interlaced concert DVD.

Back to the topic : I converted Excalibur to Xvid using your method (SeparateFields, SelectOdd) and the results are quite good.
Why quite, because it's a long movie (135 min) on 1CD and so the bitrate was 600Kbits. But the result on still scene is good and on moving scene some postprocessing should help (it's blocky).

[xly]

Looking for a good solution to transfer my personal DV PAL clips to Mpeg2, I have tried your filter with the following Avisynth command. I was impressed by the result.

LoadPlugin("D:\AVIDEO\VDub1410\plugins\TomsMoComp.dll")
SegmentedAviSource("c:\--pool--\E\2ACarn00.avi")
SeparateFields
TomsMoComp(1,-1,1)

By the way SegmentedAviSource is an useful command to load a sequence of files captured with Scenalyser.

Surprisingly the Mpeg2 clip was much better with TmpgEnc than with CCE which shows some light vertical lines, visible with Zplayer zoom.
I continue to investigate.

Thanks and congratulations

xly

It's more fun to compete !!

[trbarry]

I already know that porn DVD are mainly interlaced.

Vlad -

Actually, to make myself a liar, it turns out the one I was talking about above was really progressive. DVD2AVI said it was 100% interlaced but it was (I think) 24p HDTV progressive video source, DeComb handles it just fine with no deinterlacing. And it wasn't even CSS encrypted. ;)

I hope that both become a trend in DVD production.

- Tom

[trbarry]

xly -

If you are looking for the highest possible quality (instead of saving bit rate and disk space) on video source then you might be better off using TomsMoComp as a deinterlacer instead of a doubler.

Maybe try just using:

LoadPlugin("D:\AVIDEO\VDub1410\plugins\TomsMoComp.dll")
SegmentedAviSource("c:\--pool--\E\2ACarn00.avi")
TomsMoComp(1,15,1) # or TomsMoComp(0,15,1)

to deinterlace the entire frames with motion comp deinterlace.

- Tom

[xly]

Thanks for yr quick reply and comment.

Finding the best way of encoding of DV (Pal interlaced) to Mpeg2 is a challenge for all our community.

To do so I am testing the combination of Avisynth commands/filters by loading them both in VirtualDub and/or Zplayer. This gives a first indication: visual quality, frame/field size, frame rate.

I then encode to Mpeg2 the pseudo-avi avs file with TmpgEnc and/or CCE and I compare the results. In addition to the visual checking, I check the details of pixel structure obtained with the zoom feature of Zplayer.

Obviously the Mpeg2 encoder adds its own effect in the encoding process. The difficulty of this exercize is to find the best matching between Avisynth filtering AND mpeg2 encoding. One solution may be better in a pure avi world (DV to divx for example)and less effective when combined with mpeg2 encoding.

I use se for my tests a clip showing a baby sit in his cart in a garden. It gives a mixture of several difficulties : horizontal lines (the car in the back side), oblique lines (the cart), text ( the cart name), details ( grass and flowers), delicate skin, typical DV "edge" artifact(baby socks).I've attached a self-extracting bmp image to illustrate this kind of scene.

DV encoding is a challenging effort as you must fight against the "mirage effect" with horizontal lines floating, the scale effect of oblique lines, the pixellisation of plain surfaces, the blurring of details etc

Considering all these difficulties your solution is not at par with mine at this moment. Don't ask me why.I am still investigating and I shall report to you.

xly

It's more fun to compete !!

[xly]

I did not notice that Domm forun did not accepted *.exe (self-extracting) files. Please find this bmp file in a zip format.

xly

[xly]

I have compared the 2 following Avisynth sequences A and B

A:
SeparateFields
AssumeFieldbased
TomsMoComp(1,-1,1)

B:
TomsMoComp(1,15,1)

A gives a progressive+50 fps clip with a top quality.
B gives an excellent progressive+25 fps clip

A: and B: have similar filesize.


xly

[frank]

Hmm...

A:
SeparateFields.SelectEven
TomsMoComp(1,-1,1) # doubles the lines

Now you'll get 25 fps and the same filesize.

[xly]

Thanks for your Hmmm

In fact my A: formula does not make any miracle : only a good 50 fps progressive image without de-interlacing artefacts, to feed the Mpeg2 encoder which seems to be happy with that !

When going to 50 fps by keeping the 2 fields of an interlaced frame transformed into frames you don't lose information and you keep the initial fluidity. It visually gives a smoother image than by skipping one of the two fields , mainly for the oblique lines.

TmpsMoComp (1,-1,1) probably does nothing else than doubling the lines which is not its main interest ...I agree with you.

Anyway, the DV-2-Mpeg2 conversion being a challenging issue, and I am waiting for challengers.

xly

Its' more fun to compete !!

[frank]

The only thing I can't agree is that you say A and B have the same filesize.

Yes, you doubled the framerate and don't lose any information - but you added redundant information by doubling the lines!
With doubled framerate 50 fps at the same size you'll get the twice file length.
Then I'd prefer de-interlacing.

When watching interlaced video on PC screen the BOB de-interlacing routine doubles automaticly the framerate as you wanted.

[xly]

Frank,you can check yourself that the final size of 50 fps Mpeg2 file is the same as the 25 fps one. I check the fps with BitRateViewer but I can't bet that it gives a correct number.

Of course, I can change the parameters of my PC player (ZPlayer as mentioned before), but this should influence the comparisons. I also compare with my Dvd Player as well.

At the end of the day we will probably come to the conclusion that all these quality comparisons are nonsense as it is a matter of personal feeling. Indeed as long as there is no objective scientific instrument to mathematically measure pixel by pixel the differences between two files, source and output, every method can be considered as the best.

It's just a matter of discussion.And how can we support all these brilliant who continously work to improve the quality of filters if there is no mean to evaluate the improvement... That is another story !

Thanks for your help

xly

It's more fun to compete !!

[lark]

what am i missing?
isn't it obvious that the filesize is about the same,
because the lenght (in seconds) of the clips is the same
and the codec targets for a bitrate/s....?

regards
t :)

[xly]

You are right and wrong !

In fact the file sizes and of course the bitrates are the same although I have selected the Tmpg CQ mode, not the CBR !

Anyway I noticed with BitrateViewer that the Quant factors were different 9,23 for 50 fps ( case A) and 6,35 for 25 fps ( case B)!!
As far as I know a lower Q factor means a better encoding !!

This confirms that when considering a DV2Mpeg2 conversion we have to find the best matching between the Avisynth filters pre-processing and the Mpeg2 encoding parameters !

But in line with your remark I wonder if finally it not better to leave the lines "undoubled" to keep the fluidity effect of 50 fps without doubling the source bitrate.

xly

It's more fun to compete !!