TurnsTile 0.3.2 - Mosaic/palette functions

[Robert Martens]

TurnsTile 0.3.2

Avisynth 2.5 - Windows - x86
Source - GitHub


Use TurnsTile() to achieve a blocky, low-res mosaic effect on your input, and the companion CLUTer() to apply an arbitrary color palette. Several sample tilesheets, and one sample palette, are included.

Both functions support RGB32, RGB24, YUY2, and YV12.

Old versions:
0.3.1 - GitHub
0.3.0 - GitHub
0.2.1 - GitHub
0.2.0 - GitHub
0.1.0 - GitHub

Previous OP follows.

----

As intimidating as it feels to join this side of the AviSynth world, I'm going to dip my toes into the waters of plugin development and offer this, the result of about three weeks of fiddling with Visual C++. TurnsTile takes an input video in RGB32, RGB24, or YUY2 formats and builds a mosaic based on it, pulling image data for each tile from either the video itself (for basic mosaic operation) or from a custom sheet that you provide (to provide something similar to a palette effect, though true palettizing is still only a plan I've got for future releases).

Sample stills (see a few posts down for one showing off 0.2.0's updated 'res' parameter) don't show off the effect as well as actual video, so take a look at a collection of samples I strung together in this Youtube clip: http://www.youtube.com/watch?v=WFJreg-LW50

If you just want to try it out, everything you'll need is included in the download. Everyone else settle in for another of my history lessons, I'm about to drift off on a tangent.



I've been a fan of the rogue-like computer game Dwarf Fortress since its first public release in 2006, and although my interest is intermittent (I'm not a very good player, it's easy to get discouraged) I come back to it now and again. The game's graphics are old-school pseudo ASCII text, and over the months I spent developing SimpleSlugUpscale, I got the itch to use my newfound script writing abilities to try and simulate the effect in AviSynth. A few weeks back I put together a script, employing Gavino's wonderful GScript and GRunT plugins, that accomplished what I was after, successfully chopping up the game's modified CP437 tilesheet (hence "pseudo" ASCII; the game uses OpenGL to draw its tiles) and applying the characters to the input video.

The only problem was the 0.04 frames per second render speed. Frankly I'm not sure "frames" deserves that 's'.

Some careful rethinking of my approach allowed me to shuffle the ScriptClip ugliness to a better place, to do as little work per frame as possible, and I managed to tweak the script up to 0.8~0.9 fps. This, mind you, for a 640x480, 30p input clip. "Improved by an order of magnitude" if you're in advertising, "still laughably unusable" if you're the rest of us. I spent some time trying to convince myself that writing a plugin was stupid, and performance wouldn't see much improvement, if any. The Crop and Stack filters I used in the script are fairly low cost, as I understand, and the GScript loops are compile time constructs that don't themselves impact runtime performance. RGBDifference/AverageLuma are more demanding, but still very well optimized by people who know a lot more about this than I do.

Eventually, I got over my self-doubt and set to digging through the Filter SDK, SimpleSample (thanks to Simon Walters and everyone else who worked on that!), and the AviSynth Development board. After a while, despite fourteen years of false starts learning C/C++, I managed to wrap my head around the principles involved and get a plugin to compile.

Apparently, the nature of the effect I was after is a bit demanding for a user created script, no matter how efficient the underlying code, because with the same clip and tilesheet, the plugin version of my idea running on the same system hits well over a hundred frames per second. Of course, now that I've gone through the process of creating TurnsTile, I would hardly be surprised to find that a more experienced scripter could do what I failed to without having to write a plugin. It was useful C++ experience, though, so I chalk this one up to a win.

Should you want to try and turn out frames a bit faster, I've tested this plugin with SEt's AviSynth 2.5.8 MT, and it seems to work in mode 1. I can't make any promises (I'm still trying to understand the idea of "reentrant" functions, and how they provide, or at least help thread safety), but it's worth trying if you're really in a rush. Just make sure you have a fire extinguisher handy.

Planar support is going to have to wait until my brain catches up to the idea, as will assembly optimizations. One thing at a time, I'm sure I'll get there eventually.

Included in the 'extras' folder in the archive are a few goodies I thought might be of interest:

-A handful of solid color gradient tilesheets to get you started
-The barebones tilesheet generator I wrote to create said tilesets
-AnimatedTiles.avs, a script that blends through a rainbow of the aforementioned sample sheets, to be used with AVISource as another tilesheet (as I did with the highway footage in the sample clip)
-A tilesheet pieced together from the others, meant to provide a "poor man's thermal camera" effect (as per the third video sample), along with the script that built it
-TurnsTile_Script.avs, the positively tectonic script version of TurnsTile 0.1.0 that motivated the plugin (requires GScript and GRunT)

All told, this code is undoubtedly naive, and I'm sure you'll find some real groaners in the source, but it seems to do the intended job well enough that I thought someone else might like to play with it.

[osgZach]

Aww coool, Predator-vision!

The ASCII art effect is nice too

[Gavino]

Thanks, Robert.
An instructive example of the power and flexibility of the script language on one hand, and its performance limitations vs specifically crafted plugin code on the other.

Originally Posted by Robert Martens

I spent some time trying to convince myself that writing a plugin was stupid, and performance wouldn't see much improvement, if any. The Crop and Stack filters I used in the script are fairly low cost, as I understand, and the GScript loops are compile time constructs that don't themselves impact runtime performance.
...
Apparently, the nature of the effect I was after is a bit demanding for a user created script, no matter how efficient the underlying code, because with the same clip and tilesheet, the plugin version of my idea running on the same system hits well over a hundred frames per second.

The overhead of the scripted version comes from two things:
1) Repeated chaining of StackHorizontal/Vertical.
Although each single operation is relatively fast, there are 225 separate Stack instances (assuming the default no of tiles), so that's a lot of data being copied from place to place on each frame.
2) Use of ScriptClip which means the Avisynth parser being invoked for each tile of each frame (in fact twice per tile since Eval is used).

In contrast, the plugin code can build up the tiles in situ in a single video frame, and without either parsing or invocation of other filters.
The comparison is slightly unfair, since the plugin uses just one pixel (or two, depending on settings) to determine which tile to select, whereas the script version uses AverageLuma on the entire tile. However, even if the code was extended to do the equivalent of AverageLuma, it would still be very much faster.

I've tested this plugin with SEt's AviSynth 2.5.8 MT, and it seems to work in mode 1. I can't make any promises (I'm still trying to understand the idea of "reentrant" functions, and how they provide, or at least help thread safety)

Since your GetFrame code does not write to any global or class variables, it can be called by several threads at the same time (and for different frames) without any interference, making it suitable for Mode 1.

[Robert Martens]

Originally Posted by Gavino

Use of ScriptClip which means the Avisynth parser being invoked for each tile of each frame (in fact twice per tile since Eval is used).

That Eval, for the 'diffstring' variable, is actually something I only added while preparing the package for release. Previously I'd been testing the script version with either RGBDifference or AverageLuma hardwired in, as appropriate for a given clip. My stated performance numbers were based on that, and I can imagine it's only slower with Evaluate thrown into the mix. Me and my brilliant last minute ideas.

Originally Posted by Gavino

The comparison is slightly unfair, since the plugin uses just one pixel (or two, depending on settings) to determine which tile to select, whereas the script version uses AverageLuma on the entire tile. However, even if the code was extended to do the equivalent of AverageLuma, it would still be very much faster.

Although I was aware of at least one plugin that lets users grab individual pixel values, I never took the time to try using it in the script. I suppose I'm glad to hear it wouldn't have helped much. Something I did toy with was resizing the clip to its minimum before running the analysis; at 640x480, with 16x16 tiles, I'd Point or Bilinear resize the clip down to 40x30 first, then crop to 1x1 for each tile and analyze that. Didn't seem to offer much of a speedup, though, and made the script a bit more complicated.

Thanks for the feedback, very informative! Especially the note on multithreading, that explanation makes perfect sense.

While I'm at it, I wasn't sure if I should wait for jmac698 to post something here, or if I should address the issue in the CCC thread, but I'm replying already so I may as well touch on this:

Originally Posted by jmac698

I would describe the effect as thus; an image contains a grid where each square represents a replacement for a brightness, this is applied to the video. So a gradient of grey would show a normal video, but a tile of ASCII would make an ASCII video (like in VLC). Weird colors make a cool thermal cam effect.
Is that right?

That's essentially the way it works, but a gradient of grey will show a grey video, as in the first example clip shown in the Youtube video. I don't really "replace" anything in the original clip, strictly speaking, I build a new one out of pieces of the tilesheet. By default, it's based on an average of R, G and B for RGB footage, or Y1 and Y2 for YUY2 clips, but you can change that with 'mode'. Any individual component can be used as the basis for the tile index decision.

Some sort of actual blending might be good for the future, mixing the tile contents with the section of the clip behind them (namely pulling color from the background to tint the tiles in the output), but I don't do that yet.

[Robert Martens]

TurnsTile 0.2.1

Sorry to report no YV12 support yet, but I tinkered with this some more and added a couple of things:

-An external tilesheet is now optional. If you just pass in one clip, a basic mosaic effect will be applied to it.

-The behavior of 'res' has been changed. Instead of being an arbitrary divisor, it now serves as a bit depth simulator. The output range is cut up into 2^res pieces, and each value (the tile index, when a tilesheet is used, or the RGB/YUV component when only one clip is provided) is rounded accordingly. For example:



-'lotile' and 'hitile' have been added, in case you have a need to limit the tiles selected from your sheet. Maybe you have an odd number of tiles, with some blank spots at the top or bottom of the image, or you just want to limit which tiles are used without having to create a new sheet in an image editing program. Whatever the case, use these to dictate your desired bounds.

-Tile size is now automatically calculated based on the input clips. I try to hit 16x16, but reduce it as necessary to get factors of the clip dimensions involved, and dropping as far as necessary to achieve square tiles. For example, 640x480 will default to 16x16, 1920x1080 will be 15x15 (assuming no tilesheet; with one of the 256x256 files I bundled into the zip, it'll be 8x8), and 797x512 will be 1x1 (though you could get 1x16 if you really want to). If you provide a tilesheet, it comes into the equation as well, and the dimensions will be factors of the width and height of both clips. Somewhat confusing to explain, but it should make the plugin slightly less stressful to use.

SILENT UPDATE: Just caught and quickly fixed an obnoxious bug in the auto calculation code; I wasn't checking the minimum width, so YUY2 clips with mod 1 height would try to use 1x1 tiles but fail, throwing an error. 0.2.1 confirms that the requested tile dimensions adhere to the minimums.

[jmac698]

I would like an effect where I use a certain palette for the tiles. You could directly simulate oldschool computers this way. CGA for example is just the basic colors and combinations.
Also it would be great if you could write an article on how you learned to make a filter. I still can't do it in C++ but I made an article for C.
That it took you 3 weeks is exactly my complaint. I needed to get something done and still it took a week to write a simple plugin in C. That's why I think scripting is far better for getting stuff done, even then it takes hours. So there should be a way to make scripting more powerful, or some plugin template where everything is ready and you just write the pixel code. It should be possible to give an avisynth script function definition and write a full plugin template based on it like turnstile(clip c, clip tile, int res) or whatever.

[Robert Martens]

Originally Posted by jmac698

I would like an effect where I use a certain palette for the tiles. You could directly simulate oldschool computers this way. CGA for example is just the basic colors and combinations.

Absolutely! I agree, true palette functionality would be great, and is something I plan on tackling next. I don't think it'll be too difficult, but I've said that about a lot of things, so we'll see how it goes.

Off the top of my head, I think the approach would involve taking a tilesheet as input (or, perhaps, a third clip, so you could have a tilesheet and palette at the same time) and presorting the tile indices according to the average RGB value for a tile, thereby giving me a base to use for selecting the closest match. Along with that I could have a variety of presets for things like CGA, NES, Atari, and other assorted systems. The idea's a work in progress, so take that with a grain of salt.

Originally Posted by jmac698

Also it would be great if you could write an article on how you learned to make a filter. ...
That it took you 3 weeks is exactly my complaint.

That's not a bad idea, I think I'll take a crack at a tutorial. There are some holes in my knowledge of software development/C++ that make me question my authority to tell anyone how to do this, but obviously I figured out how to get something useful accomplished, so I might be able to work up an article of some sort.

I really don't think the three weeks is too bad, though. I suppose when you consider how much free time I have compared to most people, my three weeks could be more like everyone else's month and a half, but it still strikes me as a reasonable time frame. SimpleSlugUpscale 0.1 took me a month to write, and it's a script.

Most of the three weeks I spent on TurnsTile was thinking my way through the algorithm. With a specific project in mind, tackling the unfamiliar aspects of programming in general, C++ in particular, broad image processing concepts, and AviSynth-specific conventions was really only a matter of Googling anything I didn't know. Between that, the Filter SDK pages, and SimpleSample, I pieced the framework together fairly easily. The overwhelming majority of my time was, in fact, spent figuring out the pixel processing code.

The plugin template idea is intriguing, but I can't pretend to have enough knowledge just yet to comment one way or the other on whether it's feasible.

[Robert Martens]

Originally Posted by jmac698

I would like an effect where I use a certain palette for the tiles. You could directly simulate oldschool computers this way. CGA for example is just the basic colors and combinations.

Done and done. Version 0.3.0 is finally finished, less than a week shy of two months later. This probably should have been at least two releases, I don't know how to pace myself intelligently.

-Actual paletting, albeit using only Euclidean Distance as the measure, is now present in the form of CLUTer(), which lets you apply an arbitrary palette to your input clip. As per the docs, however, keep your color count reasonable, I'd estimate that a hundred unique values or so is the top end of bearable. The approach I've taken is rather blunt, and doesn't take well to large palettes. I've included an AviSynth script that generates the CGA colors, as laid out on the Wikipedia page, but you're welcome to roll your own.

-TurnsTile finally supports YV12, which looks almost as good but runs much faster. Both functions support all four 2.5.x colorspaces.

-An interlaced parameter is present in both functions, in case of interlaced footage. Especially important for YV12 processing, since the two functions, at some point, each need to read the right luma/chroma samples for a given pixel.

-The 'res' option has been fixed, and now rounds correctly. Previous releases had been simulating one more bit per component than the user requested, the new setup addresses that.

-lotile and hitile now work in clip-only mode in TurnsTile, controlling the minimum and maximum component values, respectively. A trivial effort, really, I don't know why I thought I couldn't do it before.

-Countless code changes, large and small. Got rid of one gigantic, embarrassing inclusion guard mistake, eliminated useless #defines, switched tile index/component scaling from on-the-fly to using a precalculated vector. All sorts of goodies. Review the source if you're interested.

[Robert Martens]

Nothing major for 0.3.1, but a quick troubleshooting session with a user revealed an unfortunate dependency on the Visual C++ CRT DLLs, which has now been eliminated. While I was at it, I fixed another mistake in my project settings, and the plugin now does not require SSE/SSE2 support. Probably not a huge issue in 2011, but it was still an unnecessary demand, and there was no noticeable performance improvement anyway.

[Robert Martens]

Okay! Enough screwing around. For a year and a half now I've been letting myself get distracted by all sorts of other projects, and between that and my unconscionable version control practices I haven't gotten a damn thing done with this plugin.

Armed with what Git I learned working on an unrelated project, I've abandoned the sloppy repo I had going for the next major TurnsTile release. I've started from scratch, adding in the first five releases as they were but then proceeding to commit in something like a responsible manner. The code for Avxsynth and Vapoursynth support is largely ready to go, but before going back and cleanly applying those changes to the new repository I've decided to assemble a quick fix for a bug I can't believe I never noticed. TurnsTile 0.3.2 allows you to use the interlaced parameter along with a tilesheet, something that's apparently been broken since September of 2011. I was almost convinced I was doing this right.

I've also set myself up on GitHub, so if you want to follow along as I inch my way toward 0.4.0 (or 1.0.0, or whatever I end up calling it) feel free: https://github.com/ItEndsWithTens/TurnsTile