Palette manipulation

[ajk]

Hi,

I've been playing with a little thermal imaging camera. It can be used as a simple USB video device, in which case the output is a grayscale image. In many cases, it is useful to display the result in some manner of false colour to highlight the temperature scale. There are proprietary apps for all this of course, but I'd like to try and work with the raw data.

So, this is the image as captured from the camera (Raspberry Pi on my desk here):

https://ajk.kapsi.fi/projects/avisynth/ir-original.png

In GIMP, it is straightforward enough to apply a colour gradient which covers all brightness values from 0-255:

https://ajk.kapsi.fi/projects/avisynth/ir-gimp.png

I looked into how to achieve this in Avisynth. The most promising plugin appeared to be TurnStile, which includes a palette-changing filter called CLUTer. So, I exported the gradient as a separate image (https://ajk.kapsi.fi/projects/avisynth/ir-palette.png) and fed that to the filter and got this:

https://ajk.kapsi.fi/projects/avisynth/ir-cluter.png

Clearly there is some other processing than just a direct copying of the colour values. I don't entirely understand what the plugin does here, or if it can work for my purpose.

I also looked into generic image processing filters such as mt_lut and rgba_rpn. These are certainly able to pick out a particular shade and process it, but I haven't been able to devise any script which isn't essentially 256 individual colour replacements :)

For my purposes it would probably be enough to divide the luma range into four segments and apply a different gradient to each. So something like mapping values 0-63 to #000000-#91009c (linearly interpolated), with other values for the other three segments.

Any suggestions on getting one of these approaches to work, or new ideas? I suppose there is always the option to write a plugin from scratch, which I might eventually do :)

[Didée]

Processing color is much more difficult than just luma, at least for me.

For the case you showed, I tried to replicate it (not in Gimp, but Picture Window):

https://thumbs4.imagebam.com/58/43/a1/MEZNYJL_t.png (https://www.imagebam.com/view/MEZNYJL)

Not exactly the same, but seems reasonably close.

For this particular mapping, it's perfectly possible (though not perfectly easy) to set up some LUT conglomerats for masktools that would do exactly this. But rather without any control knobs to do like "more-this-and-less-that" ...

If anyone has better ideas, speak out loud.

/edit: alternatively, just some color presets could be masked-merged together by luminosity ranges - not as elegant, but more easy to do.

[poisondeathray]

You can create a cube LUT and apply in avs+ with DGCube or AVSCube

Gimp doesn't have a way to export cube LUT directly based on effects (maybe it does now, I don't use gimp that often) ; but there was a way you can use a G'MIC plugin with haldclut , and then convert haldclut to a cube lut using a python script. (Or other video/image editing programs can often export cube LUT's directly)

Of course that is not interactive in avs+ either; the interractivity with dials/knobs will be in the other program before exporting the cube LUT. ie. There won't be "live" feedback in avs+, as the transforms will be "baked" into the cube

https://github.com/mikeboers/LUT-Convert


python "hald_to_cube.py" input.png output.cube


Here is the zipped cube if you wanted to try it . You can go with a larger LUT to more closely approximate colors but that will slow down processing speed
https://www.mediafire.com/file/ljtec34zl3wdvz2/haldclut_64_to_cube.zip/file


ImageSource("ir-original.png")
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBP16")
DGCube(last, "path\haldclut_64_to_cube.cube", lut="full", in="full", out="full")


https://i.postimg.cc/kgBJm6s9/dgcube-lut64.png (https://postimages.org/)

[ajk]

Thanks Didee and poisondeathray!

So there are indeed options to explore here. I will look into these tools/plugins and see what is most convenient to work with. Having real-time adjustments isn't that important to me, but of course a simple workflow is always preferable over one that includes many steps.

One would think this is a simple operation (at least computationally speaking), but I suppose the concept of a palette applies more to image processing rather than video, which is why something like GIMP has more tools in this category.

[Didée]

Looking at the Gimp color gradient with R-G-B-isolated histogram() showed some interesting properties:
- green seems close to a linear projection
- red looks like an inverse-gamma projection (inverse-gamma == gamma from white>>black instead of black>>white)
- blue is a bit more crazy, the brightmost part ("heat" region yellow>>white) is almost linear, for the lower dark region seems it can be approximated by two waves of inverse-gamma

So, modelling the channels with just Avisynth-internal levels() (plus Overlay() and invert()):

https://thumbs4.imagebam.com/28/e5/77/MEZOGS3_t.png (https://www.imagebam.com/view/MEZOGS3)(Gimp) <> https://thumbs4.imagebam.com/9e/5c/13/MEZOGS4_t.png (https://www.imagebam.com/view/MEZOGS4)(Avisynth levels() )

Doing it this was, there are some small discontinuities in the almost-black parts of each channel, however it doesn't really matter.

Stone Age Avisynth with wood club and stone axe:

imagesource("G:\Downloads\ir-original.png")
#GIMP = Imagesource("G:\Downloads\ir-gimp.png")

#XYZsource("whatevever.ext") # maybe your YUV source
#converttoRGB(matrix="Rec709")

src=last

red = src.invert().levels(0,0.275,255+0,0,255+64,false).invert()

green1 = src.levels(88,1.0,227,0,235,false)
green2 = src.levels(227,1.0,255,235,255,false)
greenX = src.levels(227,1.0,228,235,0,false)
green = green2.overlay(greenX,mode="subtract",pc_range=true)
\ .overlay(green1,mode="lighten", pc_range=true)

b1 = src.invert().levels(199,0.35,255,101,255-20,false).invert()
b2 = src.levels(64+8,0.35,128,155,0,false)
b3 = src.levels(216,1.0,255,0,250,false)
b12 = b1.overlay(b2,mode="darken",pc_range=true)
b123= b3.overlay(b12,mode="lighten",pc_range=true)

MergeRGB(red,green,b123)

#stackvertical(GIMP.subtitle("Gimp gradient").converttorgb32(),last.subtitle("Levels elaboration").converttorgb32())

return(last)


result:

https://thumbs4.imagebam.com/7c/24/b3/MEZOHQM_t.png (https://www.imagebam.com/view/MEZOHQM)

Orange/yellow-heat comes out a bit more saturated, but maybe that's even nice. ;)

[ajk]

That is a really cool approach! Using Levels() and other internal functions is certainly a step in the direction I would like to see, external tools are always a bit of a hassle. Of course, as you demonstrate, this still requires a detailed analysis of the gradient properties and careful thought for how to best implement it using the available functions.

I'm looking into what it takes to make an actual filter out of this. Will post some results later.

[ajk]

Okay, so I spent a bit of time in Visual Studio and came up with a filter. Copying the colour data over from one image to another only takes a couple dozen lines of code.

Now I can take this source image:

https://ajk.kapsi.fi/projects/avisynth/applypalette-plugin/images/sample.png

And this image which describes the palette:

https://ajk.kapsi.fi/projects/avisynth/applypalette-plugin/images/palette1.png

Apply the palette to the image using:

ImageSource("sample.png")

ApplyPalette(ImageSource("palette.png"))

And get the following result:

https://ajk.kapsi.fi/projects/avisynth/applypalette-plugin/images/result1.png

Or, alternatively use a different palette to get a different result:

https://ajk.kapsi.fi/projects/avisynth/applypalette-plugin/images/palette5.png

https://ajk.kapsi.fi/projects/avisynth/applypalette-plugin/images/result2.png

I have the first iteration of the plugin available here: ApplyPalette-0.1.zip (https://ajk.kapsi.fi/projects/avisynth/applypalette-plugin/ApplyPalette-0.1.zip)

Only works on RGB24 or RGB32 sources for now. Might add support for at least planar RGB later. Another idea would be to supply a range of colours instead of a sample image, so that the palette can be entirely generated within a script.