Filters to cartoonize and decolor and "undetail"

[TCmullet]

Is there an Avisynth filter, free or cheap, that can take a live action video and turn it into an animated cartoon? That is, turn it into a line drawing of all the people and objects and fill in simple colors for everything? Also, converting full or simple colors to greyscales would be desirable.

Second, might there be a filter, dependent or independent of the above filter, that could create a black and white line drawing only of the video? Perhaps you're thinking I mean simply converting to greyscale. No, I include the idea of somehow allowing the level of details to be adjustable so that at the highest end of retention, you'd see full facial expressions (for example), but at the lowest end, you'd see nothing but outlines of all people and objects.

Are these too tall an order for what Avisynth can do?

[butterw2]

No idea what's natively available for Avisynth.

Some realtime gpu video shaders are available for mpc-hc.
https://forum.doom9.org/showthread.php?t=181584&page=3
- reducing color palette (posterization)
- converting to greyscale
- edge detection
They could theoretically be used in avisynth via AvisynthShader.

VLC dx11 also has a cartoon filter that you can encode with.

[johnmeyer]

I haven't used it for years, but there is this for VirtualDub:

http://compression.ru/video/cartoonizer/index_en.html

[butterw2]

Based on the featured sample images, the virtualdub cartoonizer seems quite hit and miss. It should be possible to do better.
If it is possible to do a cartoon transform in photoshop, it should be possible to get a similar result in avisynth with a video scene.
Your results will however depend on the input scene and parameter values. A complex algorithm is likely required to get a temporaly stable, near pixel perfect result.

The realtime vlc cartoon filter isn't bad (Tools > filters > video effects > colors > Gradient +Edge + Cartoon). The source code is in gradient.c, so I guess it could be ported to avisynth.

More or less detail is built-in to all edge detection methods (ex: Sobel). Output is thresholded.

[butterw2]

The VLC buit-in filter uses a simple Y' (Luma) method (edge + colorize). The effect runs easily (up to 1080p input) in realtime on a single cpu core, the results are best on well lit hd video, but as can be seen on the 720p input screenshot the results are not perfect. Better results are possible with clean high bitrate 4K input.

https://raw.githubusercontent.com/butterw/bShaders/master/assets/vlc_gradient-edge-cartoon_720p.png

The simple colorize method compresses smoothed Luma in 4 fixed bands. The edges (in black) are produced by a sobel filter thinned with angle processing.
Grayscale input is also handled. No tuning parameters are made available.

https://code.videolan.org/videolan/vlc/-/blob/ded62a069f457ae919fae30b60d920a8ac3c92b4/modules/video_filter/gradient.c

I may port this to a mpv gpu pixel shader. I also have other realtime shaders for toon/comic-book stylised effects. Does anyone have suggestions on how to achieve higher quality results ?

- Luma methods are sensitive to lighting (changes). Avoid fluctuating light sources.
- Shadows and compression artifacts in dark areas also tend to show up (and fluctuate).
- A (removable) frame border artifact is present.
- Results could probably be improved by adding dithering to limit visible banding.

[StainlessS]

Judging by above sample image, maybe could benefit from a little bit of blurring following the cartoonizing to soften those edges,
[probably spatial and temporal blur, or maybe HqDn3D], but then I'm not really used to looking at cartoons.

[butterw2]

Thanks, temporal methods aren't typically possible in video player shaders, but I'll give it a look in Avisynth.
Just resizing down the screenshot to 75% gives quite a nice result. Of course screenshots only tell part of the story...

Ideally, I would also like to be able to upscale from 720p to 1080p. Edge detection will perform better on the upscaled image, but it might also require some final smoothing.
---

Here's another lightweight shader bPosterize(levels=4) applied to the smoothed image, which performs a slightly improved color Posterization in rgb (helps avoid greens in skintones).

https://raw.githubusercontent.com/butterw/bShaders/master/assets/bPosterize2-4_rgb_x1280.jpg

PS: Is there a way to scale image display size on the forum ?

[johnmeyer]

There is a completely different approach to this, one which gives you a LOT more flexibility than any of the "cartoonizer" plugins. I've used it myself, so I know it works.

What you do is export your video as an image sequence. Then, open one of those images in Photoshop (or your favorite image editor -- I use an ancient program called Photoimpact). Use the image editor's plugins to do the cartooning. Mine has such a plugin. You can use more than one plugin if you don't get quite the look you are going for using just one plugin.

Then, use the batch feature in your photo editor to apply that same sequence of changes to all the other photos that you exported. Almost any decent photo editor has a batch facility that will let you do this.

Finally, reassemble the images back into a video. You can do this with most NLEs. Virtualdub can also do it.

[butterw2]

If such a Photoshop plugin that can do cartooning as desired (different art styles are possible), it means it could also be implemented in Avisynth. It seems there just hasn't been any great interest to do so, so far.
Considerations on source and output resolution, lighting, output palette, parameter adjustment and temporal stability would still be relevant though.

Simple comic book style shaders are quite popular for realtime video. I'm not sure how they compare to the best pro-plugins probably available as Premiere or similar add-ons.

[StainlessS]

Anybody know how Polar Express was done ? [thnik that is the name of the Tom Hanks thing].
(Maybe I get it when we come out of Lockdown).
Maybe was hand painted [over live action], with some kind of auto copy for similar frames.

[poisondeathray]

Kuwahara for avisynth (x86 only , but can run in mp_pipeline x64)
https://forum.doom9.org/showthread.php?t=170788
http://www.wilbertdijkhof.com/Kuwahara_v11.zip

[butterw2]

bStipple comic book shader (mpv .hook and .hlsl):

https://raw.githubusercontent.com/butterw/bShaders/master/assets/mpv_bStipple_x1280.jpg

This effect uses an increased contrast black and white image, adds blue noise stippling (which is useful to cover up artifacts) and then blends the result with the color image.

Works well with low-res input <=720p.
The code is very simple, except for the part relating to noise generation which wouldn't be needed with an external blue noise .png (the shader can be used to generate this texture, but higher quality implementations might be available).

[StainlessS]

butterw2, I like that effect, nifty :)

[butterw2]

bHatch (.hlsl, .hook) shader maps an increased contrast input image (6 levels) to 4 color palette with 2 hatching levels.
https://raw.githubusercontent.com/butterw/bShaders/master/assets/bHatch_6levels_4col_pop_x1280.png

EDIT: was (4 levels, 2 solid colors + hatching) https://raw.githubusercontent.com/butterw/bShaders/master/assets/4color_hatch_resize_x1280.jpg .
Shader was used post-resize (1080p) and the result was resized down to 720p providing some smoothing in edges and palette.
It's probably possible to figure out a better combination of colors and hatching (reverse hatch \\ can be used). It would also be possible to combine selected colors, dots/hatching, edges from the different methods presented here.
Because this shader inherently causes some loss of detail, it works best on close-ups (ex: Tiktok videos).

For reference, the source frame (720p) (https://github.com/butterw/bShaders/blob/master/assets/original_exterior_49.05_x1280.png)

[butterw2]

Pencil-sketch with Edge_Sharpen shader (edge threshold: 0.15, bg: grey200)
https://raw.githubusercontent.com/butterw/bShaders/master/assets/pencil-sketch_EdgeSharpen_x1280_grey.png

Different edge detection algorithm exists (with an ajustable edge threshold), some can be sensitive to source artifacts.

Frei-chen Edge detection (more detailed):
https://raw.githubusercontent.com/butterw/bShaders/master/assets/mpv_freichen_x1280_grey.png

[TCmullet]

Gosh, thank you all for the contributions! I'm glad you all got more out of this than I did! It turns out that the project I hoped to tackle may not be possible, for reasons totally "political" of sorts. But if I did become able to do it, let me ask you further. Butterw2, would it be possible to virtually eliminate all or most details that are not part of an "edge"? Basically, the outlines of people and objects are what is wanted, and perhaps other edges like clothing necklines, etc. I want to see that this is a human and probably the gender (if obvious), but not their identity nor details of what things look like OTHER than their shapes via the edges. Almost like a silhouette of sort. (But not a black-filled silhouette.) Need to see what people are doing without any details of their own showing on them or the objects they are handling. This might be like a movie's storyboarding, I think. Maybe even more obfuscated than that. Basically, animated outline drawings.

[butterw2]

It's possible to threshold higher but then parts will go missing (the dark lines will remain). The outlines are clean with anime, but with real life video content, it's more messy.

EDIT: to keep only the thick dark lines, something like this would be required:
edge_detect(threshold: keep only the dark lines)
erode, median filter, dilate: eliminate thin lines

[johnmeyer]

The process I outlined above can certainly do what you ask. Here is an example of one frame:

Original image
https://i.imgur.com/MxS8q29.jpg

Image with just pen outline
https://i.imgur.com/7yGYMil.jpg

I forgot to remove the last little bit of background, but with the image editing program it is trivial to do that.

Getting only edges from an image can be tricky. I chose this image because the background is fuzzed out and shows what happens when the edges become less distinct.

This can be automated. It took me less than a minute to do this, and you can process the images pretty quickly, depending on the resolution.

[Reel.Deel]

Cool thread, some weeks ago I was looking around for some hlsl filters that I could use in AviSynth, the selection is limited :(. BTW, thanks butterw2, I came across your filters :).

Here are two examples of things I found.

Using AviSynthShader with the following hlsl filters:

Cartoon Shader by Andrew Price (http://www.andrewalexanderprice.com/cartoonshader.php):
https://i.ibb.co/M14tDzh/cartoonshader.png

Gameboy shader by kchapelier (https://github.com/kchapelier/mpc-hd-shaders):

https://i.ibb.co/9rSvsHs/gameboy-effect-shader.png

[butterw2]

Dark Sobel_Edges + Saturated Color (cartoon enhanced reality):
https://raw.githubusercontent.com/butterw/bShaders/master/assets/Sobel_Sat_Color_x1280.jpg

bSobel_Edge.hlsl
- T_Sobel 0.15
- SatFactor 1.8

[butterw2]

@Reel.Deel

I would like to test AvisynthShader, could you please post your script ?

[Reel.Deel]

@Reel.Deel

I would like to test AvisynthShader, could you please post your script ?

Here you go:

input = FFImageSource("original_exterior_49.05_x1280.png")

c = ConvertToShader(input, Precision=1)
Shader(c, "gameboy.hlsl", output=1)
ExecuteShader(last, c, Precision=3, OutputPrecision=1)
ConvertFromShader(last, Precision=1, Format="RGB32")

That is the script I used for the examples I uploaded on the last page.

[butterw2]

# Contour
Blurring prior to applying the Sobel filter cleans and thickens the edges. I used the following for a neon glow shader:

in pre-resize RGB:
- Gaussian Blur (sigma=1.5)
- Luma Sobel Edge (Threshold 0.3)
- Gaussian Blur (3x3) + Neon Output(Cyan)


# Colors
A simple 4-level Luminosity map can give a decent result for greyscale cartoon colors.

Luma maps often have issues with video encoding artifacts (banding/dancing noise present in shadows gets highlighted). Using uncompressed input is likely beneficial here. Tiktok filters also have realtime preview, meaning the lighting conditions can be adjusted as required prior to capture/encoding.

[butterw2]

Edge: Output of Sobel in Luma without Thresholding
in Y(UV), inverted with Chroma, darker lines means stronger detection.
Y pre-processing: Gaussian Smoothing Filter (3x3)
https://raw.githubusercontent.com/butterw/bShaders/master/assets/mpv_invSobel_Luma_noThreshold.jpg

[johnmeyer]

Edge: Output of Sobel in Luma without Thresholding
in Y(UV), inverted with Chroma, darker lines means stronger detection.
Y pre-processing: Gaussian Smoothing Filter (3x3)
That is an amazing effect. I assume you used reel.deel's script, but since there are so many blurring filters, I'd love to see the exact script you used, if you don't mind sharing. Thanks!

[butterw2]

It's just the (edge) output of a Sobel filter in Luma. What's unusual is that it's not thresholded and that it's inverted. Keeping the chroma, makes the output look more natural.

Input smoothing filter is optional, I don't typically use it with video input.
Y = 1-Sobel_edge(Y), keep UV.
If there is a Sobel_edge function available, this can run natively in avisynth.

This realtime shader runs in mpv video player. I will be adding a couple of tweaks beneficial for my use cases in the next update.

[johnmeyer]

Thanks!

[butterw2]

Edge detection kernels (3x3): Prewitt (used by Edge_sharpen.hlsl), Frei-chen, Sobel, Scharr

Sobel calculates 2 convolution kernels, Freichen-chen uses 9, including 4 edge detection kernels and a mean. But once you factor in kernel normalization, Sobel and Scharr give similar results, and this is true overall for all these methods, which is not to say that implementation specific tweaks can't provide some improvements, especially for effects.
EDIT2: I've now also implemented a 5x5 Sobel shader in mpv.


Input spatial smoothing (ex: gaussian blur(sigma=0.5)), scaling method (if applicable) have some impact on detection.
An erosion shader can optionally be applied to the output edges to thin the line width.
With a clean HD video input, and simple shader processing, the results can be quite uncanny. Avisynth may have further options available (ex: temporal smoothing).

The main issues encountered with video are as follows:
- Detection sensitive to lighting (and lighting change).
- Black Bars/Logos/text watermarks need to be removed, as their edges get detected.
- Some elements with intricate motifs/textures (ex: clothing) get overdetected, this can lead to glimmering with motion.
- Strong detections on eyes (iris contour, pupil, reflections) can be disturbing.
- Shadows get detected.
- While at the same time, some sections of the main outline can be weak or missing.

EDIT:
- with Avisynth Shader (dx9 hlsl) there is significant overhead vs. previewing an avs script + applying shaders in a video player such as mpc-hc (hlsl) or mpv (glsl). You need to use Avisynth Shader if you want to post-process in avisynth or to encode video.
- In a video player realtime 1080p60 should be achievable. Output images can be obtained directly through screenshots: https://forum.doom9.org/showthread.php?p=1931990#post1931990
- mpc-hc dx9 hlsl shaders run in .rgba, and Avisynth Shader requires .bgra for its I/O, so some shader require code adaptation to get correct color processing:
https://forum.doom9.org/showthread.php?p=1932280#post1932280
- It should be possible to write hlsl code that uses yuv colorspace with Avisynth Shader (untested). Mpv .hook glsl shaders can operate in rgb or source yuv colorspace.

[butterw2]

For native avisynth edge detection solutions I think the following is what you would use:

http://avisynth.nl/index.php/Scharr using MaskTools2/Mt edge (3x3 kernel):
function scharr(clip c) {
scharr_x = c.mt_edge("3 0 -3 10 0 -10 3 0 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr_y = c.mt_edge("3 10 3 0 0 0 -3 -10 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr = mt_lutxy(scharr_x, scharr_y, yexpr=mt_polish("((x*x)+(y*y))^0.5"), u=1, v=1)
return scharr
}

Avisynth+ also has a built-in general convolution function (3x3 or 5x5).

EDIT: Asd-g/TCannyMod can directly be used (3x3 kernel, input: 8bit planar) with good results
LSmashVideoSource(fname, prefer_hw=3).TCannyMod(mode=1, sigma=0, sobel=true, chroma=2).Invert("Y")

EDIT2: Avisynth+ script with different output modes: https://github.com/butterw/bShaders/blob/master/edge/bCartoonize.avs

https://raw.githubusercontent.com/butterw/bShaders/master/assets/mpv_LSobel.inv.dk.jpg
I've tried to bring back the original dark colors with a simple threshold on Luma. The dark/light threshold is clearly visible on the ground here and would need to be smoothed out.
In order to make the edges visible, the blacks are washed out, which can be an issue when a lot of the shot is in shadow.
The issue is combining colors (dark/light Luma) and edge_detection(weak/strong) while managing to avoid threshold artifacts.