Filters to cartoonize and decolor and "undetail"

[TCmullet]

Is there an Avisynth filter, free or cheap, that can take a live action video and turn it into an animated cartoon? That is, turn it into a line drawing of all the people and objects and fill in simple colors for everything? Also, converting full or simple colors to greyscales would be desirable.

Second, might there be a filter, dependent or independent of the above filter, that could create a black and white line drawing only of the video? Perhaps you're thinking I mean simply converting to greyscale. No, I include the idea of somehow allowing the level of details to be adjustable so that at the highest end of retention, you'd see full facial expressions (for example), but at the lowest end, you'd see nothing but outlines of all people and objects.

Are these too tall an order for what Avisynth can do?

[butterw2]

No idea what's natively available for Avisynth.

Some realtime gpu video shaders are available for mpc-hc.
https://forum.doom9.org/showthread.php?t=181584&page=3
- reducing color palette (posterization)
- converting to greyscale
- edge detection
They could theoretically be used in avisynth via AvisynthShader.

VLC dx11 also has a cartoon filter that you can encode with.

[johnmeyer]

I haven't used it for years, but there is this for VirtualDub:

http://compression.ru/video/cartoonizer/index_en.html

[butterw2]

Based on the featured sample images, the virtualdub cartoonizer seems quite hit and miss. It should be possible to do better.
If it is possible to do a cartoon transform in photoshop, it should be possible to get a similar result in avisynth with a video scene.
Your results will however depend on the input scene and parameter values. A complex algorithm is likely required to get a temporaly stable, near pixel perfect result.

The realtime vlc cartoon filter isn't bad (Tools > filters > video effects > colors > Gradient +Edge + Cartoon). The source code is in gradient.c, so I guess it could be ported to avisynth.

More or less detail is built-in to all edge detection methods (ex: Sobel). Output is thresholded.

[butterw2]

The VLC buit-in filter uses a simple Y' (Luma) method (edge + colorize). The effect runs easily (up to 1080p input) in realtime on a single cpu core, the results are best on well lit hd video, but as can be seen on the 720p input screenshot the results are not perfect. Better results are possible with clean high bitrate 4K input.



The simple colorize method compresses smoothed Luma in 4 fixed bands. The edges (in black) are produced by a sobel filter thinned with angle processing.
Grayscale input is also handled. No tuning parameters are made available.

https://code.videolan.org/videolan/v...ter/gradient.c

I may port this to a mpv gpu pixel shader. I also have other realtime shaders for toon/comic-book stylised effects. Does anyone have suggestions on how to achieve higher quality results ?

- Luma methods are sensitive to lighting (changes). Avoid fluctuating light sources.
- Shadows and compression artifacts in dark areas also tend to show up (and fluctuate).
- A (removable) frame border artifact is present.
- Results could probably be improved by adding dithering to limit visible banding.

[StainlessS]

Judging by above sample image, maybe could benefit from a little bit of blurring following the cartoonizing to soften those edges,
[probably spatial and temporal blur, or maybe HqDn3D], but then I'm not really used to looking at cartoons.

[butterw2]

Thanks, temporal methods aren't typically possible in video player shaders, but I'll give it a look in Avisynth.
Just resizing down the screenshot to 75% gives quite a nice result. Of course screenshots only tell part of the story...

Ideally, I would also like to be able to upscale from 720p to 1080p. Edge detection will perform better on the upscaled image, but it might also require some final smoothing.
---

Here's another lightweight shader bPosterize(levels=4) applied to the smoothed image, which performs a slightly improved color Posterization in rgb (helps avoid greens in skintones).



PS: Is there a way to scale image display size on the forum ?

[johnmeyer]

There is a completely different approach to this, one which gives you a LOT more flexibility than any of the "cartoonizer" plugins. I've used it myself, so I know it works.

What you do is export your video as an image sequence. Then, open one of those images in Photoshop (or your favorite image editor -- I use an ancient program called Photoimpact). Use the image editor's plugins to do the cartooning. Mine has such a plugin. You can use more than one plugin if you don't get quite the look you are going for using just one plugin.

Then, use the batch feature in your photo editor to apply that same sequence of changes to all the other photos that you exported. Almost any decent photo editor has a batch facility that will let you do this.

Finally, reassemble the images back into a video. You can do this with most NLEs. Virtualdub can also do it.

[butterw2]

If such a Photoshop plugin that can do cartooning as desired (different art styles are possible), it means it could also be implemented in Avisynth. It seems there just hasn't been any great interest to do so, so far.
Considerations on source and output resolution, lighting, output palette, parameter adjustment and temporal stability would still be relevant though.

Simple comic book style shaders are quite popular for realtime video. I'm not sure how they compare to the best pro-plugins probably available as Premiere or similar add-ons.

[StainlessS]

Anybody know how Polar Express was done ? [thnik that is the name of the Tom Hanks thing].
(Maybe I get it when we come out of Lockdown).
Maybe was hand painted [over live action], with some kind of auto copy for similar frames.

[poisondeathray]

Kuwahara for avisynth (x86 only , but can run in mp_pipeline x64)
https://forum.doom9.org/showthread.php?t=170788
http://www.wilbertdijkhof.com/Kuwahara_v11.zip

[butterw2]

bStipple comic book shader (mpv .hook and .hlsl):



This effect uses an increased contrast black and white image, adds blue noise stippling (which is useful to cover up artifacts) and then blends the result with the color image.

Works well with low-res input <=720p.
The code is very simple, except for the part relating to noise generation which wouldn't be needed with an external blue noise .png (the shader can be used to generate this texture, but higher quality implementations might be available).

[StainlessS]

butterw2, I like that effect, nifty

[butterw2]

bHatch (.hlsl, .hook) shader maps an increased contrast input image (6 levels) to 4 color palette with 2 hatching levels.


EDIT: was (4 levels, 2 solid colors + hatching) https://raw.githubusercontent.com/bu...size_x1280.jpg .
Shader was used post-resize (1080p) and the result was resized down to 720p providing some smoothing in edges and palette.
It's probably possible to figure out a better combination of colors and hatching (reverse hatch \\ can be used). It would also be possible to combine selected colors, dots/hatching, edges from the different methods presented here.
Because this shader inherently causes some loss of detail, it works best on close-ups (ex: Tiktok videos).

For reference, the source frame (720p)

[butterw2]

Pencil-sketch with Edge_Sharpen shader (edge threshold: 0.15, bg: grey200)


Different edge detection algorithm exists (with an ajustable edge threshold), some can be sensitive to source artifacts.

Frei-chen Edge detection (more detailed):
https://raw.githubusercontent.com/bu...x1280_grey.png

[TCmullet]

Gosh, thank you all for the contributions! I'm glad you all got more out of this than I did! It turns out that the project I hoped to tackle may not be possible, for reasons totally "political" of sorts. But if I did become able to do it, let me ask you further. Butterw2, would it be possible to virtually eliminate all or most details that are not part of an "edge"? Basically, the outlines of people and objects are what is wanted, and perhaps other edges like clothing necklines, etc. I want to see that this is a human and probably the gender (if obvious), but not their identity nor details of what things look like OTHER than their shapes via the edges. Almost like a silhouette of sort. (But not a black-filled silhouette.) Need to see what people are doing without any details of their own showing on them or the objects they are handling. This might be like a movie's storyboarding, I think. Maybe even more obfuscated than that. Basically, animated outline drawings.

[butterw2]

It's possible to threshold higher but then parts will go missing (the dark lines will remain). The outlines are clean with anime, but with real life video content, it's more messy.

EDIT: to keep only the thick dark lines, something like this would be required:
edge_detect(threshold: keep only the dark lines)
erode, median filter, dilate: eliminate thin lines

[johnmeyer]

The process I outlined above can certainly do what you ask. Here is an example of one frame:

Original image


Image with just pen outline


I forgot to remove the last little bit of background, but with the image editing program it is trivial to do that.

Getting only edges from an image can be tricky. I chose this image because the background is fuzzed out and shows what happens when the edges become less distinct.

This can be automated. It took me less than a minute to do this, and you can process the images pretty quickly, depending on the resolution.

[Reel.Deel]

Cool thread, some weeks ago I was looking around for some hlsl filters that I could use in AviSynth, the selection is limited . BTW, thanks butterw2, I came across your filters .

Here are two examples of things I found.

Using AviSynthShader with the following hlsl filters:

Cartoon Shader by Andrew Price:


Gameboy shader by kchapelier:

[butterw2]

Dark Sobel_Edges + Saturated Color (cartoon enhanced reality):


bSobel_Edge.hlsl
- T_Sobel 0.15
- SatFactor 1.8