Analyze complexity of a clip based on colors in it and movement and get a value out ?

[Yanak]

Hello,

I'd like to analyze short video clips and get a value out of the analyze based on the number of colors the clip use and the movement/ scene changes it have from one frame to another but don't know where to start with this.

The less colors are used in the clip and the less changes from one frame to another there will be the smaller the output score will be,
the higher the number of colors used in the clip and the more changes from one frame to another there will be the higher the score will be.

Output value can be any range, just needs to be a single value reflecting the complexity of the clip based on the number of colors it have and the scenes changes / movement from one frame to another.

At his point I'm a bit lost and not really know where to start looking for achieving this, maybe motion plugins can achieve something like this ( never used them much beside a couple of quick testings long time ago) or maybe there is some already existing plugins or functions able to do the job, overall i'd like to have a way to measure the complexity of an input clip in terms of colors changes from one frame to another.

so if anyone have any tips or some guess to point me into a direction please.

Thanks a lot in advance

[johnmeyer]

Download StainlessS "RTStats" package and go through the readme file. He provides a huge number of tools that analyze video clips.

You should also look at the section of the AVISynth documentation where you find "AverageLuma" and a whole bunch of other functions that provide information on a clip's luma and chroma values.

Using these tools you should be able to create some sort of metrics that will give you different values for colorful clips compared to muted ones.

[Yanak]

I was looking into StainlessS stuff but was not sure it could be the best method to do this.

Will study this and also AverageLuma function and see what i can get out.

Thank you very much.

[StainlessS]

ApparentFps can give you instantaneous apparent framerate as a measure of 'activeness' around current frame.
Perhaps something like, sum for every frame and divide by the number of frames.
That plugin is (I think) only based on luma.
Mobile:

[johnmeyer]

Just to be clear, I was NOT recommended Averageluma for anything. Instead, I was suggesting that you look at the other functions located in the same part of the AVISynth doc as that function. (I wasn't at my editing computer when I wrote that and could not recall the names of all the other functions.)

I am now at my main editing computer. The built-in AVISynth functions that you'll find interesting include these:

YPlaneMedian(clip [, int offset = 0])
UPlaneMedian(clip [, int offset = 0])
VPlaneMedian(clip [, int offset = 0])
BPlaneMedian(clip [, int offset = 0]) AVS+
GPlaneMedian(clip [, int offset = 0]) AVS+
RPlaneMedian(clip [, int offset = 0]) AVS+

YPlaneMin(clip [, float threshold = 0, int offset = 0])
UPlaneMin(clip [, float threshold = 0, int offset = 0])
VPlaneMin(clip [, float threshold = 0, int offset = 0])
BPlaneMin(clip [, float threshold = 0, int offset = 0]) AVS+
GPlaneMin(clip [, float threshold = 0, int offset = 0]) AVS+
RPlaneMin(clip [, float threshold = 0, int offset = 0]) AVS+

YPlaneMax(clip [, float threshold = 0, int offset = 0])
UPlaneMax(clip [, float threshold = 0, int offset = 0])
VPlaneMax(clip [, float threshold = 0, int offset = 0])
BPlaneMax(clip [, float threshold = 0, int offset = 0]) AVS+
GPlaneMax(clip [, float threshold = 0, int offset = 0]) AVS+
RPlaneMax(clip [, float threshold = 0, int offset = 0]) AVS+

YPlaneMinMaxDifference(clip [, float threshold, int offset = 0])
UPlaneMinMaxDifference(clip [, float threshold, int offset = 0])
VPlaneMinMaxDifference(clip [, float threshold, int offset = 0])
BPlaneMinMaxDifference(clip [, float threshold, int offset = 0]) AVS+
GPlaneMinMaxDifference(clip [, float threshold, int offset = 0]) AVS+
RPlaneMinMaxDifference(clip [, float threshold, int offset = 0]) AVS+From the Wiki:

This group of functions return statistics about the distribution of pixel values on a plane (Luma, U-chroma and V-chroma, respectively). The statistics are, in order of presentation: maximum, minimum, median and range (maximum - minimum difference).
threshold is a percentage, stating how many percent of the pixels are allowed above or below minimum. The threshold is optional and defaults to 0. In v2.61 an offset argument is added, which enables you to access the statistics of other frames than the current one. I think this matches almost exactly what you said you want to do.

[poisondeathray]

How about numbers of unique colors ? Does avisynth or StainlessS' functions have something like gimps color cube analysis, or imagemagicks' identify ?

e.g

identify.exe -format "%k" img.png


eg. gimp color cube analysis #unique colors
https://docs.gimp.org/en/plug-in-ccanalyze.html




What is the background information ? Why are you wanting this info ?

For the other "characteristics" , besides motion vectors, how about some sort of "noise" estimation measure ? From encoding perspective, random noise is very complex

Just brainstorming some ideas

[Yanak]

@ StainlessS:
i'll look into it but don't think this is what i need, i need to take in account the differences from one frame to the next one and so on.

There could be 2 similar frames for example one at the start and one at the end but those should get flagged at max value, not like if they were following each other.

@johnmeyer:
yes i was looking into this , things like "YDifferenceFromPrevious(clip)" , then i had to go out but it's something i need to read and study in the next days.

@poisondeathray :
I made a powershell script that will work into Staxrip and use Gifski (https://github.com/ImageOptim/gifski) to create HQ gifs

The powershell script gets infos from the input video and filters selected in Staxrip, like FPS, resolution, trimed parts in the preview video window if there is trimming etc, then send the avs script of Staxrip into ffmpeg to extract the frames needed for the gif creation, then Gfski proceed the frames and makes a very nice HQ gif.

There is a way to estimate the output size of the gif, but this is varying a lot depending the number of colors in the footage, the movement etc, like seen here :
https://github.com/ImageOptim/gifski/issues/28#issuecomment-370617837

So i am trying to get a kind of measure value for the complexity of the video, something i could rely on and vary depending the complexity in colors and changes from one frame to another inside a clip.

Then once i'll have this complexity index I will run a loads of tests and log stuff to in the end try to get a multiplicator value for the size estimation based on the "complexity value" of the input footage, to hopefully get a bit more accurate results for the estimated output sizes in Gifski, all this based on the "complexity index value " of the video that i am trying to define here.

Not sure if it's clear or not.

[poisondeathray]

Not sure if it's clear or not.

Yep it's clear. It's clear that we need a GIF replacement badly. Now. Today. Stat. :D . Look at the sizes of the gifski examples. whoa!



But a "complexity estimation" might be useful for other purposes too...

[raffriff42]

There's also MotionMask (http://avisynth.nl/index.php/MaskTools/MotionMask) which (if you take its total luminance) would give you an estimate of the motion. But for your specific purpose, the GIF encoder is the best estimator - I would simply make test encodes (short scenes, fast settings). Pretty soon you will develop an eye for things that won't encode well.

[johnmeyer]

The various "YDifference..." functions are useful for finding big changes between frames, e.g., scene changes. They will easily give you the motion metric you are looking for. For instance, the higher YDifferenceFromNext, the more movement between frames.

However, the YDifference... functions work on changes between frames ("interframe"). For the other things you want to do, you want functions that work within a frame ("intraframe"), like the ones I highlighted in my last post. By using several of those functions together, you should be able to create an approximation of a luma and/or chroma standard deviation function. This seems like what you need for the second part of what you are trying to do.

[Yanak]

Yep it's clear. It's clear that we need a GIF replacement badly. Now. Today. Stat. :D . Look at the sizes of the gifski examples. whoa!

But a "complexity estimation" might be useful for other purposes too...
I know, I know, but Gif is not dead :p ( APNG is nice but not widely supported sadly )

and here the big sizes tested are uncommon for most daily uses to be honest, this said the output this tool gives is nothing like any other gif composer i seen before in terms of quality.

Useful for things like Twitter allowing a 15Mb sized gif ( then transcoded internally into a video and displayed with a gif tag on it... logic 404 ) this still ends up in better quality than a high res video uploaded there that gets trashed into a soup of pixels by them video encoder for example... a QH gif as long as you can keep it under the upload size limits is a nice alternative there, same on some other sites.

And probably that yeah a complexity index might serve for some other uses too.

@raffriff42 :
I'll look into this tomorrow probably, when I'll have enough time to dig into all this calmly.

Thank you guys.

edit
@johnmeyer :
saw your post after typing this,
yes seems like one good option, in my powershell script inside Staxrip i'll just have to analyze the video and get an index out of it.

Will need time to work on this and get a script working but i'll spend my next days free time digging all this.

Thanks again.

[StainlessS]

There could be 2 similar frames for example one at the start and one at the end but those should get flagged at max value, not like if they were following each other.

Not sure if I got that, but ApparentFps works on unique or dupe (by thresh), similar frames at either end are not taken into account, only adjacent frames.

EDIT: was just chuckng it in the mix, you dont seem to know what you want, either.
EDIT: If you divide instantaneous FPS result by FrameRate, you will have a measure in range 0.0->1.0 representing 1 seconds worth of clip activity around current frame.

[Yanak]

I'll give it a try, only looked quickly on the wiki when i answered so maybe i misunderstood, need to test it and see exactly how it works, need to do this for all proposed options and see what i can get out of this a good and reliable way to measure videos and get a kind of complexity index value.

I know what i want ( i think :p, maybe i am not good enough at explaining it) , but don't know ( yet ) how I will achieve it, at least i have loads of things to explore and test in the next days and i think i can come up with something, i hope :p

Thank you.

[raffriff42]

I was actually thinking not of MaskTools (http://avisynth.nl/index.php/MaskTools)' MotionMask, but MVTools (http://avisynth.nl/index.php/MVTools)' MShow and MMask. Here is an old post of mine, showing usage of each.
https://forum.doom9.org/showthread.php?p=1757541#post1757541

[Yanak]

The first script in your link seems promising to estimate movement, will need to experiment and see if i can make it triggered on very small artifacts, it seems to detect the little artifacts on static scenes for the little i quickly tested but i will need it to do this very aggressively, never used MVTools much, I need to read documentation to get the hand on it and start experiments.

If i can get this translated directly into at least a txt file with a kind of total sum of motions values per frames, one frame per line in the txt file, it will be good enough as then later i can simply ask my powershell script to make the sum of all values in the text file and have a global value for the movement in the clip, then need something similar for the colors used and i think i will have something to work with.

I'll also test "YDifference..." tomorrow ( 2+am here ) and some other methods proposed and see if i can get something working.

Loads of testing and experiments awaiting me in the next days, and a lot of documentation to read too.

Thanks a lot for all this guys, i was not expecting so many answers and directions to look at, really great help :)

[raffriff42]

You need to understand what a GIF encoder does. And you need to experiment with different sources, different software, and different encoder options, to get a feel for the resulting size and quality tradeoffs.
https://en.wikipedia.org/wiki/GIF#Palettes
GIF is palette-based: the colors used in an image (a frame) in the file have their RGB values defined in a palette table that can hold up to 256 entries, and the data for the image refer to the colors by their indices (0–255) in the palette table...

Each frame can designate one index as a "transparent background color": any pixel assigned this index takes on the color of the pixel in the same position from the background, which may have been determined by a previous frame of animation...

Many techniques, collectively called dithering, have been developed to approximate a wider range of colors with a small color palette... These techniques sacrifice spatial resolution to approximate deeper color resolution.. the dithering patterns often interfere with the compressibility of the image data, working against GIF's main purpose.

https://en.wikipedia.org/wiki/GIF#Animated_GIF
The size and color quality of animated GIF files can vary significantly depending on the application used to create them. Strategies for minimizing file size include using a common global color table for all frames (rather than a complete local color table for each frame) and minimizing the number of pixels covered in successive frames (so that only the pixels that change from one frame to the next are included in the latter frame). Simply packing a series of independent frame images into a composite animation tends to yield large file sizes.
(another important strategy is reducing the number of colors while still looking acceptable -- this is the main difference I've seen between GIF encoders, not that I've tried them all or anything)

Note "motion" for a GIF encoder is different from "motion" for MPEG-2, h.264 or h.2654 encoders -- these latter ones can re-use data from previous (and subsequent) frames even though the pixels have moved somewhat. GIF can't do that. It only knows that pixels from the previous frame that haven't changed at all (part of a a static background for example) can be re-used. https://en.wikipedia.org/wiki/Motion_compensation
Motion compensation exploits the fact that, often, for many frames of a movie, the only difference between one frame and another is the result of either the camera moving or an object in the frame moving. In reference to a video file, this means much of the information that represents one frame will be the same as the information used in the next frame...

[Yanak]

I know the very basics of gif, thanks for the reminder tho, always good to read, but gifski doesn't use a single palette for all the animation like for example FFmpeg does when creating gifs using the "palettegen" option followed by the "paletteuse" option and one of the 3 or 4 Dither methods it offers which give rather nice gif's for quality already ( and smaller ones usually :p but Gifski quality is on another level ) , for example this tiny tool here is what i was using previously after modifying it a tiny bit : https://github.com/MaverickTse/ffGif , integrated directly into Staxrip via a custom menu sending the .avs and filters applied in Staxrip sent to it directly,

but Gifski creates a new palette for each frame of the animation using pngquant ( one of the reason it's slow to create gifs i guess ) then combine colors between frames and use some dither too i think.

This said i tested a few things already and found out that the "YDifferenceFromPrevious()" in avisynth is the best index i can have to measure differences from one frame to the next one which is perfect for the job in regards to gif,

I output the results on a text file and use my powershell script to sum all the "YDifferenceFromPrevious()" values generated on each frame, from the ps script i run avsmeter and the .avs script having this added to it using Staxrip commands to add a filter via powershell:
WriteFile("H:\text.log", "YDifferenceFromPrevious()")
But i may need to find a better solution than avsmeter to generate the log file, it analyze the script first before benching it, so a video of 200 frames ends up with a log of 400 values once done, not a big deal and easy to divide by 2 in my powershell script :
Get-Content "H:\text.log" | % { $YDifferenceFromPrevious += [math]::abs($_) }
$movement_index = ($YDifferenceFromPrevious / 2)
Or since i have a variable for the total number of frames i can simply ask to sum from line 1 to line x , x being my number of frames variable,but well i'd rather get a way to evaluate all the video input i select more simply than this, without using avsmeter, just generating the log file with the values i need, not sure how i can do yet but still working on this.

Now i need to find something for the colors index and StainlessS RT_Stats doc is something i am still studying for now, saw "RT_DBaseRgbColorCount" but not tested and only vaguely read about it for now as missing time, but will continue with this and see if it can achieve what i need, with those 2 indexes ( need then to be scaled scaled and combined) i will have what i need i think.


My first idea was indeed to run loads of tests on lot of different inputs and collect data to find a pattern depending the different inputs, but i'd better go with some indexes like "YDifferenceFromPrevious()" for "movement" to finalize this and get a better estimation in the end, i'll just have to log the indexes values during my tests runs and then try to define some multiplictor/formula based on this.

Thanks a lot :)

[poisondeathray]

I know the very basics of gif, thanks for the reminder tho, always good to read, but gifski doesn't use a single palette for all the animation like for example FFmpeg does when creating gifs using the "palettegen" option followed by the "paletteuse" option and one of the 3 or 4 Dither methods it offers which give rather nice gif's for quality already ( and smaller ones usually :p but Gifski quality is on another level ) , for example this tiny tool here is what i was using previously after modifying it a tiny bit : https://github.com/MaverickTse/ffGif , integrated directly into Staxrip via a custom menu sending the .avs and filters applied in Staxrip sent to it directly,

but Gifski creates a new palette for each frame of the animation using pngquant ( one of the reason it's slow to create gifs i guess ) then combine colors between frames and use some dither too i think.



A bit OT, sorry ...

It does create high quality gifs (that's a bit of an oxymoron, but hey...) . But the very first test I made had errors not visible with other methods. I think it's the "combine colors between frames thing." 3/56 frames had splotchy errors

[VS_Fan]

... found out that the "YDifferenceFromPrevious()" in avisynth is the best index i can have to measure differences from one frame to the next one ...

My first idea was indeed to run loads of tests on lot of different inputs and collect data to find a pattern depending the different inputs, but i'd better go with some indexes like "YDifferenceFromPrevious()" for "movement" to finalize this and get a better estimation in the end, i'll just have to log the indexes values during my tests runs and then try to define some multiplictor/formula based on this.
The decimation stage of IVTC plugins use more appropriate methods to measure the difference between adjacent frames, with or without considering chroma planes. They use those metrics to identify and remove duplicated frames.

These plugin functions can be used in a special dryrun mode, to output those metrics without decimating any frame:

TDecimate for Avisynth(+) (http://avisynth.nl/index.php/TIVTC/TDecimate):
Tdecimate(mode=4, output="Tdecimate_output.txt") #, display=True)

VDecimate for Vapoursynth (http://www.vapoursynth.com/doc/plugins/vivtc.html#vivtc.VDecimate):
measured_clip = core.vivtc.VDecimate(processed_clip, dryrun=True)
Vapoursynth won't generate a text file but will attach the following properties to every frame: VDecimateDrop, VDecimateMaxBlockDiff, VDecimateTotalDiff. You could use python to extract them to a text file.

[Yanak]

A bit OT, sorry ...

It does create high quality gifs (that's a bit of an oxymoron, but hey...) . But the very first test I made had errors not visible with other methods. I think it's the "combine colors between frames thing." 3/56 frames had splotchy errors
Did not noticed this but might depend of the footage used or i could have missed something too, the gifs i made outside the tests runs and actually used were clean as far as i could see, maybe something to report to the creator of this tool if there is some weird outputs, he's regularly patching it when problems are reported.

I'll check myself to see if there is things i can notice, since i started to try integrate this into a powershell script running from Staxrip i created loads of gifs but mostly for testing purposes, not necessary looking at most of them tho.

And yes it's a bit of an oxymoron indeed ^^


@VS_Fan :
Thanks for the tip, i downloaded it and gave a quick try using :
Tdecimate(mode=4, blockx=4, blocky=4, chroma=true, output="H:\Tivtc.log")

Then in my PS script i simply used this to parse the log and sum the columns 2 & 3 separately and get 2 distinct values :
Get-Content "H:\Tivtc.log" | Select-Object -Skip 2 | foreach {
-split $_ | select -index 1 | % {$2nd_column_sum += [math]::abs($_)}
-split $_ | select -index 2 | % {$3rd_column_sum += [math]::abs($_)}
}

This said the blocks method even small like i set them is maybe not the best option, i don't know yet, i think i will give a few tries using those 2 values + the YDifferenceFromPrevious() sum index i already done and for now run just a few tests when i'll have time and see if there is something i can correlate between all this when running login tests and putting the puzzle pieces together.

Still need time to look into StainlessS RT_Stats function "RT_DBaseRgbColorCount" and see if i can make something with this too.

Thanks a lot.