Zscene - Scene change detection for Vapoursynth

[Adub]

Repo: https://github.com/adworacz/zscene
Version: 0.1

In a bit of "if you can't beat 'em, join 'em", I threw together a quick plugin to handle scene change detection. More specifically, it doesn't actually do any scene change detection itself (yet), but farms out to much more accurate tools like av-scenechange.

All that's currently implemented is a "ReadScenes" function that (you guessed it) reads the scene data produced by external tools and adds the pertinent scene change properties to each frame.

Use it to replace the (much outdated) "misc.SCDetect" function for automatic scene change detection.

In the future, I'll likely implement some custom scene change detection algorithms to do this all inline, but for now relying on scene change detection that's literally used in the AV1 compression algorithm is enough for most use cases.

[Myrsloik]

Originally Posted by Adub

Repo: https://github.com/adworacz/zscene
Version: 0.1

In a bit of "if you can't beat 'em, join 'em", I threw together a quick plugin to handle scene change detection. More specifically, it doesn't actually do any scene change detection itself (yet), but farms out to much more accurate tools like av-scenechange.

All that's currently implemented is a "ReadScenes" function that (you guessed it) reads the scene data produced by external tools and adds the pertinent scene change properties to each frame.

Use it to replace the (much outdated) "misc.SCDetect" function for automatic scene change detection.

In the future, I'll likely implement some custom scene change detection algorithms to do this all inline, but for now relying on scene change detection that's literally used in the AV1 compression algorithm is enough for most use cases.

Finally a sequel to scxvid. Scene change detection has now come full circle.

[Z2697]

Personally I don't think av-scenechange is good.
Sure it's better than simple pixel difference thresholding or smth, but we have mvtools which is online and faster (last time I checked it).
Taking some sort of RD cost into account (av-scenechange is taken from rav1e, so I think there must be some) can be a desired feature depending on the following process though.

[Adub]

Finally a sequel to scxvid. Scene change detection has now come full circle.

Right you are. Only took 3 major generations of codec development...

Personally I don't think av-scenechange is good.

Any particular data you have to back this statement? I agree that the MVTools may have advantages if you're already running an MAnalyse pass (since the scene detection built from those vectors is essentially free at that point).

av-scenechange uses 3 different metrics for scene change detection, as described in the source code: https://github.com/rust-av/av-scenec...ard.rs#L19-L21

And to clarify the av-scenechange / rav1e relationship - rav1e was the original source of the scene change detection, but it was split out as a separate module to av-scenechange and now rav1e has been updated to depend on av-scenechange directly. So they are using the exact same code at this moment.

[Z2697]

Originally Posted by Adub

Any particular data you have to back this statement?

I don't have. It's more like a personal preference. I don't want to consider encoding cost in the scene detection for filtering.
Plus the ability to run online.
And the speed, well I tested again, av-scenechange can be faster, even much faster if the fast mode is used, but MVTools is still fast enough.
Although, I must say they are closely related - the motion and the cost, and worth considering in many cases.

[Selur]

I'll wait till zscene incorporates av-scenechange, since I only need the scene detection info attached to the frames not in an external file.
=> looking forward to when misc.SCDetect can be replaced.

[Z2697]

Originally Posted by Selur

I'll wait till zscene incorporates av-scenechange, since I only need the scene detection info attached to the frames not in an external file.
=> looking forward to when misc.SCDetect can be replaced.

Several alternatives to misc.SCDetect already exist though.
If we forget about the outdated XviD based ones, there're MVTools and styler00dollar's AI models (works on downscaled video so quite fast actually), and others I don't know of.
(I don't mean zscene shouldn't be one of them)

Speed-wise, I don't think there's anything that can beat misc.SCDetect because it's so simple.

[Adub]

I'll wait till zscene incorporates av-scenechange, since I only need the scene detection info attached to the frames not in an external file.

Just to be clear - Zscene *does* attach the scene detection info to the frames, as the standard props. It just reads the JSON file emitted by av-scenechange to do it.

[Selur]

Sorry, yes, but you would still have to use something like:

vspipe -c y4m source.vpy - | av-scenechange -o scenes.json -

to create the json, right?

[Adub]

Yes, that's correct, although depending on your use case, and the features (like ffmpeg support) that av-scenechange is compiled with, you can feed your source file directly to it with something like this:

av-scenechange -o scenes.json foobar.mkv

So you can avoid the use of vapoursynth for the initial scenes collection if desired.

But in sum, you do need to run av-scenechange on your source somehow to produce the JSON that Zscene can then read.