propagate per-frame metadata through format conversions?

[cheyrn]

I want to tag frames so that I can recognize them in a transformed video as a form of debugging.

For example:

Video A frame 23 is the frame in question.

Video A is manipulated in various ways and stored as Video file B and the program ends

In a different session, on a different day, etc. Video B is manipulated in various ways and stored as Video file C and the program ends

I want to be able to tell from vapoursynth which frame in video file C was once frame 23 in video file A.

I was thinking that I could encode a number in pixel values at a certain place on the frame and use it as a tag. That assumes I can rerun the transforms again without the encoding after I have identified what the frame mapping was.

My knowledge of video formats is very little. Is there a typical way to store per-frame metadata that would survive transformation between some different video formats?

[LoRd_MuldeR]

Is there a typical way to store per-frame metadata that would survive transformation between some different video formats?

This is called digital watermarking (http://en.wikipedia.org/wiki/Digital_watermarking). And there are techniques available that embed information in the video or audio data in an (almost) imperceivable way that survives rather strong re-compression or even analog transfers.

However, I don't know if any such techniques are freely available. At least I'm not aware of any free tools that do this.

For your purpose, it would probably be much easier to simply dump the per-frame metadata into a separate file at the very end of your script, so they can be loaded again the next time...

[feisty2]

This is called digital watermarking (http://en.wikipedia.org/wiki/Digital_watermarking). And there are techniques available that embed information in the video or audio data in an (almost) imperceivable way that survives rather strong re-compression or even analog transfers.

However, I don't know if any such techniques are freely available. At least I'm not aware of any free tools that do this.

For your purpose, it would probably be much easier to simply dump the per-frame metadata into a separate file at the very end of your script, so they can be loaded again the next time...

will digital watermarks survive blurring, strong denoising, etc?

[LoRd_MuldeR]

will digital watermarks survive blurring, strong denoising, etc?

Well, it depends, of course.

Usually the "strength" of the digital watermark is a parameter of the embedding process, which controls the trade-off between "robustness" and "imperceptibility". The more you tweak towards "robustness", the more modifications (re-encoding, blurring, denoising, noise addition, etc) can be survived - at the cost of stronger quality degradations (by the watermark itself). If you tweak towards "imperceptibility", you get better visual quality, but the watermark will be more fragile.

Of course you can always "break" the digital watermark in one way or another, if you really want to - for example by very strong filtering or by re-encoding at extremely low bitrates - but the resulting video will be pretty much useless then.

(BTW: Sometimes it's even desired that the digital watermark gets destroyed by any modification, or by very subtle modifications. This is called a "fragile watermark". It's used to verify the "integrity" of the video)

[feisty2]

@LoRd_MuldeR
Thx for answering, it just hit me that even visible watermarks (transparent kind) can be removed tho, I remember there's even a filter for it, first scan the watermark, then it will be removed
As for digital watermark, if it stays constant on every frame, I think it can be detected by some motion filter, once we found where it is, we can remove it like the visible ones
Edit: if it jumps everywhere between the frames, a temporal denoise filter will assume it's just some random temporal noise, and will remove it

[LoRd_MuldeR]

As for digital watermark, if it stays constant on every frame, I think it can be detected by some motion filter, once we found where it is, we can remove it like the visible ones

But it doesn't need to be constant on every frame at all. You shouldn't think of the digital watermark as a fixed pattern that is simply "overlayed" to the video.

Instead, think of it as a certain "measure", that is defined in a specific way, which yields a value of zero for "normal" input video. Or, at least, the measured value will be close to zero with very high probability for "normal" video. The process of embedding the digital watermark then aims to modify the video in a special way that pushes this "measure" into either the positive or negative direction. How exactly we modify the video doesn't matter at all and can be completely different on every frame, as long as we'll be able to "measure" a positive value (can be interpreted as a binary information of '1') or a negative value (can be interpreted as a binary information of '0') afterwards. That's already enough to "embed" your binary data.

[feisty2]

Oops, got it, so it's not an actual "watermark", but more of some inserted information :D
Well, as long as we can't see it, how bad ass it is shouldnt be a problem tho

[Myrsloik]

Apply the frame number using ModifyFrame right after the source. Then right before set_output() you use FrameEval to write the frame numbers to a file (or many files).
Second time around you load and apply the frame numbers and save them to a file again at the end.

So it's definitely possible to do using frame properties. The bad news is that you have to use the two most complex functions to accomplish it. I'm not close to a computer right now so i can't write an example script for you at the moment.

There's no real need to modify the video itself.

[cheyrn]

I don't think I understand the idea of using an external file, or it wouldn't work for some scenarios.

If there are pixels that encode a frame number, as long as no transformation distorts the pixels in a way that makes it so you can't derive the number from the pixel values, adding the watermark once would allow it to be retrieved after n transformations into different video formats.

Imagine that you don't have infinite storage and you occasionally delete files between backups that you find later that you wish you had not deleted. If you are relying on having an external file with frame mappings for every transformation, you could easily break the chain of mappings so that you can no longer tell which frame is which.