Testing AviSynth+ frame properties in FFmpeg/mpv

[qyot27]

I had posted about this in the AviSynth+ development thread back on September 23rd, but it was suggested that I start a thread about it if I want to not have it get lost in there.

I'll just copy-paste the original message below, as it covers pretty much everything:

Okay, so I'm going to need those of you that are really invested in the particulars of the color representation in AviSynth+ to test this for me, as I don't have nearly the amount of samples needed for this:
FFmpeg (https://www.mediafire.com/file/ig4oqq4i9lfs8cu/ffmpeg_r103791%252B10.7z/file)
mpv (https://www.mediafire.com/file/hp1n6zhvqevieop/mpv_r49201.7z/file)

This build of FFmpeg has a patch (https://github.com/qyot27/FFmpeg/commit/2771580eae058a642384ef02187620afb7787663) that reads some of the frame properties from AviSynth+ and populates them in what I hope is the correct fashion. The mpv build is provided because it's more comfortable than ffplay, and because unlike ffplay, it doesn't require additional filtering steps to work on the additional information (read: mpv knows the script is outputting HDR and can tonemap it automatically for SDR displays if that's how you have it set up).

Will require one of the test builds of AviSynth+ that pinterf has provided from after April, since there had been a bug in 3.7.0 (and probably the earlier versions that were also inside interface version 8) preventing the C interface from accessing frame properties; if you use 3.7.0, it will crash*. You'll also need a source filter that sets frame properties (I used the test build of ffms3000); propSetInt does work to force the properties to a given value, but that's artificial and I want more real-world examples just in case there's a problem with how the equivalencies between the frame props and FFmpeg's color enums were set up.

*this will be resolved in the final version of the patch, since it's going to require bumping to interface version 9 just to make the detection in FFmpeg work correctly (but bumping to version 9 should probably go hand in hand with more new API functions and a way of future-proofing against bugs like this).

What frame properties is it reading?
_FieldBased (the one that doesn't have to do with color; people interested in interlaced video, take note of this one)
_ChromaLocation
_Primaries
_Transfer
_Matrix
_ColorRange

I used the LG New York HDR UHD 4K Demo (https://4kmedia.org/lg-new-york-hdr-uhd-4k-demo/) for verifying that mpv with the patch can pick up and use the information instead of seeing the script as BT.709. But there's obviously a lot more entries under all those different frame properties, and I have no clue if the way I set this up actually can handle all of them the way they're ostensibly supposed to (admittedly, some of that is also up to the source filters setting the correct value and the other filters in the script not screwing it up somehow, but making sure the script->libavformat chain works as intended is the goal here). I also didn't have any interlaced content on hand to really test the field detection on.

[kedautinh12]

I wonder why you don't pull request to original ffmpeg??

[qyot27]

This is so users that actually do use AviSynth can vet the change and report problems, so the patch that eventually gets submitted to upstream FFmpeg has been properly tested. The rest of the FFmpeg devs and maintainers aren't going to test a litany of samples through AviSynth.

[FranceBB]

Well the MPV build is broken: fonts are messed up and the lower part of the player, the one with the bar which is supposed to be clickable to move across the video is broken.
Now, about FFMpeg, I've done a few tests since I'm very much interested in the project.

Test 1:

Source XDCAM-50

FFMpegSource2("00000.mxf")


ffmpeg.exe -i "\\mibctvan000\Ingest\MEDIA\temp\AVS Script.avs" -pix_fmt yuv422p -vcodec mpeg2video -s 1920:1080 -aspect 16:9 -vf setfield=tff -flags +ildct+ilme+cgop -b_strategy 0 -mpv_flags +strict_gop -sc_threshold 1000000000 -r 25 -b:v 50000k -minrate 50000k -maxrate 50000k -bufsize 36408333 -g 12 -bf 2 -profile:v 0 -level:v 2 -color_range 1 -color_primaries 1 -color_trc 1 -colorspace 1 -c:a pcm_s24le -ar 48000 -af loudnorm=I=-24:LRA=12:tp=-2 -y "\\mibctvan000\Ingest\MEDIA\temp\pre-final_output.mxf"

pause




it just hangs there...

Avisynth 3.7.1 Test 20 x64 CUDA

I also tried with FFProbe, but to no avail...

ffprobe.exe -show_format -show_streams -i "\\mibctvan000\Ingest\MEDIA\temp\AVS Script.avs"

pause

https://i.imgur.com/pkljZ0e.png
https://i.imgur.com/QvhARe2.png


On the other hand, AVSPmod works just fine, so it must be your FFMpeg build... :(
This happened on Windows 10 Enterprise x64

Test2:

I've then tried with an XAVC Intra Class 480 written by the Sony MXF Development Kit in BT2020 HLG:

FFMpegSource2("P:\00_INGEST_MAM\ENCODER\SAMPLE\SHOWCASE 4K\TEST 4K ATMOS.mxf")

And again, FFProbe hangs and the encode doesn't start with FFMpeg...


Test 3:
I've now tried with the same file, but with ffms2 instead of ffms3 and with Avisynth 32bit and FFMpeg 32bit, but the same result applies...

[emcodem]

Hmm i cannot get this to work. First the framedata variable in avisynth_create_stream_video function appeared uninitialized to me, so i set it to 0 for testing but then avs_get_frame_props_ro always crashes avisynth.dll (along with ffmpeg of course). Tested only avisynth+ 3.7.0 so far. My test avs script has only colorbars(), not more.

[VoodooFX]

Hmm i cannot get this to work. ... Tested only avisynth+ 3.7.0 so far.

You need Avisynth+ 3.7.1 test build (https://forum.doom9.org/showthread.php?p=1953460).

[emcodem]

You need Avisynth+ 3.7.1 test build (https://forum.doom9.org/showthread.php?p=1953460).

This build crashes at the very same spot, avs_get_frame_props_ro

[pinterf]

@qyot27, I have sent some working examples on using frame properties with C interface. Unfortunately 3.7.0 was broken in that regard (= instant crash), 3.7.1 builds from late April or May must be good enough for tests.

[qyot27]

Hmm i cannot get this to work. First the framedata variable in avisynth_create_stream_video function appeared uninitialized to me, so i set it to 0 for testing but then avs_get_frame_props_ro always crashes avisynth.dll (along with ffmpeg of course). Tested only avisynth+ 3.7.0 so far. My test avs script has only colorbars(), not more.
Don't. It's initialized just fine the way it was. What you did was force it to think all things have frame properties whether they've been set or not, and then it chokes when it doesn't see them (which is the reason it would continue to not work even when you updated to a 3.7.1 test build). Colorbars does not set frame properties.

@qyot27, I have sent some working examples on using frame properties with C interface. Unfortunately 3.7.0 was broken in that regard (= instant crash), 3.7.1 builds from late April or May must be good enough for tests.
Yep, I mentioned the issue with 3.7.0 and the need for the post-April test builds in the OP, along with the eventual need that the patch will check for interface version 9 once we've added the required new functions to avoid the issue in the future.

The C interface uses the same AVISYNTH_INTERFACE_VERSION as the C++ interface, as it's more or less a package deal (https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/include/avisynth_c.h#L88). Maintaining parity between the interfaces means if something gets added to the C++ interface, the equivalent should be added to the C interface (or vice-versa), or else the process of playing catch-up is going to require bumping the interface version again.

In the frame properties access situation, a separate check, something like AVISYNTHPLUS_INTERFACE_VERSION that continuously ticks upward like AVISYNTH_INTERFACE_VERSION does, or an AVISYNTHPLUS_INTERFACE_BUGFIX_VERSION that resets to 0 every time there's a bump in the main INTERFACE_VERSION would be the solution, with corresponding functions in the C++ and C interfaces to check the value of those versions.

If something like that existed already, it wouldn't require bumping to interface version 9 for FFmpeg to work with it, because it could check for version 8, and then check for the bugfix version to see if it's a properly functioning version 8 (as the way I understand the actual problem is that it wasn't actually an API break that happened with the fix in git, the C interface's frameprop functions as written just didn't work to start with). But adding it now means we have to bump to version 9, and if we're doing that, there's probably other functions we should add just to make the bump more substantial (like the non-clip-based version checks you mentioned on Github).

Well the MPV build is broken: fonts are messed up and the lower part of the player, the one with the bar which is supposed to be clickable to move across the video is broken.
My guess is that that's purely because I don't use libass' DirectWrite functionality. It's using fontconfig, so mpv probably needs the extra subfont.ttf and fonts.conf stuff to supplement it in the mpv/ subdirectory where you'd put the rest of the player config.

RE: all the 'hanging', either an exact XDCAM-50 sample is going to be necessary to reproduce it or you'll have to try running under gdb (not that it would do much, since these aren't debug builds, but there might be something that gets picked up). Because that could very well be something else doing it.

FFmpegSource2("freeMXF-mxf1a.mxf")
Sample from http://freemxf.org/samples/index.html

Using FFMS2 2.23 (from 2016, since that's more than old enough to not be aware of AviSynth+'s frame properties):
E:\Documents_C>ffmpeg -i test_mxf.avs
ffmpeg version r103791+10 master-447cf53774 HEAD-616bfba5e1
contains: avs_intfix datetime mergedin plain_pkgconfig silent_invoke vapoursynth_alt versioninfo
Copyright (c) 2000-2021 the FFmpeg developers
built on Sep 23 2021 03:17:02 with gcc 11.1.0 (GCC)
libavutil 57. 6.100 / 57. 6.100
libavcodec 59. 9.100 / 59. 9.100
libavformat 59. 5.100 / 59. 5.100
libavdevice 59. 0.101 / 59. 0.101
libavfilter 8. 9.100 / 8. 9.100
libswscale 6. 1.100 / 6. 1.100
libswresample 4. 0.100 / 4. 0.100
libpostproc 56. 0.100 / 56. 0.100
Input #0, avisynth, from 'test_mxf.avs':
Duration: 00:00:10.64, start: 0.000000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p(pc, gbr/unknown/unknown, progressive), 720x576, 25 fps, 25 tbr, 25 tbn
At least one output file must be specified

Using the FFMS3000 test build from August 17th (https://forum.doom9.org/showpost.php?p=1949983&postcount=2669) (which does support frame properties):
E:\Documents_C>ffmpeg -i test_mxf.avs
ffmpeg version r103791+10 master-447cf53774 HEAD-616bfba5e1
contains: avs_intfix datetime mergedin plain_pkgconfig silent_invoke vapoursynth_alt versioninfo
Copyright (c) 2000-2021 the FFmpeg developers
built on Sep 23 2021 03:17:02 with gcc 11.1.0 (GCC)
libavutil 57. 6.100 / 57. 6.100
libavcodec 59. 9.100 / 59. 9.100
libavformat 59. 5.100 / 59. 5.100
libavdevice 59. 0.101 / 59. 0.101
libavfilter 8. 9.100 / 8. 9.100
libswscale 6. 1.100 / 6. 1.100
libswresample 4. 0.100 / 4. 0.100
libpostproc 56. 0.100 / 56. 0.100
Input #0, avisynth, from 'test_mxf.avs':
Duration: 00:00:10.64, start: 0.000000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p(tv, bt470bg, progressive), 720x576, 25 fps, 25 tbr, 25 tbn
At least one output file must be specified

[FranceBB]

Ok, so I've made some more tests.
On Avisynth prior to interface version 9 it doesn't work, but with Avisynth 3.7.1 Beta 20 x64 it works.

Test 1:

FFVideoSource("TEST.mxf")

https://i.imgur.com/IHF7oQS.png

General
Complete name : TEST.mxf
Format : MXF
Format version : 1.3
Format profile : OP-1a
Format settings : Closed / Complete
File size : 1.11 GiB
Duration : 1 min 0 s
Overall bit rate : 159 Mb/s
Encoded date : 2019-12-10 11:00:12.000
Writing application : nablet XAVC Export 4.5.9.2313.1
Writing library : Sony MXF Development Kit (Win32) 4.17.0.146.1

Video
ID : 2
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High 4:2:2 Intra@L5.1
Format settings, CABAC : No
Format settings, GOP : N=1
Format settings, wrapping mode : Frame
Codec ID : 0D01030102106001-0401020201323001
Duration : 1 min 0 s
Bit rate : 150 Mb/s
Maximum bit rate : 400 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 FPS
Standard : Component
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 10 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.722
Stream size : 1.05 GiB (94%)
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : HLG
Matrix coefficients : BT.2020 non-constant


Works ok.

Test 2:

FFVideoSource("SeqMod2.ts ")

https://i.imgur.com/nu0wpAz.png

General
ID : 13124 (0x3344)
Complete name : SeqMod2.ts
Format : MPEG-TS
File size : 4.78 GiB
Duration : 25 min 9 s
Overall bit rate mode : Variable
Overall bit rate : 27.2 Mb/s

Video
ID : 336 (0x150)
Menu ID : 21862 (0x5566)
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5@Main
HDR format : SMPTE ST 2086, HDR10 compatible
Codec ID : 36
Duration : 25 min 10 s
Bit rate : 25.3 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 FPS
Standard : Component
Color space : YUV
Chroma subsampling : 4:2:0 (Type 2)
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.122
Stream size : 4.44 GiB (93%)
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant
Mastering display color primaries : Display P3
Mastering display luminance : min: 0.0000 cd/m2, max: 1000 cd/m2
Maximum Content Light Level : 1000 cd/m2
Maximum Frame-Average Light Level : 400 cd/m2


so it seems to be working for HLG and PQ, then I tried with some different color curves (i.e transfer characteristics) like ZLog2:

Test 3:

FFVideoSource("ZCAM E2 H265 ZLOG2 HDR.MOV")

and apparently ffms3000 failed to index the video...

EDIT: Fixed, now it does, but it behaves just like LWLibav does.

I tried with:

LWLibavVideoSource("ZCAM E2 H265 ZLOG2 HDR.MOV")

but unfortunately it filled it with BT709 only without any clue about ZLog2...

https://i.imgur.com/c7Sb6yY.png

General
Complete name : ZCAM E2 H265 ZLOG2 HDR.MOV
Format : MPEG-4
Format profile : QuickTime
Codec ID : qt 2004.06 (qt )
File size : 378 MiB
Duration : 40 s 800 ms
Overall bit rate mode : Variable
Overall bit rate : 77.8 Mb/s
Encoded date : UTC 2020-10-17 16:28:51
Tagged date : UTC 2020-10-17 16:28:51
com.zcam.camera.prepend : v0
CameraMaker : Z CAM
CameraModel : E2
com.zcam.camera.sn : 410A0030030
com.zcam.camera.fw : 0.96
com.zcam.camera.movieType : MOVIE
com.zcam.camera.lut : Z-Log2
com.zcam.camera.iso : 2500
com.zcam.camera.storage : LEXAR CFAST 128GB CARD
com.zcam.camera.aperture : F4
com.zcam.camera.lensType : EF70-200mm f/4L IS USM
com.zcam.camera.lensFocalLength : 200mm
com.zcam.camera.focusDistance : 16435mm
com.zcam.camera.whiteBalanceMode : Manual
com.zcam.camera.whiteBalanceKelvin : 5600K
com.zcam.camera.whiteBalanceTint : 0
com.zcam.camera.sharpness : Strong
com.zcam.camera.noiseReduction : Weak
com.zcam.camera.projectFps : 25
com.apple.proapps.shootingRate : 25
com.zcam.camera.captureVfr : 100
com.apple.proapps.cameraName : B
com.apple.proapps.reel : 001
com.zcam.camera.battery : 15.0V
com.zcam.camera.temperature : 49.3°C
com.zcam.camera.shutterAngle : 90°
com.zcam.camera.splitIndex : 0
com.apple.proapps.clipID : B001C0064_20201017182851_0001
com.zcam.camera.hash : 39a3

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5.1@Main
HDR format : SMPTE ST 2086, HDR10 compatible
Codec ID : hvc1
Codec ID/Info : High Efficiency Video Coding
Duration : 40 s 800 ms
Bit rate : 74.5 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.359
Stream size : 362 MiB (96%)
Language : English
Encoded date : UTC 2020-10-17 16:28:51
Tagged date : UTC 2020-10-17 16:28:51
Color range : Limited
Mastering display color primaries : R: x=0.060000 y=0.330000, G: x=0.300000 y=0.150000, B: x=0.640000 y=0.600000, White point: x=0.312700 y=0.329000
Mastering display luminance : min: 0.0000 cd/m2, max: 0 cd/m2
Codec configuration box : hvcC


Test 4:

I then tried to see what would happen if the user was gonna convert internally from PQ to HLG:

FFMpegSource2("SeqMod2.ts")

ConvertYUVtoLinearRGB(Color=0,OOTF=false)
ConvertLinearRGBtoYUV(Color=0,HDRMode=1,HLGLw=1200,HLGColor=2,OOTF=false)

and what is reported to FFMpeg is still BT2020 HDR PQ despite the stream now being HLG:

https://i.imgur.com/yAMm9ix.png

so I guess that developers will have to update their plugins to get and change the info about the stream and there should also be a way for the user to change those info within the stream before delivering the stream to FFMpeg.

AVSPmod should also now be able to display those info in the bottom right part.

[FranceBB]

Test 5:

testing now field parity on XDCAM-50 files:

FFVideoSource("\\mibctvan000\Ingest\MEDIA\temp\00000.mxf")

Top field first interlaced is detected and passed through correctly:

https://i.imgur.com/tAGsOza.png

General
Complete name : \\mibctvan000\Ingest\MEDIA\temp\00000.mxf
Format : MXF
Commercial name : XDCAM HD422
Format version : 1.3
Format profile : OP-1a
Format settings : Open / Incomplete
File size : 709 MiB
Writing application : Omneon OP1a Muxer

Video
ID : 2
Format : MPEG Video
Commercial name : XDCAM HD422
Format version : Version 2
Format profile : 4:2:2@High
Format settings : BVOP
Format settings, BVOP : Yes
Format settings, Matrix : Default
Format settings, GOP : M=3, N=12
Format settings, picture structure : Frame
Format settings, wrapping mode : Frame
Codec ID : 0D01030102046001-0401020201040300
Bit rate mode : Constant
Bit rate : 50.0 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.965
Time code of first frame : 00:00:00:00
Time code source : Group of pictures header
GOP, Open/Closed : Closed
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709


Test 6:

and here's the problem, if the user bobs it and makes it progressive...

FFVideoSource("\\mibctvan000\Ingest\MEDIA\temp\00000.mxf")

Bob()

it's still reported as interlaced TFF

https://i.imgur.com/CIPPIaY.png

however if I try IsFieldBased() it returns false inside Avisynth:

FFVideoSource("\\mibctvan000\Ingest\MEDIA\temp\00000.mxf")
Bob()
IsFieldBased() ? Text("It's field based") : last

https://i.imgur.com/Ee6mPRa.png

which means that things are working correctly in Avisynth from the Avisynth point of view, they're just not passed correctly to FFMpeg as it seems that no matter what the user does, changes ain't passed through...

Same goes if I use things like:

FFVideoSource("\\mibctvan000\Ingest\MEDIA\temp\00000.mxf")
Bob()
Converttoyv24(matrix="PC.709")

instead of reporting "Full PC Range" it still reports Limited TV Range...

https://i.imgur.com/OB8XwVb.png

[StainlessS]

however if I try IsFieldBased() it returns false inside Avisynth:

http://avisynth.nl/index.php/Clip_properties
Video: Interlacing

bool clip.IsFieldBased

Returns true if the clip is field-based. What this means is explained here.


"Here" is link pointed to
http://avisynth.nl/index.php/Interlaced_fieldbased
Interlaced versus field-based video

Currently (v2.5x and older versions), AviSynth has no interlaced flag which can be used for interlaced video.
There is a field-based flag, but contrary to what you might expect, this flag is not related to interlaced video.
In fact, all video (progressive or interlaced) is frame-based, unless you use AviSynth filters to change that.
There are two filter who turn frame-based video into field-based video: SeparateFields and AssumeFieldBased.

It flags separated fields. [or AssumeFieldBased. was used]

[VoodooFX]

Ok, so I've made some more tests.
http://avisynth.nl/index.php/Clip_properties

It's about the frame properties (http://avisynth.nl/index.php/Internal_functions#Functions_for_frame_properties), not the clip properties.

[StainlessS]

It's about the frame properties, not the clip properties.
So FranceBB should test the frame property, not clip property [?]
And would only [at present] work there for builtin bob [?]. [until other bobbers set frame properties].

[VoodooFX]

So FranceBB should test the frame property, not clip property [?]
And would only [at present] work there for builtin bob [?]. [until other bobbers set frame properties].
Yes.
I doubt that bob() sets any frame properties, maybe it should only let them pass through [atm], I didn't tried anything with them yet.

There (https://forum.doom9.org/showpost.php?p=1951684) I asked you about them, maybe they could replace some rt_stats db thingies.

[pinterf]

Nothing sets frame properties inside Avisynth at the moment. Plugins are way ahead of Avisynth core in this regard.

[emcodem]

Yes, i can confirm that no stock filter will set any frame props. This means, no matter which filter we use in the script, the frame properties will not change to the ones that the source plugin (maybe) did set.

I created a test file set with some different color props and interlacing, mid stream changes here: https://wetransfer.com/downloads/25eed323d7d1b253ed8b6e108ecc806820211006153519/cf11be87d3a79da7f390cd64b5748f8320211006153538/1d6674?utm_campaign=WT_email_tracking&utm_content=general&utm_medium=download_button&utm_source=notify_recipient_email

But after all, i guess testing with actual files is superfluous in this case as we would only test if the frame props from ffms3 and a handfull other external plugins support frameprops already.

To the question if it makes sense to forward the props to ffmpeg even if only a bunch of source filters supports them but no internal filters do support this:
In our usecase, ffastrans, which means fully automated generation of scripts, it really makes a lot of sense. The reason is simple: when we insert a filter programatically, we can also insert the needed "propSet" so that in the end Avisynth will forward the correct Metadata to ffmpeg. In fact this is the only clean way for us to get the props of the finally returned clip. Sure there are many workarounds possible but this would be the best integrated way.

On the other hand, for manual usecases, it will most likely be really problematic when avisynth forwards only the props that the source plugin did set instead of the finally generated clip's props. Users would have to insert propSet lines (as they certainly have to insert assumeField/framebased and assumeB/Tff currently) to hint ffmpeg on what their finally generated clip is. The question which i cannot answer yet is if it is better to have "no values set" in ffmpeg than "wrong values set", there is a chance that it turns out to be the equal.

After all i can at least confirm that it works as intended, i just tested using propSet with 0 or 1 and all of them forced some reaction in the prop detection of ffmpeg.

propSet("_ColorRange", 0) #pc,tv
propSet("_Matrix", 0) # 0 = gbr
propSet("_Primaries", 0) # 0 = gbr
propSet("_Transfer", 0) #
propSet("_FieldBased", 0)
propSet("_ChromaLocation", 0)

Next tests could be to set all possible values for the properties and see if they report in ffmpeg the same value that was set in Avisynth but would such tests really make sense or do we maybe even have to wait with that patch until all stock filters are setting the frameprops?

[emcodem]

@qyot27 where we want to go with this now? ...i'll be happy to help wherever i can but honestly i think the frameprops don't make sense unless all the stock filters use it as expected. This is needed to set some kind of standard that filter developers can follow from my perspective.
My backgrounds on this is that it is more or less the same with the frame/fieldbased property. I mean yeah, some plugins still get it wrong but after all if they do it "wrong", one can at least open a feature request or bug report, point to avisynth internal filters and say "look, here is how this is intended to work"...
Anyway, i'd like to ask you to at least apply the correction of my patch for the interlaced detection because it is already part of ffastrans and at this very moment it passes wrong values to ffmpeg.

[MysteryX]

I'm writing a script and wanted to account for frame properties.


propGetxxxxx: This filter can only be used within run-time filters


What's the point of having frame properties if scripts can't use them?

[StainlessS]

propGetxxxxx: This filter can only be used within run-time filters

What's the point of having frame properties if scripts can't use them?
Scriptclip et al are runtime filters.

A script session during 'compile stage' [pre frameserving] is really just there to stitch together filters into a filter graph,
ie calling filter constructors and chaining them together, at that stage [without some cheating] there aint any frame numbers,
nor associated frame properties.

[MysteryX]

so ... in VapourSynth we cheat and read properties from the first frame for simplicity. In Avisynth, any usage of frame properties must be in ScriptClip?

When you need to pass ColorRange to create a LUT table, or other complex filters, it's kind of a dead-end. Creating a LUT on every frame killed performance. For something simple like calling Tweak with the right coring parameter.

[StvG]

so ... in VapourSynth we cheat and read properties from the first frame for simplicity. In Avisynth, any usage of frame properties must be in ScriptClip?

When you need to pass ColorRange to create a LUT table, or other complex filters, it's kind of a dead-end. Creating a LUT on every frame killed performance. For something simple like calling Tweak with the right coring parameter.

Read frame prop from the first frame, save it to temp file, read the file. An example (RT_Stats, CallCmd used) :BlankClip()
propset("_reba", 4)
temp_dir = RT_GetSystemEnv("temp") + "\" + "temp.txt"

ScriptClip(function [temp_dir] ()
{
RT_TxtWriteFile(String(propGetInt("_reba")), temp_dir)
last
})
RT_AverageLuma(n=0, w=1, h=1)

var = Value(RT_ReadTxtFromFile(temp_dir, 1, 0))
Subtitle(String(var))
ScriptClip(function [temp_dir] () { CallCmd(close="cmd /c del " + temp_dir, hide=true) })

[StainlessS]

Nuttin' stopping you from something like this [the cheating stuff]

AviSource(...)
current_frame = 0
x=Averageluma # or whatever frame property thingy
...

[StainlessS]

This works

BlankClip()
propset("_reba", 4)

# ...

# cheatin' stuff

current_frame=0 # pretend frameserving in runtime filter
var = propGetInt("_reba")
Subtitle(String(var)) # EDIT: And then drop-though to frame serve to Player, Vdub etc

[MysteryX]

I had written the VapourSynth code to respect per-frame color-range and matrix -- in the theoretical case that it could ever be useful.

But I came to the conclusion that in such rare case, you'd be better off running a filter that auto-splits the file into segments to process them individually, because nearly no filter would support respecting it per-frame anyway.

And for theoretical or imaginary benefits, you pay a performance cost, and cannot use LUT or any hard-to-initialize filter based on frame property values.

Is there any Input (video source) or Output (encoder/player) that supports variable matrix and range?

[FranceBB]

Today I had a very bad behavior with frame properties.
A source encoded by a broadcast hardware recording port passed some dynamically changing metadata; in other words, they have been constant up to a certain point and then they changed 'cause the input stream into the port changed and the saved metadata in the file changed accordingly.
When I indexed it with ffms3 and worked with the file inside Avisynth with several filters like dfttest etc it went through up to a certain point, then the encoding just stopped.
It didn't crash, the RAM was still allocated, but the encoding just stopped, without any sign of moving/proceeding.
I repeated the test over and over again, it was always stopping at the very same point.
This happened with FFMpeg opening the Avisynth Script and encoding.
I tried to re-encode the source in HuffYUV lossless with FFMpeg and then Index with ffms3 and encode, same behavior, stopped mid encode.
I tried to re-encode the source in H.264 lossless with FFMpeg and then Index with ffms3 and encode, same behavior, stopped mid encode.
I tried to re-encode the source in FFV1 lossless with FFMpeg and then Index with ffms3 and encode, same behavior, stopped mid encode.
Then, I realized it could have been because of the dynamically changing metadata, so I re-encoded with FFMpeg in FFV1 lossless but stripping out the metadata like so:

-c:v ffv1 -slicecrc 1 -c:a pcm_s24le -ac %i_a_channels% -pix_fmt yuv422p10le -map_metadata -1 -s %i_width%:%i_height% -ar 48000

and then indexed with ffms3 and encoded and... it worked like a charm.
So the question is: is there a way to make ffms3 ignore and strip out the metadata / frame properties entirely to prevent this kind of thing from happening and behave like the old good ffms2 + Avisynth 3.7.0?

[FranceBB]

Nice one, StainlessS!
I'm reading about http://avisynth.nl/index.php/Internal_functions#propSet

Looks like it works!
So now the user is accountable to get and set the right frame properties for every encode every time.
What a bummer DX

Jokes aside, it works, this is an HDR PQ:

FFVideoSource("\\mibctvan000\Ingest\MEDIA\temp\trim.m2ts")

https://i.imgur.com/CJcJov7.png

and this is after the conversion to HLG:


FFVideoSource("\\mibctvan000\Ingest\MEDIA\temp\trim.m2ts")


ConvertYUVtoLinearRGB(Color=0,OOTF=false)
ConvertLinearRGBtoYUV(Color=0,HDRMode=1,HLGLw=1200,HLGColor=2,OOTF=false)
propset("_Transfer", 18)
propDelete("ContentLightLevelAverage")
propDelete("MasteringDisplayMaxLuminance")
propDelete("MasteringDisplayMinLuminance")
propDelete("MasteringDisplayPrimariesX")
propDelete("MasteringDisplayPrimariesY")
propDelete("MasteringDisplayWhitePointX")
propDelete("MasteringDisplayWhitePointY")
propDelete("ContentLightLevelMax")


https://i.imgur.com/T20bGUF.png


So what is cool about this is that we CAN not only make use of the properties like automatizing tonemapping by reading the MaxCLL value and set it as a variable, but we can also pass the right info to whatever we're frameserving to so that it's gonna make use of those info.

The "bummer" is that the user MUST set those correctly every time and has to do that manually...

or... just cheat and turn Avisynth 3.7.1 into Avisynth 3.7.0 with:

propClearAll()

hahahhahahahahahaha

[videoh]

I'm trying to implement the AVS+ frame properties for DGDecodeNV but getting confused. I took avisynth.h and the avs folder from pinterf's GIT. I have in GetFrame():

PVideoFrame frame = env->NewVideoFrame(vi);

Then I try to get props:

AVSMap* props = m_decoder.has_at_least_v8 ? env->getFramePropsRW(frame) : nullptr;

But there is an error because getFramePropsRW() wants an AVSFrameRef*. But I never heard of an AVSFrameRef.

Can someone please straighten me out on how to do this? Thank you.

[pinterf]

I'm trying to implement the AVS+ frame properties for DGDecodeNV but getting confused. I took avisynth.h and the avs folder from pinterf's GIT. I have in GetFrame():

PVideoFrame frame = env->NewVideoFrame(vi);

Then I try to get props:

AVSMap* props = m_decoder.has_at_least_v8 ? env->getFramePropsRW(frame) : nullptr;

But there is an error because getFramePropsRW() wants an AVSFrameRef*. But I never heard of an AVSFrameRef.

Can someone please straighten me out on how to do this? Thank you.
Use the central AvisynthPlus repo (https://github.com/AviSynth/AviSynthPlus), mine is not a fresh one.

[videoh]

Thank you, pinterf. I'll try that in the morning and report back.

There's one other matter I can work around but thought to mention. If I have a filter that supports VS and AVS, then I have to include both header files. But then they both define ptUnset etc. in enums, causing a compiler error.

[pinterf]

For this very reason the names were changed in current avs header.

[videoh]

Use the central AvisynthPlus repo (https://github.com/AviSynth/AviSynthPlus), mine is not a fresh one. That's working great. Thank you.

[Reel.Deel]

@qyot27

Would it be possible for AviSynth+ to pass a timecode file (VFR) to FFmpeg?

[tormento]

Would it be possible for AviSynth+ to pass a timecode file (VFR) to FFmpeg?
If interested, I posted a message in the AVS main development thread and in MKVToolnix one too.

AFAIK, for now, it's easier to encode as if it is a CFR and then apply a timestamps file to MKV.

[Reel.Deel]

AFAIK, for now, it's easier to encode as if it is a CFR and then apply a timestamps file to MKV.

I know how to insert timecodes into mp4/mkv files. My use case is different. Creating animated PNGs/GIFs/Webp with AviSynth/FFmpeg.

[qyot27]

@qyot27

Would it be possible for AviSynth+ to pass a timecode file (VFR) to FFmpeg?
Theoretically, the AviSynth demuxer could be adjusted to use the given values of the frame timing (I assume that's the _AbsoluteTime, _DurationNum, and _DurationDen frame properties, or whatever can be used to modify them per-frame), and iterate it over every frame instead of doing what it currently does, which is just set the framerate of the entire clip using the num/den rational (http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavformat/avisynth.c;h=350ac6d11d0d4b34482172a59a7cbdc7310d4026;hb=30322ebe3c55d0fb18bea4ae04d0fcaf1f97d27f#l246).

The logic for that would likely have to shift location significantly, probably out of avisynth_create_stream_video and into avisynth_read_packet_video, or constitute a hybrid where the current approach works in concert with whatever frame properties are set as overrides. It's largely the same reason that I didn't add the MasteringDisplay and whatnot properties - as those don't have to be static, making sure they're set right is trickier (also, AFAIK, FFmpeg doesn't set those as properties, but as side metadata, requiring a much larger amount of changes to even get the AviSynth demuxer to work with that concept to start with).

Can it be done? Yeah, probably. It's currently above my head, though (and I rarely ever have to deal with VFR as it is, so ensuring it does what it's supposed to do would be up to the community). If someone wants to figure it out and contribute, by all means, have at it.

[qyot27]

FFmpeg now supports reading two more frame properties - _SARNum and _SARDen, which are then combined into what FFmpeg reads as the full SAR value.

And because of how...tricky...that one would be when dealing with extraordinarily common resizing tasks, there is now a brand-new mechanism:
>ffmpeg -hide_banner -h demuxer=avisynth
Demuxer avisynth [AviSynth script]:
Common extensions: avs.
AviSynth demuxer AVOptions:
-avisynth_flags <flags> .D......... set flags related to reading frame properties from script (AviSynth+ v3.7.1 or higher) (default field_order+range+primaries+transfer+matrix+chroma_location)
field_order .D......... read field order
range .D......... read color range
primaries .D......... read color primaries
transfer .D......... read color transfer characteristics
matrix .D......... read matrix coefficients
chroma_location .D......... read chroma location
sar .D......... read sample aspect ratio

So using the old LG 4K HDR New York demo,
>ffmpeg -hide_banner -loglevel 40 -i test_hdr.avs
Stream #0:0: Video: rawvideo, 1 reference frame (Y3[11][10] / 0xA0B3359), yuv420p10le(tv, bt2020nc/bt2020/smpte2084, progressive, left), 3840x2160, 25 fps, 25 tbr, 25 tbn

And with -avisynth_flags:
>ffmpeg -hide_banner -loglevel 40 -avisynth_flags +sar-field_order-chroma_location -i test_hdr.avs
Stream #0:0: Video: rawvideo, 1 reference frame (Y3[11][10] / 0xA0B3359), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 25 tbn

>ffmpeg -hide_banner -loglevel 40 -avisynth_flags none+matrix -i test_hdr.avs
Stream #0:0: Video: rawvideo, 1 reference frame (Y3[11][10] / 0xA0B3359), yuv420p10le(bt2020nc/unknown/unknown), 3840x2160, 25 fps, 25 tbr, 25 tbn

>ffmpeg -hide_banner -loglevel 40 -avisynth_flags all -i test_hdr.avs
Stream #0:0: Video: rawvideo, 1 reference frame (Y3[11][10] / 0xA0B3359), yuv420p10le(tv, bt2020nc/bt2020/smpte2084, progressive, left), 3840x2160, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 25 tbn

[qyot27]

@qyot27

Would it be possible for AviSynth+ to pass a timecode file (VFR) to FFmpeg?

Nearly three years later...

$ ../ffmpeg_build/bin/ffmpeg -i test.avs -vf vfrdet -f null -
ffmpeg version N-117370-g8f3957c41c Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 13 (Ubuntu 13.2.0-23ubuntu4)
libavutil 59. 41.100 / 59. 41.100
libavcodec 61. 21.100 / 61. 21.100
libavformat 61. 9.100 / 61. 9.100
libavdevice 61. 4.100 / 61. 4.100
libavfilter 10. 6.100 / 10. 6.100
libswscale 8. 4.100 / 8. 4.100
libswresample 5. 4.100 / 5. 4.100
libpostproc 58. 4.100 / 58. 4.100
Input #0, avisynth, from 'test.avs':
Duration: 00:03:16.95, start: -0.667000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p(progressive), 512x384, 21.83 fps, 29.97 tbr, 1k tbn
Stream #0:1: Audio: pcm_f32le, 24000 Hz, stereo, flt, 1536 kb/s
[Parsed_vfrdet_0 @ 0x5a1d9609e7c0] VFR:-nan (0/0)
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> wrapped_avframe (native))
Stream #0:1 -> #0:1 (pcm_f32le (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[out_#0:1 @ 0x79f294003140] The "all_channel_counts" option is deprecated: accept all channel counts
Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf61.9.100
Stream #0:0: Video: wrapped_avframe, yuv420p(progressive), 512x384, q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn
Metadata:
encoder : Lavc61.21.100 wrapped_avframe
Stream #0:1: Audio: pcm_s16le, 24000 Hz, stereo, s16, 768 kb/s
Metadata:
encoder : Lavc61.21.100 pcm_s16le
[Parsed_vfrdet_0 @ 0x79f29c002600] VFR:0.782667 (2348/652) min: 33 max: 668 avg: 47
[out#0/null @ 0x5a1d96b50f40] video:1289KiB audio:18402KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 3001 fps=0.0 q=-0.0 Lsize=N/A time=00:02:20.37 bitrate=N/A speed= 141x


To zero in on the important bits there (https://superuser.com/a/1487417):
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p(progressive), 512x384, 21.83 fps, 29.97 tbr, 1k tbn
...
[Parsed_vfrdet_0 @ 0x79f29c002600] VFR:0.782667 (2348/652) min: 33 max: 668 avg: 47

The very experimental FFmpeg test build:
ffmpeg_N-117370-g8f3957c41c.7z (https://www.mediafire.com/file/dcz6pin3s1zavh2/ffmpeg_N-117370-g8f3957c41c.7z/file)
Because the test build is intended to test the feature, vfr mode is enabled by default, but can be turned off by -avisynth_flags -vfr.

By 'very experimental' I mean that it comes with some very obvious issues that will need addressing. These are documented in the patch's commit message (https://github.com/qyot27/FFmpeg/commit/8f3957c41c1586badf9bb27090c240e52006fee8).