How to get I-frames on scene change points (good scene change detection)?

[edward_andrew]

Hello!
I'm remastering some old DV footage in Virtual Dub, deinterlacing with QTGMC and encoding with x264 vfw v.2.00 (output to MP4 file) / external encoder x264 CLI r3081 . Video resolution 720x576, 50 fps, multiple shots of 10-30 seconds length (completely different scene every shot) in the same video file.
Without thinking or reading enough documentation I used the recommended parameters for x264 from an old post, like bframes 16, keyint 500 (10x framerate). After finishing the project I had the surprise that all the I-frames were placed at the Keyint interval, ignoring completely the scene change (verified in Avidemux).
After digging in old Doom 9 posts like:
http://forum.doom9.org/showthread.php?t=146955
http://forum.doom9.org/showthread.php?t=144207
I found a code section posted by Akapenguin which described a mathematical relationship between scenecut, keyint, distance to the previous I frame and P/I ratio.
By chance I found that lowering the max consecutive B-frames (from 16) I could get I-frames exactly at scene change points with default scenecut value. I set min-keyint to 1 to prevent any interference from there.
After much experimentation on a segment with 7 different scenes I found that:
1.For Keyint 500
a) only B-frames 0 and 1 worked as desired with scenecut 40
b) B-frames 3 scenecut 9-11 will get al hard edits
c) B-frames 4 scenecut 7-10 will miss only 1 hard edit
2.For Keyint infinite
a) B-frames 3 scenecut must be 9-32
b) B-frames 4 scenecut must be 9-23
Increasing or decreasing scenecut value outside that range made the encoder to miss progressively more scene changes until all 7 were missed.
I tried to change rc-lookahead (to 500), me, ref, but it seems that those parameters does not influence the I-frames placement.
I would like to use smaller keyint, without ruining the placement of I-frames, as some GOPs reach over 12 seconds. I suppose B-frames 3 or 4 are enough.
Beside B-frames, min-keyint, keyint and scenecut, what other tweakable parameters may influence the placement of I-frames at scene change points?

My actual encoding settings (CLI):
--crf 15 --demuxer raw --input-res %(width)x%(height) --fps %(fpsnum)/%(fpsden) --preset slower --bframes 3 --scenecut 21 --no-deblock --subme 10 --psy-rd 1.00:0.15 --merange 32 --chroma-qp-offset -3 --keyint infinite --min-keyint 1 --no-mbtree --qpmax 22 --pbratio 1.20 --no-dct-decimate --no-fast-pskip --opencl -o "%(tempvideofile)" -

[ChaosKing]

A poor mans version: encode every scene individually and then mux everything together. There are scene detection plugins for avisynth / vapoursynth. Not sure if it can be achieved with x264 alone. FFMpeg could maybe solve this too.

[Boulder]

x265 is as bad as x264 at finding scene changes (same code), so I've been using StainlessS' DBSC script to get a QP file to feed to x265. It should also work for x264, I think the format is the same.

https://forum.doom9.org/showthread.php?t=171624

[benwaggoner]

It wouldn't be surprising since if practical min-keyint=bframes due to the b-frame lookehead. You might see if lowering --rc-lookahead to below bframes also improves placement.

I know x265 will use non-IDR I-frames in mid-GOP scene changes; can't remember if that is enabled in x264. If not, it would be a good feature to backport.

If you know where you want IDR and i-frames, you can always specify those in a qpfile.

There are other reasons why very small GOPs may not be optimal. There's overhead with every IDR. It definitely is problematic if doing adaptive streaming, for example, as there is also overhead with every fragment request. Default --min-keyint is higher for good reason.

[benwaggoner]

x265 is as bad as x264 at finding scene changes (same code), so I've been using StainlessS' DBSC script to get a QP file to feed to x265. It should also work for x264, I think the format is the same.

https://forum.doom9.org/showthread.php?t=171624
There are some new scene cut algorithms in x265, although reports have been mixed at best.

Bear in mind that "scene cut" detection isn't actually optimized for detecting hard edits. It is to find the lowest-distortion place to insert an IDR so that there is always and only one between min-keyint and keyint.

They don't always place them where we'd want them for navigation. And that can vary per scenario. Should the keyframe come at the all-black frame before a fade from black, or at the first full frame after the frame ended? Different optimal choices for different applications.

[Boulder]

Wouldn't it be silly (quality wise) to not add an I-frame where there is a hard edit and all references would finish anyway? I've seen the encoders miss some very clear ones, and yes, the new methods in x265 are worse as they are more like alpha versions of what could be.

[edward_andrew]

Thank you all for your suggestions.
I'll first try to adjust min-keyint = bframes, rc-lookahead < bframes and post the results. This might fix wrong I-frame placement with flash and strobing.
The encoding of every scene separately is a pain, with more than 120 scenes per tape, but clearly the safest. I'll keep it as a last resort.
Scene detection Vapoursynth plugins must be tweaked too, depending on the source, like x264.
I don't know how to use a qpfile, and if I set the I-frames myself I could as well encode each scene separately ( I have to find visually the scene change anyway).

Wouldn't it be silly (quality wise) to not add an I-frame where there is a hard edit and all references would finish anyway?

I was wondering about this too, only I didn't know to name the scene change "hard edit".

Conclusion:
Increasing the min-keyint from 1 to [B-frames] (in my case 4) had no effect on I-frames placement on hard edit.
For Keyint infinite, 500 and 250 and B-frames 4 decreasing rc-lookahead to 4 and lower had no effect on I-frame placement, the useful Scenecut interval remained the same as with default value of rc-lookahead.
For Keyint 250:
a)B-frames 3 scenecut 8-9 get all hard edits
b)B-frames 4 scenecut 8-11 will miss only 1 hard edit
B-frames 3 or lower combined with keyint infinite (or at least 500 ) seems the best option for me. I have to test the choice on an entire tape.

[poisondeathray]

I don't know how to use a qpfile

It's text file with framenumber and it's frametype and quantizer . Use capital "I" for IDR frame

https://en.wikibooks.org/wiki/MeGUI/x264_Settings#qpfile

[edward_andrew]

I encode in CRF mode. To obtain a qpfile I suppose that I have to run a first pass to analyze the video. Then I edit the qpfile marking the desired hard edit points as I-frame. CRF mode will use my modified qpfile?

[Boulder]

Nope, the qpfile needs to be created manually, or by that utility script I pointed to, and it can be used in a multipass encode or CRF mode encode. It just tells the encoder which frame type decisions you want to override.

As you already use Avisynth, it's fairly simple to setup that monster of a script and get at least most of the scenecuts right. The script can be opened for example in VirtualDub to verify how it has worked through the source.

[Boulder]

I believe this would already do. For this detection part, you can use a fast bobber like YADIFMod2(mode=3) instead of QTGMC.

Open in VDub, and once it finishes, the working directory should contain a qpfile and you can check the scene changes visually.

This version of DBSC.avsi should work out of the box, I had to fix it a bit compared to the version in that old thread. Just save it with an .avsi extension to your plugins folder and make sure you have the dependencies mentioned in the file as well.
https://pastebin.com/y682ekiW



#LOAD SOURCE HERE

VideoFileName = "draft" # so the qpfile will be draft.qp.txt
InfoDB = VideoFileName+"_Info.DB" # INFO DBase
ScanDB = VideoFileName+"_Scan.DB" # SCAN DBase
RECYCLE_INFO_DB= True
RECYCLE_SCAN_DB= True

DBSC_CreateInfo(InfoDB,RecycleDB=RECYCLE_INFO_DB) # Create INFO DB (REQUIRED), for fast scanning
DBSC_Scan(InfoDB,ScanDB,RecycleDB=RECYCLE_SCAN_DB) # Detect Scene Changes and store in ScanDB

DBSC_WriteQPFile(ScanDB,VideoFileName+".qp.txt")

Return DBSC_StackSceneChanges(Last,InfoDB,ScanDB) # Check detections START/END of scene STACKED.

[benwaggoner]

Wouldn't it be silly (quality wise) to not add an I-frame where there is a hard edit and all references would finish anyway? I've seen the encoders miss some very clear ones, and yes, the new methods in x265 are worse as they are more like alpha versions of what could be.
There may be valid references from before a hard edit that can apply later. Motion search isn't object tracking, but finding matches where lower bitrates can be found in inter prediction than intra coding. Some surprising vectors can turn out useful.

But in general yes, if min-keyint is preventing an IDR frame from being used, a non-IDR I frame would be used.

[edward_andrew]

@Boulder
Thank you for the script, I'll test it (VapourSynth).


Edit:

The formula b-frames 3, min-keyint 1, keyint infinite, scenecut 21 (average of 9-32) worked acceptably on an entire tape of 1 hour. I had only 4 missed hard edits out of 112, 2 obvious hard edits without any reason and 2 hard edits in places where there was a very short interval between hard edits, of 3-10 frames, and a lot of motion.
It's still unclear to me the behavior of Scenecut parameter. From my experiments I saw that the lower the value, the better the detection and smaller file size, correlated with a number of B-frames of 3 or less.

[edward_andrew]

Motion search isn't object tracking, but finding matches where lower bitrates can be found in inter prediction than intra coding.

About motion search and scene change - I used a lot Deshaker made by Gunnar Thalin. During the first pass Deshaker used motion analysis and scene detection. Every scene change was marked as n_scene in the log file.

@Boulder - in the qpfile is enough to specify only the frames that I want to be I-frames, and nothing else?
Say
0 I 15
136 I 15
167 I 15
195 I 15
...............
is enough? An let x264 decide what to do with the rest of the frames. What is the mathematical relation between the quantizer for IDR frames and CRF value ?
I noticed that during the editing I can write down the scene change points manuallly.

[Boulder]

@Boulder - in the qpfile is enough to specify only the frames that I want to be I-frames, and nothing else?
Say
0 I 15
136 I 15
167 I 15
195 I 15
...............
is enough? An let x264 decide what to do with the rest of the frames. What is the mathematical relation between the quantizer for IDR frames and CRF value ?
I noticed that during the editing I can write down the scene change points manuallly.
You can leave the actual QP value out, so it's just the frame number and 'I'. The encoder will then figure out the QP as if it had made the frame type decision itself.

[edward_andrew]

The qpfile method worked fine with x264 CLI. x264 wfv does not accept --qpfile argument. Overriding the I-frame placement increased the bitrate a little, but from all that I tried is the best. No missed scene change.
Virtual Dub can export to file all edits, like "AddRange([frame],[number of frames]) . With a Python script I can convert all edit points to qpfile format.
For the quantizer on chaneru.com is recommended -1 to let x264 decide.

[benwaggoner]

Are we still using Video For Windows apps? I was at the event Microsoft announced DirectShow to deprecate VfW. 26 years ago! They also promised a Mac version too.

VfW wasn't even out for years before it was deprecated. But yet, it lives on, somehow.

tl;dr can you switch to a non-VfW workflow?

[jpsdr]

Also, i don't know where you can find qpfile spec, but if i remember properly, there is a difference between I and i in the qpfile. From what i've understood, use I only on frames where you want to put chapters, use i on the others, it will be more efficient for the bitrate. (But correct me if i understood unproperly).
You can also don't put the -1, it's the default setting.

[Selur]

--qpfile <string> Force frametypes and QPs for some or all frames
Format of each line: framenumber frametype QP
QP is optional (none lets x264 choose). Frametypes: I,i,K,P,B,b.
K=<I or i> depending on open-gop setting
QPs are restricted by qpmin/qpmax. source: x264 --fullhelp

[benwaggoner]

I forgot about K. That's probably the best default to use unless doing something specific. Adaptive streaming, where the desired minimum GOP duration is generally at least two seconds, could benefit from using "I" or "i" as appropriate.

The .csv logs from x265 give a scenecut rating for each frame indicating how appropriate for an IDR each is, which could be quite helpful in making these determinations. It also includes luma min/mean/max for each frame, so one could figure out what is the first or last frame of a transition and make that an IDR or I.

I know that .csv logging got contributed to x264 some years ago, but it didn't make it into the mainline and I don't recall seeing any build that uses it.

[edward_andrew]

tl;dr can you switch to a non-VfW workflow?

Old habits die hard.
The call to load the viddeo in Vapour Synth is DirectShowSource(). I don't know how to replace that and the QTGMC deinterlacer, with similar image quality.
To edit DV recordings I used a lot Virtual Dub. I tried to switch to AviDemux but it did not display the frame number, I could not adjust the size of the video preview, and I had a problem with the color conversion from RGB to YUV that I could not fix (its x264 encoder lacked the option to Keep Input Colorspace, and AviDemux lacked the Color Depth option).
In Virtual Dub I could set the color depth for input and output to YV12 and then no more color space conversions and wrong colors.
Good to know that is no need to specify the QP at all in the qpfile, less hassle.
I do not use open GOP, sometimes I need to extract certain scenes later after final encode.