VD2 and FFMPEG Update

[GillesH]

Simple question, please :
In the new version of VirtualDub2 (Version 20/43073), the FFMPEG Plugins/Codecs have been updated with the latest version FFmpeg 4.1 of November 6, 2018. Yes or No ?
Thanks

[shekh]

Not updated

[GillesH]

Thank you shekh for your answer and your great work on VirtualDub2.
If I asked this question, it is because we are seeing encoding anomalies in H264 with the new version of Windows 10/1809.
With the version of Windows 10/1803, there is no problem of encoding with the plugins of VD2.
Therefore, I assumed that an FFMPEG update could solve this problem.
But I'm not sure. It's just a guess.

[shekh]

Encoding H264 is handled by x264-8.vdplugin which is built around libx264, not FFMpeg. Anyway, it has very little connection with the OS, have no idea how it can cause issues.

[poisondeathray]

If I asked this question, it is because we are seeing encoding anomalies in H264 with the new version of Windows 10/1809.
With the version of Windows 10/1803, there is no problem of encoding with the plugins of VD2.

What kind of anomalies ? Can you describe them ?

Is this on the same hardware ? Or different computers ? Did you rule out system specific causes ? eg. overheating, bad memory, bad psu etc.. ?

[GillesH]

Hello,
To be more specific, you may know an application called FILM9 that I co-authored.
This application is completely free and uses the great tools that are Avisynth and VirtualDub.
Since late October, we have used VirtualDub2 (thanks to Shekh).
That said, several people have reported anomalies with the H264 encoding.
We suspect Version 1809 of W10, but without any certainty.
There may be other reasons like RAM, CPU and others ... This problem is very recent.
On my PC which is still in V1803, I have no problem.
Here. We seek to help those who encounter this problem.

For information, here is the link for FILM9 :
http://contact41766.wixsite.com/film9
Thanks for your advice

[poisondeathray]

Hello,
To be more specific, you may know an application called FILM9 that I co-authored.
This application is completely free and uses the great tools that are Avisynth and VirtualDub.
Since late October, we have used VirtualDub2 (thanks to Shekh).
That said, several people have reported anomalies with the H264 encoding.
We suspect Version 1809 of W10, but without any certainty.
There may be other reasons like RAM, CPU and others ... This problem is very recent.
On my PC which is still in V1803, I have no problem.
Here. We seek to help those who encounter this problem.

For information, here is the link for FILM9 :
http://contact41766.wixsite.com/film9
Thanks for your advice



I'm asking for a better description of " several people have reported anomalies with the H264 encoding."

Because that will help to narrow down what the underlying problem might be

eg. Is it misorderered frames ? macroblocking artifacts ? wrong colors etc... or something else entirely , such as stops encoding halfway, wrong number of frames ?

ie. What do you mean by "anomolies" ? . It's a very vague description

Does everyone report the same type of "anomolies" or different ?

[GillesH]

Hello,
I understand your questions. It's logic. I will try to be more precise.
FILM9 uses Lagarith, ProRes, FFV1 and H264 encoding capabilities.
Everything works perfectly with its Codecs EXCEPT THE H264.
With the H264, we have sometimes illogical operating disparities :

1st example: With 2 completely identical PCs (I7-2600K, 3.40Ghz, 16GB RAM)
PC1 / W10 V1803 : H264 encoding works normally. It's slow, but it work.
PC2 / W10 V1809 : crash at the beginning of the encoding (see error message). Whatever the number of pixels to process.

VirtualDub Error : Video compression error: An unknown error occured (may be corrupt data). (error code -100)

Other examples reported by the users of which we do not know the characteristics of the PC :
- with H264 encoding 1280x720 (or lower): it works
- with a H264 1920x1080 encoding : Crash at the beginning of the encoding.

Of course, we assume saturation or poor indexing of memory.
We know the limitations of 32 bits tools.

But this disparity of malfunctions, ONLY in H264, could suggest that one can improve.
But where ? Are there parameters that need to be adjusted ? Are there any priority updates to propose ? etc ...

In any case, thank you for your advice.

[shekh]

A quick hack to help with memory is to enable "large address aware" flag. You can use extra\auxsetup.exe, "Enable LAA" button.

[GillesH]

Thank you Shekh, but this LAA patch was already installed. It is very useful but not enough.

[poisondeathray]

FILM9 uses Lagarith, ProRes, FFV1 and H264 encoding capabilities.
Everything works perfectly with its Codecs EXCEPT THE H264.
With the H264, we have sometimes illogical operating disparities :

1st example: With 2 completely identical PCs (I7-2600K, 3.40Ghz, 16GB RAM)
PC1 / W10 V1803 : H264 encoding works normally. It's slow, but it work.
PC2 / W10 V1809 : crash at the beginning of the encoding (see error message). Whatever the number of pixels to process.

VirtualDub Error : Video compression error: An unknown error occured (may be corrupt data). (error code -100)

Other examples reported by the users of which we do not know the characteristics of the PC :
- with H264 encoding 1280x720 (or lower): it works
- with a H264 1920x1080 encoding : Crash at the beginning of the encoding.

Of course, we assume saturation or poor indexing of memory.
We know the limitations of 32 bits tools.

But this disparity of malfunctions, ONLY in H264, could suggest that one can improve.
But where ? Are there parameters that need to be adjusted ? Are there any priority updates to propose ? etc ...

In any case, thank you for your advice.


Very likely memory related , because the 1280x720 encode completes, but 1920x1080 fails.

h264 encoding in the commonly used configurations will utilize more memory than the others, mainly because of long GOP encoding. Many frames are held in memory at one time. You can reduce the memory consumption by adjusting some settings - shorter lookahead values, shorter GOP's (default is 250 , reduce keyint value), but you will reduce compression efficacy

prores, lagarith are I-frame only (intra encoding, single frames), and ffv1 is usually short GOP's or I-frame. As expected, they consume less memory

Or use 64bit process, or try to split up the processes

I don't know why different w10 version behaves differently. But the slowness is probably from swapping memory (pagefile)

[GillesH]

Agree with you on a memory problem.
Your idea of adjusting the GOP is interesting.
There is no direct setting in the x264-8.vdplugin interface.
But, that must be possible in the "Extra Command Line".
I do not master this possibility very well.
Could you give us an example of a weaker GOP to set.
Thank you.

[shekh]

In "Select video compression" dialog use "Force keyframes every .. frames".

[hello_hello]

I think it'd be better to reduce the maximum gop size instead, as at least then x264 can still try to place keyframes where it makes sense, such as the first frame of a scene.

Adding the following to the "extra command line" section would give you the x264 default maximum gop size (10 seconds at 25fps).

--keyint 250

You probably wouldn't want to go any lower than 24 or 25 (1 second).
The default lookahead for the "medium" speed preset is 40. For "slow" it's 50 and for "slower" it's 60. You could try restricting it to something less.

--rc-lookahead 20

[Groucho2004]

I think it'd be better to reduce the maximum gop size insteadInstead of what?

[hello_hello]

Instead of what?

See the post immediately before my previous one (#13).

[Groucho2004]

See the post immediately before my previous one (#13).What shekh recommended does exactly the same as "--keyint xxx".

So, you're basically saying that it would be better to reduce the GOP size instead of reducing the GOP size.

Also, reducing the GOP size and lookahead value was already recommended by poisondeathray.

[hello_hello]

What shekh recommended does exactly the same as "--keyint xxx".

So, you're basically saying that it would be better to reduce the GOP size instead of reducing the GOP size.

I don't use VD2 regularly so not for 1000000000000000000000000 years would I have assumed a setting labelled "force keyframes every" would set the maximum gop size rather than "force keyframes every" according to the specified interval (as though a qp file was being used for x264). The "force keyframes every" option could possibly use a rename, although maybe it works as advertised with other codecs.

The help file even implies the "force keyframes every" option isn't the same as maximum gop size.

"Many modern codecs do adaptive key frame placement and more complex data rate regulation, and place all their settings in their configuration dialog."

Also, reducing the GOP size and lookahead value was already recommended by poisondeathray.

And GillesH then asked for an example and instructions for adding them to the x264 command line, which is why shekh suggested using the "force keyframes every" option.
It seems he was allowed, or did you send shekh a PM to let him know he repeated one of poisondeathray's recommendations?

[GillesH]

Thank you for your advice to everyone.
We have done many tests, but for the moment nothing really meaningful.
We agree that the problem is related to memory management and is directly related to the GOP.
In FILM9, the H264 encoding is programmed into a VD2 ".vdscript" file.
Shekh's advice is immediately visible in the line "VirtualDub.video.SetCompression(0x34363258, X, 10000.0, "x264-8.vdplugin");"
This has an influence. Different values were tested. A value of X = 10 seems to give the best results, but it is not enough to lift some crash.
The tips for "Extra Command Line" are less obvious to implement. We did not see any impact in the line "VirtualDub.video.SetCompData(9336, "xxxxxxxx");"
It would take a specific example of a extra command line to, for example, halve the influence of the GOP and see if there is a real difference.
Thanks

[hello_hello]

That's why I can't see myself ever using Win10. It sounds like a Windows update broke something.

Maybe it'd be easier to encode in 2 steps for a while. Output a lossless intermediate file using huffyuv or Lagarith etc, then re-encode the lossless file with x264.

PC2 / W10 V1809 : crash at the beginning of the encoding (see error message). Whatever the number of pixels to process.

If it's happening as soon as you start encoding, it's possibly not a memory issue. I see film9 requires the K-Lite Mega Codec Pack. Maybe the Windows 10 update broke/changed something there?

[Groucho2004]

The "force keyframes every" option could possibly use a renameWhy? It does exactly what it says on the tin - setting a key frame every xxx frames.

[GillesH]

FILM9 is already working in 2 steps with a Lagarith intermediate file.
The encoding is, of course, in the 2nd step.
As has been said, everything works fine except with H264/FullHD encoding.
And only with some clips. Probably those who have hectic images that forces the GOP to work more.
Therefore, the impact of the GOP should be reduced to limit the memory bug.
For the problem of versions W10 : leave aside for the moment. Not everything is for the last 1809.

[hello_hello]

Why? It does exactly what it says on the tin - setting a key frame every xxx frames.

Forcing a keyframe every xxx number of frames and setting xxx as the maximum distance between keyframes aren't the same thing.

For "force keyframes every" to work exactly as the option name suggests, it'd have to also set --no-scenecut or --scenecut 0, but it doesn't.

[hello_hello]

FILM9 is already working in 2 steps with a Lagarith intermediate file.
The encoding is, of course, in the 2nd step.
As has been said, everything works fine except with H264/FullHD encoding.
And only with some clips. Probably those who have hectic images that forces the GOP to work more.
Therefore, the impact of the GOP should be reduced to limit the memory bug.
For the problem of versions W10 : leave aside for the moment. Not everything is for the last 1809.

I didn't realise that. I assumed you were editing/filtering/encoding in a single step.
To me though, that makes the problem more odd, given all you're doing at the x264 stage is re-encoding a lossless file. I run XP, so I only have 4GB RAM, with only 3.5GB being usable (due to the video card RAM) and I don't have a problem encoding 1080p, even with a bit of slow Avisynth filtering included in the process.

If that's what you're doing, do you need to use FILM9 for the x264 encoding? You could open the lossless file with most GUIs to re-encode it.

[Groucho2004]

Forcing a keyframe every xxx number of frames and setting xxx as the maximum distance between keyframes aren't the same thing.

For "force keyframes every" to work exactly as the option name suggests, it'd have to also set --no-scenecut or --scenecut 0, but it doesn't.I don't know how else to get this into your thick skull. Last try:

Clip encoded with VD2/H.264 8 bit using "Force keyframes every 39 frames":
Encoding settings : cabac=1 / ref=5 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=hex / subme=8 / psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=39 / keyint_min=3 / scenecut=40 / intra_refresh=0 / rc_lookahead=39 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00


Clip encoded with VD2/H.264 8 bit using command line option "--keyint 39":
Encoding settings : cabac=1 / ref=5 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=hex / subme=8 / psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=39 / keyint_min=3 / scenecut=40 / intra_refresh=0 / rc_lookahead=39 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00

[hello_hello]

GillesH,
Is it only VD2 running during the x264 encoding part of the process?

I just opened a video with VD2 via an Avisynth script and resized it to 1920x1080 as a quick test. It encoded with x264 fine (I only encoded 1000 frames). It converted to huffyuv in an AVI container without problems and also to huffyuv in an MKV (using VD2). Both lossless files encoded with x264's default settings without an error (using VD2).

[hello_hello]

I don't know how else to get this into your thick skull

The irony there is quite funny.

Even if you're not clever enough to understand, I haven't denied the "force keyframes every" option sets --keyint for x264. Although due to the option name I didn't realise that's what it did originally, which is why I said it could possible use a rename, but --keyint doesn't necessarily force keyframes to be an exact distance apart. Without disabling scene detection, --keyint only sets the maximum distance between keyframes.

You might end up with the same result much of the time for low keyint values, as the maximum distance will often be reached before a scene change, but you only have to encode with x264 once or twice to realise --keyint 250 doesn't give you keyframes exactly 250 frames apart.

I just opened a video I encoded yesterday with --keyint 250, picked a random keyframe and then looked at the distance to the next one. They were 135 frames apart. How would the VD2 option have forced them to be 250 frames apart instead?

[shekh]

I just opened a video I encoded yesterday with --keyint 250, picked a random keyframe and then looked at the distance to the next one. They were 135 frames apart. How would the VD2 option have forced them to be 250 frames apart instead?

Do you expect "force keyframes" option to put keyframes on exactly the specified interval (and allow extra keyframes on scene changes)?
Honestly I did not put much thinking in it. For me linking this option to "--keyint" sounds reasonable. I needed to do something with the option (either disable it or make it do something), and implementing any behavior other than --keyint is just more effort and maybe unnecessary confusion. As I understand, forcing keys at fixed frames is only good for precise cutting afterwards.

[shekh]

GillesH,

It seems you pass message about crashes from somebody else (rather than yours own).
If the person who encounters crash can isolate reproducible case, he/she should report it directly and prepare files, crash reports, logs.
This way I may find out what the problem really is and whether it is possible to improve something.

[hello_hello]

Do you expect "force keyframes" option to put keyframes on exactly the specified interval (and allow extra keyframes on scene changes)?

"Force keyframes every xxx frames" seems to imply the keyframes will always be the same distance apart.
It might work that way for other formats, but now I know it sets the maximum keyframe distance without disabling scene change detection for x264, I know what to expect. Only setting --keyint probably makes more sense most of the time.

As I mentioned previously, if the option also set "--scenecut 0" for x264, it'd work as the name seems to imply, but that's probably not too important.

Although for x264 at least, you could possibly have the best of both worlds without too much work. The current option for specifying --keyint and automatically enforcing "--scenecut 0", with an additional option to enable x264's scene detection. Or the other way around. If something like that involves too much work, I'd probably leave it as it is (or maybe have the option's name change when appropriate to indicate it sets --keyint instead).

Edit: Currently the option is greyed out when selecting a codec for which it doesn't apply, so maybe you could duplicate the option and call it --keyint, and grey out the appropriate option(s) as it'd apply to the selected codec.

Just some thoughts....

[Groucho2004]

The irony there is quite funny.

Even if you're not clever enough to understand, I haven't denied the "force keyframes every" option sets --keyint for x264.
Yes, you have:
I think it'd be better to reduce the maximum gop size instead
Instead of what?
See the post immediately before my previous one (#13).
What shekh recommended does exactly the same as "--keyint xxx".Your statement "I think it'd be better to reduce the maximum gop size instead" clearly says that they're not the same. The fact that you didn't know they were the same does not change that.

[hello_hello]

Yes, you have:

You're beginning to look silly by continuing with this, but here's what I said including the part you left out.

Even if you're not clever enough to understand, I haven't denied the "force keyframes every" option sets --keyint for x264. Although due to the option name I didn't realise that's what it did originally, which is why I said it could possible use a rename...

And after you pointed out my mistake I said:

I don't use VD2 regularly so not for 1000000000000000000000000 years would I have assumed a setting labelled "force keyframes every" would set the maximum gop size rather than "force keyframes every" according to the specified interval (as though a qp file was being used for x264).

What don't you understand about that? Does it sound like I didn't realise I was mistaken?

Ever since then you've been claiming that setting a --keyint and "forcing keyframes every xxx frames" are the same. The option might set --keyint, but as I've said several times, setting --keyint doesn't force keyframes every xxx frames as the VD2 option suggests, it just sets the maximum distance between them. Have you used x264 before?

Why? It does exactly what it says on the tin - setting a key frame every xxx frames.

It doesn't say "set a keyframe every xxx frames". It says "force a keyframe every xxx frames". You can't even quote the tin correctly.

Here's the question you ignored. I can only assume you're so intent on proving me wrong you were blinded to it, but I'll try again.

I just opened a video I encoded yesterday with --keyint 250, picked a random keyframe and then looked at the distance to the next one. They were 135 frames apart. How would the VD2 option have forced them to be 250 frames apart instead?

[StainlessS]

Stop squabbling girls, nuff is e-nuff :)

[Groucho2004]

Stop squabbling girls, nuff is e-nuff :)Agreed.

@hello*2
Time is too precious to argue with you. Your tendency to mis-quote and deflect is annoying and pathetic.

[hello_hello]

@hello*2
Time is too precious to argue with you. Your tendency to mis-quote and deflect is annoying and pathetic.

And yet two days after your last post you returned specifically to spend your time arguing again (and parrot what I said about mis-quoting). Before you leave while pretending not to notice my question for a second time, would you please give me an example of where I mis-quoted you to show you're not blatantly lying.

[GillesH]

It seems you pass message about crashes from somebody else (rather than yours own).
If the person who encounters crash can isolate reproducible case, he/she should report it directly and prepare files, crash reports, logs.
This way I may find out what the problem really is and whether it is possible to improve something.

Hi Shekh,
Yes, about FILM9, we have user feedback on some issues.
But the analysis of these problems is not always easy to identify.
Also, in general, these users do not attend Doom9.

That said, I confirm that the setting of the "keyint" has improved the x264 encoding for formats FullHD (1920x1080) which is very greedy in memory.
The crash has disappeared. This setting in the "Force keyframes every" window is convenient.
This makes it possible to enter the value in the line "VirtualDub.video.SetCompression" without difficulty.

Thanks for your advices.

[Groucho2004]

would you please give me an example of where I mis-quoted youI never wrote that you mis-quoted me. Try to read my post again before accusing me of lying.

[hello_hello]

I never wrote that you mis-quoted me. Try to read my post again before accusing me of lying.

Yeah... "tendency to mis-quote" takes the specifics out of it. Plenty of "implication" to hide behind of course (otherwise why say it?) while hoping nobody notices you're just name-calling because you've got nothing else. That's fine though. It's a big forum and there's a search function. Show me twice where I've mis-quoted anyone and I'll admit you're not a liar. Once isn't a tendency. Three barely qualifies, but I'll settle for two to make it easy..... although could you throw in an example of where I've "deflected"? I think you might be confusing your failure to understand with deflection.

Whoops.... it's been so long since you contributed to discussing the actual topic I almost forget to ask again....

I just opened a video I encoded yesterday with --keyint 250, picked a random keyframe and then looked at the distance to the next one. They were 135 frames apart. How would the VD2 option have forced them to be 250 frames apart instead?

[StainlessS]

Come on girls, stop it, tis getting a tad tedious.

[hello_hello]

StainlessS,
I count six posts over eight days not on the topic of GillesH's problem or VD2's "force keyframes every" option (related). Two posts from Groucho2004, two replies from me, and two from the armchair moderator. You must have a very low threshold for tedium. ;)