how to call ffmpegsource V2 ?

[halsboss]

Had a look at the FFmpegSource2 Documentation and wasn't sure how to apply specifically for re-indexing if necessary. I wasn't clear on how to get FFVideoSource() to accept an index from ffmsindex.exe for example.

The issue: Over at http://forum.videohelp.com/topic363395.html (since the doom9 forums were down for a bit) an issue is outlined with a xvid .AVI clip. Briefly, a problem .AVI something to do with badly indexed keyframes, or more particularly non packed bitstream .AVI returning frames out of order under certain circumstances.

That's unusual and very very unhelpful when trying to use temporal filters and/or MT, as the output becomes jerky.

jagabo and poisondeathray kindly diagnosed it and mentioned that ffmpegsource to open the .avi may deal with it.

So, can anyone please clarify (a tad more than the doco) with an example if possible, how to use FFVideoSource() (on such a problem .avi) and then how to separately use the external ffmsindex.exe and feed it's result into FFVideoSource() in case I need to do that ?

Lastly, how does FFIndex() fit into that picture (ie how/when to use it) ?

Hopefully this clarification will be of use to other less technically inclined avisynth users too.

Thanks.

[halsboss]

Interestingly FFVideoSource seems to work on that source inconsistently "after a fashion". If you open the .avs in Virtualdub
loadplugin("C:\Program Files\AviSynth 2.5\plugins-zzz\FFMS2.dll")
FFVideoSource("G:\DVD\problem\zzz-problem-frames-34-35.avi")
AssumeFPS(25)
and single-step through it all the way past frames 32,33,34,35,36,37 then you'll see the "new scene" begin on frame 33 for the duration of that session.

If you close Virtualdub and open the same .avs again and then click directly on frame 42 and single-step backwards you'll see the "new scene" begin on frame 34 for the duration of that session.

So, If I knew how to apply the indexing maybe I'd get consistent results ?

[halsboss]

Worked it out. Hopefully correctly. Finally, something gives a consistent result, right or wrong FFMpegSource V2.00beta4 from http://forum.doom9.org/showthread.php?t=127037 is consistent and usable on the odd source exihibiting the aforementioned problem in post #1.
loadplugin("C:\Program Files\AviSynth 2.5\plugins-zzz\FFMS2.dll")
FFIndex("G:\DVD\problem\problem-full.avi", indexmask=7, dumpmask=0, overwrite=true)
FFVideoSource("G:\DVD\problem\problem-full.avi")
AssumeFPS(25)

The FFIndex(..., overwrite=true) forces re-creation of a full index with the same filename/extension plus appended extension ".ffindex". In this case a 500Mb xvid .avi results in only about an 8Mb ".ffindex" file. The FFVideoSource(...) seems to detect the ".ffindex" and use it. It takes about a minute for the FFIndex(...) to do it's thing which is acceptable in the scheme of things to get a reliable source for avisynth to use.

The original problem, of out-of-sequence frames being returned, goes away and you get a repeatable result whether stepping forward or backward through the frames in the non-bitstream-packed xvid .AVI.

I guess a thing to be aware of is a 2-pass HCEnc may force re-creation of a new index. If you use HCEnc, I guess the "lossless" function is handy - it's better than great for slow scripts in any case !

A HUGE thankyou to poisondeathray and jagabo without whose direction a usable solution would never have arisen.

[Myrsloik]

This shouldn't happen or work. Is it possible that you first created an index using beta 3?

[halsboss]

This shouldn't happen or work.
Oh dear, I'm sorry to read that it shouldn't happen or work. That means I must have done something wrong and may be back to square 1 with the issue. I'll check my re-encodes tonight and see if the jerkiness at scene changes, which highlights the underlying issue, has gone away. It seemed to in the initial tests.

Edit: yes the jerkiness in the result seemed to disappear implying the out-of-sequence frames issue went away

Is it possible that you first created an index using beta 3?
Unfortunately, I've only downloaded it for the first time just the other day and used it, and it was the latest V2beta4.

I did notice that all 4 cores are now being flogged during encodes whereas they used to run at about 60% (implying waiting for non-cpu resources like disk). I'll check a test cpu/encode time of avisource vs FFVideoSource over the next day or so.

[halsboss]

Results for HCEnc023 1-pass , 7500 frames identical script/source excepting the xxxSource statements

*PROFILE BEST
*AUTOGOP 15
*CLOSEDGOPS
*DC_PREC 10
*MATRIX FOX1
*BITRATE 9200
*MAXBITRATE 9400
*CQ_MAXBITRATE 2

SetMTmode(mode=5,threads=4) # start with mode=5 forAVIsource http://forum.doom9.org/showthread.php?p=1067216#post1067216
SetMemoryMax(256)
... source ...
SetMTmode(mode=2,threads=4)
ConvertToYV12(interlaced=FALSE) # for Deblock_QED_MT2 and DeHalo_alpha
Deblock_QED_MT2() #default quant1=20
ConvertToYUY2(interlaced=FALSE) # Ensure YUY2 for the rest including Convolution3D
Convolution3D(0, 3, 4, 3, 4, 2.8, 0)
lanczos4resize(resizeWidth,resizeHeight).LimitedSharpenFaster(smode=4,strength=100)
Converttoyv12()
SetPlanarLegacyAlignment(True)
Distributor() # use this when using HC and SetMTmode, per http://forum.doom9.org/showthread.php?p=1063622#post1063622

HCEnc023 results cut'n'pasted from the log (the times don't look right, but it IS cut'n'paste I assure you. The elapsed time was about 3 mins 10 secs and all 4 cores of the Q9450 belted along at 85%-100%) -

FFIndex(...,indexmask=7,dumpmask=0,overwrite=true)
FFVideoSource(...)

pass 1 encoding time: 0:03:08 (188.45 s)
fps: 39.8
nr. of gops: 555
nr. of frames: 7500
nr. of I-frames: 555
nr. of P-frames: 2666
nr. of B-frames: 4279
average bitrate: 8138
minimum bitrate: 1163
maximum bitrate: 9361
bytes in bitstream: 305254021
bits in bitstream: 2442032168
average Quantizer: 3.588
total CPU time: 0:11:20 (188.45 s)
total elapsed time: 0:03:36 (216.22 s)

AviSource(...)

pass 1 encoding time: 0:03:11 (191.09 s)
fps: 39.2
nr. of gops: 556
nr. of frames: 7500
nr. of I-frames: 556
nr. of P-frames: 2664
nr. of B-frames: 4280
average bitrate: 8143
minimum bitrate: 1163
maximum bitrate: 9361
bytes in bitstream: 305448191
bits in bitstream: 2443585528
average Quantizer: 3.586
total CPU time: 0:10:58 (191.09 s)
total elapsed time: 0:03:13 (192.75 s)


Conclusion ? Not a huge difference given the FF indexing time is probably included in the elapsed time. Well acceptable (to me, at least) if the issue in post 1 goes away.

[halsboss]

Well, it was too good to be true. It did something all funny to some frames. It DID say it was a beta.

Refer clipped source 200k http://rapidshare.com/files/193348627/clipped-original.avi and funny output from HC (recompressed to avi) 130k http://rapidshare.com/files/193346274/clipped-destroyed.avi also clipped from a full encode.

[Myrsloik]

Both links point to the same file. Is this intentional?

[halsboss]

Nope. Sorry about that. Link corrected.

[halsboss]

jagabo really helped with the mpeg2modifier recommendation.

Used mpeg4modifier to make the AVIs packed bitstream and all problems went away opening the avis with avisynth with avisource. Even an avi I had problems with which was supposedly packed bitstream I tried to unpack then pack again and hey presto it worked just fine - go figure.

Is there any other ways to open AVIs the "non-VFW" way ? I'd have thought problems with xvid AVIs and avisynth would have been more common. Seems like a bit of a "hole" in avisynth in the modern environment and I'd strongly prefer avisynth remain relevant :)

[halsboss]

mpeg4modifier of itself was a good idea while it lasted. I saw a packed bitstream xvid in .AVI and thought "that'll be OK" but it turned out to be a little odd. Unpacked with mpeg4modifier and repacked with it, and still had problems with it.

The only reliable way I found to get around it was via a combination of mpeg4modifier and the latest ffmpeg -
(1) unpack it with mpeg4modifier
(2) convert to a lossless ffv1 .AVI (ffmpeg handles xvid's b-frames in AVI OK by the looks, from non-packed-bitstream AVIs)
(3) then process the ffv1 AVI with avisynth.

Command lines :-
"C:\SOFTWARE\MPEG4Modifier\mmcl.exe" --unpack --always-write "input.avi" "input-unpacked.avi"
"C:\SOFTWARE\ffmpeg\ffmpeg.exe" -y -i "input-unpacked.avi" -f avi -vcodec ffv1 -acodec copy "input-ffv1.avi"

[halsboss]

ANSWER:
The Packed Bitstream AVI fix ONLY works if you use the latest XVID codec and not ffdshow's xvid decoder. It works then.

From the other thread http://forum.doom9.org/showthread.php?t=145262
Always use packed bitstream with avi and use xvid to decode instead of ffdshow. Unticking decoding xvid in ffdshow (both vfw and non-vfw) and installing the latest xvid makes the problem go away.

[henryho_hk]

Kinda losing track..... is ffmpegsource working or not?

[Myrsloik]

No idea but this sounds like a very odd bug. (if it is a bug)

[halsboss]

I haven't tested it since I had issues. If it's based on ffmpeg, the recent ffmpeg commandline gave a message about illegal packed bitstream when I fed a packed bitstream AVI into it, so I'm not sure. I fed the ffmpeg commandline an unpacked AVI (after mpeg4modifier) and it didn't spit, so I venture a guess ffmpegsource may be similar ?