VC1Source Plugin

[Warren]

Hi guys, this is my first foray into AviSynth plugin writing and I've had some success after a little bit of a learning curve.

There is still no easy way to decode a VC-1 Elementary Stream into raw pictures without going through a long convoluted directshow chain of filters in AviSynth so I decided to make a VC1Source plugin, sort of like the MPEG2Source plugin from dgIndex.

Using some reference VC-1 decoder code that also parses the VC-1 Elementary Stream I now have a plugin that will decode pictures, albeit slowly (~4.5fps on a Core2Duo E6600). The problem i am having currently is to do with framerate, the VC-1 ES reports the FPS as 29.97 even though I know for a fact that the video is 24 FPS since it comes from a HD DVD. Now it does set a pulldown flag in one of the data structures, but I'm not sure exactly how that is supposed to work.

Does anyone have any ideas or comments?

[IanB]

Perhaps putting up some source code for comment and some reference URL's might inspire some discussion.

As a guess pulldown possible means do telecine (3:2 pulldown).

[Warren]

http://pastebin.ca/331855 is a log of some of the flags for each frame that is decoded using my plugin. Notice that every 13 frames two consecutive frames have repeat field flag set to 1 - this is where judder occurs if you view the resulting YUV I420 file. Note that the clip use is progressive, not interlaced.

[Mug Funky]

wow! i thought it'd be much longer before something like this comes up.

i'll be following this thread closely :)

btw, an AVCsource would be cool too... dshow isn't all that reliable. all in good time though - i'm just teasing.

[IanB]

So you are proposing the field pulldown sequence would be this00 11 12 23 33 44 55 56 67 77 88 99 9A AB BB CC Cd de ee ff ...i.e. 26 frames results in 66 fields. Which sort of implies the input progressive rate was 23.61275 fps.

How does Directshow handle the stream, particularly the time indexes it assigns to each frame.

[crypto]

...the VC-1 ES reports the FPS as 29.97 even though I know for a fact that the video is 24 FPS since it comes from a HD DVD

I also noticed this. The HD-DVD streams are also flagged as pulldown and interlaced. Very strange. This might also explain why Toshiba players are struggling with 24p playback as it would require a reverse pulldown.

@IanB
No, 26 frames exeed the HD-DVD spec.

[zambelli]

I also noticed this. The HD-DVD streams are also flagged as pulldown and interlaced. Very strange. This might also explain why Toshiba players are struggling with 24p playback as it would require a reverse pulldown.
AFAIK, all VC-1 encoded movie titles on HD-DVD are 23.976p with 3:2 pulldown, creating 29.97i compliant decoded streams.

[IanB]

@IanB
No, 26 frames exceed the HD-DVD spec.I doubt it, I see no limit for wierd pulldown cycle length in any HD-DVD specs.

If I had said a "repeating cycle of 13 film frames results in 33 interlaced fields", would you have been happier.

This is of course at odds with what I (and Zambelli and you) were expecting to see, i.e. a normal 3:2 pulldown with 4 film frames resulting in 10 interlaced fields.

Unfortunatly Warren is not sharing his references or code so we do the best clairvoyant interpolation of the data possible.

Maybe the stream has wierd flags, maybe Warren's code is bugged, maybe I am reading the log incorrectly. Whatever it is the logfile provides has a 13 frame cycle anomoly that I would like explained.

[crypto]

AFAIK, all VC-1 encoded movie titles on HD-DVD are 23.976p with 3:2 pulldown, creating 29.97i compliant decoded streams.

Thanks for confirming. Now to the difficult part. What parameters would you set for WME9? 29.97i ? Where does the pulldownflag come from? I did not find any setting or does interlaced imply pulldown?

I setup a HD-DVD profile, that I posted here (http://forum.doom9.org/showthread.php?p=946142#post946142) I believe now, that I have to add interlaced.

[Warren]

I doubt it, I see no limit for wierd pulldown cycle length in any HD-DVD specs.

If I had said a "repeating cycle of 13 film frames results in 33 interlaced fields", would you have been happier.

This is of course at odds with what I (and Zambelli and you) were expecting to see, i.e. a normal 3:2 pulldown with 4 film frames resulting in 10 interlaced fields.

Unfortunately Warren is not sharing his references or code so we do the best clairvoyant interpolation of the data possible.

Maybe the stream has weird flags, maybe Warren's code is bugged, maybe I am reading the log incorrectly. Whatever it is the logfile provides has a 13 frame cycle anomaly that I would like explained.

I'm using SMPTE's reference decoder so my decoder is not bugged (unless theirs is, which I somewhat doubt seeing as they made the spec). I'm currently using r6 of their decoder but am going to update my code to r7 tonight. You can grab that from http://www.smpte-vc1.org/AHG/Test/Decoder/FDS1/

[Warren]

Thanks for confirming. Now to the difficult part. What parameters would you set for WME9? 29.97i ? Where does the pulldownflag come from? I did not find any setting or does interlaced imply pulldown?

I setup a HD-DVD profile, that I posted here (http://forum.doom9.org/showthread.php?p=946142#post946142) I believe now, that I have to add interlaced.

Please stay on topic.

[Warren]

I'm using SMPTE's reference decoder so my decoder is not bugged (unless theirs is, which I somewhat doubt seeing as they made the spec). I'm currently using r6 of their decoder but am going to update my code to r7 tonight. You can grab that from http://www.smpte-vc1.org/AHG/Test/Decoder/FDS1/

It seems there is very little change from r6 to r7 and it is nothing that would affect the way the frames are coming out. I'm going to do a dump with their reference decoder.exe and compare the YUV output to that of my plugin to eliminate my plugin glue code from the list of suspects.

Update:

I just checked the YUV output of their reference decoder.exe and it produces the same judder pattern. I quite frankly don't know where to go from here....

Advice?

[Warren]

Well I found the problem and fixed it. It seems that the SMPTE Reference VC-1 decoder (up to r7 at least) has a bug in it where it will ignore the first frame in each Sequence (~13 frames) and not only the first frame of the video like it's supposed to.

My beta tester is giving the plugin a run through and if it passes I'll upload a binary here for people to enjoy.

[IanB]

Well at least thats explained :D

[Warren]

Sorry for the silence, I've been trying to figure out how to figure out how many frames are in the stream. I may have to end up scanning the entire stream - not sure.

[IanB]

I am sure Don will have much sympathy for you over this issue, hence the DGIndex component.

Iff! there are reliable timestamp packets in the file you might only need to count the frames before the first timestamp and after the last timestamp in the file.

Reverse scanning the packets from the end of file to find that last timestamp will test your metal.

Likewise to do frame accurate seeking without an index on a VBR stream. Guess a file position based on average bitrate, then binary search for a slightly early timecode then count forwards for the right frame.

For bonus points you could remember the index points as the stream plays so any backwards seeking would be fast and accurate, any forwards seeking could use the above algorithm or do a hard scan forwards if there are no timecodes.

[Warren]

I wonder if this information is contained in the .map files or if the HD DVD players implement an algorithm like you suggest.

[Neo Fagin]

For what it's worth I'm one of the people 'beta testing' this plugin as it were.

I've done a few tests, and finally tried reencoding an entire feature with the plugin. The source was the movie "The Thing" demuxed from EVO with evodemux 0.541 to an elementary vc-1 stream. This was the result:

F:\Documents and Settings\encode>f:\x264\x264.exe --pass 2 --bitrate 5305 --stats "g:\thing.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid \
--b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "F:\x264\eqm_avc_hr.cfg" \
--progress --no-psnr --no-ssim --output "g:\thing.mkv" "g:\thing.avs" --trellis 1

avis [info]: 1280x544 @ 23.98 fps (156380 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
WARNING: Hypothetical Reference Decoder Underflow i=0 F=789639
WARNING: HRD: Buffer underflow, bit count = 1398928
WARNING: Hypothetical Reference Decoder Underflow i=0 F=700690
WARNING: HRD: Buffer underflow, bit count = 884952
WARNING: Hypothetical Reference Decoder Underflow i=0 F=700689
WARNING: HRD: Buffer underflow, bit count = 898496
WARNING: Hypothetical Reference Decoder Underflow i=0 F=700689
WARNING: HRD: Buffer underflow, bit count = 1372192
WARNING: Hypothetical Reference Decoder Underflow i=0 F=799012
WARNING: HRD: Buffer underflow, bit count = 1046576
WARNING: Hypothetical Reference Decoder Underflow i=0 F=773289
WARNING: HRD: Buffer underflow, bit count = 774760
WARNING: Hypothetical Reference Decoder Underflow i=0 F=747026
WARNING: HRD: Buffer underflow, bit count = 759472
WARNING: Hypothetical Reference Decoder Underflow i=0 F=700689

[...around 200 more of the same error...]

WARNING: HRD: Buffer underflow, bit count = 1192360
WARNING: Hypothetical Reference Decoder Underflow i=0 F=1003460
WARNING: HRD: Buffer underflow, bit count = 1190344
WARNING: Hypothetical Reference Decoder Underflow i=0 F=14745600
WARNING: HRD: Buffer underflow, bit count = 89821568
x264 [info]: slice I:1036 Avg QP:11.73 size:105568a 0:00:00
x264 [info]: slice P:48614 Avg QP:11.98 size: 50272
x264 [info]: slice B:106730 Avg QP:12.78 size: 16601
x264 [info]: mb I I16..4: 48.7% 45.8% 5.5%
x264 [info]: mb P I16..4: 5.5% 11.2% 0.4% P16..4: 27.6% 19.1% 20.3% 0.2% 0.2% skip:15.5%
x264 [info]: mb B I16..4: 6.7% 1.5% 0.0% B16..8: 14.5% 2.0% 9.9% direct: 1.4% skip:63.9%
x264 [info]: 8x8 transform intra:41.8% inter:59.9%
x264 [info]: direct mvs spatial:95.5% temporal:4.5%
x264 [info]: ref P 41.6% 23.9% 11.3% 15.7% 7.5%
x264 [info]: ref B 29.0% 24.8% 14.7% 19.0% 12.5%
x264 [info]: kb/s:5304.9

encoded 156380 frames, 4.81 fps, 5305.03 kb/s

The movie encoded fine up to 17:36, after which the frame at 17:36 was duplicated ~115,000 times for the remainder.

The first pass ratefactor was 18.28 so I do not believe this problem occured during it.

Don't criticize/suggest stuff about my x264 commandline please unless you think it's the cause of the problem. ;) I've used the same commandline for about two dozen reencodes from MPEG-2 and h264 sources and yes I know some options are hardly worth enabling etc.

[Isochroma]

@Neo Fagin: remember that x264 and most other encoders seek forward/backward in the stream, probably unpredictably. Since the decoder probably isn't ready for such seeking, you'll have to decompress the entire stream to an intra- only codec (Huffyuv/Zlib etc.)

Using that method, we can test the decoder's sequential decode function. When that function is stabilized, then we can work on the seeking, which once completed will allow non intra encoders to work.

[Warren]

Hrm, I guess I should try and implement seeking in the code then - this should be fun.

[Isochroma]

Before writing new code, it might be instructive to look at what has been done with a similar case - MPEG-2 in VOB. The premiere solution in that area is DGIndex/DGMPEGDec.

It seems that for seeking, EVOB and VOB are similar; their cargo elementary streams aren't seekable until an index is built, whether the compression is MPEG-2, AVC, or VC-1.

Thus rather than duplicate work, it might be an efficient use of resources to put things that already exist together. We have some pieces of the puzzle:

1. DGIndex/DGMPEGDec: An already mature solution that provides seekable decoding of MPEG-2 in TS and VOB.

2. demux.exe (drmpeg) / EVOB demux (pelican9). These tools can demux EVOBs.

3. VC1Source plugin: decodes VC-1.

Let's re-examine the older case again: VOBs. Does anyone take a VOB and demux the MPEG-2 ES to a file, then take that file and decode it? Not many, if any: people index it and then use mpeg2source() in their AVS. Demuxing first just wastes time and doubles storage requirements, and is totally unnecessary for MPEG-2 frameserving.

So why should EVOB be any different? Indexing will be required in any case, because many filters require seeking, and all inter-intra codecs require it. So there are really only two options: build an external indexing app from scratch, or...

neuron2, pelican9, drmpeg, and Warren get together and add EVOB demux, index and decode functionality to the DGtools, which saves maximum effort (framework already exists), and provides a single well-known interface to all VOB decoding. The small matter of AVC decode can be solved by integrating already publicly-available code from ffmpeg/ffdshow.

The pieces of the puzzle are already mostly complete, it is only a matter of fitting them together. I've every confidence this can be done with moderate effort and reasonable cooperation.

[IanB]

@Isochroma,

You have many good points, I just don't quite agree with jumping on the index pass bandwagon yet!.

Having to do an index pass is a bit of a pain and solutions that lessens that requirment I would consider a plus. One downside to index pass solutions is dealing with live streams.

Maybe in the end an index pass will be the most practicle but lets not stiffle any of the unborn brilliant ideas just yet.

[veffremov]

Thanks for confirming. Now to the difficult part. What parameters would you set for WME9?

It should come as no surprise, that HD-DVD titles are encoded using a different encoder, than WME9. I'm not sure WME9, has the capability to insert repear field flags in the bitstream.

However, if you are fammiliar with the bitstream syntax (DESCRIBED in teh SMPTE docs), you can encode as progressive 23.97 and post process the bitstream to insert the repeat field flags to form a propper 3:2 pattern.

[Isochroma]

@IanB: Indexing will be required; remember, EVOB is no different from VOB, there are no shortcuts or tricks which will work here and not with VOB.

There is no way to get around the indexing requirement, the raw streams are just as unseekable as MPEG-2 streams in VOB.

Half-baked solutions will probably work in some cases but not all, causing endless problems for users with missing or incorrect frames, depending on the filters in their script and codecs used for compression. This will result in a nightmarish avalanche of people complaining about various problems, all caused by simply not doing the job properly in the first place.

Now is the time to ensure that a proper solution is in place, so that further development can rest on a firm foundation.

For example, with a correctly working indexer and file format like .d2v, it is possible to test the decoder's correct functionality. Without an index, frames must be decompressed to raw or non-delta compressor before comparing, wasting much time and storage space.

[Warren]

Why exactly does the stream need to be seekable anyway? Avisynth has an internal frame buffer that will store a few back frames for plugins that require it, but if you're just transcoding with a few filters in between what needs to be able to seek?

I'm not interested in making a plugin so people can do NLE, watch a movie or seek around willy-nilly to look at things. If there is a valid, concrete need for seeking that would require indexing I'll look at adding it but if there isn't I won't be even touching it. The source code for dgindex/mpeg2source is a bloody mess and I don't want to try sifting through it if I can avoid it.

[Isochroma]

I can assure you that my suppositions are not trite.

Encoders may and often do request many frames backwards or forwards, depending on how many b-frames. That is what is important. Various filters can request frames too, and for example my motion-compensated denoising script usually pulls 6 back and 6 forward. The forward frames aren't cached yet, and some of the backward ones won't be because of RAM and setmemorymax() limits.

You can't count on AviSynth's caching feature to cover up inaccurate frame seeking. For example, I use setmemorymax() to limit the cache to something small in my scripts. Others may not have sufficient physical RAM.

For many reasons, it is not sound programming practice to rely on AviSynth's cache to cover up accuracy issues. There is no easy way out of this short of full indexing, or it would have been done already with MPEG-2 in VOB, which has the same issue.

[Mug Funky]

maybe indexing per GOP could make things quicker. so long as the decoder knows where each frame request will take it and what GOP that will land in, all should be well (shouldn't it?). i don't know nearly enough about these things to be able to speak with authority on this subject...

fwiw, seekability is very important do me as well... i use a lot of conditional filters with multiple clips switched between each other. that means seeking over a possibly quite long range of frames (it might take 1000 frames from 1 clip, then switch to the alternate one).

there's also filters like smartdecimate and tdeint+eedi that take multiple clips as arguments. these are very commonly used.

of course, if frame-accurate seeking is possible without an index, that'd be good too.

[foxyshadis]

Encoders like x264 and xvid have frame-buffers, as does anything through vfw; they don't seek within the actual input stream.

There is mpegsource, and of course directshowsource, and both do tend to fall apart for mpeg ps anytime even moderately complex scripts get introduced as single-frame seek errors are magnified by extra processing. To say nothing of trying to get scripts with trims to co-operate. Index the input or output to lossless, either way you have to do extra to make up for a lack of indexing.

[Eastermeyer]

Some news on this ?

[Warren]

Sadly, no. I'm in the middle of a move right now and haven't had time to work on the plugin.

Right now I'm unsure about where to go from the current build. I could implement seeking but people are getting some underrun errors in the reference decoder with retail HD DVD VC-1 video and the reference decoder is quite slow. I looked into using FFMPEG's VC-1 decoder but it's not quite ready for primetime yet - it artifacts a lot.

Any thoughts?

[Mug Funky]

you could always work on the seeking and parsing part while waiting for the kinks to be ironed out of the decoder side by someone else. looking at the stunning success of AACS :), there'll be plenty of material for the ffmpeg guys to work on, so it's just a matter of time before their VC-1 decoder is ready for public consumption.

or you could get settled in at your new place and chill for a while first :)

[Guest]

The source code for dgindex/mpeg2source is a bloody mess and I don't want to try sifting through it if I can avoid it. That's not true. Yes, there are more globals than some people like in DGIndex, but it's certainly not a "bloody mess". And DGDecode is very clean and without globals at all (to support multiple instantiation). I have been able to enhance and extend the DGIndex source code for a long time now, and I've been commenting it as I go, so it's now better commented than lots of other open source code. It's as complex as it needs to be to do what it does. I see it as low class to try to excuse your unwillingness to confront a difficult problem by insulting someone else's efforts.

Furthermore, I have been able to use it as a model to implement H.264 decoding, including both non-frame-accurate seeking as in DGIndex, and frame-accurate seeking as in DGDecode. The basic seeking model is just fine and could be usable for VC1 as well. It can be understood easily without looking at any specific source code, so your excuse is also a red herring.

Lots of Avisynth filters require frame-accurate seeking. Decimate() is a prime example. You can't rely on the Avisynth cache, because not only is it limited, but it requires that you have already decoded linearly the frames that you seek, and that is often not the case. There's no point to an Avisynth source filter that doesn't implement frame-accurate seeking. GetFrame() is the name of the game!

[infernix]

I too have been testing this VC1Source plugin and I've only been able to fully decode one (1) EVO stream correctly. All others fail, ending up in a frameloop of 2 or 3 frames, starting somewhere around 30% for the rest of the stream.

FATAL ERROR:
FATAL ERROR: DecodeACRunLevel: AC coef array overrun
WARNING: Buffer not emptied: 1404186 bits left after 7196 frames decoded, 7196 frames output
WARNING: Hypothetical Reference Decoder Underflow i=0 F=9485229
WARNING: HRD: Buffer underflow, bit count = 89415520


This was on linux, with avs2yuv feeding to two mencoders through FIFOS for an x2641pass and a ffvhuff encode.

I have tried triticals memory usage friendly avisynth build as well, but to no avail. It's a real shame, since ffvc1 isn't ready for primetime yet and this is the only alternative on Linux.

[Warren]

Tried to add some more debugging to the plugin to try and fix these buffer issues but I'm having issues with avisynth & vc1Source on Vista - it seems the plugin can't open files. Guess I'm going to have to install WindowsXP inside Parallels to test this.

[Gannjunior]

Hello,
what about the seeking problem on hddvd vc1 source?

i'm trying to do a backup of my hdddvd and i'm frameserving vc1 video stream by graphtedit and directshowsource method.
tried with something like this:

DirectShowSource("G:\yourvideo.grf",fps=23.976,framecount=xxxxxx,seekzero=false,seek=true,audio=false)
and also a script without seek command...

i did an encode at low resolution (40 minutes) to have an avi file to use to observe and evaluate the video scenes (otherwise it is impossibile to observe the video by avs, 'cause it is too slow) to get a new avs script optimized by trim (choosing best resize and filter according to scenes typology...).

I obtained something like 12-15 trim fragments and i decide to encode in megui only one of them to do a test of my h264 codec setting and i realized that the resulting encoding is not the trim i forced in avs...

i hope i've had been clear...any idea?

thank you!
ciao

[IanB]

Well it looks like you are getting a taste of "NOT frame exact seeking".

Try this test script :-A=DirectShowSource("G:\yourvideo.grf",fps=23.976,framecount=xxxxxx,audio=false)
B=AviSource("G:\yourvideo.avi")
StackHorizontal(A, B)Load it into VirtualDub and scrub around the time line.

Oh dear the left and right picture which should always match apparently do not.

[Gannjunior]

hi Ian,

i tried the script. and, if I, for example, go to frame 151000 (deeply in film) the xvid encoded file (B) matches correctly but the A (directshowsource) goes wrong...i underline that the A source is the script that i used to encode (correctly) B...
(ah..of course the seeking to go to 151000 frame is very slow - 6/8 minutes).

[IanB]

Unfortunatly, until someone writes a frame exact VC1 reader you will need to linearly transcode to a lossless, seekable, intermediate format.