Decimate Speed Increase in 2nd pass?

[zettai]

I apologise if this idea has been suggested before but I haven't seen it discussed elsewhere.

I was thinking about the processing that decimate does and I was wondering 'why do it twice?'. Lots of us do 2-pass encodes using decimate and it seemed unnecessary to me to work out the frames for decimation in both passes.

Would it be possible to add a logging feature to decimate where during the first pass you could have say Decimate(logtofile="C:/decimate.log") and then in the second pass have Decimate(log="C:/decimate.log")or something similar.

Of course, I dont know what speed increase this could offer but it might be of some value to those doing 2 pass encodes.

For me, I know this could also have other uses. I often do video editing from avisynth with decomb and it would be great if I could do a dummy 1st pass to speed up the decimation when editing.

Also, I was wondering if the same principle could be added to other frame detection filters such as copysame et al.

Feel free to discuss, flame, etc...

[sh0dan]

Sounds reasonable. Donald should just make a few checks to see if the frames are the same, ie. store framecount, size and a perframe CRC - if any differ, just recalculate these frames.

Thinking of it, perhaps avisynth should have an internal int getCRC(PClip clip, int frame) - that way it could be used, if other filters should require the same functionality.

[Guest]

Well, it's not just Decimate(). It's also Telecide(). I suppose something could be done. This has been discussed before. I'm afraid it's just not a high priority for me because I never do 2-pass encoding. Horrors!

However, if enough people start badgering me, probably something could be done. But first let me put the new-generation field matching and duplicate detection in place.

Seems to me calculating CRCs would negate the gains. Better to just have an explicit two-pass mode, where I store the decisions made in a file and just read back from the file during the second pass. For Telecide() I'd still have to reassemble the fields but I'd save the matching analysis. For Decimate, I'd just store the frames to delete and save almost all of the work.

[meleth]

Consider yourself badgered :P

[Guest]

How would you expect the 2-pass thing to work? How about if there is a single parameter, such as "use logging". During the first pass, the log file would not be present and normal analysis would be done and the log file created. Then during the second pass Decomb would see the log file and use it.

Now I'm wondering if reading/writing the log file will be slower than redoing the analysis!

[zettai]

@neuron2

Yes, for that reasion you would have to have a user defined option such as log=1 for write mode (1st pass) and log=2 for read mode (2nd). Decomb could possibly detect the source to see if there's a log file for it but I don't like that idea - it should be manual.

Of course in the end quality is better than speed - so you keep working on wonderful algorithms, this is low priority - it's just an idea :)

[meleth]

Automatically detection would be prefered and i guess that could be solved in a number of ways.

If it could look up the bitrate currently used it could probably use that(I only use nandub), or i could always live with a tempfile/registry entry indicating if it's on the first or second pass.

[zettai]

I can't see automatic detection ever being reliable. What if you stop the 1st pass half way etc? I think theres just too many issues with an automatic detection of a log file.

As for the "will it be quicker" issue, I don't know as I dont know the exact procedure of the filter. I just dont have the technical experience, so if anyone else can make claims for or against then that's fine, it's just an idea.

[meleth]

Well if it can access the bitrate that should be good enough, atleast when using nandub. As for automatic detection of a log file it's quite simple.

If !file
firstpass()
createfile()
else
rmfile()
secondpass()

This would work good enough for the automatic (default behavior) ofcourse making it possible to override it with a parameter shouldn't isn't that big of deal either.

[zettai]

Originally posted by meleth
Well if it can access the bitrate that should be good enough, atleast when using nandub.

yeah, but are presuming that everyone uses the same method for encoding things, which is very very restrictive.

I really dont think any automatic method would be better than a manual method.

here's an example - I make a script, load it into vdub to have a look at it and as vdub acesses it a log file is made for those frames. I then come back to it later, start a first pass and decomb uses the log file which only contains information for a few frames. It's just too buggy - a system that would work for one encoding method could easily fall down for another. Manual is much better and really isnt that much of a hassle.

The only thing that would really need an automatic method is an auto 2-pass program but that could easily just be programmed rewrite the avs file for a 2nd pass anyway, so I don't see the problem.

[meleth]

Well I did say that making it manually possible to override the automatic detection should be possible. All that should really need is a simple if statement.

Also as I mentioned below it shouldn't create the file/flag/whatever indicating it should do the second pass until the first pass has been completed. I assume here that there is a way for decomb to determine that the encoding is complete, either by a framecount or if avisynth does something when it's done that can be used.

[int 21h]

What if I'm encoding an avs in TMPGEnc or CCE, how would you detect the bitrate there?

Also, what if I just load up the avs in Virtualdub and scroll through it?

What if I play the avs in mplayer2?

How would the 'detection' work then? I think you'd need to ditch the detection idea and simply rely on a quick frame index at the beginning of the log file (i.e. X total frames), then a quick get frames at the initialization of Decomb. But even then, what if I did a first pass over an entire file, and then wanted to encode just a section, shouldn't it be smart enough to seek in the log?

As you can see, there are some issues that would have to be conquered to implement this... and we don't even know what kind of yield it would generate, it may only save 1fps every 6000 seconds. This needs to be researched more to find out if its worth it, I encourage you to run some tests (Yes, you, everyone that wants this idea), to try and figure out what sort of speed increase is likely. As I mentioned in the other thread, time permitting, I will attempt to analyze it with Vtune, and get back to you.

[sh0dan]

@neuron:
CRC32 or CRC16 is _very_ fast, and would eliminate checks for first pass/second pass - if any CRC fails - ignore the file and recalculate - framecount is probably almost just as good, but not as safe (if a filter is added, etc).

Found this code, that also reads from a file.


while(bSuccess && dwBytesRead)
{
// Register use:
// eax - CRC32 value
// ebx - a lot of things
// ecx - CRC32 value
// edx - address of end of buffer
// esi - address of start of buffer
// edi - CRC32 table
__asm {
// Save the esi and edi registers
push esi
push edi

mov eax, dwCrc32 // Load the pointer to dwCrc32
mov ecx, [eax] // Dereference the pointer to load dwCrc32

mov edi, ptrCrc32Table // Load the CRC32 table

lea esi, buffer // Load buffer
mov ebx, dwBytesRead // Load dwBytesRead
lea edx, [esi + ebx] // Calculate the end of the buffer

crc32loop:
xor eax, eax // Clear the eax register
mov bl, byte ptr [esi] // Load the current source byte

mov al, cl // Copy crc value into eax
inc esi // Advance the source pointer

xor al, bl // Create the index into the CRC32 table
shr ecx, 8

mov ebx, [edi + eax * 4] // Get the value out of the table
xor ecx, ebx // xor with the current byte

cmp edx, esi // Have we reached the end of the buffer?
jne crc32loop

// Restore the edi and esi registers
pop edi
pop esi

mov eax, dwCrc32 // Load the pointer to dwCrc32
mov [eax], ecx // Write the result
}
bSuccess = ReadFile(hFile, buffer, sizeof(buffer), &dwBytesRead, NULL);
}


get the rest from this site (http://www.codeguru.com/algorithms/crc32.html)

[int 21h]

Downloaded the 3.9 source, and compiled to generate the Program Database (needed for symbol lookup), then benchmarked through Vtune 5.0. No surprises really. Made a script with Telecide(), Decimate(cycle=5), and ran through a chapter of a vob in Virtualdub via Mpeg2dec.dll.

Raw results are here: http://phreak404.virtualave.net/Decomb.pdf

60% of Decomb's total execution time was spent in Deinterlace. You can figure out any % you want to by dividing by 3853 (That is the total number of samples taken during the 4 minutes it took to encode the vob)

IMO, adding this would not at all speed up total execution.

[Guest]

Phwew. :) Thanks for the results, int 21h!