If I add a fprintf in the encoding source....

[Wbtz]

Hello!

I am currently exploring XVID for my project. Being new in the area of encoders and decoders I would like to ask a question: Let's say I would like to edit the bitstreamputbits() function to write the bitstream to a file. The thing I am not sure about is if this were possible. That is because, if virtual dub were to use xvid to encode a movie, will the fprintf function work along side it? (This means during encoding, a file which i specified will be written as well with the bits I want).

Not sure if i made myself clear!

[Nic]

Yes it would if you add everything correctly & make sure to close the file at the end.

Wouldnt fwrite be better than fprintf if your writing out a bitstream? :)

Cheers,
-Nic

[Wbtz]

Hey Thanks Nic!

I am not proficient enough in C. fprintf only writes a text file? fwrite writes a binary file i suppose?

Anyway, how do you guys test the source code once u edited the code? Meaning, if i edited the code, do i install the xvid.inf file again? It will overwrite the old one? And if i want to have different test versions?

static void __inline
BitstreamPutBit(Bitstream * const bs,
const uint32_t bit)
{
if (bit)
bs->buf |= (0x80000000 >> bs->pos);

BitstreamForward(bs, 1);
}

I suppose i should open a file in the if(bit), do i need to follow the way it writes to the buffer? I dun get the code above..can someone explain it to me? Thanks!

[-h]

If all you want to do is capture the output of the encoder, why not edit the vfw code instead? All xvidcore does is write all its output to a buffer supplied by vfw, then vfw returns that buffer to the calling application (VDub then writes it to an AVI file).

If you wrote out the contents of frame.bitstream (the length of which is frame.length) in vfw/src/codec.c -> compress() you'd achieve the same thing, and it'd work a lot faster than individual writes for every bitstring. And it'd save you a lot of trouble :)

-h

[Wbtz]

Editing the VFW's code is pretty new. So i will just fwrite the frame.bitstream from the compress() function of codec.c ? And if i wanted to write lets say 1 megabyte to each file (A 10 meg file produces 10 files etc). Is that still possible with VFW?

If it were possible. How then can i decode? Is it with VFW too?

[-h]

What is the end goal of this project?

There are probably easy ways to solve the entire problem, instead of creating extra work at every step.

-h

[Wbtz]

The end goal is pretty experimental like.

The idea is to speed up streaming. Just like how download accelerator
and other download managers do it. Segment the file and download from different servers. Similarly, I want to apply it to video but perhaps on a frame by frame basis. So 10 consecutive frames are each stored in separate files. So file 1 has frame 1, 11 , 21 etc etc.

That is my main purpose. But the thing is all frames have different sizes. Not sure if i can write each frame to each file. What do u suggest? Streaming function will be implement in the decoder( i suppose) and will just get the 10 different files from the 10 different servers.

[Wbtz]

I have got another question. If I were to write the bitstream from bitstream.h in the bitstreamputbit(), will this binary file of bitstream be playable in windows media player?

Or do I have to do it in the VFW's source?

[-h]

Hm. It sounds like you just want a streaming server capable of load balancing between multiple servers.

Or alternately, the decoder would (presumably) keep track of how many bytes of input it has read. It could then use traditional multi-source download techniques to get the next 100 KB from one server, another 100 KB from another server and so on. So long as no gaps appeared in the reconstructed file (i.e. one of the connections timed out and the code wasn't smart enough to try another mirror), it could be decoded linearly without any hiccups.

The problem with relying on the raw encoder output is that it is just that - raw output. It needs a container to be useful, such as AVI or MP4 or OGM. If you just have a raw bitstream, you don't know how many bytes each frame used, how many frames you have, or other useful information.

It will be a headache to trap the bitstream calls from xvidcore - you'd have to trap both BitstreamPutBit() and BitstreamPutBits(), then duplicate their functionality and buffers and write to a separate file. It would be much easier to do from within vfw, since it receives a buffer filled with the current frame's data, the data's length in bytes, the keyframe status of the frame and several other pieces of data.

-h

[Wbtz]

Yes, load balancing as well! So it would be easier to tap from "LRESULT compress(CODEC * codec, ICCOMPRESS * icc)" in codec.c of vfw.

For each time the compress() function is called, it is one entire frame with its auxilliary information?

Are frame.length, frame.bitstream the only things i need to write to?

Which part of the function does it write these info?

if (!WriteFile(codec->twopass.hints, &frame.hint.hintlength, sizeof(int), &wrote, 0) || wrote != sizeof(int) || !WriteFile(codec->twopass.hints, frame.hint.hintstream, blocksize, &wrote, 0) || wrote != blocksize)
{
DEBUGERR("couldn't write to hints file");
return ICERR_ERROR;
}

Is it the above?

Sorry -h, I am getting confused. Also if linux was used for encoding, it would not use vfw anymore..then what would it require?


I Found this on MSDN:
The ICM_COMPRESS message notifies a video compression driver to compress a frame of data into an application-defined buffer.

ICM_COMPRESS
wParam = (DWORD) (LPVOID) &icc;
lParam = sizeof(ICCOMPRESS);
Parameters
icc
Pointer to an ICCOMPRESS structure. The following members of this structure specify the compression parameters: lpbiInput, lpInput, lpbiOutput, lpOutput, lpbiPrev, lpPrev, lpckid, lpdwFlags, dwFrameSize, and dwQuality.
The driver should also use the biSizeImage member of the BITMAPINFOHEADER structure associated with lpbiOutput of ICCOMPRESS to return the size of the compressed frame.

lParam
Size, in bytes, of ICCOMPRESS.
Return Values
Returns ICERR_OK if successful or an error otherwise.



Which are the variables to write to the binary file using fwrite?

[Wbtz]

Hey -h, I read the code for VFW, it prepares the video for writing by virtualDub and understood a little. Let me paint the impression in my mind:

frame.bitstream in compress() now has the output compressed data to write.

frame.length has the size of the compressed data

However, how about things such as flags for the AVI index? Must they be written too? What else are they? Any order of writing?

I have inserted this code:

outfile = fopen("test.dat", "a+");
fwrite(frame.bitstream, sizeof(BITMAPINFOHEADER), frame.length, outfile);


VirtualDub would crash! Think something is wrong but cant figure out where! Please help!:confused:

[Wbtz]

Hi!

I have never used Linux before and was wondering after you "make" the xvid codec, do you still have to use a third party program like virtual dub to encode or will you produce a .exe file as the encoder?

[-h]

Hm. XviD on linux requires a program capable of using the libxvidcore.so file to encode video, like transcode or mencoder. I've not used linux before, so I'm not sure what to tell you.

In terms of the data you need, all you need for each frame is the frame number, the frame data and the length. Write the frame data (frame.bitstream) to a file (whose name is the frame number) whose length is frame.length. Then whatever decoding program you write will have to request these files.

I can post some code for vfw/codec.c soon, but have an appointment right now.

-h

[Wbtz]

Thanks a million zillion -h!!!!!

You are my saviour!!!!!

Eagerly awaiting....lol:p

[-h]

Unfortunately it is likely that I won't be able to do anything until Sunday, when I'll return from a thanksgiving trip to see some family.

You might want to remind me around that time..

-h

[Wbtz]

Sure! How do i remind you? PM? Email?

[Wbtz]

Hi -h!
I PM-ed you!
Thanks!!!!!!!:D

[Wbtz]

Hi!

VirtualDub still crashes with my code. Where is my mistake?

Thanks!

[-h]

I'll give it a go this afternoon. I'm just finishing up that lossless codec.

Should only be 10 or so lines of code. I wouldn't use it with B-frames though.

-h

[Wbtz]

Hi!

I modified it to :

outfile = fopen("file.out", "a");
fwrite(frame.bitstream, frame.length, 1, outfile);

This time round, virtual does not have an access violation but produced a 100mb file for a 800kb video. LOL... Guess i will wait for your code then -h! thanks!

[Wbtz]

Hi -h!

I am not using B frames too. It becomes too complex. Will just rely on the I and P frames. How is the code coming along? Thank you very very much!

-Wbtz

[Wbtz]

Hi -h!
Awaiting your help...or anyone else?

Thanks!;)

[-h]

Well, this did the trick for me. I only tested the dev-api-3 version though.

In vfw/codec.c, modify the code around line 561 (after the call to xvid_encore()) to look like this:

case XVID_ERR_FORMAT :
return ICERR_BADFORMAT;
}
// begin new code
{
HANDLE f;
char s[100];
int dummy = 0;

wsprintf(s, "%06d%c", codec->framenum, frame.intra ? 'i' : 'p');

f = CreateFile(s, GENERIC_WRITE, 0, 0, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);

if (f == INVALID_HANDLE_VALUE) {
OutputDebugString("couldn't create file to write frame to");
} else {
WriteFile(f, frame.bitstream, frame.length, &dummy, 0);
CloseHandle(f);
}
}
// end new code
if (frame.intra==1)
{
codec->keyspacing = 0;

It will write files with 6-digit names to the current directory, and each name will end with i or p depending on whether it's an i-frame or not.

I don't know what you can do about audio, since that's handled elsewhere, or about playback, since you'll have to load the files in some custom program (or modify XviD's vfw frontend to load files instead of data sent from the calling program).

-h

[Wbtz]

Oh My Gosh!

It worked for me too. I have never come across all these commands. I googled it and found it on the MSDN library.

These are windows functions available from the windows.h library?

Another question: If these files were merged can it be said to be a valid .avi file? The reason i ask is because the total file size is smaller than the actual .avi produced.

Finally, a million thanks -h, though I don't even know what was happening.

Oh yes, a small question on decoding then... the reverse process should be done in the decompress() function in codec.c then?

[-h]

Yes, those are the actual windows file routines (fopen, fread, fwrite, etc. all map to window's CreateFile, ReadFile, WriteFile functions).

Those files can't be merged to create an AVI stream, since the AVI side data is missing (such as index markers, keyframe flags, audio data, etc.). If you want to prevent the AVI from having the same video data written (and use up a lot of disk space), you can set outhdr->biSizeImage to be 0 further down in the function.

Decoding will be difficult of course, but sure the decompress function could read and decode the files instead of the data sent by the calling program.

It is a messy way of doing things, and there's no way to handle audio similarly.

-h

[Wbtz]

Hi -h,

Thanks so much for your help.

I think I will not touch audio at all and not to mention B frames.

If I want to do decoding I must use the windows file routines as well in decompress() ? My guess is I have to disable data calls from then calling program first before trying to read the files.

Finally, if the decoder manages to read the individual files then will the avi side data still be present to be able to be played properly? Or do i also have to write the avi side data to a file when encoding..if possible.

Thanks.

-edit-
I understand what you mean. I omit the frame info from the original AVI(so it only contains avi side data) and modify the decoder such that when that movie is played frame info is taken from the separate files?

Did i get this right?

[Wbtz]

This is part of the decompress function
LRESULT decompress(CODEC * codec, ICDECOMPRESS * icd)
{
XVID_DEC_FRAME frame;

frame.bitstream = icd->lpInput;
frame.length = icd->lpbiInput->biSizeImage;

frame.image = icd->lpOutput;
frame.stride = icd->lpbiOutput->biWidth;

if (~((icd->dwFlags & ICDECOMPRESS_HURRYUP) | (icd->dwFlags & ICDECOMPRESS_UPDATE) | (icd->dwFlags & ICDECOMPRESS_PREROLL)))
{
if ((frame.colorspace = get_colorspace(icd->lpbiOutput)) == XVID_CSP_NULL)
{
return ICERR_BADFORMAT;
}
}
else
{
frame.colorspace = XVID_CSP_NULL;
}


I checked the readfile windows function. It requires that I know the number of bytes to be read. How can I determine the number of bytes to be read from a file since I do not know frame.length ?


frame.bitstream = icd->lpInput;
frame.length = icd->lpbiInput->biSizeImage;
frame.bitstream and frame.length should be from the files produced right? Other than that do i need to take note of anything else?

Thanks!

[Wbtz]

// begin new code
{

frame.length = icd->lpbiInput->biSizeImage; //Removed 1st line, this is 2ndline
HANDLE f;
char s[100];
int dummy = 0;

wsprintf(s, "%06d%c", codec->framenum, frame.intra ? 'i' : 'p');

f = CreateFile(s, GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);

if (f == INVALID_HANDLE_VALUE) {
OutputDebugString("couldn't read file");
} else {
ReadFile(f, frame.bitstream, frame.length, &dummy, 0);
CloseHandle(f);
}
}
// end new code


Hi -h, I have tried to implement this in the decompress() function of codec.c in VFW but it doesn't work. Where is the fault? Puzzled...thanks!!
:scared:

[-h]

I can't access my computer right now, but you want to use GetFileSize() to find out how much data to read from the file in question. You should use that length instead of frame.length.

-h

[Wbtz]

// begin new code
{

HANDLE f;
char s[100];
int dummy = 0;

wsprintf(s, "%06d%c", codec->framenum, frame.intra ? 'i' : 'p');

f = CreateFile(s, GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);

frame.length = GetFileSize (f, NULL); // Modified from line1

if (f == INVALID_HANDLE_VALUE) {
OutputDebugString("couldn't read file");
} else {
ReadFile(f, frame.bitstream, frame.length, &dummy, 0);
CloseHandle(f);
}
}
// end new code

--------------------------------------------------------------------------------


Somehow it still doesnt play...
The avi file that is without the frame information contains the other avifile headers etc right?

[Wbtz]

I get the following error.

'intra' : is not a member of 'XVID_DEC_FRAME'

should be from the line

wsprintf(s, "%06d%c", codec->framenum, frame.intra ? 'i' : 'p');

if xvid_dec_frame does not have the frame.intra then what should be done. One option i suppose should be i dont include the i and p frames in the filename

Suggestions?

[-h]

Yeah since the AVI file includes information about keyframes, you don't need the i or p appended to the file names. That may solve your problems.

-h

[Wbtz]

Tried it. No errors during compilation but on playing using WMP (with ffdshow installed) the following error appears:

ClassFactory cannot supply requested class (Error=80040111)

:(

[-h]

You could only play the file you created with a modified build of XviD - why would you want it to use ffdshow?

-h

[Wbtz]

Meaning I cannot use media player or any other player to play the file?

[-h]

You can use any player, so long as the player calls the modified XviD dll and *not* ffdshow or any other MPEG-4 decoder to play the file. Otherwise, that decoder will not know to read files from the hard drive instead of from the AVI.

-h

[Wbtz]

Ok.. i played it in virtualdub instead. Only a black screen though. Means at least it can be played? I suppose virtualdub uses the xvid decoder directly to play right?

[-h]

Yep, that should work. If you want to debug the process, I'd suggest creating a string with the information you'd like to see and outputting it with OutputDebugString(string); - then you can view the messages being sent with dbgview from www.sysinternals.com.

-h

[Wbtz]

Debugview is a good tool! Thanks!

What could be the problem actually? Since it displays only a black screen?

Currently in compress()
{
HANDLE f;
char s[100];
int dummy = 0;

wsprintf(s, "%06d", codec->framenum);

f = CreateFile(s, GENERIC_WRITE, 0, 0, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);

if (f == INVALID_HANDLE_VALUE) {
OutputDebugString("couldn't create file to write frame to");
} else {
WriteFile(f, frame.bitstream, frame.length, &dummy, 0);
CloseHandle(f);
}
}

In decompress()



XVID_DEC_FRAME frame;
frame.bitstream = 0;
{
HANDLE f;
char s[100];
int dummy = 0;

wsprintf(s, "%06d", codec->framenum);

f = CreateFile(s, GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);

frame.length = GetFileSize (f, NULL); // Modified from line1

if (f == INVALID_HANDLE_VALUE) {
OutputDebugString("couldn't read file");
} else {
ReadFile(f, frame.bitstream, frame.length, &dummy, 0);
CloseHandle(f);
}
}

I had to make frame.bitstream = 0 initially becos it produced an error : local variable 'frame' used without having been initialized

Is this why there is a black screen?

[Wbtz]

Hi -h,

Sorry to disturb you once again.

I can't seem to get around the local variable 'frame' used without having been initialised error in the decompress() function. Only if I initialise the frame.bitstream = 0 then only will the error go away. Why is this so? Is this the reason for the black screen throughout the whole movie?

[-h]

If you set frame.bitstream to 0, the ReadFile() call will fail because the pointer it's supposed to be writing to is 0 - invalid. Allocate some memory (frame.bitstream = malloc(50000); or so) so there is some buffer that you can read the frame data into, otherwise there's nothing for the core to decode.

Don't forget to free(frame.bitstream); after decoding of the frame is finished.

-h

[Wbtz]

Hello!
Edited the decompress() but still a black screen when playing the encoded video in virtualdub.

LRESULT decompress(CODEC * codec, ICDECOMPRESS * icd)
{
XVID_DEC_FRAME frame;

//frame.bitstream = icd->lpInput;
//frame.length = icd->lpbiInput->biSizeImage;

//start of code
frame.bitstream = malloc(50000);
{
HANDLE f;
char s[100];
int dummy = 0;

wsprintf(s, "%06d", codec->framenum);

f = CreateFile(s, GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);

frame.length = GetFileSize (f, NULL); // Modified from line1

if (f == INVALID_HANDLE_VALUE) {
OutputDebugString("couldn't read file");
} else {
ReadFile(f, frame.bitstream, frame.length, &dummy, 0);
CloseHandle(f);
}
}
//end of code

frame.image = icd->lpOutput;
frame.stride = icd->lpbiOutput->biWidth;

if (~((icd->dwFlags & ICDECOMPRESS_HURRYUP) | (icd->dwFlags & ICDECOMPRESS_UPDATE) | (icd->dwFlags & ICDECOMPRESS_PREROLL)))
{
if ((frame.colorspace = get_colorspace(icd->lpbiOutput)) == XVID_CSP_NULL)
{
return ICERR_BADFORMAT;
}
}
else
{
frame.colorspace = XVID_CSP_NULL;
}

switch (xvid_decore(codec->dhandle, XVID_DEC_DECODE, &frame, NULL))
{
case XVID_ERR_FAIL :
return ICERR_ERROR;

case XVID_ERR_MEMORY :
return ICERR_MEMORY;

case XVID_ERR_FORMAT :
return ICERR_BADFORMAT;
}
free(frame.bitstream);
return ICERR_OK;
}

Any mistakes from the above?

Ohyes also..there has been a warning from the onstart which is

LINK : warning LNK4075: ignoring /EDITANDCONTINUE due to /INCREMENTAL:NO specification

But I ignored it. Hope its not this problem!

[Wbtz]

harlow -h...

I have discovered that the readfile does not work. What I did was i encoded into the different frames and then deleted them leaving only the AVIFILE with no frame info. Then I attempted to play and only achieved the same result: the black screen (in virtualdub). Was wondering if there is an error in my statement.

Also the debugview shows no outputdebugstring to the system meaning it does not even pass through the decompress() function in codec.c?

Very stunned now..the filters used to play the xvid avi are video renderer and xvid codec so it should work rite?:confused:

[-h]

You deleted the frames that you wrote during the encoding stage? If you delete the frame data, you'll have nothing to display during decode..

I'd try putting an OutputDebugString() call at the very start of decompress(), to make sure it's being called. But I'm sure it is.

Also, are you decoding with the XviD .ax DirectShow filter, or the XviD .dll VfW codec? The code you're writing will only affect the behaviour of the .dll, so if the .ax is on your system, it will be over-riding anything you do when played in Media Player.

-h

[Wbtz]

Hi -h,


LRESULT decompress(CODEC * codec, ICDECOMPRESS * icd)
{
XVID_DEC_FRAME frame;

//frame.bitstream = icd->lpInput;
//frame.length = icd->lpbiInput->biSizeImage;

OutputDebugString("Can you see me");

//start of code
frame.bitstream = malloc(50000);
{
HANDLE f;
char s[100];
int dummy = 0;

wsprintf(s, "%06d", codec->framenum);

f = CreateFile(s, GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);

frame.length = GetFileSize (f, NULL); // Modified from line1

if (f == INVALID_HANDLE_VALUE) {
OutputDebugString("couldn't read file");
} else {
ReadFile(f, frame.bitstream, frame.length, &dummy, 0);
CloseHandle(f);
}
}
//end of code



This is what I have inmy codec.c VFW frontend decompress(). I deleted the frames but it was still able to plaay without any outputdebugstring to debugview. When I added the "can you see me" message, debugview still did not show any message meaning it was not called.

One thing to clarify, debugview captures these messages automatically right? Do I have to configure anything else?

I installed using the xvid.inf, it think it will install the .dll file. How about the .ax? But since I was able to encode to the various frames as files, this would mean I am already relying on the .dll file?

PS: I am playing the encoded avi file in virtualdub with debugview launched. Seems like the whole decompress() function is bypassed altogether....:eek:

[-h]

I'd try uninstalling XviD and deleting the .ax file just to be sure.

Also, put OutputDebugString() calls in driverproc.c, just after "case ICM_DECOMPRESS :" - if that doesn't trigger debug output, it means xvid.dll isn't being called.

-h

[Wbtz]

ok...still doesnt work!!!!

Uninstalled Xvid from add remove progs.

Deleted xvid.ax

Added OutputDebugString("test"); to after case ICM_DECOMPRESS...

Played it in virtualdub...still a black screen...and no output to debugview!!!!!!why???? konked out oredi.

Anymore suggestions? Getting really desperate as the deadline is near...

PS: Tried with audio...the audio is still playable!

[-h]

It sounds like you've encoded the file with the FourCC set to DX50, and then upon decoding it's calling DivX 5.x to decode. I can't think of any other explanation.

I may try to do this from scratch tomorrow (on my day off), though I still can't see any gain from writing frames to files and reading them from multiple servers - you're going to have to write a *lot* of additional code (say several thousand lines) to have a useful decoder.

-h

[Wbtz]

Hi -h,

the FourCC is set to XviD.

I realise there is tonnes of coding to be done especially the socket programming area of which I have no choice. I have read up on this already and have done some programming in this area before.

Frankly, I am also doubtful that this will pay off but I have no choice haha.... I will at least have to show it doesnt work!

Really thankful you are able to help me debug. Sorry to trouble you. Thanks a zillion million!

Btw, r u a student?

[Wbtz]

hi -h,

seems the decoder is able to read the encoded files. its worse now and windows media player crashes! something to do with memory allocation?