NuEnc 0.00d & Libavcodec improvements/suggestions

[Peter Cheat]

These are the modifications to the original libavcodec, the heart of FFMPEG, QuEnc, FreeEnc, Mencoder, and now NuEnc.

Updated 20/10/04

- New faster quantiser scaling routine. [100%]
This seems to give more of a constant quality than the original code, and saves a few bits on easy-to-encode frames. Now even better than my first slow routine...

- Buffer underflow problem resolved [96%]
Buffer underflows are prevented by increasing the quantiser slightly when pending overflows are detected. Buffer underflows are checked for each frame, and before a frame is encoded, it has been checked for underflow at least thirty times. Also, no more than 90% of the estimated remaining buffer can be used, reducing the possibility of underflow further.

- Max bitrate adherence removed [0%]
Unnessesary.

- Min bitrate works correctly [100%]
Specifying a minimum bitrate doesn't screw up prediction anymore.

- Multipass encoding [90%]
To help reduce bitrate spikes, minor (negligible) improvement in quality. Optimal encoding occurs after 3 passes on average. (Quality _may_ actually appear to degrade with many passes!).

(% = estimate of how close code is to complete and final)

Try the NuEnc frontend, allows multipass encoding and has a "turbo" first pass for those who can't wait so long for an encode to finish. Heaps of other additions, check the site for more details, including forced I frame insertion for chapters. Noise reduction is currently disabled. Scene detection sensitivity increased and other changes made to improve quality :D.

Source code, and some compiles are available here: Modified Libavcodec Site (http://petercheat.host.sk/libav/)

[JDay]

Would you mind re-uploading "ffmpeg+avcodec.zip"? The zip seems to be corrupt.

[Amnon82]

yeah it is broken. I'm realy intresting in this modded version. Maybe I can use it for my EasySVCD/DVD release.

[Paced]

Originally posted by JDay
Would you mind re-uploading "ffmpeg+avcodec.zip"? The zip seems to be corrupt.

It's not broken; the same thing happened to me a few times. Make sure that your .zip file is 933kb in size - for some reason my Internet Explorer ( :devil: ) "finished" the download prematurely on several occasions. Fortunately, after several tries, I was able to get the entire 933kb :D

[Amnon82]

k, i get it!

@Peter Cheat: I'm using now your avcodec.dll in the new version 0.4 of EasySVCD/DVD. This avcodec.dll is much better. Can you write a guide for editing and compiling ghe avcodec.dll. My last compile was 2.55 MB. I think there was all the mpeg4 stuff in it. What is the easiest way to get rid of it?

[JDay]

Yeah, it seems to be all there. I was able to get it ok , using IE. In Firefox (what I normally use), I had to start the download a few times, since it kept finishing prematurely (no errors given). Got it in the end though. I'll start testing, now.

[JDay]

Well, found a problem with this version. I encoded a small clip with DVD-RB and QuEnc 0.51 D2 test. With the avcodec.dll that came with QuEnc, I got good quality, and ended up with a file about 68.2mb (68.1mb with CCE; 02m,27s @ 4177kbps). I tried this version on that clip (and a few neigboring cells from the disc) and ended up with a rather ugly 12.3mb file.

Here are some bitrate viewer screenshots of that clip:

CCE:
http://img13.exs.cx/img13/8995/CCE-BV.png
QuEnc (Old):
http://img20.exs.cx/img20/5426/QuEnc-Old.png
QuEnc (Peter Cheats's):
http://img20.exs.cx/img20/3213/QuEnc-New.png

Hope this helps! I'll do some testing on some other source and target material later tonight.

Oh, by the way, the tested disc was Hellboy (R1). That particular clip was progressive, 23.976fps, 720x480, 16:9.

Edit: Fixed image links.

[dragongodz]

did some quick tests with QuEnc 0.53. i even changed the b offset to 1.0 (b factor is already 1.50 for 2 pass). yep not good. all high quants (ranged 26-30). so no good for QuEnc anyway.

peter cheat - is there any other settings you can think of that need to be changed in QuEnc ?

[dragongodz]

just tried using ffmpeg and near the end of the first pass a heap of underflows were reported. no problem since it is just the first pass.
second pass though crashed at about the point those underflows were reported. so somethings not right there.

[Peter Cheat]

I didn't actually do any tests with FFMPEG :o

I use Mencoder for all my testing. Mencoder is good, you just have colour problems when using the internal resizing & with XviD sources.

My DLL does NOT work with QuEnc 0.53. I tested it, and the output was far too small, and blocky. Obviously QuEnc should be compiled with new avcodec.lib for it to work.

@Amnon82 A 2.55MB file is about the right size. I use UPX (Ultimate Packer for eXecutables) to make it smaller. It is widely used.

I have written code for a third pass, and it works. Initial testing shows that max bitrate is not exceeded anymore, but this is with a very short 1500 frame clip. Also, the spikes in second pass have been reduced by fixing up scene detection code (it was enabled in second pass meaning that some p-frames were changing to I-frames, making prediction very hard indeed).

The 3-pass code will be uploaded at some stage soon, but more files needed modification. Since I did the changes for Mencoder, I'll need to change FFMPEG first. I'll also test FFMPEG before uploading it this time.

[Amnon82]

I wrote also a gui for mencoder. So I have only to replace the three modded files with ther orginals in the source of MPEnc.

Lets see how it works ...

[Peter Cheat]

Originally posted by dragongodz
just tried using ffmpeg and near the end of the first pass a heap of underflows were reported. no problem since it is just the first pass.
second pass though crashed at about the point those underflows were reported. so somethings not right there.

The version of FFMPEG I have been using seems broken. Crashes when writing the last line of the log file. Not sure whether I did that, or not (I dont think so :D ).

Anyway, 3-pass code has been done, and is updated on my site. It works well.

Comparing the number of times the max bitrate was exceeded using same 4min clip (music video)-
TMPEGEnc 2.59 2-pass: 30 times
Mencoder (modified libavcodec) 2-Pass: 40 times
Mencoder (modified libavcodec) 3-Pass: 0 times

The maximum bitrate is now adhered to very well in 3-pass. Not perfect when doing low res, low bitrate encodes, but flawless so far with DVD specs.

The modified avcodec.dll works with QuEnc 0.54 (QuEnc is now compiled with latest libavcodec sources). Three pass is not available with it though...

[dragongodz]

The modified avcodec.dll works with QuEnc 0.54
cool. will have to test it some time during the week. hopefully others will aswell and report back. :)

QuEnc is now compiled with latest libavcodec sources
well actually QuEnc 0.52 and 0.53 used CVS checkout from early august while for 0.54 i went back to the 12th july version. thought that would be easier for people to get since it can be directly downloaded from FFMpegs page.

[JDay]

I encoded my test clip with this latest version (using QuEnc). Results were much better than last time (see my post above) but I still noticed something odd. With all other encoders (and other avcodec builds), the q. level roughly follows the same pattern as the bitrate. Here, it seemed that it stayed mostly constant except when the bitrate dropped below the q. level, in which case it increased slightly. I'm no expert in this area, so I'm not exactly sure what this means, its just something that caught my eye. Bitrat-viewer screenshot is below, which can be compared with the previous shots in my post above.

http://img53.exs.cx/img53/2818/QuEnc-8-30-04.png

[dragongodz]

JDay - what settings did you use, such as max bitrate etc ?

[JDay]

Originally posted by dragongodz
JDay - what settings did you use, such as max bitrate etc ?

I think I kept all the default setttings. Mpeg2, high quality, vbr, 2 pass, 12 - gop size, 2 - max b frames, 8 - dc precision, 9800 - max bitrate, all advanced settings unticked, @4177kbps.

Edit: And could someone please give me some info on the q. level and how that relates, or should relate, to the bitrate? Thanks.

Edit 2: Also, I see that the QuEnc encodes have a linear quantscale, while cce and tmpg encodes are listed as nonlinear. What does this mean, exacly? Is this significant?

[Peter1234]

Peter Cheat,
Your new compiles of avcodec.dll seem to have the SSE2 problem discovered by VMesquita in FreeEnc (see http://forum.doom9.org/showthread.php?threadid=80544&highlight=SSE2). On my P4 system your avcodec.dll seems to crash about half of the time. He solved this by disabling SSE2 with a patch. I do not know details.

[dragongodz]

ok did a quick test on a very short (just over 1 minute) clip at dvd size.
average bitrate 3000, max bitrate 4300.

my compile and tmpgenc never exceeded max bitrate, slightly undersized. peters exceeded max 3 times and average bitrate exceeded slightly. i even tested by changing the b quant factor to what peter says on his page and no difference.
also i notice his compile has the pulsing with playback while mine doesnt.

so its getting there but not quite ready yet. :D

JDay - i did not get the flat quant level you did but then again i set much lower average and max bitrates. the q level is quantisation level. that is how much each frame is quantisised. the higher the quant the smaller the frame size(ignoring motion vector info etc) and also the less detail retained. think of it like blurring, the more blurring the less fine info needs to be kept, result smaller size.
ok before anyone jumps on that let me say i am just trying to explain it in the simplest terms that anyone can understand, ok ?



seem to have the SSE2 problem
He solved this by disabling SSE2 with a patch
or you could simply compile it with cygwin or even try the mem-align-hack as i suggested to vmesquita(which i dont think he did try).

[JDay]

Thanks dragongodz, that's what I thought. But if a higher q. level means a smaller frame size, you'd expect it to fluctuate in inverse of the bitrate, correct? However, as you can see on my previous BV screenshots of CCE and the standard avcodec, it seems the bitrate and q. level follow the same path. Am I missing something?

Anyway, I did some more tests, using the same clip and settings as before, but trying some different bitrates (2000, and 8000). I found that on my encode at 2000kbps, the q. value was an inverse of the bitrate. At 8000kbps, the q. level and bitrate followed roughly the same path. Given these results, and my encode earlier with a nearly flat q level, it would appear that there is some sort of bitrate threshold, near which the q. level flattens, and beyond which, it inverts.

Here's some more BV screenshots of those encodes:
http://img53.exs.cx/img53/3571/QuEnc-2000.png
http://img53.exs.cx/img53/5571/QuEnc-8000.png

[Peter Cheat]

Originally posted by Peter1234
Peter Cheat,
Your new compiles of avcodec.dll seem to have the SSE2 problem discovered by VMesquita in FreeEnc (see http://forum.doom9.org/showthread.php?threadid=80544&highlight=SSE2). On my P4 system your avcodec.dll seems to crash about half of the time. He solved this by disabling SSE2 with a patch. I do not know details.

Oops, I know about this problem, will be fixed next compile (have to disable SSE2)

@JDay
The rate control has been modified to give a more of a constant quality. A parameter will be available (soon) to adjust the modulation. The constant Q level in this case is caused by the Max bitrate adherence in this particular case it seems. CCE and TMPEnc use logarithmic scale quantisation, supposed to be better, but turning a linear scale to pseudo-logarithmic wouldn't be hard, and although it is said that logarithmic is better, its all subjective

I would like to get something clarified (as the SVCD, and DVD specs are unclear).

Is max bitrate:
- the average bitrate over each second (like in bitrate viewer)
- the average bitrate over a gop (mention before somewhere?)
- or the maximum a frame can be

I have used the latter, but it makes for crap encodes as I Frames are slaughtered with high quants.

[dragongodz]

JDay - i was trying to make it simple so didnt mention other factors. things that effect size aswell is actual frame content. simple example - frame 1 is completely black, frame 2 is bright with lots of colours and lines etc. do both frames at the exact same quant. can you guess the sizes of the frames ? the black frame will be tiny while the colourful one will be much larger. so yes it can be inverse some frames and then not others or even completely one or the other but that depends upon what the footage is. the bitrate line is more important so dont worry to much about the quant line for now. :)

have to disable SSE2
why not just compile with cygwin or at least try the mem-align-hack ?

Is max bitrate:
- the average bitrate over each second (like in bitrate viewer)
- the average bitrate over a gop (mention before somewhere?)
- or the maximum a frame can be
well since the actual dvd specs cost a lot of money and then you have to sign a non-disclosure agreement......who can say ? :)
all we can go by is actual experience. heres a little arguement over it
http://club.cdfreaks.com/lite/t-76710.html
now lets assume thats right and its per second. the problem also arrises from the fact that people have reported problems when you have frames spike above the max aswell which suggests the 9.8Mb is also the max per frame.

oh and about I frames. they contain no motion vectors etc so they should actually be able to be a lower quant for the same size as P and B frames. :)

[Inc]

Nice, cant wait to test the new mod of libavcodec. :)

A very nice test sequence is the beginning of KillBillVol1.

Right at the beginning you see the Miramax Inro where the camera moves from the ocean up to the city-skyline where then the Miramax Logotype appears.

This movement over the water(ocean) means a very complex luma changing pixel contend which does let jump up at other encoders the bitrate or the quantizer if max bitrate ist set to low.

This sequence is directly followed by a totally black screen sequence. Here the encoder can show how he reacts on such changes ;-)

I used that sequence and other sliced parts of that movie to test FreeEnc, Qenc, mencoder and CCE.

Greets
Inc.

PS: Also for high peaking tests you can use "Poltergeist" at the scene where you can see the full of snow television screen.

[Peter Cheat]

Originally posted by dragongodz
oh and about I frames. they contain no motion vectors etc so they should actually be able to be a lower quant for the same size as P and B frames. :) [/B]

Hmm. Then why not encode with only I frames? Because I frames are much larger than P,B frames. Why? Because I frame contain the information about the whole frame (basically JPEG compressed). P frames are the differece between previous I and P frames. B frames are the differences between the P frames on either side (before and after) of the frame.

Hence, I frames almost always biggest, and get considerably bigger with lower quants...

Apparently, there are NO software encoders that strictly comply with the DVD standard. What is the chance of making libavcodec able to make strictly compliant to play with 100% of DVD players? I don't know but all odds are against it...

[dragongodz]

Then why not encode with only I frames? Because I frames are much larger than P,B frames
yes i know that. look at other encoders though and see what quant they are using for I frames comparitivly and you will see what i am talking about.

P frames are the differece between previous I and P frames. B frames are the differences between the P frames on either side (before and after) of the frame.
again not telling me anything i dont already know. the motion vectors do happen to add to the size of the frame though so generally(especially low bitrate encoding) you will see them use the same or higher quants than the I frame preceding them. notice no pulsing with other encoders ?

Apparently, there are NO software encoders that strictly comply with the DVD standard. What is the chance of making libavcodec able to make strictly compliant to play with 100% of DVD players? I don't know but all odds are against it...
hmm well the problems with CCE as i have said before seem to be when using 1 or 2 pass, its possible more doesnt have a problem. also CCE using a lower max bitrate(using a max of 9000 and up seem to be the problem) does create compliant streams, well it fixed the problems some people had. i dont know how many people make dvds with Tmpgenc or Procoder so there is no real way to say how much problem they have. 100% is the target even if it is unreachable. if you can get avcodec to hit bitrates consistantly the same as Tmpgenc and CCE then that should be damn good. :)

[Peter Cheat]

libavcodec has better potential than both TMPGEnc and CCE. The motion search is more sophisticated I believe (looking at the motion vectors chosen by all three).

I noticed that the pulsing seems to be caused by the lack of I frames compared with TMPEnc and CCE. By reducing the scene change threshold (and increasing the number of I frames), the first pass size (at const quant) actually is REDUCED!. This even surprised me. I have updated avcodec.dll with a good scene threshold value hard-coded in for first-pass (2nd pass uses first pass frame choices).

Looking at CCE and TMPEnc, both encoders use the same or lower quants for I frames. Using TMPEnc 2.59, you can see the frame sizes when you have enabled the writing of the log file. This clearly shows that I frames are the largest frame in the GOP 99% of the time. Sometimes a P-Frame is even larger in high motion.

How can I get libavcodec to hit bitrates consistently the same as CCE and TMPEnc? CCE and TMPEnc both use different bitrates for the same scene (view with bitrate viewer). The point is not to copy/imitate other encoders. Libavcodec is very different in many ways. The rate control is just going to be another of those things that makes it different. I beleive that with some effort, libavcodec can become better than commercially available encoders. It already performs better at lower bitrates than both CCE and TMPEnc (TMPEnc creates blocky material, CCE makes it so noisy and ugly its unbearable).

Everyone seems to use "Bitrate Viewer" to look for bitrate spikes/max bitrate. "Bitrate Viewer" looks at every second of video (24,25,29.97 etc frames, depending on frame rate), AVERAGES the bitrate over this second, and then reports this as the bitrate at this second. It doesn't report the "real" max bitrate, meaning it doesn't report when an individual frame exceeds the max bitrate. The DVD (and SVCD) specs seem are simply not specific. SOME ORIGINAL MOVIES DO NOT EVEN WORK IN DVD PLAYERS. SOME MOVIES DO NOT PLAY ON WINDOWS 2000/XP (without a fix) This is because hardware manufacturers interpret the standard one way, companies that author original DVDs interpret the standard another way and hence we have a problem!!! Some manufacturers knowingly create their hardware out of spec. 100% compatibility is at the moment an impossibility.

Visit http://www.dvddemystified.com/dvdfaq.htm and take a look for yourself. The standard should be clearly rewritten to combat this issue. Also, the official specs should be distributed for FREE everywhere, not that you have to pay large amounts of money to see a document (that is vague and poorly written).

Btw, I updated my modified avcodec.dll using the "Bitrate Viewer" max bitrate model. Maybe this will work better. Picture pulsing seems to be resolved now, but I haven't done much testing with this.

Im out,
Peter Cheat

[Nic]

Hi Peter,

Good work :) Could you update the source code on your website? I'm finally finding a little time to be playing around with it myself :)

Thanks,
-Nic

[Peter Cheat]

FFMPEG sources are now updated. Its best to do a clean compile to use with QuEnc, as just using the DLL is really dodgy. avcodec.h needed to be modified for implementation of 3-pass.

FFMPEG may crash at the end of 1st and 2nd pass, seems to be a log file writing problem. This means the last line os the log is cut off, the output stream is fine.

[dragongodz]

Peter i will continue to discuss this by PM. that way i can be more to the point. :)

[RobertR]

@Peter Cheat:
Can you please post your full source of ffmpeg/mplayer ? I'd like to make a diff out of it and apply it over what i get from ffmpeg/mplayer cvs (i'm trying to use libavcodec for dvd backup under Linux).

Last week Paul Curtis on ffmpeg-devel mailing list posted a patch that

This patch inserts the correct information into the MPEG2 stream for the creation of DVDs. I am using 'dvdauthor' as the benchmark for reating DVD VOB files.

This patch creates PCI/DSI (NAV) packets after each GOP start. This means that forwarding and rewinding should stop on a complete frame of video, and should eliminate the "blocking". In addition, this also
removed the problem of audio being out of sync after a rewind or
fast-forward.

The command line to create a DVD complaint MPEG is:
ffmpeg -i your_inputfile -target dvd outputfile


From what i've read on ffmpeg-devel ML this patch was not put into cvs. It should be in ML archives (see ffmpeg.org for details). If You can't find it i can post it on my webpage.

Could You consider cooperating with ffmpeg team to incorporate your patches back to main source tree? (i know i've already asked this but this was w/o any echo :)

[dragongodz]

i see this is being discussed in the FFMpeg mail list so i thought i better make some comments to clear some things up.

forced scene detection is not the solution to the pulsing. try QuEnc 0.54 with its normal avcodec.dll and you will see no pulsing even when scene detection is disabled. how ? i changed I frame factor to force I frames to a lower quant. originally you would get things like I(3),B(1),B(1),P(1), where the numbers are the quant for that frame. the higher quant I frame is what causes the pulse. look at any other encoder and that will only happen on extremely rare occasions. instead you are more likly to see I(1),B(2),B(2),P(1) etc. you will notice it also doesnt suddenly flood with underflows because of the change.
why nobody has noticed that before i have no idea. :D

also about max bitrate. i have PM'd Peter 2 different links which suggest the max bitrate is 9800kbit per second AND a max spike/burst(for VBR obviously) of 9800Kbit per frame for dvd.

i believe mpeg2enc had reasonably good rate control(though i never really used it, just what i have read) so maybe it would be worth looking at that and maybe xvid 1.1 branch to see what they are doing aswell. it may help to give ideas etc on how to improve avcodecs rate control.

[RobertR]

I've mailed ffmpeg mailing list about this. Note however that i don't plan to be a forum2mailinglist gateway ;)
Michael Niedermayer wasn't too enthusiastic about Peter Cheat changes.
I've done some simple tests with Peter Cheat's patches under linux and results were visually bad. I'll rerun this testversion again with exact same settings as dargongodz is using in his latest QuEnc.
For now dragongodz your settings give the best results for me (just need to get rid of macroblocks all over gradients).

[Peter Cheat]

Originally posted by RobertR
I've mailed ffmpeg mailing list about this. Note however that i don't plan to be a forum2mailinglist gateway ;)
Michael Niedermayer wasn't too enthusiastic about Peter Cheat changes.

I'm glad he is. My changes are just experiments to improve the codec in the future, not to use now. It completely sux at the moment. I fixed some code yesterday that improved the picture significantly. My changes are for testing to see what happens. You need to sometimes make one step back before making two steps forward.

My aim is to change quantiser distribution so that lower quants are given to spatially complex frames, and higher quants given to temporally complex frames (you can see detail well during motion). At the moment, libavcodec doesn't do this. There are also many other issues that need resolving.

@Dragongodz
I have a clip that pulses when encoded with the original QuEnc 0.54 dll. The pulsing is strange as the pulse is cause by noise from the P,B frames before an I frame, which suddenly disappears when the I frame comes up (its a slow-moving scene). Hmmm.

[RobertR]

Originally posted by Peter Cheat
I'm glad he is. My changes are just experiments to improve the codec in the future, not to use now. It completely sux at the moment. I fixed some code yesterday that improved the picture significantly. My changes are for testing to see what happens. You need to sometimes make one step back before making two steps forward.

My aim is to change quantiser distribution so that lower quants are given to spatially complex frames, and higher quants given to temporally complex frames (you can see detail well during motion). At the moment, libavcodec doesn't do this. There are also many other issues that need resolving.

Please please please do us all a favor and subscribe to FFmpeg mailing list. It's rather low volume. All the developers are there so you can present your point of view.
I'd also like to ask you to either post a diff to sources or full sources of libavcodec/mplayer/ffmpeg/whatever-u-are-changing. I've spent most of evening butchering my mplayer sources so they will compile (i;m using cvs almost daily updated). If you lack web space i can provide you with some :)

[Nic]

Keep up the good work Peter, I'm sure you'll get to your goals. I'll wait till your next source code release with your new bugfix and then release a new QuEnc for people to test your changes :)

@RobertR: Please stop repeating your request to Peter, I'm sure Peter will feel more happy about his code being picked apart by those on the ffmpeg list when he's finished with it. Rather than people criticising what is unfinished code.

Cheers,
-Nic

[RobertR]

@Nic
Point taken.

@Peter Cheat
I'm very very sorry for pushing.
I just thought that cooperating even at early stages can just get Your code better not to mention the chances to fold it into main stream sources (as i understand Michael Niedermayer is the one who governs which patches are accepted and which are not). All i really want is free MPEG2 encoder that can beat CCE and can run on diff operating systems. FFmpeg is very promising in both areas.
For now i'll just shut up and keep testing :)

[dragongodz]

I have a clip that pulses when encoded with the original QuEnc 0.54 dll. The pulsing is strange as the pulse is cause by noise from the P,B frames before an I frame, which suddenly disappears when the I frame comes up (its a slow-moving scene). Hmmm.
thats a different type of pulsing. thats more like rate control running out of bits for the later frames in the GOP. so bit distribution would need to be refined to fix that properlly i would think. some more tweaking to settings may help reduce it aswell but i dont have time to at the moment.
the normal pulsing is bad I frames ,which can be seen using the normal FFMpeg settings, with any type of footage. the change to settings i used fixes that only, atleast in the tests i did. :)

[Peter Cheat]

@RobertR
All my modifications of sources is basically butchering the current CVS. The reason for not becoming a member of the mailing list is too not stir up a huge fuss and flame wars. To them, I'll be some try hard n00b wrecking all of their hard work (in their eyes), which is not what want. Copying my sources over the latest CVS should work without problems (but no guarantees there).

What I am trying to do is address issues that have been ignored for some time. But before I started modifying code, I knew little about video compression, very little C (but I know Java pretty well) and very little about rate control. I had to read dozens of documents before i could get started on the code, and I had to read the current code line by line to work out how it worked exactly. This all took time. Now I am just testing out different algorithms for rate control etc. They are unfortunately, tuned to specific video scenes that I test with right now. A generalised rate control algorithm is the aim. Atm, it isn't far from this, it just needs refinement.

The idea is that I listen to input and feedback, and make changes/improvements that the general encoding public want done (this mean you). The more input/feedback I get, the better the code will be.

Currently, I can't update with anything new as I am halfway through a rewrite of the bit allocation for 2nd & 3rd pass (note that the third pass atm is a bit of a hack). This will improve the quality of some low-motion scenes that are currently attacked with too high quants. This will be finished by Wednesday (I hope). I don't actually have any free time right now, I code during breaks and late at night.

My final aim is to have decent code by Christmas that I can submit to the CVS. At the moment, it isn't even close...

The only part 100% done is the 'compressibility' parameter used to predict the future frame size/3rd pass frame size. This is a good approximation based on the fact the the relationship between texture bits and quantisation level is linear (linear quantisation). This IS better than how it is originally done, BUT cannot be implemented without a complete rewrite of rate control.

I'm out,
Peter Cheat

[mean]

For the crash, it may be due to a win32 quirk
When you switch from reading to writing on a file open in rw mode
you MUST do a fseek(0,SEEK_CUR)
in between

Don't know if that's the problem here, but it took me some time to figure it out on another stuff i'm working on.

HTH

[dankescheen]

Hello L&G,
a short test ( 4:3 PAL-clip, 2616Frames, 720x576 - Encoder-Settings: AVG3100 / max8000 / GOPsize 15) gives me strange results :

V0.54 / avcodec Peter Cheat (2pass)

AVG 8658 Max 18684 Filesize 136 MB !!!

Very bad quality - pulsing & blocky i-frames


V0.54 / avcodec Peter Cheat (1pass)

AVG 2401 Max 3797 Filesize 38 MB

quality is OK


V0.54 / avcodec memaligned (2pass)

AVG 2483 Max 5101 Filesize 38,7 MB

quality is OK


V0.54 / avcodec original (0.54package) (2pass)

AVG 2447 Max 4499 Filesize 38,7 MB

quality is OK


V0.51 / avcodec original (0.51package) SinglePass !!

AVG 2439 Max 4534 Filesize 38,4 MB

quality is OK


Sorry boys, I can't see any positive effect in the enhancements ....
I never had any problems with Nic's last version. I always set the max. Bitrate at 8000 and the AVG between 3000 - 5000. (Yep, my Excel-Sheet works great). SinglePass is OK for me, because more passes only take more time without a real quality difference.

Cheers & keep up your work :)

[Amnon82]

I wait for christmas ... santa :D

[Peter Cheat]

Originally posted by dankescheen
Hello L&G,
a short test ( 4:3 PAL-clip, 2616Frames, 720x576 - Encoder-Settings: AVG3100 / max8000 / GOPsize 15) gives me strange results :

V0.54 / avcodec Peter Cheat (2pass)

AVG 8658 Max 18684 Filesize 136 MB !!!

Very bad quality - pulsing & blocky i-frames

Yes this does happen. My last update does not work correctly with QuEnc. The changes I have made the DLL significantly different. This is why QuEnc needs to be compiled with the updated library to work correctly. But this is not important for now. My changes seem to suck for DVD resolutions, and are better for SVCD resolutions, where the content is relatively low motion with few high to very high motion sequences.

Rate control will be changed completely by Wednesday, it is half done, bits are now distributed to the GOP as a whole, and bits are then prioritised with I being highest, P frames next, and B frames last. The code is better structured, and now that I am looking at mpeg2enc, will be similar to that.

[Nic]

So how's it all going Peter? Still working on it?

-Nic

[Peter Cheat]

Assigning bits to frames did not work properly. You end up with poor quality overall. I'm going back to a more traditional approach, and will upload asap maybe with an example encode.

[dragongodz]

though assigning bits to frames is possibly good for CBR. something for the future when VBR is working better. :)

[Peter Cheat]

The problem was that the prediction seemed to be off. So it wouldn't work for CBR either. But I found an error in my code which would have been a cause to the problem. I'll try it out again if when I get some time (tomorrow) because, as you said it could be good for CBR. Kill two birds with one stone, so to speak.

[RobertR]

I've tested your ffmpeg version under Windows (could You please post your sources too? my ability to test anything under windows is somewhat limited). This time i was impressed with quality of encode (I aim at DVD backup so am using bitrates around 4000 kbit/s). I have to admit that last time i used command line that was posted on Your website and results were umm.. erm.. baaad. Anyway this time encode was just perfect. I also compared encodes made by 'normal' libavcodec (made with mencoder under linux) and 'your' encode using Bitrate Viewer.
Average bitrate was set to 4000kbit/s and max bitrate to 8000 kbit/s. With normal libavcodec bitrate never reached 5500 kbit/s, modified libavcodec it went up to 6000kbit/s (might be even more i'm writing it from memory). Bitrate curve in 'your' encode was more like bitrate curve in the original cell, while 'normal' libavcodec created more flat bitrate curve.
For me it's awsome :) (at least on the sample i could test under windows ;)

[Peter Cheat]

It is slowly improving. The sources will be uploaded soon. I won't upload it yet, I am still trying to change the rate control to a bit assigning algorithm rather than quantiser assigning algorithm. The only advantage of the bit assigning algorithm will be speed, but this is probably necessary if you try to encode a long sequence as the rescaling routine will take a long time, and they may happen quite frequently, especially during high motion scenes. Assigning bits will mean that quantisers will not have to be rescaled, bits will just have to be added/subtracted proportionaly during the encode as the predictor is not 100% accurate. I have other things to do at the moment so this work has been put to the side momentarily, but I will update very soon.

[dragongodz]

good to hear from you Peter.

turning in to major task isnt it ? you can probably see why not alot has been done with it for some time. :D

[Peter Cheat]

I think that the reason that nothing was done to it was because it required almost a complete rewrite to stop the max bitrate from being exceeded and to prevent underflow. Many people who spent a large amount of time creating quantiser blurs that follow differentiable functions will probably pissed off to see this code absolutely destroyed and replaced with less mathematically elegant code that just does the job. If I didn't have to try to work out how the code work originally, it would have saved me a lot of time! There is only a few hours of coding to have it finished, I just have assignments and other things to do, which have higher priority.
*Goes back to writing 40+ page report*

[dragongodz]

I think that the reason that nothing was done to it was because it required almost a complete rewrite to stop the max bitrate from being exceeded and to prevent underflow
actually i did read that Michael(of FFMpeg) has partially written a different rate control from scratch aswell. his plan is/was to have it selectable(so you could choose old rate control or the new one). i think your version will be ready before his is though as he seems to be working on other parts of FFMpeg ,such as the snow codec, most of the time. :)

less mathematically elegant code that just does the job.
ye no doubt some will complain its not elegant or neat etc etc etc but who cares. its the end result that matters first. once its working people can clean it up or whatever they like.