Is 3 passes really enough??

[HKT3020_1]

I don't understand why the encoding took so long, I had CPU resources to spare. I logged only 49% of CPU use and on average the speed on CCE was 0.51-0.53. :mad:
[23:41:49] Phase II ENCODING started
[03:30:44] Phase II ENCODING completed in 1669 minutes.
[03:30:44] Phase III, REBUILD started.
[03:46:22] Phase III, REBUILD completed in 16 minutes.
The DVD looked great by the way, thanks for all the help. :p

[Boulder]

Avisynth is not multithreaded and with slow scripts most of the CPU usage comes from Avisynth processing. That is, CCE spends most of its time waiting for Avisynth to serve the frames to encode.

Welcome to the 24hr club :p

[jdobbs]

Wow. I'm not sure I have that much patience.

[jdobbs]

There have been some posts about the difficulty with Saving Private Ryan in the past. (I don't have this disc).
I wonder if anyone ever has tried redistribution on that difficult disc, and if so, did it help?I tried this again with REDISTRIBUTION using CCE. It actually came out worse than the one that wasn't redistributed. The bitrates didn't change dramatically (a surprise to me), and two of the segments came out too low, causing blockiness.

I'm testing now with HC.

[Sharc]

I obtained good results - for my eyes - for SPR (PAL) with the following settings:
- Encoder: CCE (for redistribution and encoding)
- Filter: cpu=4 (efficient suppression of macroblocks, with loss of some details in flat areas)
- Redistribution enabled, 2-pass encoding
- Matrix: Number 6 of RME (for redistribution and encoding)
- Q_final for redistribution = 107
- Cell 9 manually tweaked to 1300 (instead of 805, which is apparently too low, causing blockyness)

Overall reduction was 62.5%.
I selected Q_final for redistribution because I noticed significant differences in the redistribution profile between Base_Q (34) and Q_final (107) for HC/lumgain1 during my tests. Q_final produced similar profiles for CCE and HC, even for different matrices, so I felt more comfortable using Q_final.

[tom942]

@Sharc

Where do you get "Q_final", with RB-OPT?.

[Sharc]

Yes.
You might also get it from DVD-RB using OPV.
I didn't try the encoding with Base_Q (34) profile of DVD-RB, so I don't know how the result would have been. From earlier tests I found the risk of a cce crash during redistribution to be lower with a higher Q. That's another reason why I went straight to Q_final (=107 for matrix #6).

[JohnGalt]

@Sharc

That error should have been addressed in 1.26.3, unless you're talking about some other problem.

- Added an option to the MODE menu in which you can choose to always use HC as the encoder for REDISTRIBUTION. This prevents a rare CCE problem that can happen when high bitrate matrices are in use.

[Sharc]

@JohnGalt
Yes.
I found the HC profile with Base_Q (34) to be a little "flat", and in particular much different from the profile with HC or CCE for Q=107 and the said matrix. But as I said, I didn't try the encode with Base_Q and HC for redistribution, so I cannot judge the result. May be I will try later.
I just posted a setting which I found to produce a reasonable result for a difficult disc (long duration, film grain, smoke, dust, water, action). There might be better settings, though.

[jdobbs]

Could you show me an example of the difference between the distribution (in bitrate) between the two Q values? I wouldn't have thought it would be significant.

[Sharc]

Here 2 graphs for comparison:
Both using HC for redistribution, but with different Q.
Matrix for redistribution was mpeg standard in this case.
I have seen this dependence of Q in other cases as well.

If, however, the Q is selected such as to yield the same file size (Q_final for example for a DVD-5), the redistribution profile becomes almost independent of the matrix and the encoder. That's the reason why I normally prefer Q_final for redistribution. Because Base_Q and Q_final are for standard matrices often similar, the difference in the redistribution profile is normally not so obvious. The effect is more pronounced for high bitrate matrices.

[kumi]

I can confirm everything Sharc has said about the Base_Q<->Q_final discrepancies. It's the reason I use RBOpt for any projects that don't use the default matrix, since it calculates Q_final and applies it to redistribution.

I wonder if jdobbs would consider adding a Q calculation step to the redistribution process. It would make it safe to redistribute using custom matrices.

[jdobbs]

I'd have to be convinced that it would actually make it better (I'm not yet). I'd suggest in the example given that cell 9 looks really, really bad (mine did, if this is SPR -- and it had the higher of the two bitrates). Big swings aren't necessarily a good thing.

[Sharc]

I agree that big swings do not simply guarantee best results. However, I have seen - for high bitrate matrices - that Base_Q (or any Q which is significantly lower than Q_final) produces very flat curves (CBR like), much flatter than the standard 2-pass VBR mode. It also appears that these are the cases where CCE in tendency aborts and HC slows down - in particular when the VBV check is enabled. This made me put a little question mark.

Cell 9 in SPR appears to be an exceptional case which requires manual tweaking. Perhaps one should introduce a minimum bitrate parameter in the rebuilder.ini (e.g. 1300) to prevent redistribution from suggesting unrealistic low bitrates?

Added:
Could it be that a low Q in combination with a high bitrate matrix simply drives the bitrate to excessive high values? What does the encoder do then? Set the bitrate to the maximum compliant with the mpeg standard? Could this be the reason for "flat" redistributions? Or does the encoder crash? .... ??

[archaeo]

Just a note - attachment is still pending approval - still not able to see what Sharc has posted - thanks

[Wombler]

I obtained good results - for my eyes - for SPR (PAL) with the following settings:
- Encoder: CCE (for redistribution and encoding)
- Filter: cpu=4 (efficient suppression of macroblocks, with loss of some details in flat areas)
- Redistribution enabled, 2-pass encoding
- Matrix: Number 6 of RME (for redistribution and encoding)
- Q_final for redistribution = 107
- Cell 9 manually tweaked to 1300 (instead of 805, which is apparently too low, causing blockyness)

Overall reduction was 62.5%.
I selected Q_final for redistribution because I noticed significant differences in the redistribution profile between Base_Q (34) and Q_final (107) for HC/lumgain1 during my tests. Q_final produced similar profiles for CCE and HC, even for different matrices, so I felt more comfortable using Q_final.

I've tried SPR again myself and your settings are definitely better than my original effort.

I'm glad I suggested this film as it's proving very interesting for the experts and educational for me.:)


Wombler

[Wombler]

Cell 9 in SPR appears to be an exceptional case which requires manual tweaking. Perhaps one should introduce a minimum bitrate parameter in the rebuilder.ini (e.g. 1200) to prevent redistribution from suggesting unrealistic low bitrates?


Is it not a bit difficult though to determine either if the bitrate is unrealistically low or if the source just happens to be highly compressible?

Credits/intros or fade-ins/fade-outs can in certain instances drop to very low bitrates without any substantial deficit.

If you specify a minimum bitrate that's possibly too low then you risk not optimising the remainder of the film.

I suppose the key would be finding a happy medium between the two.

Sounds like a good idea though.


Wombler

[Boulder]

I wonder if that ninth cell is the one that required some manual intervention (=using CCE and tweaking the .vaf file after the first pass) before it looked good when encoded. The encoders seemed to allocate too few bits there no matter what the matrix or the other settings were.

SPR also has a very low avg bitrate for such a bitrate-hungry material, which will make things even more difficult. I have one CBR-encoded TV series DVD where redistribution also makes one scene look really bad (smoke, flat greenish colours). Then again, the other scenes look much better than with a non-redistributed encode. It's a kind of "choose your poison" decision there.

[Sharc]

@Wombler:
It is a compromise, yes.
The lowest redistributed cell bitrate I have seen so far without producing artefacts is around 1000 kbps (dark scenes, like in WTC). Even end credits are normally higher. If one goes below 1000, even simple dark scenes tend to collapse. If we would now lift the 1000 to "luxury" 1300 that wouldn't cannibalize the rest of the movie much, since there are not many "1000 scenes" anyway.
Just some thoughts. Might require more tests though.

[Sharc]

I wonder if that ninth cell is the one that required some manual intervention (=using CCE and tweaking the .vaf file after the first pass) before it looked good when encoded. The encoders seemed to allocate too few bits there no matter what the matrix or the other settings were.
Yes, exactly. It helped to manually set the average bitrate of that cell to 1300 - the original redistributed bitrate was in the range 800 .... 1200.
1300 brought it just around the corner, sort of a threshold effect.

[jdobbs]

The idea, though, is to create a process that doesn't require manual intervention. The majority of users only want to check the option, and not have to review the data to see if it is good or bad. The process should result in the assumption of "good".

The problem with setting a "minimum" bitrate is that there are so many exceptions. What happens when all the segments are under the minumum? What happens when the total of the ones under the minimum creates an oversize? When you raise the bitrate for one that doesn't meet the minimum you have to steal the bits from somewhere else... where? etc and so forth...

[Sharc]

Sure, if the process can be automated, the better.
Actually I thought it could be automated:
After the prepare the bitrates are known. The ones below the minimum would be raised to the minimum, the required bits would be stolen in proportion from the rest. I agree, not so straightforward. I also do see the problem with the "right" value for the minimum. It depends for example on the encoder.

I am just about to encode SPR with HC, matrix #6, redistribution with Q_final=107, and cell 9=680 (!). The test encode of cell 9 at 680 kbps looked surprisingly good .... may not require much tweaking, if at all ....

[kumi]

Just a thought: are we sure there isn't a CCE-specific problem when redistributing? I wonder if someone with SPR has tried redistributed it with HC, and checked whether or not cell 9 needs manual bitrate adjustment.

[Sharc]

See graphs above for HC. The low bitrate for cell 9 is not specific to CCE. CCE and HC produce similar low values for that cell. HC however seems to cope much better with the very low bitrate. Just about to run an encode with HC ....

[jdobbs]

Cell 9 is an exceptionally dark cell, which contributes to the low bitrate... it distributes the same way (low) with CCE, HC, and Mencoder, so it's the process -- not the encoder that is the source of the problem.

[Fishman0919]

@jdobbs

Won't it just be easy and more accurate to do a full 2 pass encoding with the avg bitrate you need with whatever encoder you would like to use (that way you can see how that encoder deals with the entire movie) then get the bitrate for each segment from the full 1 file encoding.

It would be a much longer process but it would also be the most accurate.

[jdobbs]

That would work -- but I'll tell you honestly, the minimal benefits you get from REDISTRIBUTION really aren't worth all that effort. You'd be better to just not use it in a case like SPR. It does cell 9 (and the rest of the movie) just fine. I'm have occasional doubts as to whether I should have ever even included REDISTRIBUTION in DVD-RB. It is very definitely overused. It solves the problem of CBR sources or poorly encoded originals -- but that's a very rare occurance. Others, of course, may have different opinions.

[JohnGalt]

So it's best not to leave redist. on all the time? I've used the option for several backups now, and I've been quite pleased with the results thus far. Granted, Lynch's 'Inland Empire' looked rather shoddy, but no worse than the original -- the lo-fi DV was, after all, an aesthetic choice on the director's part.

[Sharc]

Not a 1-click method, but what I usually do is:
- Select a matrix which is slightly "lower bitrate" than the matrix of the original. This is often matrix #2 or #3 of RME. It helps to preserve the sharpness.
- Do redistribution and 2-pass encoding with that matrix, using Q_final.
- Very exceptionally I test or even tweak suspicious low bitrate segments after the redistribution -- cell 9 of SPR with cce has actually been the only exception in my experience where manual tweaking was really requested.....

IMHO this gives better results than any other > 3-pass standard method - what the original question of this thread was about.

Sure, the degree of noticeable improvement compared to the standard 2- or 3-pass mode depends on the source material and on individual perception/preference. It is sometimes difficult to improve something which is already very good .... :)

[jdobbs]

So it's best not to leave redist. on all the time? I've used the option for several backups now, and I've been quite pleased with the results thus far. Granted, Lynch's 'Inland Empire' looked rather shoddy, but no worse than the original -- the lo-fi DV was, after all, an aesthetic choice on the director's part.
You're fine leaving it on... except in the rare instance such as SPR. I'm adding some code for the next release the will help prevent radical changes like the one in cell 9.

[Wombler]

You're fine leaving it on... except in the rare instance such as SPR. I'm adding some code for the next release the will help prevent radical changes like the one in cell 9.

Ahh excellent. It's good to see something productive has come out of this thread.

Although DVD Rebuilder is that good now that it must be getting more and more difficult to find areas to improve on.


Wombler

[Sharc]

@jdobbs:
Would it be too difficult to add a parameter under rebuilder.ini options to select Q_Final rather than Base_Q?
To me it makes a significant difference on the quality and effect of the redistribution, especially for non-standard (high bitrate) matrices. I personally don't care much about the extra time required for Q_final calculation. Q_final needs not even to have the same accuracy as for OPV - if speed of calculation should be important.

[jdobbs]

I can add it as a hidden option, but is there any way you can show some PSNR or other data that might support the idea that it improves the quality? I've done a quite a bit of testing -- and it just doesn't seem to be supported from what I've seen. It seems to be something along the lines of "measuring with a micrometer and cutting with an axe".

I guess it doesn't hurt to add it though.

[jdobbs]

Ok. For the next release I've added a "REDIST_USE_FINAL_Q=n" parameter to the "[Options]" area of the INI. If set to 1, a set of prediction passes is run against the main movie vts (same as OPV). The Q from that prediction is used as the FINAL_Q value for REDISTRIBUTION. Not sure if it will definitively improve quality in any way -- but it surely won't hurt it, so "C'est la vie"...

[Sharc]

Thanks a lot. I guess I will set the new parameter to 1 .... :)

[jdobbs]

Yeah, I probably will too. :)

I'm testing it now. I had to make a couple of adjustments for when "Always Use HC..." was checked.

[kumi]

That's brilliant, thank you jdobbs :D

[JohnGalt]

Ok. For the next release I've added a "REDIST_USE_FINAL_Q=n" parameter to the "[Options]" area of the INI. If set to 1, a set of prediction passes is run against the main movie vts (same as OPV). The Q from that prediction is used as the FINAL_Q value for REDISTRIBUTION.

When the next release comes out, perhaps someone could write a short guide explaining all this? I've been following the redist. threads and have a vague handle on it, but I'm rather lost with this latest turn in the discussion regarding Q_FINAL vs. BASE_Q and so on.

Also, does this redistribution method take filters into acc't (or does it matter)? I almost invariably use Half+Half + KISS on extras.

[jdobbs]

Yes, filters are taken into account.

One of these days I need to write a complete user guide...

[stereo]

When the next release comes out, perhaps someone could write a short guide explaining all this? I've been following the redist. threads and have a vague handle on it, but I'm rather lost with this latest turn in the discussion regarding Q_FINAL vs. BASE_Q and so on.

Also, does this redistribution method take filters into acc't (or does it matter)? I almost invariably use Half+Half + KISS on extras.

I second that. It would be great to have a short guide, because lately, I too have been pretty lost in the redist. area.

On that note: It would be great to have a guide - or a sticky - outlining all the hidden options that are now available.

[auldyin]

I echo the comment by stereo

Cheers

[Wombler]

One of these days I need to write a complete user guide...

That would be really useful as there's very little available elsewhere for the more technical stuff.


Wombler

[AGKnotUser]

You could try something like RemoveGrain(mode=5,modeU=17).TemporalSoften(2,3,4,8,2) instead of Degrain for faster processing. You need the RemoveGrain plugin, TemporalSoften is an Avisynth internal function.

Do you still need to add cpu=4?

[Boulder]

I'd still use that.

[AGKnotUser]

You could try something like RemoveGrain(mode=5,modeU=17).TemporalSoften(2,3,4,8,2) instead of Degrain for faster processing. You need the RemoveGrain plugin, TemporalSoften is an Avisynth internal function.

I read in the RemoveGrain docs: "RemoveGrain modes 1-10,17-25 are for truely progressive input only". Should Mode=5 be changed to another value for interlaced?

[Boulder]

You shouldn't use any mode of RemoveGrain directly with an interlaced source. There's a lot of threads which discuss dealing with interlaced sources.

[AGKnotUser]

Thank you, I'll look them up.

[Wodan]

Do you still need to add cpu=4?

where do i put cpu=4? in the rebuilder.ini ? also should the vbr bias always be set to 0? same with qual prec?

[Sharc]

In the rebuilder.ini under [Options] include the line
MPEG2SOURCE_OPTS=cpu=4

[HKT3020_1]

I wasn't exactly sure if a new thread should've been created. I'm new when it comes to mobile encoding with DVD-RB. I used the default options & the iPod preset 16:9 (640x360) 800kbs and saw heavy pixelation during fast moving scenes on the latest Season of Boston Legal. Any suggestions from long time mobile encoding users?
I have the latest 4GB iPod Nano if that helps. :confused: