Herbie: Fully Loaded [Archive]

magic144

28th October 2005, 06:12

magic144

28th October 2005, 06:16

fyi

so we're on the same page

did an RB run with movie-only (no menus or extras) and kept the English soundtrack (6 ch) and English DC soundtrack (2ch) as well as the English subtitles (Stream 01)...

also, here's my "Versions" output: - I used CCE for the encode...

- Versions :
-- CCE SP Version: 2.70.2.1
-- HC Version: 0.15.0.0
-- DECODER Version: UNKNOWN
-- AVISYNTH Version: 2.5.5.0

magic144

28th October 2005, 06:29

if I also might venture
the problem 'may' be related to the DISC=yes at the start of the immediate next cell (Cell 07) as listed by IfoEdit in the SP/ILVU/DISC/SA column

hope this puts you on the trail

m

fyi on my Aspire Digital AD-900 player, the same point plays almost glitch-free (a tiny stutter if you stare long and hard enough!)

magic144

28th October 2005, 06:46

as you will see
this is EXACTLY the place where IfoEdit identified the Layer Break in the original fileset (from the original IFO file)

I can hear your stomach churning at the layer-break comment already...

m

magic144

28th October 2005, 07:04

next pointless (or not) comment

the DVD2One output of the same disc (same streams) plays back OK on the Sony - only noticeable artifact is a tiny tearing (combing-like effect) between frames at the "layer break", almost too quick to notice

magic144

28th October 2005, 08:25

ok next point of interest

ran the RB output thru VobEdit to demux
and then thru muxman to remux
using ChapterXtractor to read the chapter points from the source

the output DVD plays fine on the Sony
and there's no sign of a Layer Break transition in the output - what does this mean???

m

jdobbs

28th October 2005, 14:57

@jdobbs

hello again
here we go again

I've just run Herbie: Fully Loaded (Canadian R1) thru movie-only backup
and I'm seeing SCR values <= DTS and PTS in the resultant VTS_01_1.VOB

it's weird coz it's only for a brief spell between lba=58160 and lba=69388

anyway, possibly irrespective of that, I've got a hangup/gets-stuck on the Sony player at about 41:47 which is almost exactly where a cell change occurs (about the beginning of cell 7, see attached jpg)

over to you for analysis

m The SCR is supposed to be less than DTS and PTS...

1. What does IFOEDIT say for that point after the rebuild?
2. What happens if you add "LAYER_BREAK_REMOVAL=0" to the REBUILDER.INI "[Options]" area and reaccomplish the REBUILD?

magic144

28th October 2005, 16:50

err yeah, sorry my head was obviously scrambled
I meant SCR was > either (or sometimes both) DTS and PTS for that interlude

I'll try another build with your LAYER_BREAK_REMOVAL idea and let you know the outcome

what do you mean by Q1
"1. What does IFOEDIT say for that point after the rebuild?"
I included the jpg for you to see exactly what IfoEdit showed it as... or did you mean what does IfoEdit show the muxman rebuild point as?

in any case, here is the IfoEdit display for the muxman rebuild... as you can see, the cell structure has changed to make every cell a chapter

I should mention that there are no SCR discrepencies with the muxman remuxed version (even though there were those noted SCR discrepencies in the RB output original that muxman was working with)

jdobbs

28th October 2005, 17:30

You'd said:this is EXACTLY the place where IfoEdit identified the Layer Break in the original fileset (from the original IFO file) so I had assumed the jpg you'd posted showed the values from the original IFO. So you're saying it is from the one that was already competed by REBUILDER?

magic144

28th October 2005, 17:39

sorry if there's a confusion

the attachment from post#1 in this thread (Clipboard.jpg) is from the RB rebuild.

the attachment from post #4 in this thread (Clipboard2.jpg) is from the ORIGINAL IFO.

the attachment from post #8 in this thread (Clipboard3.jpg) is from the muxman remuxing of the RB rebuild (i.e. it is the IFO AFTER muxman has remuxed the original RB output)

hope this clarifies

do you have the disc? I'd be interested if you can reproduce the problem(s) - I explained the rebuild I did in post #2

m

jdobbs

28th October 2005, 17:42

No... but I can pick it up.

Can you tell me what percentage of reduction it showed in the log? I've added a correction for v1.02 and was wondering if it is related.

jdobbs

28th October 2005, 19:44

Well I just finished running this movie, R1 using the Movie-Only mode and keeping the audio you called for... it plays fine. I also scanned the entire set of output files -- no SCR anomalies found.

Are you leaving something out? Preprocessing? What version of DVD-RB?

magic144

28th October 2005, 21:36

did you keep the subtitles too (English)??

FYI, I don't do any preprocessing as I know it interferes with your debugging abilities and I want you and I to be on the same page :-)

I'm also attaching a jpg of the Input Settings page

I'm about to run it thru again
here's the Preparation log...

[13:27:53] Phase I, PREPARATION started.
- CCE SP 2.70.2.1 encoder selected.
- "Movie Only" mode is enabled.
- VTS_08: 2,612,922 sectors.
-- Scanning and writing .D2V & .AVS files
-- Processed 145,168 frames.
-- Building .AVS and .ECL files
- Reduction Level for DVD-5: 95.8%
- Overall Bitrate : 6,705/5,364Kbs
- Space for Video : 3,964,480KB
- HIGH/LOW/TYPICAL Bitrates: 7,771/3,021/5,364 Kbs
[13:31:52] Phase I, PREPARATION completed in 4 minutes.

jdobbs

28th October 2005, 21:50

Could you sent your REBUILDER.INI to dvd-rb@dvd-rb.com? That way I can make sure I have exactly the same settings as you.

jdobbs

28th October 2005, 22:16

By the way... my preparation log looks exactly like yours... all sizes, rates, etc. are the same.

Carpo

28th October 2005, 22:36

could it be bad rip (drive old ?)- or hard drive issues (fragmentation bad sectors ?)

jdobbs

29th October 2005, 00:21

Nope... I was just able to force it. I'm working on it now.

magic144

29th October 2005, 00:40

eh? force what?

I'm still running my retest, the results of which I'll post here asap...

ps - after I'm finished this iteration, I'll try your suggestion of the
LAYER_BREAK_REMOVAL=0
line in my ini file

m

jdobbs

29th October 2005, 01:15

It won't help. Here's what I found:

The original was done at an incredibly high bitrate... and according to bitrate viewer is out of spec... it has a peak of 10382Kbs for video alone and an average of 9396Kbs -- either of which gets ridiculously high when you add the 448Kbs+192Kbs for audio.

I think bitrate viewer may be giving the bitrate incorrectly -- maybe its including the repeated fields... but...

Since this was Movie-Only mode, the reduction level was only 95.8% -- keeping the bitrates very high. Even worse, I found that the output of CCE reached a peak of 9640 (and that is definitely without the repeated fields), even though the maximum bitrate was set to 9000.

I'm looking at options to see what I can do...

magic144

29th October 2005, 01:26

err,
are you saying a very high bitrate has caused these effects?
are you seeing the SCR anomaly now? or do you mean the layer break issue?

my rebuild is at 62% so it's still in progress

if the bitrate was too high, how come demuxing and remuxing with muxman didn't flag any errors?

magic144

29th October 2005, 01:35

and shouldn't CCE do what it's told with a provided maximum bitrate?

btw, thanks for you continued work and feedback - I appreciate it, and hopefully if it contributes to continual product improvement, others will too (even if doing so unknowingly in the future!)

jdobbs

29th October 2005, 01:35

The output is falling out of spec... 9640 + 448 + 192 = 10280Kbs (plus a little for the subs). The maximum acceptable is 10080.

I had made the assumption that CCE would hold to its maximum bitrate. Now I need to go back and do some adjustments to try and account for that. I think that's what is throwing your Sony off...

It seems that these "stutter" problems are related to Movie-Only encodes in which the output bitrates are close to or greater than the original bitrates -- and, interestingly enough, they go away when using HC. I think the CCE bitrate maximum may be the cause. Luckily those kinds of high bitrates are pretty rare.

I'm going to run it again using CCE Basic (which is what I normally use). I think it may keep its max bitrate better.

magic144

29th October 2005, 01:41

at the (point of the original) layer break you mean? where my Sony hangs...

it didn't hang on the muxman rebuilt image - surely that hasn't altered the bitrate at that point

just to reiterate, I hadn't noticed any other problems with the Sony playback - just the hanging at the orig layer break point (the SCR probs were simply highlighted/noted by the ChkTim prog I got from this forum from a previous issue - I can't tie them to anything I can see on playback - in fact I don't know how to correspond an LBA address with a real track time)

magic144

29th October 2005, 01:43

just as a related item

is there an ini file override for specifying a manual value for max bitrate
(it might come in handy for this kind of thing!)

e.g. if you are calculating 9000, could a user specify something less manually?

jdobbs

29th October 2005, 02:07

I have no answer for the layer_break thing.. I think that if I fix this it may be indirectly the cause for what you're seeing.

Where DVD-RB ran into problems was in the reintegration of the original audio. There was a GOP that got so big that adding the audio at its original SCR points made it impossible for the GOP to fit within its time slice. So even with the minimum 146.286 clock steps between sectors, the total time accumulated overran the next GOP... that's really rare and is a side effect of the massive bitrate.

You can change the maximum bitrate with a setting under "[Options]" in REBUILDER.INI.

max_bitrate=xxxx

The default value is 9000.

jdobbs

29th October 2005, 02:11

I'm working on putting in some code that will determine the peak bitrate on the original, and use that to calculate a new maximum dynamically. I think that may fix this.

jdobbs

29th October 2005, 02:15

Go ahead and try the LAYER_BREAK_REMOVAL=0 solution and see what effect it has.

After you add it to REBUILDER.INI, just redo the REBUILD in three click mode (you don't need to do PREPARE or ENCODE)...

magic144

29th October 2005, 02:17

can I just ask
what problems have you seen at your end?
have you in fact found/reproduced an SCR discrepency now?

was this "too big" GOP around the point of my layer-break failure? or potentially an unrelated issue?

jdobbs

29th October 2005, 02:20

It plays fine on my end... but the Sony players seem to have no tolerance at all and a stream that is even slightly out-of-spec has been the source of problems on them before.

Yes I reproduced the SCR discrepancy. My original test wasn't using CCE SP because I thought it was a muxing problem... it is, but it was caused by the mixture of an extremely high bitrate (average 7771Kbs) and CCE's bitrate overrun (9640Kbs).

magic144

29th October 2005, 02:29

I have certainly read a lot about Sony player problems on this forum!

I used to swear by Toshiba, but the last 2 players I had were pants. A few discs were just unplayable with them. The Sony was the first player I had which played all my problematic store-bought DVDs sans problemo. The only gripe I have about it right now is it DOES have trouble with cadence - quite often you'll ffwd and be watching several seconds of combing coz it can't figure out the pulldown properly.

Well, maybe I'll forego trying your layer_break ini file instruction, if you think it will be redundant. Let me know what else I could try to give you any more ideas.

m

magic144

29th October 2005, 03:00

as perhaps could be expected, the re-build using the same params produced identical results

I don't think disk fragmentation is an issue worth exploring - frankly that sounds highly unlikely - wouldn't that simply mean that the file's contents were spread far and wide, and not that they were in error!!??

will let you know about the sans-subtitle rebuild shortly (still going on my laptop)

jptheripper

29th October 2005, 16:35

jdobbs, since dvd-rebuilder has the potential to be perfect, if in the rebuild stage an bitrate spike is detected, would there be anyway to have rebuilder make cce reencode the cell?

jdobbs

29th October 2005, 18:34

Reencoding wouldn't do any good -- it would spike exactly the same again.

This "problem" evolved into some pretty interesting research related to requantization of previously quantized material (as happens in DVD Backup). I spent most of the night analyzing this and several other DVD streams -- then spent most of today programming . The result is "Dynamic Peak Analysis" which I've added for the soon-to-be-released next version. Based upon my tests I think it will help more than just this problem (the reintegration of audio streams and the resulting overruns) -- it will prevent wasted bandwidth and improve quality.

magic144

29th October 2005, 18:48

sounds awesome - I look forward to trying it out!
do you ever sleep????

fyi, I'm about to do the layer_break rebuild so I'll let you know v. soon - I noticed that there is another SCR discontinuity at the start of Cell 2, but the Sony doesn't seem to choke on that (???)

what did you use to determine the CCE bitrate of 9640? I'm looking at the first file (VTS_01_1.VOB) with BitRate Viewer and it indicates a peak of 10929 - will this figure include the audio too??

jdobbs

29th October 2005, 19:11

In going back and doing some of this calculations manually, I'm not very confident in what I'm seeing with bitrate viewer as a measure for determining compliance. It seems to be reporting the playback bitrate rather than the encoded bitrate (which on sources using pulldown can be quite different, and for hybrid sources is even less useful).

When I mentioned it earlier I was checking against demuxed MPEG video streams.

jptheripper

29th October 2005, 19:20

when i said reencode, i mean with different parameters, i.e. different max bitrate sent to cce to avoid the spike.

i wonder if the difference between encoded bitrate compliance and playback bitrate compliance is why some players balk on some disks and others dont, is there different standards?

magic144

29th October 2005, 19:21

is there a straightforward way to work it out manually then? or is it convoluted - I wouldn't mind knowing how it's done

magic144

29th October 2005, 19:25

btw, did you figure out why muxman would be able to remux it satisfactorily? or are you not familiar with that prog too much - it seemed to do it OK without any complaints about mux rate (maybe it doesn't check compliance?)

only problem with the muxman solution however, was that the subtitle stream gets bungled somehow (most of it seems to get lost on the repack - don't know why)

just burning the layer_break test disc now - results imminent!

jdobbs

29th October 2005, 19:27

is there a straightforward way to work it out manually then? or is it convoluted - I wouldn't mind knowing how it's doneYou need some tools.

I just use code that I have that finds the start codes in the MPEG stream, you can then measure the size of the frame or GOP (depending upon how you want to look at it) and then calculate the bitrate based upon that size and the framerate.

jdobbs

29th October 2005, 19:31

I'm not really that familiar with muxman... but since its written by MPUCODER I have no doubts about its quality. I'd guess it checks for compliance. I'm not sure, though, whether compliance is checked against individual frames or GOPs (hmm... I should know that). It appears that bitrate viewer reports frame bitrates.

My problems here are related to my decision (way back with DVD-RB first started) to keep the audio/subtitle streams intact. Muxman remuxes the audio/substreams so it wouldn't run into that issue.

magic144

29th October 2005, 19:54

ok, the layer_break rebuild works better on the Sony

although it makes it pause for a second or so as if there was a physical layer break, which is, of course, impossible on a single-layered DVD-R!!!...

hopefully your new Dynamic Peak Analysis will alleviate the need to leave this layer break in the output and this disc will work like all the other DVD-RB successes!

I will try it when the next version comes out and let you know

thanks again for your tireless efforts

magic144

29th October 2005, 20:01

forgot to mention:

obviously, of course, the SCR discrepencies still exist on the layer_break rebuild - I was simply reporting on the layer break itself!

magic144

29th October 2005, 22:34

wowee!! 1.02 is here already!!

I'm running Herbie thru now... results to follow...

jdobbs

29th October 2005, 23:10

Shhh... it's still mailing. ;)

jdobbs

29th October 2005, 23:45

magic144

30th October 2005, 00:54

hmm, I thought that whole layer break issue had been put to bed a few releases back

RB used to leave the LB in the image, but then you put in some processing to remove it - I've processed lots of discs since then and never had an LB problem again until this little bugger :-)

I hope this build takes care of it!- I'm at about 62% and counting... (and crossing fingers) - you had said you thought the one problem might have been inducing the other...

the SCR issues did make a couple of noticeable glitches my 1.01 proc of Herbie, but that's nothing compared to the huge glitch of the layer break...

fyi, I tried the same disc on my Philips DVP642 and it didn't glitch at the LB - there's something about Sony's and clock references that don't tolerate each other very well...!!!

I'll update you when the new build finishes

magic144

30th October 2005, 02:49

well, 1.02 build removes all SCR probs
but the layer break freeze is still there

I don't know why, or what's special about this particular video
(except it seems to be across 4 VOBs for some reason)

cell 2 has a DISC=yes discontinuity showing in IFOEdit, same as cell 7
and both cells are starting new VOBs

but for some reason, cell 7 really throws the sony for a loop

can you see anything in the characteristic of the original layer break (cell 7) which is leading to this problem?? - it's a nuisance because it means I'd have to run RB with a special [Options] line for these special-case films (which I'd only find out about after I burnt the first attempted build image!), and even then, I'd have to put up with a stupid layer break pause for a layer break that isn't physically there!!

good work on the bitrate analysis - that's totally awesome

magic144

30th October 2005, 22:18

just an update on this disc

I finally made a full movie-only disc with 2 audio tracks, the English subtitle track and NO layer-break issues with the Sony

Step1 - Run the disc thru RB 1.02pro (WITHOUT special-case layer-break processing, i.e. WITHOUT the special-case [Options] line) - keeping 2 audio tracks and English subs

Step 2 - Demux the video, both audio and the subpicture streams - THIS is where I was having huge problems (tho I assumed it was Muxman doing something faulty - shame on me! - that's what happens when you ASS-U-ME things) - there is something different about the subtitle track on this disc and VOBEdit doesn't demux it correctly - it APPEARS to have demuxed it OK, but it doesn't work with Muxman (or DVDAuthorGUI which I also tried) - the breakthru came when I found another free tool called PGCDemux which produced a .sup file which worked!!! (thankyou PGCDemux!!!)

Step 3 - Remux all the streams with Muxman (free edition works just fine) - use IFOEdit to reimport the colours for the subs from the original's IFO (with which you can also use IFOEdit to export the txt file for the chapter points, which you can edit as you like)

at the end of this, all is well - the disc is re-authored and does not suffer from the layer-break freeze on the Sony - if you look at the new IFO, the cells have been reassigned so there is precisely 1 cell per chapter (the original had a couple of extra cells, including 1 at the layer break)

I guess I'll have to use this process again with problematic RB builds at this weird layer-break anomaly - hopefully jdobbs can figure out what is causing this hang/freeze some time in the future (he's rather good)

cheers all

magic144

1st November 2005, 03:08

just to follow up

apparently, I've read somewhere else, that VobEdit doesn't (always?) properly handle demuxing subs for titles with multiple VOB IDs - and Herbie has about 4, hence I guess all the subs get lost after the first one and cause woes in remuxing

just something to keep an eye out for in future

Video Dude

1st November 2005, 03:53

You might want to try removing the layer break before using DVD-RB.

Disclaimer: jdobbs does not support preprocessing.

2Cool's IfoEdit Layer Break Removal Guide (http://forum.doom9.org/showpost.php?p=263784&postcount=6)

Running the DVD through DVDShrink with "No Compression" will also remove the layer break (make sure you have that option enabled in the Preferences).

I always remove the layer break before using DVD-RB. I've always done that and never had any freeze or stutter where the layer break once was.

magic144

1st November 2005, 04:25

Video Dude,

as you will see from, e.g., this thread

http://forum.doom9.org/showthread.php?t=77696&page=3

there is something much more involved going on with the Sony "layer break" playback problem than just adjusting the break cell's Seamless Playback bit

for one thing, the SCR is usually reset at that point and therefore discontinuous (DISC=yes), but I don't believe jdobbs is attempting to change any of the clocking (SCR values) in the subsequent packets, so pretending they're seamless when they're not won't work (at least with the Sony, which seems to pay a lot more attention to SCR than many other players) - it's as if it suddenly comes across a totally unexpected value and gets totally lost

I might spend some time next weekend playing around with it to see what can be done, but I have no sophisticated in-depth knowledge of mpeg2 or DVD structures/internals

mpucoder might know a lot more about this, I dunno if he's knocking around anywhere and able to comment? it certainly seems that on a muxman remuxed build, all of the packets have been reassembled with a brand new SCR sequence which is continuously increasing throughout the duration