BD Rebuilder Beta - Bug Reports Only

[Sharc]

Yes. The originals were directly from ripped BDs. The comparison was done within AVISYNTH, they were AVC, 1080p, YV12, 8 bit. The encode settings were the command lines generated by BD-RB for output to BD (which puts compliance requirements on GOP size, maximum bitrate, etc).

Now I wonder how NVENC's standard bitrate encoding compares to X264. I suspect it will fall behind since it doesn't have a true two-pass option. But that will have to be something for a different day.
Another thought: Did your clips for the PSNR and SSIM tests include black letterbox borders? I am asking because black borders will bias the results in favour of the poorer encoder. For a fair encoder comparison any borders should be cropped off I think, as even a poor encoder can encode these perfectly.

[mikeq]

mikeq: could be a tsMuxeR muxing issue.
The reported offsets seem to match multiples of 23.976fps frame durations.
What has BDInfo to say about the source disc ?

Seems like it is - I've opened a bug on the tsmuxer site.
If I extract the streams and rebuild the BluRay - I get a similar problem.

[jdobbs]

Another thought: Did your clips for the PSNR and SSIM tests include black letterbox borders? I am asking because black borders will bias the results in favour of the poorer encoder. For a fair encoder comparison any borders should be cropped off I think, as even a poor encoder can encode these perfectly.I don't believe that is true. The black borders might increase the overall scores, yes. But I it would do so equally for both encoders. For example, in a completely black screen (at the beginning of the movie) both encoders get a perfect SSIM of 1.00 for those frames and a deviation of 0.0000 and a score of 113.0590 dB in PSNR. The same would hold true for the black border areas -- equally for both encoders.

I would also add that an accurate evaluation would compare the two encoders as they would be used. You wouldn't trim the borders in a BD backup (if you did the output would be non-compliant) or in a true 1:1 encode of an original.

[Sharc]

I don't believe that is true. The black borders might increase the overall scores, yes. But I it would do so equally for both encoders. For example, in a completely black screen (at the beginning of the movie) both encoders get a perfect SSIM of 1.00 for those frames and a deviation of 0.0000 and a score of 113.0590 dB in PSNR. The same would hold true for the black border areas -- equally for both encoders.

Hmmm .... let me expand my thoughts with an (overly) simple model:

Assume a picture with 50% black and 50% active picture area.
Now take 2 encoders ('Good' vs 'Poor'). Both encode the static black part perfectly, say with quality score 100.
The 'Good' encodes the active picture with quality 80, while the 'Poor' encodes the active picture with quality 60.

Case 1: with borders
Good (average) = (100+80)/2 = 90
Poor (average) = (100+60)/2 = 80
Difference Good - Poor = 10

Case 2: borders cropped
Good (average) = 80
Poor (average) = 60
Difference Good - Poor = 20

Hence for case 2 the quality differeence between the 2 encoders is more prominent.
In case 1 the black borders mask the deficiency of the poor encoder.

Am I totally mislead?

[jdobbs]

Hmmm .... let me expand my thoughts with an (overly) simple model:

Assume a picture with 50% black and 50% active picture area.
Now take 2 encoders ('Good' vs 'Poor'). Both encode the static black part perfectly, say with quality score 100.
The 'Good' encodes the active picture with quality 80, while the 'Poor' encodes the active picture with quality 60.

Case 1: with borders
Good (average) = (100+80)/2 = 90
Poor (average) = (100+60)/2 = 80
Difference Good - Poor = 10

Case 2: borders cropped
Good (average) = 80
Poor (average) = 60
Difference Good - Poor = 20

Hence for case 2 the quality differeence between the 2 encoders is more prominent.
In case 1 the black borders mask the deficiency of the poor encoder.

Am I totally mislead?But... if you go back to my tests -- you'll see that I chose a NVENC CQM value in which the SSIM and/or PSNR values are equal to or greater than that of X264. That would mean that at that value and size NVENC represented in the table would have to be achieving a BETTER score (using the logic of your post). If the borders were equal -- then that means the encoded picture portion would have to be a higher quality (via the metric). The output sizes don't show significant differences, so that would indicate roughly equal encoder quality.

See what I mean?

[Sharc]

But... if you go back to my tests -- you'll see that I chose a NVENC CQM value in which the SSIM and/or PSNR values are equal to or greater than that of X264. That would mean that at that value and size NVENC represented in the table would have to be achieving a BETTER score (using the logic of your post). If the borders were equal -- then that means the encoded picture portion would have to be a higher quality (via the metric). The output sizes don't show significant differences, so that would indicate roughly equal encoder quality.

See what I mean?
Yes, I think I see what you mean. You basically encoded for equal ("equal" within practical limits) PSNR (or SSIM) by adjusting CRF or CQM of encoders A and B respectively, means same objective final quality for A and B according to the metrics. You kept the borders -- as one would do in real BD-RB backup scenarios. I don't doubt the test method and the results at all, so don't get me wrong.
Eventually we may now want to compare the resulting file size and find that encoder A produced a file size which is say few % less than B (at 'same' PSNR or SSIM). Hence we conclude that encoder A is more efficient than B. I just wrapped my head around the question whether the conclusion would have been the same with the black borders cropped off because the borders would bias the result unequally for A and B. Confusing myself ... ;-)
Many thanks for the enlightening and useful tests.

[jdobbs]

Yes, I think I see what you mean. You basically encoded for equal ("equal" within practical limits) PSNR (or SSIM) by adjusting CRF or CQM of encoders A and B respectively, means same objective final quality for A and B according to the metrics. You kept the borders -- as one would do in real BD-RB backup scenarios. I don't doubt the test method and the results at all, so don't get me wrong.
Eventually we may now want to compare the resulting file size and find that encoder A produced a file size which is say few % less than B (at 'same' PSNR or SSIM). Hence we conclude that encoder A is more efficient than B. I just wrapped my head around the question whether the conclusion would have been the same with the black borders cropped off because the borders would bias the result unequally for A and B. Confusing myself ... ;-)
Many thanks for the enlightening and useful tests.There is a point (CRF=27.5 and CQM=32.5) where the NVENC output was 10% bigger. But, conversely, there was also a point (CRF=16 and CQM=16) where is was 12% smaller.

Just for S&G, maybe I'll go back and strip the borders to see what happens.

[Sharc]

There is a point (CRF=27.5 and CQM=32.5) where the NVENC output was 10% bigger. But, conversely, there was also a point (CRF=16 and CQM=16) where is was 12% smaller.

Just for S&G, maybe I'll go back and strip the borders to see what happens.
I made a test with my 1050ti. The result is not nearly as good as what you got with your 1660.
I aligned for same PSNR as a reference, using the commandline from BD-RB.

x264:
CRF=22 PSNR=44.81dB SSIM=83.00824601 filesize=3'307'459k 1.000

NVEncC:
CQM=25 PSNR=44.82dB SSIM=82.53126698 filesize=4'561'774k 1.379


For same PSNR the video filesize for NVEnc was 38% higher than for x264. For same SSIM it would be even more.
No borders in my test movie, but I don't think that this matters much. The big difference seems to be the HW.

[Mike-uk]

I made a test with my 1050ti. The result is not nearly as good as what you got with your 1660.
I aligned for same PSNR as a reference, using the commandline from BD-RB.

x264:
CRF=22 PSNR=44.81dB SSIM=83.00824601 filesize=3'307'459k 1.000

NVEncC:
CQM=25 PSNR=44.82dB SSIM=82.53126698 filesize=4'561'774k 1.379


For same PSNR the video filesize for NVEnc was 38% higher than for x264. For same SSIM it would be even more.
No borders in my test movie, but I don't think that this matters much. The big difference seems to be the HW.

1050ti does not support B-fame where as the 1060 does

[Sharc]

1050ti does not support B-fame where as the 1060 does
1050ti supports B frames for h.264 (AVC) very well. I am using it all the time.
It does however not support B frames for h.265 (HEVC).

[jdobbs]

I made a test with my 1050ti. The result is not nearly as good as what you got with your 1660.
I aligned for same PSNR as a reference, using the commandline from BD-RB.

x264:
CRF=22 PSNR=44.81dB SSIM=83.00824601 filesize=3'307'459k 1.000

NVEncC:
CQM=25 PSNR=44.82dB SSIM=82.53126698 filesize=4'561'774k 1.379


For same PSNR the video filesize for NVEnc was 38% higher than for x264. For same SSIM it would be even more.
No borders in my test movie, but I don't think that this matters much. The big difference seems to be the HW.Interesting. I wonder what has changed that could make that much difference?

Maybe that specific source? My scores were the averages across several discs. But, I don't think any of the individual scores that were used were that far off the average.

[cartman0208]

In this Wiki article (https://en.wikipedia.org/wiki/Nvidia_NVENC#Sixth_generation,_Turing_TU10x/TU116) 15% bitrate savings are mentioned, so yes, it must be the HW ... it does not explain 38% difference though ...

[Sharc]

@jdobbs, cartman0208:
It seems that the discrepancy is partially attributed to the GOP size.
For the tests I set the vKeyint in the alternate.txt to 48, and the x264 encoded accordingly.
Apparently the --bluray in the NVEncC commandline forced a shorter GOP of 30 which seems to have penalized the NVEncC.
Encoding at same GOP of 24 (default) the difference of the filesize between x264 and NVEncC for similar PSNR (or SSIM) becomes less: about 25..30% rather than the 38% as before.

[Sharc]

Interesting. I wonder what has changed that could make that much difference?

Maybe that specific source? My scores were the averages across several discs. But, I don't think any of the individual scores that were used were that far off the average.
The +38% filesize for NVEncC can be attributed do different GOP sizes, which was 48 for x264 (my mistake, '--bluray-compat' does not overrule it). It was however forced to 30 by '--bluray' for NVEncC which penalized NVEncC.
Redoing the test for GOP 24 (blu-ray compliant, BD-RB default) reduced the filesize difference to 20 ....30%.
Adding --lookahead 24 to the NVEncC commandline brought another improvement for NVEncC, so eventually I ended up with about 12% ... 16% file size penalty for NVEncC for 'equal' PSNR/SSIM in my tests.

Example (for my 1050ti):
x264 CRF=22.0, PSNR=44.82dB, SSIM=82.803, filesize=100% (reference filesize)
NVEncC CQM=25.8, PSNR=44.95dB, SSIM=82.613, filesize=126%
NVEncC CQM=25.5, PSNR=44.97dB, SSIM=82.756, filesize=115%, with '--lookahead 24' added to the NVEncC commandline.

[jdobbs]

Has anyone had any luck using the NVENCC command line options for iVTC? In BD Rebuilder, I currently force AVS input when doing iVTC (which means the encode is roughly the same speed as X264). I thought I'd try using the vpp options in NVENCC and see how well they work. When I try "--vpp-decimate cycle=5" for encoding on a source that uses the duplicated frame method -- I get a terrible jumpiness (compared to an AVS using tivtc, for example). VPP doesn't seem to find the duplicated frame very well at all.

[Sharc]

When I try "--vpp-decimate cycle=5" for encoding on a source that uses the duplicated frame method -- I get a terrible jumpiness (compared to an AVS using tivtc, for example). VPP doesn't seem to find the duplicated frame very well at all.
Should it be cycle=6 perhaps for 30/25 conversion?

[jdobbs]

Should it be cycle=6 perhaps for 30/25 conversion? If trying to convert from NTSC to PAL, yes. These are sources that were originally FILM (virtually everything falls into that category) that were converted to NTSC for broadcast. Many TV shows (especially older ones) were originally shot at 24fps and then telecined. I'm trying to get them back to FILM format.

[Sharc]

If trying to convert from NTSC to PAL, yes. These are sources that were originally FILM (virtually everything falls into that category) that were converted to NTSC for broadcast. Many TV shows (especially older ones) were originally shot at 24fps and then telecined. I'm trying to get them back to FILM format.
If it is telecined film, the IVTC involves 2 steps AFAIK:
a) field matching -> restores the progressive film frames
b) decimation -> removes the duplicates
In avisynth it is
TFM()
TDecimate()
I understand that the '--vpp-decimate' can only do step b).
I didn't try though, it's just my interpretation.

Edit:
Oh, there seems to be an IVTC method which does not require field matching:
https://forum.doom9.org/showthread.php?t=176657

[Mike-uk]

MKV import it creates a pseudo disc ?? it also converts .mkv to .m2ts without re encoding ?? does this mean i can burn the pseudo to a disc and it will play without spending the time to re encode it first ???

[Sharc]

MKV import it creates a pseudo disc ?? it also converts .mkv to .m2ts without re encoding ?? does this mean i can burn the pseudo to a disc and it will play without spending the time to re encode it first ???
BD-RB extracts the streams from the imported .mkv and creates an intermediate blu-ray file structure in the IMPORTS folder.
If the size fits to a disc you can directly burn this pseudo structure of the IMPORTS folder to a disc without re-encoding, and if you are lucky it's even compliant - depending on the streams in the .mkv source. However, to ensure that the final result is fully blu-ray compliant (size and format wise) a subsequent re-encoding step is normally required which involves much more (i.e. re-encoding the streams as necessary) than just packing everything into an .m2ts container.

[Mike-uk]

BD-RB extracts the streams from the imported .mkv and creates an intermediate blu-ray file structure in the IMPORTS folder.
If the size fits to a disc you can directly burn this pseudo structure of the IMPORTS folder to a disc without re-encoding, and if you are lucky it's even compliant - depending on the streams in the .mkv source. However, to ensure that the final result is fully blu-ray compliant (size and format wise) a subsequent re-encoding step is normally required which involves much more (i.e. re-encoding the streams as necessary) than just packing everything into an .m2ts container.

ok great cheers thats explains and answeres my queastion :)

[laserfan]

As we approach the twelfth anniversary of this thread and the "beta" of BD-RB, I would like to extend my best wishes to jdobbs & family for a Merry Christmas and a Happy New Year!

I think the first movie I tried w/BD-RB might have been Live Free or Die Hard and even though the conversion was at a measly 3.5KBps or something bitrate, it turned-out awesome and looks great to this day. Of course, in 2008 this was big stuff as we were sorta feeling our way around in those days. But I digress... see you in the New Year.

[Sharc]

Wow ...... it's even incredible 16 years including DVD-Rebuilder, more than 20'000 posts and countless working hours.
Thank you jdobbs, Merry x-mas and a Happy and Healthy 2021.

[Mike-uk]

Hope you have a good holidays Jdobbs, and thanks again for all you hard work on BD-Rebuilder , and to all the other people on here that have helped

[jdobbs]

Thanks everyone. I hope you all have a happy holiday season.

[Mike-uk]

hmm ok did and encode to bd25 with Nvencc used setting highest (very slow), i see from lastcmd.txt it is using --vbr why are we not using --vbrhq which then also enables multipass ?? also lookahead is not enabled ??

--codec h264 --preset quality --bluray --qp-min 0 --vbr 17504 --aq-temporal --keyfile "G:\BD REBUILDER\WORKFILES\VID_00351.CHP" --sar 1:1 --aud --pic-struct --vbv-bufsize 30000 --max-bitrate 35000 --gop-len 24 -o


yet setting "high quality (default)" has --multipass 2pass full enabled

--codec h264 --preset default --bluray --qp-min 0 --multipass 2pass-full --vbr 17504 --aq-temporal --keyfile "G:\BD REBUILDER\WORKFILES\VID_00351.CHP" --sar 1:1 --aud --pic-struct --vbv-bufsize 30000 --max-bitrate 35000 --gop-len 24 -o

[jdobbs]

hmm ok did and encode to bd25 with Nvencc used setting highest, i see from lastcmd.txt it is using --vbr why are we not using --vbrhq which then also enables multipass ?? also lookahead is not enabled ??

--codec h264 --preset quality --bluray --qp-min 0 --vbr 17504 --aq-temporal --keyfile "G:\BD REBUILDER\WORKFILES\VID_00351.CHP" --sar 1:1 --aud --pic-struct --vbv-bufsize 30000 --max-bitrate 35000 --gop-len 24 -o


yet setting "high" has --multipass 2pass full enabled

--codec h264 --preset default --bluray --qp-min 0 --multipass 2pass-full --vbr 17504 --aq-temporal --keyfile "G:\BD REBUILDER\WORKFILES\VID_00351.CHP" --sar 1:1 --aud --pic-struct --vbv-bufsize 30000 --max-bitrate 35000 --gop-len 24 -oIn my tests it slows the encode down... but doesn't improve the picture quality (per SSIM evaluation). In fact, the SSIM value slightly worsened when it was set (for that particular circumstance [--preset quality], not all settings).

All of the BD-RB settings are based on actual testing and output results -- not "what it should do".

[Mike-uk]

ah ok cool if youve tested with multipass and no difference then thats all good :)

[lithiumus]

hey JD, I'm using the Edit and blanking feature to remove various screens for Full BD / UHD backup (no re-encode) but the smaller ones around 1Mb or smaller do not show up for blanking. Is there any way to get all the M2TS files to show up so I can choose to blank them?

[jdobbs]

hey JD, I'm using the Edit and blanking feature to remove various screens for Full BD / UHD backup (no re-encode) but the smaller ones around 1Mb or smaller do not show up for blanking. Is there any way to get all the M2TS files to show up so I can choose to blank them?Have you set the following two options?

MIN_M2TS_SIZE=0
MIN_PLAYLIST_MINS=0

[jdobbs]

I have updated the first post of this thread with a link to the latest version of BD Rebuilder (v0.61.19). Changes for this release:- Corrected an issue in which attempts to
encode secondary video with NVENCC would
result in another encode of the primary
video stream by incorrect reference.
- Fixed an issue in which --vpp-deinterlace
might incorrectly be used in --avs mode
during NVENCC encodes, causing an encode
failure.
- Corrected an issue in which setting a
FIXED_CRF (with the hidden option) and
choosing CRF for full backup encoded was
preventing "keep original video" from
occurring.
- Added a workaround to an odd error that
can randomly occur in NVENCC when trying
to encode a single frame.
- Fixed an issue in which 30fps sources that
are being inverse telecined my still result
in a keyint (gop size) of 30.
- Reenabled the MULTIPROCESS hidden option.
Note: MULTIPROCESS is disabled when the
NVENCC encoding option is selected.
- Other minor corrections and cosmetic fixes.

[ggtop]

Thank you very much and happy New Year.
All the best for you and your beloved. Stay healthy,

ggtop

[lithiumus]

Have you set the following two options?

MIN_M2TS_SIZE=0
MIN_PLAYLIST_MINS=0

I thought I set these in the INI file... looks like I should set it in the config via the app instead... worked as soon as I edited and saved it within the app!

Any option to avoid extracting and remuxing when no re-encode is required or is it necessary to remux regardless? Thanks again and Happy New Year!

[jdobbs]

I thought I set these in the INI file... looks like I should set it in the config via the app instead... worked as soon as I edited and saved it within the app!

Any option to avoid extracting and remuxing when no re-encode is required or is it necessary to remux regardless? Thanks again and Happy New Year!It becomes a real goat-rope if you try to do it without extracting first. It's much cleaner to have a group of elementary stream that are used to reconstruct it.

[lithiumus]

It becomes a real goat-rope if you try to do it without extracting first. It's much cleaner to have a group of elementary stream that are used to reconstruct it.

Yeah that makes total sense!

One last question, what's the difference between

Full Backup with Edit Mode (Manual Blanking)

vs

Movie & Menus (Auto-Blank Extras) with Edit Mode (Manual Blanking)

Using the Options:

BLANK_THRESHOLD=0
MIN_M2TS_SIZE=0
MIN_PLAYLIST_MINS=0

I found that I get ALL the streams for blanking if I use
Movie & Menus (Auto-Blank Extras) with Edit Mode (Manual Blanking)

But missing streams using
Full Backup with Edit Mode (Manual Blanking)

[Michi]

I would like to test multiprocess on UHD/x265, but only HD/x264 is enable on multiprocess.

How can I enable x265 on multiprocess?

[jdobbs]

I would like to test multiprocess on UHD/x265, but only HD/x264 is enable on multiprocess.

How can I enable x265 on multiprocess?You can't. It was never supported. I may have a look at it -- but there were good reasons why I didn't implement it originally.

[Michi]

With some films with a slightly lower bit rate, the cpu load constantly changes from high to low. Two encoding processes would fix the problem.

I use the preset "High Quality CFR" for uhd on my AMD Threadripper 1950x.

[ggtop]

I have updated the first post of this thread with a link to the latest version of BD Rebuilder (v0.61.19). Changes for this release:
- Reenabled the MULTIPROCESS hidden option.

Thank you for MULTIPROCESS coming back. Just running an encode.

@all: DGDecNV users should use build 222 or higher as it massively increased indexing performance. It's running at approx. 300MB/s here where it used to be at around 100 MB/s with previous builds. It's like PacMan eating up the progress bar :-)

ggtop

[videoh]

@all: DGDecNV users should use build 222 or higher as it massively increased indexing performance. It's running at approx. 300MB/s here where it used to be at around 100 MB/s with previous builds. It's like PacMan eating up the progress bar :-)

ggtop Thank you. Glad you like it. Use 223 because it improves demuxing for MKV when you have mechanical hard disks involved (must set option 'Optimize for HDD').

[ggtop]

Thank you. Glad you like it.

Yes, definitely. Thank you very much. I then saw 223 is out, but your binaries area holds the latest version only anyway. Always a good idea to follow the improvements. Keep up the good work, too.

ggtop

[MrVideo]

DGDecNV users should use build 222 or higher as it massively increased indexing performance.
Huh? The version I am using is 2053 and I downloaded that on 4/26/2019.

[Mark_Venture]

Huh? The version I am using is 2053 and I downloaded that on 4/26/2019.
When running the app, under about it tells you the build number.

I was on 2053 but an older build. I just re-downloaded tonight, and yes 2053 build 223 is faster.

[ggtop]

Huh? The version I am using is 2053 and I downloaded that on 4/26/2019.

Link to the update thread: http://rationalqm.us/board/viewtopic.php?f=5&t=463&start=250

Link to DGDecNV: http://rationalqm.us/dgdecnv/dgdecnv.html
That one has a link to the binaries area

Simply redo what you've done on 4/26/2019 :D

ggtop

[cartman0208]

With some films with a slightly lower bit rate, the cpu load constantly changes from high to low. Two encoding processes would fix the problem.

I use the preset "High Quality CFR" for uhd on my AMD Threadripper 1950x.

How about using 2 instances of BDRB (with different output folders), if you want the CPU fully utilized? That way you can get 2 Jobs done at the same time.

[videoh]

Huh? The version I am using is 2053 and I downloaded that on 4/26/2019. There is the major build number 2053 and the minor build number called the slipstream number. Why have things seemed to stall at major build 2053? Crackers seem to follow only the major build number.

Subscribe to the Binaries Notification thread and you'll get an email whenever a new slipstream is released. It's definitely worth upgrading to the latest slipstream. Be careful to regenerate your INI file if you are coming from a significantly older version.

[jdobbs]

There is the major build number 2053 and the minor build number called the slipstream number. Why have things seemed to stall at major build 2053? Crackers seem to follow only the major build number.

Subscribe to the Binaries Notification thread and you'll get an email whenever a new slipstream is released. It's definitely worth upgrading to the latest slipstream. Be careful to regenerate your INI file if you are coming from a significantly older version.Hadn't tried DGDecNV on my new GTX-1660, so I thought I'd give it a try for the hell of it and downloaded the newest release. For the particular job I was running it is twice as fast as Directshow/LAV Filters (with CUVID enabled). Nice. Didn't expect that. I got 132fps on an X264 encode compared to a little over 60fps.

[Mark_Venture]

Hadn't tried DGDecNV on my new GTX-1660, so I thought I'd give it a try for the hell of it and downloaded the newest release. For the particular job I was running it is twice as fast as Directshow/LAV Filters (with CUVID enabled). Nice. Didn't expect that. I got 132fps on an X264 encode compared to a little over 60fps.

Yeah I was amazed even with the upgrade over the build of DGDecNV I had downloaded in early December.

Before 223, alt movie MKV with NVEnc with my GTX1660ti/i7-10900k it was taking about 30-35 min per movie. With 223 and Optimize for HDD checked, and the same settings... the 5 movies I did last night took 10-15 min each!

[videoh]

Great to hear guys! We're making performance improvements across the board for DG tools. Currently in progress: DGIndex.

Also looking to solve the internationalization issues (non-Unicode code base).

[thecrowler]

There is the major build number 2053 and the minor build number called the slipstream number. Why have things seemed to stall at major build 2053? Crackers seem to follow only the major build number.

Subscribe to the Binaries Notification thread and you'll get an email whenever a new slipstream is released. It's definitely worth upgrading to the latest slipstream. Be careful to regenerate your INI file if you are coming from a significantly older version.

In anyone could help in private with build 223, I'll be very gratefull.
Thanks for u're time, kind regards.