Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 22nd December 2018, 23:37   #6541  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 324
If you want to encode without VBV buffer, you could use --seek and --frames options, for example:
for %i in (0 1000 2000) do ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek %i --frames 1000 c%i.hevc

which expands to:
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 0 --frames 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 1000 --frames 1000 c1000.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 2000 --frames 1000 c2000.hevc

Finally you can put these 3 parts together:
mkvmerge -o c-whole.mkv c0.hevc + c1000.hevc + c2000.hevc
-----------------
example for Windows
Ma is offline   Reply With Quote
Old 23rd December 2018, 00:10   #6542  |  Link
BLKMGK
Registered User
 
Join Date: Feb 2008
Posts: 121
Quote:
Originally Posted by asarian View Post
Coding HEVC directly with (hardware-accelerated) NVENC, for instance, is an order of magnitude faster (and sometimes several orders of magnitude even) than what we can achieve with x265 itself, but you have no control over the process, and the quality is definitely lower than that of the latter.

I was more talking about OpenCL in x264; and especially used for the KNLMeansCL denoiser (QTGMC), where it speeds things up significantly. Even though I recently heard even OpenCL yields a theoretical lower quality.



As for the latter, typically you have to refresh your explorer window (a few times) to see how much is actually output already, or it will stay at 0 bytes.

As for chunks, I never used them before, so I can't really say something useful about that.
Ah, makes sense to use them for programmatic speedups and for filters I agree. I may try using my GPU a few times just to see if the results are acceptable to me but so far haven't been able to get it working the few times I've tried

This definitely wasn't a refresh issue in explorer, Chunk really doesn't appear to save any data until it hits its start frame and continues encoding after it hits its end frame. Ugh!

Quote:
Originally Posted by Ma View Post
If you want to encode without VBV buffer, you could use --seek and --frames options, for example:
for %i in (0 1000 2000) do ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek %i --frames 1000 c%i.hevc

which expands to:
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 0 --frames 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 1000 --frames 1000 c1000.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 2000 --frames 1000 c2000.hevc

Finally you can put these 3 parts together:
mkvmerge -o c-whole.mkv c0.hevc + c1000.hevc + c2000.hevc
-----------------
example for Windows
I've done this except I started and ended on keyframes with the understanding that not doing so could have poor joins. The issue with that is it requires first pulling an index and then skipping through it to build the jobs. The first test attempted I built the latter part by hand (PITA) and after encoding I couldn't detect the seams at all. Seemed a good path forward!

I then noticed chunk in the release notes. Chunk, from its description and some discussion I'd seen here, seemed tailor made for clutering but my tests sure don't seem to prove it out. I'll try your example and see if I can detect the seams, perhaps I've been too paranoid?

Thank you for the example of how to build the jobs!

Edit: Tested this method ignoring keyframes and thought I detected a hitch in the video. Sure enough VLC shows two dropped frames where the hitch occurs and I see it elsewhere too. I'll keep testing and see if the keyframes are the issue. I'll pull an index and create some jobs. I need an automated way to build jobs from the index but not at every break lol

Last edited by BLKMGK; 23rd December 2018 at 00:37.
BLKMGK is offline   Reply With Quote
Old 23rd December 2018, 01:29   #6543  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 324
I tried to add --chunk-start/--chunk-end to the example and it is like this:
Encoding in parts of 1000 frames (+100 before, +100 after):

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 0 --frames 1100 --chunk-start 1 --chunk-end 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 900 --frames 1200 --chunk-start 101 --chunk-end 1100 c1.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 1900 --frames 1200 --chunk-start 101 --chunk-end 1100 c2.hevc

mkvmerge -o chunks.mkv c0.hevc + c1.hevc + c2.hevc
Ma is offline   Reply With Quote
Old 23rd December 2018, 05:00   #6544  |  Link
BLKMGK
Registered User
 
Join Date: Feb 2008
Posts: 121
Quote:
Originally Posted by Ma View Post
I tried to add --chunk-start/--chunk-end to the example and it is like this:
Encoding in parts of 1000 frames (+100 before, +100 after):

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 0 --frames 1100 --chunk-start 1 --chunk-end 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 900 --frames 1200 --chunk-start 101 --chunk-end 1100 c1.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 1900 --frames 1200 --chunk-start 101 --chunk-end 1100 c2.hevc

mkvmerge -o chunks.mkv c0.hevc + c1.hevc + c2.hevc
That looks like a very good idea to get the encoder to take into account frames before and after the specified range in order for the encoder to be more efficient! It's not breaking on keyframes, am I wrong about their importance?

I tried your example on my sample file, I'm not spotting glitches or dropped frames in the statistics and I managed to extend it out some.

ffmpeg -i INPUT.MKV -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 3100 --frames 1200 --chunk-start 101 --chunk-end 1100 --csv stats.csv c3.hevc

If I'm wrong about the keyframes that will make life much easier. I'll try doing an entire film once I've incorporated this into my bash script, mux in sound, and check the synch.

Thank you! I really appreciate you providing examples too as it makes understanding what you're doing much simpler.
BLKMGK is offline   Reply With Quote
Old 23rd December 2018, 05:39   #6545  |  Link
nghiabeo20
Registered User
 
Join Date: Jun 2013
Posts: 19
What can SAO do? Does it improve quality/file size?
I'm targeting good quality with reasonable bitrate (5-6000 kbps for 1080p10, BD remux source) with CRF 21. Thanks!
nghiabeo20 is offline   Reply With Quote
Old 23rd December 2018, 13:42   #6546  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 59
Quote:
Originally Posted by nghiabeo20 View Post
What can SAO do? Does it improve quality/file size?
I'm targeting good quality with reasonable bitrate (5-6000 kbps for 1080p10, BD remux source) with CRF 21. Thanks!
Yes, SAO increases efficiency/compression. But it usually does it with the drawback of an slight reduction in fine detail. So its pretty great for low bitrate encodings and animation, but not so much if you wanna keep most of the detail.

Its best if you test it for yourself, and see what settings gives you the best results, but if you target that bitrate range I whould say that you might wanna turn it off.

And btw, I think crf21 will end up at a much lower bitrate for most content if you dont us tune grain or something similar. I get arround those numbers with crf18.
excellentswordfight is offline   Reply With Quote
Old 23rd December 2018, 16:46   #6547  |  Link
nghiabeo20
Registered User
 
Join Date: Jun 2013
Posts: 19
Quote:
Originally Posted by excellentswordfight View Post
Yes, SAO increases efficiency/compression. But it usually does it with the drawback of an slight reduction in fine detail. So its pretty great for low bitrate encodings and animation, but not so much if you wanna keep most of the detail.

Its best if you test it for yourself, and see what settings gives you the best results, but if you target that bitrate range I whould say that you might wanna turn it off.

And btw, I think crf21 will end up at a much lower bitrate for most content if you dont us tune grain or something similar. I get arround those numbers with crf18.
Maybe I'm doing something wrong. All I edit from the veryslow preset is these commands

Code:
bframes=6:subme=3:rd=5:qcomp=0.75:no-sao:deblock=-2:-2
however CRF 22 10bit without any tuning gives me a whopping 8500kbps for a 30s test clip from my The Hundred-Foot Journey bluray!

Last edited by nghiabeo20; 23rd December 2018 at 16:56.
nghiabeo20 is offline   Reply With Quote
Old 23rd December 2018, 16:59   #6548  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,513
The bitrate depends a huge deal on the content. Grainy or noisy stuff requires much more than some clean CGI content.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 24th December 2018, 00:24   #6549  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 59
Quote:
Originally Posted by nghiabeo20 View Post
Maybe I'm doing something wrong. All I edit from the veryslow preset is these commands

Code:
bframes=6:subme=3:rd=5:qcomp=0.75:no-sao:deblock=-2:-2
however CRF 22 10bit without any tuning gives me a whopping 8500kbps for a 30s test clip from my The Hundred-Foot Journey bluray!
As boulder said, thats very source dependant. And a 30s sample might not be representative either. But with that said i think your qcomp setting plays a role here (maybe deblock as well).
excellentswordfight is offline   Reply With Quote
Old 24th December 2018, 04:17   #6550  |  Link
singhkays
Registered User
 
Join Date: Aug 2018
Posts: 7
Can anybody give any advice if there is any benefit by taking the following approaches? and is either one of the splitting methods preferred?
  1. Split a video file on keyframes
  2. Split a video file based on the shots (i.e. different camera angles)

other than the obvious ability to parallelize the encoding task
singhkays is offline   Reply With Quote
Old 24th December 2018, 08:09   #6551  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,800
Quote:
other than the obvious ability to parallelize the encoding task
Aside from further parallelizing the encoding more there is no reason to split a source unless you want to.
When having the split a source for parallelizing the encoding I would prefer to split 'a video file based on the shots' since the source simply might have key frames placed in a fixed interval not caring about coding efficiency. That said splitting on key frames is easier.
But splitting on key frames in respect to scenes and using the segmented encoding feature of x265 is probably the best easy to implement way. Ideally you wound split at the points where the encoder would want to set key frame for coding efficiency.
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 24th December 2018, 23:28   #6552  |  Link
singhkays
Registered User
 
Join Date: Aug 2018
Posts: 7
Quote:
Originally Posted by Selur View Post
But splitting on key frames in respect to scenes and using the segmented encoding feature of x265 is probably the best easy to implement way. Ideally you wound split at the points where the encoder would want to set key frame for coding efficiency.
Thank for the advice. I have specific question on the below

Quote:
splitting on key frames in respect to scenes
How would this work? Should I split on keyframes within the scenes?

Quote:
segmented encoding feature of x265
I haven't heard about this. How does this feature work?

Quote:
Ideally you wound split at the points where the encoder would want to set key frame for coding efficiency
How do I figure out where the encoder wants to set key frames? I thought that was a user controlled option i.e. keyint
singhkays is offline   Reply With Quote
Old 27th December 2018, 13:32   #6553  |  Link
filler56789
SuperVirus
 
filler56789's Avatar
 
Join Date: Jun 2012
Location: Antarctic Japan
Posts: 831
To whom this may interest...
latest revision(s) cannot be compiled under MinGW-w64 && MSYS(2).
Some recent changes prevent the libraries from being linked to the command-line .EXE.
filler56789 is offline   Reply With Quote
Old 27th December 2018, 14:32   #6554  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 324
Quote:
Originally Posted by filler56789 View Post
Some recent changes prevent the libraries from being linked to the command-line .EXE.
Multilib builds are broken (and builds with ENABLE_HDR10_PLUS too).
Simplest solution:
Code:
diff -r 537bba0b7fdc source/encoder/encoder.cpp
--- a/source/encoder/encoder.cpp	Thu Dec 27 11:56:52 2018 +0530
+++ b/source/encoder/encoder.cpp	Thu Dec 27 14:25:51 2018 +0100
@@ -69,7 +69,7 @@
     int doviProfileId;
 }DolbyVisionProfileSpec;
 
-DolbyVisionProfileSpec dovi[] =
+static DolbyVisionProfileSpec dovi[] =
 {
     { 1, 1, 1, 1, 1, 5, 1,  2, 2, 2, 50 },
     { 1, 1, 1, 1, 1, 5, 0, 16, 9, 9, 81 },
diff -r 537bba0b7fdc source/x265.cpp
--- a/source/x265.cpp	Thu Dec 27 11:56:52 2018 +0530
+++ b/source/x265.cpp	Thu Dec 27 14:25:51 2018 +0100
@@ -690,7 +690,7 @@
                 args[argCount++] = token;
                 token = strtok(NULL, " ");
             }
-            args[argCount] = '\0';
+            args[argCount] = NULL;
             CLIOptions cliopt;
             if (cliopt.parseZoneParam(argCount, args,param, i))
             {
Ma is offline   Reply With Quote
Old 27th December 2018, 15:46   #6555  |  Link
filler56789
SuperVirus
 
filler56789's Avatar
 
Join Date: Jun 2012
Location: Antarctic Japan
Posts: 831
^ Thanks for answering.

Quote:
Originally Posted by Ma View Post
Multilib builds are broken (and builds with ENABLE_HDR10_PLUS too).
OUCH! :-/ So I will have to wait.

Quote:
Simplest solution:
Code:
diff -r 537bba0b7fdc source/encoder/encoder.cpp
--- a/source/encoder/encoder.cpp	Thu Dec 27 11:56:52 2018 +0530
+++ b/source/encoder/encoder.cpp	Thu Dec 27 14:25:51 2018 +0100
@@ -69,7 +69,7 @@
     int doviProfileId;
 }DolbyVisionProfileSpec;
 
-DolbyVisionProfileSpec dovi[] =
+static DolbyVisionProfileSpec dovi[] =
 {
     { 1, 1, 1, 1, 1, 5, 1,  2, 2, 2, 50 },
     { 1, 1, 1, 1, 1, 5, 0, 16, 9, 9, 81 },
diff -r 537bba0b7fdc source/x265.cpp
--- a/source/x265.cpp	Thu Dec 27 11:56:52 2018 +0530
+++ b/source/x265.cpp	Thu Dec 27 14:25:51 2018 +0100
@@ -690,7 +690,7 @@
                 args[argCount++] = token;
                 token = strtok(NULL, " ");
             }
-            args[argCount] = '\0';
+            args[argCount] = NULL;
             CLIOptions cliopt;
             if (cliopt.parseZoneParam(argCount, args,param, i))
             {
Since I didn't know the multilib builds are impossible for the time being,
the old error message

(something about "invalid conversion from char to char*")

just went replaced with a new one :-/
filler56789 is offline   Reply With Quote
Old 28th December 2018, 07:36   #6556  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
Hi,
Quote:
Introduce --tune animation option
--psy-rd 0.4
--aq-strength 0.4
--deblock 1:1
--bframes increased by 2
Sorry if it's a silly question but Bframes increased by 2, means +2 from the default parameter which is 4 Bframes according to the doc, so it will be --bframes 6, or from 0 so it will be --bframes 2 ?

Thank you
Yanak is offline   Reply With Quote
Old 28th December 2018, 08:23   #6557  |  Link
Wishbringer
Silent Reader
 
Wishbringer's Avatar
 
Join Date: Dec 2003
Location: Germany
Posts: 289
Or from bframes you give per commandline?
What has a higher priority?
I expect: +2 from preset, except explicit given by command, then the manual value
Wishbringer is offline   Reply With Quote
Old 28th December 2018, 09:06   #6558  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,787
You will usually first use a preset, then a tuning (so the "animation" tuning will add 2 to the number of B frames of your preset); but if you also add an explicit bframes parameter later in the command line, it will supersede previously calculated values.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 28th December 2018, 10:21   #6559  |  Link
RainyDog
Registered User
 
Join Date: May 2009
Posts: 166
Looks like we have a new AQ mode in x265 2.9+28

Quote:
It scales the quantization step size according to the spatial activity of one coding unit relative to frame average spatial activity. This AQ method utilizes the minimum variance of sub-unit in each coding unit to represent the coding units spatial complexity.
Quote:
Default 1.0.

**Range of values:** 0.0 to 3.0

+.. option:: --hevc-aq
+
+ Enable adaptive quantization
+ It scales the quantization step size according to the spatial activity of one
+ coding unit relative to frame average spatial activity. This AQ method utilizes
+ the minimum variance of sub-unit in each coding unit to represent the coding
+ unitís spatial complexity.
+
+.. option:: --qp-adaptation-range
+
+ Delta-QP range by QP adaptation based on a psycho-visual model.
+
+ Default 1.0.
+ **Range of values:** 1.0 to 6.0
RainyDog is offline   Reply With Quote
Old 28th December 2018, 12:57   #6560  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,513
Seems interesting. It would be nice to hear of the cases which this new method targets.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:32.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.