x265 HEVC Encoder

[Stacey Spears]

Looks like the spec doesnt mandate closed or open gop, both should be viable.

Section 3.2

https://web.archive.org/web/20160605030640/http://www.blu-raydisc.com/assets/Downloadablefile/BD-ROM_Part3_V3.0_WhitePaper_150724.pdf

edit. Getting the same issue here with your source, and the issue persist even with open-gop (skipping uhd-bd preset and setting the other settings manually except for closed gop). And the issue is gone using crf mode...

I have gone ahead and modified x265 to honor the open-gop settings for BD. I did a test mux and it worked. I also confirmed that another facility encodes all of their UHD BDs using open gop with the Sony encoder.

The Sony encode I provided earlier used 8 slices, which is what they use for UHD. They re-encoded with 4 slices and also adjusted the quant table to preserve more high frequency detail. It bumped the bitrate 10 Mbps. I will share it after I QC.

I am doing a 3-pass right now. Once pass 3 (2nd pass) is complete, I will probably create a QP file, from the stats file, to push up the P and B frame QPs for the shots with keyframe pulsing and then re-start pass 1 using the new QP file.

In a short test earlier, it looked like the open gop helped with the keyframe pulsing in the fence shot.

It does seem like x265 over spends on P and B frames, at least on this content. The shots are mostly static in terms of motion.

I wish someone would implement segment re-encoding, then I could fix the shots a little more easily.

[benwaggoner]

I have gone ahead and modified x265 to honor the open-gop settings for BD. I did a test mux and it worked. I also confirmed that another facility encodes all of their UHD BDs using open gop with the Sony encoder.

The Sony encode I provided earlier used 8 slices, which is what they use for UHD. They re-encoded with 4 slices and also adjusted the quant table to preserve more high frequency detail. It bumped the bitrate 10 Mbps. I will share it after I QC.

I am doing a 3-pass right now. Once pass 3 (2nd pass) is complete, I will probably create a QP file, from the stats file, to push up the P and B frame QPs for the shots with keyframe pulsing and then re-start pass 1 using the new QP file.
Wow, doesn't this increasingly feel like authoring VC-1 HD-DVD/BD titles circa 2006 :sly:?

In a short test earlier, it looked like the open gop helped with the keyframe pulsing in the fence shot.
I hope your open GOP fix makes it into the main branch! This is exactly the sort of thing Open GOP is supposed to help. And it matters all the more with short GOPs like this (1 sec) since most IDRs won't align with cuts. Open GOP is better than RADL.

It does seem like x265 over spends on P and B frames, at least on this content. The shots are mostly static in terms of motion.
You could potentially play with --ipratio and --pbratio there.

I wish someone would implement segment re-encoding, then I could fix the shots a little more easily.
I think it is actually possible with x265 if you don't mind appending in mp4box or something.
You'd use --seek and --frames to set in/out points, and set --chunk-start and --chunk-end to expand that range by a few GOPs to get VBV state to match.

If it's a --pass 2 or 3, I'm not sure if the --chunk parameters are needed; maybe VBV state of the initial state works.

It seems sufficiently advanced encoding often winds up replicating PEP, eh?

[Stacey Spears]

Wow, doesn't this increasingly feel like authoring VC-1 HD-DVD/BD titles circa 2006 :sly:?


I hope your open GOP fix makes it into the main branch! This is exactly the sort of thing Open GOP is supposed to help. And it matters all the more with short GOPs like this (1 sec) since most IDRs won't align with cuts. Open GOP is better than RADL.


You could potentially play with --ipratio and --pbratio there.


I think it is actually possible with x265 if you don't mind appending in mp4box or something.
You'd use --seek and --frames to set in/out points, and set --chunk-start and --chunk-end to expand that range by a few GOPs to get VBV state to match.

If it's a --pass 2 or 3, I'm not sure if the --chunk parameters are needed; maybe VBV state of the initial state works.

It seems sufficiently advanced encoding often winds up replicating PEP, eh?

I tried setting ipratio and pbratio to higher values like 2.0, as a test, but it still ends up lowering P and B while maintaining the high I frame QPs on the same shots.

For the short comings of VC1, PEP had some great segment re-encoding capabilities.

You can use the concat flag (don't recall the exact name off hand) in x265 to split the clip up. I know someone who does this to break the movie into chunks for parallel encoding ala PEP. But for an 8-minute video, seems odd to break-up even further. :)

I need to do some testing with the encode on a few BD players to make sure there is not a playback issue. I wish I knew why they disabled it. Could have been confusion about it being supported OR to work around a player issue.

I made a change where I could use the zones file to set --qpmax, but when I used the feature, no I frames were inserted for the rest of the encode, and I have not gotten around to debugging it. :) I was going to set a qpmax on the GOPs where the I frames went higher. Nothing like brute force!

[benwaggoner]

I tried setting ipratio and pbratio to higher values like 2.0, as a test, but it still ends up lowering P and B while maintaining the high I frame QPs on the same shots.

For the short comings of VC1, PEP had some great segment re-encoding capabilities.

You can use the concat flag (don't recall the exact name off hand) in x265 to split the clip up. I know someone who does this to break the movie into chunks for parallel encoding ala PEP. But for an 8-minute video, seems odd to break-up even further. :)

I need to do some testing with the encode on a few BD players to make sure there is not a playback issue. I wish I knew why they disabled it. Could have been confusion about it being supported OR to work around a player issue.

I made a change where I could use the zones file to set --qpmax, but when I used the feature, no I frames were inserted for the rest of the encode, and I have not gotten around to debugging it. :) I was going to set a qpmax on the GOPs where the I frames went higher. Nothing like brute force!
I used to have an Excel script that would take x264 per-frame console output (like the x265 .csv files) and turn that into a qpfile. The main use was to determine optimal IDR placement in a first pass and synchronize that across different ABR bitrates.

It'd be easier to implement that with .csv, and then you could tweak each frame's type and qp for each frame.

[Stacey Spears]

I used to have an Excel script that would take x264 per-frame console output (like the x265 .csv files) and turn that into a qpfile. The main use was to determine optimal IDR placement in a first pass and synchronize that across different ABR bitrates.

It'd be easier to implement that with .csv, and then you could tweak each frame's type and qp for each frame.

I have a python script that you run against the stats file to dump out the frame types and QPs if you like. This is used to encode the Dolby Vision enhancement layer since they need the same frame types across the two layers.

I need to write a tool that will dump the QPs on any hevc file. We wrote a tool recently that allows us to move the SEI from one hevc file to another. I use this to move the SL-HDR2 SEI messages from the file Philips modified to a re-encoded file. This way Philips does not have to reprocess the clip for me. They have done it twice already, now they don't have to. :)

[Selur]

using:

x265 --input - --output-depth 10 --y4m --profile main10 --hme --hme-search star,hex,full --limit-modes --no-early-skip --no-open-gop --opt-ref-list-length-pps --crf 18.00 --opt-qp-pps --cbqpoffs -2 --crqpoffs -2 --limit-refs 0 --ssim-rd --psy-rd 2.50 --rdoq-level 2 --psy-rdoq 10.00 --aq-mode 0 --deblock=-1:-1 --limit-sao --no-repeat-headers --range limited --colormatrix bt470bg --output "E:\Temp\2021-05-02@15_19_28_0110_01.265"

resulted in sei:

cpuid=1111039 / frame-threads=4 / numa-pools=16 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=640x352 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / no-open-gop / min-keyint=25 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=20 / lookahead-slices=0 / scenecut=40 / hist-scenecut=0 / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=2 / dynamic-rd=0.00 / ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=0 / limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / no-frame-dup / no-hme / weightp / no-weightb / no-analyze-src-pics / deblock=-1:-1 / sao / no-sao-non-deblock / rd=6 / selective-sao=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.50 / psy-rdoq=10.00 / rd-refine / no-lossless / cbqpoffs=-2 / crqpoffs=-2 / rc=crf / crf=18.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=1 / aq-strength=0.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=5 / chromaloc=0 / display-window=0 / cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / hist-threshold=0.03 / opt-cu-delta-qp / no-aq-motion / no-hdr10 / no-hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / no-scenecut-aware-qpconformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass

-> is it a bug that "no-hme" is shown or am I missing something ?

Cu Selur

[benwaggoner]

It does seem like a bug, but you can check by looking at the output of hme on/off and verify that there are differences in output and encoding time. Visual differences with HME are most likely encoding very noisy content at 4K, as the first 960x540 analysis will have averaged out lots of the noise and have picked out the underlying features a lot better.

[Stacey Spears]

And using a qpfile to specify a low QP for that frame should work as well.


So far using a QPFile to raise the QP of the P and B frames seems to be working well. For the UHD version, one shot needed it and I am also using it for two other shots where the I frame QP value is 3-6 higher than P and B. These additional shots had pulsing in the HD version.

The end result is running 4.75 passes for each encode. I have 12 versions to encode this way. :)

1. Encode to passes and stop the 2nd pass (really pass 3 for the stats file) after it finishes the last shot in question.
2. create the QP File. I use the pass 3 stats file since its QPs are closer to final than the pass 1 stats file.
3. Encode three passes using the QP File.

Will be running a mux test today with the Atmos audio just to confirm this will actually work in the end.

For the shot with visible pulsing, I am rounding the pass 3 QP and then adding 3 to it. For the other two shots, I am rounding and then adding 1.

At this point I can only assume the x265 devs are busy on x266.

Edit: It muxed.

[benwaggoner]

So far using a QPFile to raise the QP of the P and B frames seems to be working well. For the UHD version, one shot needed it and I am also using it for two other shots where the I frame QP value is 3-6 higher than P and B. These additional shots had pulsing in the HD version.

The end result is running 4.75 passes for each encode. I have 12 versions to encode this way. :)

1. Encode to passes and stop the 2nd pass (really pass 3 for the stats file) after it finishes the last shot in question.
2. create the QP File. I use the pass 3 stats file since its QPs are closer to final than the pass 1 stats file.
3. Encode three passes using the QP File.

Will be running a mux test today with the Atmos audio just to confirm this will actually work in the end.

For the shot with visible pulsing, I am rounding the pass 3 QP and then adding 3 to it. For the other two shots, I am rounding and then adding 1.

At this point I can only assume the x265 devs are busy on x266.

Edit: It muxed.
I'm glad it muxed! But yikes, that sounds painful.

If you're having to do so many versions, perhaps --abr-ladder could be helpful?

[Stacey Spears]

I'm glad it muxed! But yikes, that sounds painful.

If you're having to do so many versions, perhaps --abr-ladder could be helpful?

Isn't --abr-ladder more for layered encodes in the streaming scenario?

3 of 12 are done so far. :) I currently have the next three running their initial two pass to figure out what to use for the qp file. This is for a single project, so not the end of the world. :p If I were a streaming service, this would not be practical. The real solution is for someone to debug the issue and figure out why the rate control is opting for low P and B QPs with high I QPs. Perhaps I should see if I can get MSU to use the content for future CODEC shootouts.

BTW, off topic, is it true that Blue Star has closed many locations in Portland? So sad if true! :p

[benwaggoner]

Isn't --abr-ladder more for layered encodes in the streaming scenario?
I thought you were encoding multiple variants of the same source. If not applicable, than it is not applicable ;).

3 of 12 are done so far. :) I currently have the next three running their initial two pass to figure out what to use for the qp file. This is for a single project, so not the end of the world. :p If I were a streaming service, this would not be practical. The real solution is for someone to debug the issue and figure out why the rate control is opting for low P and B QPs with high I QPs. Perhaps I should see if I can get MSU to use the content for future CODEC shootouts.
Good stress test content is always a delight. My new favorite stress test clip is Netflix's Sol Volante. http://download.opencontent.netflix.com/?prefix=SolLevante/hdr10/

BTW, off topic, is it true that Blue Star has closed many locations in Portland? So sad if true! :p
I see four still open, but I don't recall how many there used to be. I've barely been to restaurants the last 14 months. But my fiancé is getting her second shot on Friday, so hopefully venturing into the world more soon.

[Marsu42]

Since then I've just given up on hist-scenecut.

Too bad they don't seem to care for the original method anymore

Since Steve Borho and chen moved on, the main quality check is "does it compile". Almost every submitted patch is immediately committed without discussion now, and reverted later if need be.

Unfortunately at least --hist-scenecut is not good at all. They just left it half finished.

So the expert consensus is that --hist-scenecut is broken atm, and users should simply use the traditional method - correct?

If so, it's unfortunate this isn't marked as "Experimental feature" in the command line options :-\ https://x265.readthedocs.io/en/master/cli.html

[ShortKatz]

Did x265 development kind of stoped? It is not much going on currently.

[benwaggoner]

So the expert consensus is that --hist-scenecut is broken atm, and users should simply use the traditional method - correct?

If so, it's unfortunate this isn't marked as "Experimental feature" in the command line options :-\ https://x265.readthedocs.io/en/master/cli.html
I haven't seen much feedback about it 3.5, where it was improved. From the release notes:
+1. Improved hist-based scene cut algorithm: Reduces false positives by leveraging motion and scene transition info.

Anyone retried --hist-scencut again in 3.5?

[Boulder]

Yes, we have tested it and it's worse than before. Mind you, it didn't work properly even before v3.5..

It is very much a reality that they don't seem to have many hardcore devs left. Of course, the encoder is very stable and works well, but there are these reported cases which are never investigated.

[benwaggoner]

Yes, we have tested it and it's worse than before. Mind you, it didn't work properly even before v3.5.
Worse results, but works more reliably?

It is very much a reality that they don't seem to have many hardcore devs left. Of course, the encoder is very stable and works well, but there are these reported cases which are never investigated.
I am aware that some work is going on with x265, but yeah, pretty much nothing in the public repository.

[Boulder]

Worse results, but works more reliably?

Unfortunately just worse results. It's very flaky and detects scene changes in motion too easily (transition from a slightly motion blurred frame to a crisper one triggers it). The threshold seems to affect it in the wrong way.

https://forum.doom9.org/showthread.php?p=1919698#post1919698

What comes to development, I think most of the recent (since ~year or so) stuff has been towards streaming and (very) low bitrate encodes. For regular encodes, much less so. So there's probably someone paying for those things and that's why they get done.

[nakTT]

After many month of no update, now they have made a new commit.

https://bitbucket.org/multicoreware/x265_git/commits/

[benwaggoner]

After many month of no update, now they have made a new commit.

https://bitbucket.org/multicoreware/x265_git/commits/
Documentation only, though, and a somewhat confusing one.

If the encoder is unable to reach the level it issues a warning and aborts the encode. The requested level will be signaled in the bitstream even if it is higher than the actual level.

If it is aborted, how is it signaled? If the parameters are wrong, the first frame wouldn't have been even written.

Or is it saying that if a stream is started but has a VBV violation during the encode, it'll abort, but the stream that had been written up to that point would still have the originally specified level?

If so, that would be the expected and appropriate behavior, since the frame that violated the spec never was written and so the file up to that point should still be conformant.

[filler56789]

OUCH! It does appear that, since Steve Borho and Min Chen left the x265 project, the remaining devs DON'T KNOW very-well what they're doing :-/

So... to all the regular/frequent users of x265, ¿could you please say which old version of x265 you recommend?

[DJATOM]

Definitely before ABR ladder introduced. It added complexity to codebase and it was a source of bugs for certain amount of time (maybe some bugs still not discovered). So I recommend to use 3.3 if you want less buggy encoder.

[benwaggoner]

Definitely before ABR ladder introduced. It added complexity to codebase and it was a source of bugs for certain amount of time (maybe some bugs still not discovered). So I recommend to use 3.3 if you want less buggy encoder.
Yeah. While there are promising improvements in 3.4 & 3.5, none that I've really seen tested out enough to judge, and how much of the VBV fixes were for things that got broken post 3.3.

https://x265.readthedocs.io/en/master/releasenotes.html


No one claims even no regressions with the improved 3.5 --hist-scenecut
I've not seen test results for --rskip 2, although it is conceptually sound, and seems likely to help motion graphics and cell animation.
Have heard nothing about using RADL for variable duration GOPs yet. I've not grokked the theory for why it would be better.
Nor any results from the now bidirectional --scenecut-aware-qp, or its tuning with --scenecut-window and --max-qp delta. Using it seems to give me a syntax error some of the time (which is weird). All are conceptually sound, though.
--frame-dup seems to work fine, but those are frames that were small and early-exited already, and I've not noticed significant net improvements yet. The new Sol Levante source for my Encoding Challenge (https://forum.doom9.org/showthread.php?t=175776) should be great to test it due to all the static credits and titles.


Anyone have anything to share on those?

[filler56789]

Definitely before ABR ladder introduced. It added complexity to codebase and it was a source of bugs for certain amount of time (maybe some bugs still not discovered). So I recommend to use 3.3 if you want less buggy encoder.

Thanks.

[LeXXuz]

Definitely before ABR ladder introduced. It added complexity to codebase and it was a source of bugs for certain amount of time (maybe some bugs still not discovered). So I recommend to use 3.3 if you want less buggy encoder.

Would you mind compiling an 8-bit & 10-bit version of 3.3 with optimizations for Zen2 and Zen3? :o

I'd like to have an optimized and stable version for every day use.

[masterkivat]

Would you mind compiling an 8-bit & 10-bit version of 3.3 with optimizations for Zen2 and Zen3? :o

I'd like to have an optimized and stable version for every day use.
...I second that! :thanks:

[Boulder]

Yeah. While there are promising improvements in 3.4 & 3.5, none that I've really seen tested out enough to judge, and how much of the VBV fixes were for things that got broken post 3.3.

https://x265.readthedocs.io/en/master/releasenotes.html


No one claims even no regressions with the improved 3.5 --hist-scenecut
I've not seen test results for --rskip 2, although it is conceptually sound, and seems likely to help motion graphics and cell animation.
Have heard nothing about using RADL for variable duration GOPs yet. I've not grokked the theory for why it would be better.
Nor any results from the now bidirectional --scenecut-aware-qp, or its tuning with --scenecut-window and --max-qp delta. Using it seems to give me a syntax error some of the time (which is weird). All are conceptually sound, though.
--frame-dup seems to work fine, but those are frames that were small and early-exited already, and I've not noticed significant net improvements yet. The new Sol Levante source for my Encoding Challenge (https://forum.doom9.org/showthread.php?t=175776) should be great to test it due to all the static credits and titles.


Anyone have anything to share on those?
My views on some of those:

--hist-scenecut has been proven to not work properly, it cannot be used in any productive encodes. It went worse with the bugfixes they made and then they left it as it is.

--rskip 2 seemed to help with some onion artifacts I was seeing in flat backgrounds with some noise in my Star Trek TNG 720p encodes. I didn't find any problems with it, and shouldn't more computation and analysis be better than skipping anyway? I use --rskip 2 --rskip-edge-threshold 3 as a base setting in my encoding commandline.

Just note that --rskip 2 --limit-tu 0 --ctu 64 is asking for trouble. That combination is broken, but my bug report has not been processed at all.

Scenecut-aware-qp is a bit silly implementation because it only works in 2-pass encodes. I did some tests on a modded version that allows using it in CRF mode but didn't find it useful.

[charliebaby]

3.5+10-82786fc GCC 11.1

https://www.mediafire.com/file/gsyvua78dui1zut/x265-3.5+10-82786fc_gcc111-AVX2.7z/file

[benwaggoner]

3.5+10-82786fc GCC 11.1

https://www.mediafire.com/file/gsyvua78dui1zut/x265-3.5+10-82786fc_gcc111-AVX2.7z/file
Why is your build labeled "AVX2?" Does it exclude AVX-512 or something?

[benwaggoner]

I'm starting tests for the new anime version of my Encoder Challenge (https://forum.doom9.org/showthread.php?p=1944475#post1944475).

Despite content that seems ripe for --frame-dup helping static frames of credits, it doesn't seem to be working. Looking at the docs for --dup-threshold (https://x265.readthedocs.io/en/master/cli.html#input-output-file-options), it's unclear what the scale is, or if higher values make it more or less strict. Anyone have any insights?

[fauxreaper]

I'm starting tests for the new anime version of my Encoder Challenge (https://forum.doom9.org/showthread.php?p=1944475#post1944475).

Despite content that seems ripe for --frame-dup helping static frames of credits, it doesn't seem to be working. Looking at the docs for --dup-threshold (https://x265.readthedocs.io/en/master/cli.html#input-output-file-options), it's unclear what the scale is, or if higher values make it more or less strict. Anyone have any insights?

Scale is PSNR between two frames. Higher values make it more strict. Default is PSNR 70.

[benwaggoner]

Scale is PSNR between two frames. Higher values make it more strict. Default is PSNR 70.
Awesome, thank you. I'm trying another pass of my encode with --dup-threshold of 50.

[Gravitator]

v3.5+10 - defective image output (v3.0+1 is okay).
ffmpeg -i z:\space_6s.mkv -f yuv4mpegpipe - | x265 --y4m - --preset medium --bitrate 1000 --max-merge 1 z:\test.mkv

[Ma]

v3.5+10 - defective image output (v3.0+1 is okay).

It is caused by
https://bitbucket.org/multicoreware/x265_git/commits/f280829062a61792333405dbb62adf4ed7299d82

To go back to old behavior please add
--no-early-skip --no-b-intra --limit-refs 3
options to new version of x265.

[benwaggoner]

I've yet to get --frame-dup to actually do anything, even with --dup-threshold at 20. And with source where there are lots of identical consecutive frames.

x265.exe --input SolLevante_SDRv2_1080p24_8bit.y4m --level-idc 4.0 --preset slower --tune animation --frame-dup --dup-threshold 20 --pools "-,+" --pass 1 --keyint 120 --bitrate 1000 --vbv-maxrate 12000 --vbv-bufsize 12000 --hrd --aud --colorprim bt709 --transfer bt709 --colormatrix bt709 -o SolLevante_SDR-1080p_1000_slower-dup20_p1.hevc --psnr --ssim --csv-log-level 1 --stats SolLevante_SDR-1080p_1000_slower-dup20.stats --csv SolLevante_SDR-1080p_1000_slower-dup20_p1.csv

Every frame is at least 600 bytes. They're mostly Merge or Skip, but with very low percentages of Intra and Inter blocks of various sizes in the b-frames.

Same in first and second passes.

Anyone have any insights/suggestions?

I'm trying a single-pass test.

[benwaggoner]

And still no dup frames even with --dup-threshold 10, even!

[Boulder]

And still no dup frames even with --dup-threshold 10, even!

Works as well as the threshold in --hist-scenecut then :D

[benwaggoner]

Works as well as the threshold in --hist-scenecut then :D
Yeah, it is frustrating. I'm still getting around 200 Kbps for what's a single B&W frame that's static for seconds. x265 without using --zones spends way too much on static title card and scrolling credits.

My guess is adaptive quantization isn't accounting for the benefit of SAO (ringing reduction) and tskip (efficient encoding of sharp edges) for discreet-tone content. The sequence is going down to QP 20 despite getting PSNR averaging >50. I'd think cutree would be providing more benefit as well. This kind of content looks pretty perfect at QP 40 with SAO and tskip.

[_kermit]

I'm using ffmpeg 4.4 and this command line:

ffmpeg.exe -i %1 -pix_fmt yuv420p -f yuv4mpegpipe -| x265.exe ....

sometimes I get this and then it stops:

av_interleaved_write_frame(): Broken pipe
Error writing trailer of pipe:: Broken pipe

How can this be solved?

[Gravitator]

Yeah, it is frustrating. I'm still getting around 200 Kbps for what's a single B&W frame that's static for seconds. x265 without using --zones spends way too much on static title card and scrolling credits.
Another very useful for video surveillance cameras.

[nakTT]

Documentation only, though, and a somewhat confusing one.



If it is aborted, how is it signaled? If the parameters are wrong, the first frame wouldn't have been even written.

Or is it saying that if a stream is started but has a VBV violation during the encode, it'll abort, but the stream that had been written up to that point would still have the originally specified level?

If so, that would be the expected and appropriate behavior, since the frame that violated the spec never was written and so the file up to that point should still be conformant.
Yes. Seems like the development of x265 is basically ended already.

[_kermit]

I'm using ffmpeg 4.4 and this command line:

ffmpeg.exe -i %1 -pix_fmt yuv420p -f yuv4mpegpipe -| x265.exe ....

sometimes I get this and then it stops:

av_interleaved_write_frame(): Broken pipe
Error writing trailer of pipe:: Broken pipe

How can this be solved?

wrong forum?

[rwill]

wrong forum?

Pretty much wrong thread but

av_interleaved_write_frame(): Broken pipe
Error writing trailer of pipe:: Broken pipe


most likely means that the program behind ffmpeg, in your case x265, stopped accepting input because it closed its standard input. So probably x265 exited early or crashed.

[Gravitator]

v3.5+10 - I get a refusal of encoding.

ffmpeg -i z:\space_6s.mkv -f yuv4mpegpipe - | x265 --y4m - --preset veryslow --hme --hme-search star,sea,full --bitrate 1000 z:\test.mkv

ffmpeg version 4.3.2-2021-02-20-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10.2.0 (Rev6, Built by MSYS2 project)
Input #0, matroska,webm, from 'z:\space_6s.mkv':
Metadata:
creation_time : 2019-08-01T04:39:18.000000Z
ENCODER : Lavf58.2.103
Duration: 00:00:06.22, start: 0.000000, bitrate: 7719 kb/s
Stream #0:0(eng): Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)
Metadata:
DURATION : 00:00:06.215000000
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, yuv4mpegpipe, to 'pipe:':
Metadata:
encoder : Lavf58.45.100
Stream #0:0(eng): Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc (default)
Metadata:
DURATION : 00:00:06.215000000
encoder : Lavc58.91.100 wrapped_avframe
y4m [info]: 1920x1080 fps 13978/583 i420p8 sar 1:1 unknown frame count
raw [info]: output file: z:\test.mkv
x265 [info]: HEVC encoder version 3.5+10-82786fccc
x265 [info]: build info [Windows][MSVC 1929][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.1 Cache64
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: HME L0,1,2 / range / subpel / merge : star, sea, full / 48 / 4 / 5
x265 [info]: Keyframe min / max / scenecut / bias : 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : ABR-1000 kbps / 0.60
x265 [info]: tools: rect amp rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00 rskip mode=1
x265 [info]: tools: signhide tmvp b-intra strong-intra-smoothing deblock sao
av_interleaved_write_frame(): Broken pipeB time=00:00:01.41 bitrate=596600.6kbits/s speed=2.81x
Error writing trailer of pipe:: Broken pipe
frame= 56 fps= 35 q=-0.0 Lsize= 167063kB time=00:00:02.33 bitrate=585947.1kbits/s speed=1.45x
video:29kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 569837.375000%
Conversion failed!

[benwaggoner]

v3.5+10 - I get a refusal of encoding.

ffmpeg -i z:\space_6s.mkv -f yuv4mpegpipe - | x265 --y4m - --preset veryslow --hme --hme-search star,sea,full --bitrate 1000 z:\test.mkv
Have you tried using just ffplay with the source to confirm it decodes without any errors?

Also, can you use names instead of numbers for --hme-search? I would do it as "--hme-search 2,3,4"

[FranceBB]

My new baby arrived today.
I look forward to take it for a spin with x265 (*_*)

https://i.imgur.com/CGORtyy.png

Intel Xeon Gold, 56c/112th with AVX-512 and this time is in a server room with freezing cold temperatures, so I dare AVX512 to make it hot! xD

[_kermit]

Pretty much wrong thread but

av_interleaved_write_frame(): Broken pipe
Error writing trailer of pipe:: Broken pipe


most likely means that the program behind ffmpeg, in your case x265, stopped accepting input because it closed its standard input. So probably x265 exited early or crashed.

so, when x265 crashed, it's the right thread after all? :)

I'll check a newer version then.

[Gravitator]

He doesn't like high resolution :confused:

ffmpeg -i z:\Park.mkv -f yuv4mpegpipe - | x265 --y4m - --preset veryslow --hme --hme-search star,sea,full --bitrate 1000 z:\test.mkv
ffmpeg version 2021-06-13-git-3ce272a9da-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10.3.0 (Rev2, Built by MSYS2 project)
Input #0, matroska,webm, from 'z:\Park.mkv':
Metadata:
creation_time : 2020-07-09T11:13:13.000000Z
ENCODER : Lavf58.48.100
Duration: 00:00:01.97, start: 0.000000, bitrate: 64630 kb/s
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt470bg/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1k tbn (default)
Metadata:
ENCODER : Lavc58.94.100 nvenc_h264
DURATION : 00:00:01.966000000
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, yuv4mpegpipe, to 'pipe:':
y4m [info]: 1920x1080 fps 30/1 i420p8 sar 1:1 unknown frame count
Metadata:
encoder : raw [info]: output file: z:\test.mkv
Lavf59.3.100x265 [info]: HEVC encoder version 3.5+10-82786fccc
x265 [info]: build info [Windows][MSVC 1929][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.1 Cache64

x265 [info]: Main profile, Level-4 (Main tier)
Stream #0:0x265 [info]: Thread pool created using 4 threads
: Video: wrapped_avframe, yuv420p(tv, bt470bg/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/sx265 [info]: Slices : 1
, x265 [info]: frame threads / pool features : 2 / wpp(17 rows)
30 fps, 30 tbn (default)
Metadata:
DURATION : 00:00:01.966000000
encoder : Lavc59.1.101 wrapped_avframe
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: HME L0,1,2 / range / subpel / merge : star, sea, full / 48 / 4 / 5ts/s speed=0.522x
x265 [info]: Keyframe min / max / scenecut / bias : 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : ABR-1000 kbps / 0.60
x265 [info]: tools: rect amp rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00 rskip mode=1
x265 [info]: tools: signhide tmvp b-intra strong-intra-smoothing deblock sao
av_interleaved_write_frame(): Broken pipeB time=00:00:01.16 bitrate=746497.8kbits/s speed=1.96x
Error writing trailer of pipe:: Broken pipe
frame= 56 fps= 31 q=-0.0 Lsize= 167063kB time=00:00:01.86 bitrate=733167.3kbits/s speed=1.02x
video:24kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 694187.375000%
Conversion failed!

ffmpeg -i z:\Park.mkv -vf scale="720:480" -f yuv4mpegpipe - | x265 --y4m - --preset veryslow --hme --hme-search star,sea,full --bitrate 1000 z:\test.mkv

ffmpeg version 2021-06-13-git-3ce272a9da-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10.3.0 (Rev2, Built by MSYS2 project)
Input #0, matroska,webm, from 'z:\Park.mkv':
Metadata:
creation_time : 2020-07-09T11:13:13.000000Z
ENCODER : Lavf58.48.100
Duration: 00:00:01.97, start: 0.000000, bitrate: 64630 kb/s
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt470bg/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1k tbn (default)
Metadata:
ENCODER : Lavc58.94.100 nvenc_h264
DURATION : 00:00:01.966000000
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, yuv4mpegpipe, to 'pipe:':
y4m [info]: 720x480 fps 30/1 i420p8 sar 32:27 unknown frame count
Metadata:
encoder : raw [info]: output file: z:\test.mkv
Lavf59.3.100x265 [info]: HEVC encoder version 3.5+10-82786fccc

x265 [info]: build info [Windows][MSVC 1929][64 bit] 8bit+10bit+12bit
Stream #0:0x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.1 Cache64
: Video: wrapped_avframe, yuv420p(tv, bt470bg/unknown/unknown, progressive), 720x480 [SAR 32:27 DAR 16:9], q=2-31, 200 kb/sx265 [warning]: Source height < 540p is too low for HME. Disabling HME.
x265 [info]: Main profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 4 threads
, x265 [info]: Slices : 1
30 fps, x265 [info]: frame threads / pool features : 2 / wpp(8 rows)
30 tbn (default)
Metadata:
DURATION : 00:00:01.966000000
encoder : Lavc59.1.101 wrapped_avframe
x265 [info]: Coding QT: max CU size, min CU size : 64 / 803 bitrate=124438.4kbits/s speed=0.473x
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 4 / 5
x265 [info]: Keyframe min / max / scenecut / bias : 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : ABR-1000 kbps / 0.60
x265 [info]: tools: rect amp rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00 rskip mode=1
x265 [info]: tools: signhide tmvp b-intra strong-intra-smoothing deblock sao
6frame= 59 fps= 19 q=-0.0 Lsize= 29869kB time=00:00:01.96 bitrate=124417.8kbits/s speed=0.63x
video:25kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 117719.859375%
x265 [info]: frame I: 1, Avg QP:27.58 kb/s: 8555.76
x265 [info]: frame P: 13, Avg QP:34.91 kb/s: 3119.93
x265 [info]: frame B: 45, Avg QP:39.58 kb/s: 314.22
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: Weighted B-Frames: Y:4.4% UV:2.2%
x265 [info]: consecutive B-frames: 7.1% 0.0% 0.0% 64.3% 14.3% 14.3% 0.0% 0.0% 0.0%

encoded 59 frames in 50.33s (1.17 fps), 1072.11 kb/s, Avg QP:38.35

[Gravitator]

Have you tried using just ffplay with the source to confirm it decodes without any errors?
No.

Also, can you use names instead of numbers for --hme-search? I would do it as "--hme-search 2,3,4"
The problem is in the configuration --hme-search star,sea,full

ffmpeg -i z:\Park.mkv -f yuv4mpegpipe - | x265 --y4m - --preset veryslow --hme --hme-search umh,star,sea --bitrate 1000 z:\test.mkv
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra4.6kbits/s speed=0.47x
x265 [info]: HME L0,1,2 / range / subpel / merge : umh, star, sea / 48 / 4 / 5
x265 [info]: Keyframe min / max / scenecut / bias : 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : ABR-1000 kbps / 0.60
x265 [info]: tools: rect amp rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00 rskip mode=1
x265 [info]: tools: signhide tmvp b-intra strong-intra-smoothing deblock sao
frame= 59 fps=1.0 q=-0.0 Lsize= 179213kB time=00:00:01.96 bitrate=746497.6kbits/s speed=0.0327x
video:25kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 706810.750000%
x265 [info]: frame I: 1, Avg QP:33.19 kb/s: 18844.56
x265 [info]: frame P: 12, Avg QP:44.86 kb/s: 2957.10
x265 [info]: frame B: 46, Avg QP:49.15 kb/s: 288.71
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: Weighted B-Frames: Y:2.2% UV:2.2%
x265 [info]: consecutive B-frames: 7.7% 0.0% 0.0% 53.8% 0.0% 38.5% 0.0% 0.0% 0.0%

encoded 59 frames in 634.11s (0.09 fps), 1145.94 kb/s, Avg QP:48.00

[Gravitator]

My new baby arrived today.
I look forward to take it for a spin with x265 (*_*)

VVC :cool:

[rwill]

so, when x265 crashed, it's the right thread after all? :)

I'll check a newer version then.

What you can try is to pipe ffmpeg output into a file like so:

ffmpeg.exe -i %1 -pix_fmt yuv420p -f yuv4mpegpipe - > test.y4m

and then use test.y4m as an input file to a standalone x265.exe call to see what x265 does and debug that.