Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th July 2020, 22:56   #7721  |  Link
microchip8
ffx264/ffhevc author
 
microchip8's Avatar
 
Join Date: May 2007
Location: /dev/video0
Posts: 1,846
Quote:
Originally Posted by Boulder View Post
I just noticed that --limit-tu 0 seems to be a potentially dangerous setting coupled with --rskip 2. I've encoded episodes of 'Chernobyl' and saw that the average bitrate is quite low for 1080p material. I found one scene where the encode looked really ugly and with some testing, I checked that changing from --limit-tu 4 to --limit-tu 0 made it happen. --ctu 32 also fixed the issue.
I've encoded quite a few flicks since then with limit-tu 0 and rskip 2 and have yet to see some noticable (major) distortions. Maybe your issue is sample specific
__________________
ffx264 || ffhevc || ffxvid || microenc
microchip8 is online now   Reply With Quote
Old 25th July 2020, 09:18   #7722  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,753
I made some tests to show the issue. The basic x265 settings were "--preset slower --no-sao --aq-mode 1 --aq-strength 1.0 --qcomp 0.7 --crf 20". The source was unaltered, I only cropped the black borders off in the script.

--ctu 64 --limit-tu 4 --rskip 1 : 4730.32 kbps
--ctu 64 --limit-tu 0 --rskip 1 : 4477.60 kbps
--ctu 32 --limit-tu 4 --rskip 1 : 4735.29 kbps
--ctu 32 --limit-tu 0 --rskip 1 : 4733.48 kbps

--ctu 64 --limit-tu 4 --rskip 2 : 4863.81 kbps
--ctu 64 --limit-tu 0 --rskip 2 : 2877.67 kbps
--ctu 32 --limit-tu 4 --rskip 2 : 5090.29 kbps
--ctu 32 --limit-tu 0 --rskip 2 : 5142.87 kbps

I put the original source clip here:

https://drive.google.com/file/d/1nEa...ew?usp=sharing

The problem is very visible around the middle of the clip in the scene where the woman on the phone is in the distant background. The noise turns into very ugly shapes there.

I don't know if I should use --ctu 32 also with 1080p sources with --rskip 2 just to be safe. I need to test 1440p as well.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 26th July 2020, 14:42   #7723  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,753
A similar test with 1440p (2560x1048):

--limit-tu 4 : 6963.05 kbps
--limit-tu 0 : 6825.00 kbps
--limit-tu 4 --ctu 32 : 6905.23 kbps
--limit-tu 0 --ctu 32 : 6906.79 kbps
--limit-tu 4 --rskip 2 : 7563.58 kbps
--limit-tu 0 --rskip 2 : 5934.17 kbps
--limit-tu 4 --ctu 32 --rskip 2 : 7045.77 kbps
--limit-tu 0 --ctu 32 --rskip 2 : 7099.02 kbps

The large difference can also be seen there. I didn't check how the encode looked but probably there would be scenes which would look ugly. This means it's either --limit-tu 4 or --limit-tu 0 --ctu 32 to avoid the issue. I'm just not sure which one is the better choice quality-wise.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 29th July 2020, 10:07   #7724  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,855
New upload: x265 3.4+14-gd419c7152

abr-ladder: fix memleak in copyInfo()
Fix cost estimation error for hist-based scenecut
Fix libvmaf integration issues
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 29th July 2020, 12:48   #7725  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,753
I tested the histogram based scenecut detection yesterday quite a lot. I still couldn't get it to detect proper scenecuts -- it would either miss too many, at least one very obvious one, and lowering the threshold would cause it to find scenecuts in quite strange places. My Hot Fuzz testclip triggered scenecuts in scenes where there is some motion blur and then single frames which are just slightly less blurry. Probably the sudden sharp edged text on shirts etc. are enough to tell that it's a scenecut.

If you lower the threshold enough, it will start reducing the number of keyframes which was really strange. In the csv log, you can see that a scenecut is detected but they are often coded as B/b-frames.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 29th July 2020, 13:09   #7726  |  Link
jlpsvk
Registered User
 
Join Date: Dec 2014
Posts: 240
Quote:
Originally Posted by LigH View Post
New upload: x265 3.4+14-gd419c7152

abr-ladder: fix memleak in copyInfo()
Fix cost estimation error for hist-based scenecut
Fix libvmaf integration issues
testing to encode 1917 HDR10+ again (testing with every new build)...
__________________
AMD Ryzen 9 5950X, 32GB DDR4-3200 CL16, RTX 3060, 2TB NVMe PCIE4.0, NAS with 8x16TB HDD
jlpsvk is offline   Reply With Quote
Old 29th July 2020, 13:11   #7727  |  Link
quietvoid
Registered User
 
Join Date: Jan 2019
Location: Canada
Posts: 574
Quote:
Originally Posted by Boulder View Post
I tested the histogram based scenecut detection yesterday quite a lot. I still couldn't get it to detect proper scenecuts -- it would either miss too many, at least one very obvious one, and lowering the threshold would cause it to find scenecuts in quite strange places. My Hot Fuzz testclip triggered scenecuts in scenes where there is some motion blur and then single frames which are just slightly less blurry. Probably the sudden sharp edged text on shirts etc. are enough to tell that it's a scenecut.

If you lower the threshold enough, it will start reducing the number of keyframes which was really strange. In the csv log, you can see that a scenecut is detected but they are often coded as B/b-frames.
I've posted about this before:
Quote:
Originally Posted by quietvoid View Post
So x265 3.4+9, they "fixed" --hist-threshold to avoid false positives. However it seems to do much worse than before (as well as 3.4+7, where they fixed the false positives).
I'm getting less I frames with --hist-threshold 0.005 (default 0.03) than the default scenecut algorithm..

Maybe my sample clip is not colorful enough..
Since then I've just given up on hist-scenecut. As well as scenecut-aware-qp, because they introduced QP changes before scene cuts...
quietvoid is offline   Reply With Quote
Old 29th July 2020, 13:53   #7728  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,753
Quote:
Originally Posted by quietvoid View Post
I've posted about this before
Actually that's why I tested it now, since there was that one patch related to the feature.

Too bad they don't seem to care for the original method anymore, my bug report (https://bitbucket.org/multicoreware/...sses-two-clear) has not had any activity.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 31st July 2020, 16:20   #7729  |  Link
vpupkind
Registered User
 
Join Date: Jul 2007
Posts: 63
Quote:
Originally Posted by quietvoid View Post
I've posted about this before:

Since then I've just given up on hist-scenecut. As well as scenecut-aware-qp, because they introduced QP changes before scene cuts...
Have you tried scenecut-aware-qp? Unless you pause at this frame, a human observer is typically unable to notice quality degradation within ~100ms from a strong scene cut.
vpupkind is offline   Reply With Quote
Old 31st July 2020, 17:04   #7730  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,753
Quote:
Originally Posted by vpupkind View Post
Have you tried scenecut-aware-qp? Unless you pause at this frame, a human observer is typically unable to notice quality degradation within ~100ms from a strong scene cut.
Why does that option only work with a multipass encode? You would expect that the lookahead could take care of the work also in CRF mode.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 31st July 2020, 17:48   #7731  |  Link
quietvoid
Registered User
 
Join Date: Jan 2019
Location: Canada
Posts: 574
Quote:
Originally Posted by vpupkind View Post
Have you tried scenecut-aware-qp? Unless you pause at this frame, a human observer is typically unable to notice quality degradation within ~100ms from a strong scene cut.
I have tried it, it was fine before they changed it.
The quality change AFTER a scene cut is not visible, but the idea of lowering the quality BEFORE (which is what they did) does not sound good to me.
It makes no sense.

Quote:
Originally Posted by Boulder View Post
Why does that option only work with a multipass encode? You would expect that the lookahead could take care of the work also in CRF mode.
This is another good point, I don't know how these patches get through quality checks.
It used to work just fine for CRF, and they added a condition that makes it multipass exclusive.

Last edited by quietvoid; 31st July 2020 at 17:54.
quietvoid is offline   Reply With Quote
Old 1st August 2020, 10:06   #7732  |  Link
foxyshadis
Angel of Night
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Tangled in the silks
Posts: 9,559
Quote:
Originally Posted by quietvoid View Post
This is another good point, I don't know how these patches get through quality checks.
It used to work just fine for CRF, and they added a condition that makes it multipass exclusive.
Since Steve Borho and chen moved on, the main quality check is "does it compile". Almost every submitted patch is immediately committed without discussion now, and reverted later if need be.
foxyshadis is offline   Reply With Quote
Old 4th August 2020, 21:09   #7733  |  Link
LeXXuz
21 years and counting...
 
LeXXuz's Avatar
 
Join Date: Oct 2002
Location: Germany
Posts: 716
Quick question: should SMT/Hyperthreading be enabled on a Ryzen 9 machine or not? With 2x logical cpus I had about 66% average cpu usage and lower fps than with logical cpus = cpus @ 95-100%. Same settings of course.
LeXXuz is offline   Reply With Quote
Old 4th August 2020, 21:24   #7734  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,753
HT on works fine. If the CPU usage tends to stay low, it could a parallelization issue like CTU 64 with less than FullHD resolution source.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 5th August 2020, 06:13   #7735  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,868
Quote:
Originally Posted by LeXXuz View Post
Quick question: should SMT/Hyperthreading be enabled on a Ryzen 9 machine or not? With 2x logical cpus I had about 66% average cpu usage and lower fps than with logical cpus = cpus @ 95-100%. Same settings of course.
Always on.
Atak_Snajpera is offline   Reply With Quote
Old 5th August 2020, 08:42   #7736  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,855
HEVC has more dependencies than AVC. More parts of the calculation in x265 need to wait for intermediate results of previous calculations. Not approaching 100% CPU usage is not concerning. You may be able to run two separate encoding jobs (provided you have enough RAM and the cooling can handle the heat).
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 5th August 2020, 09:00   #7737  |  Link
LeXXuz
21 years and counting...
 
LeXXuz's Avatar
 
Join Date: Oct 2002
Location: Germany
Posts: 716
Thanks for the replies guys.

Yes, I've noticed a temperature drop of about ten degrees with SMT off. But that is of no concern anyway since the cpus are watercooled.

I always use --ctu 32 for 1080p resolutions and below. Frame threads is set to auto. x.265 picks 4 threads on 3900x (12C/24T) and 5 threads on 3950x (16C/32T).

I did an encode with frame threads set to 1 but the file size was just negligible smaller than with the auto setting.

So SMT on it is. Maybe I just do 2 parallel jobs instead.
LeXXuz is offline   Reply With Quote
Old 5th August 2020, 09:32   #7738  |  Link
microchip8
ffx264/ffhevc author
 
microchip8's Avatar
 
Join Date: May 2007
Location: /dev/video0
Posts: 1,846
Quote:
Originally Posted by LeXXuz View Post
Thanks for the replies guys.

Yes, I've noticed a temperature drop of about ten degrees with SMT off. But that is of no concern anyway since the cpus are watercooled.

I always use --ctu 32 for 1080p resolutions and below. Frame threads is set to auto. x.265 picks 4 threads on 3900x (12C/24T) and 5 threads on 3950x (16C/32T).

I did an encode with frame threads set to 1 but the file size was just negligible smaller than with the auto setting.

So SMT on it is. Maybe I just do 2 parallel jobs instead.
Try with --pmode
__________________
ffx264 || ffhevc || ffxvid || microenc
microchip8 is online now   Reply With Quote
Old 5th August 2020, 10:53   #7739  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,753
I run my 3900X with two framethreads (-F 2). I don't know if it will increase quality (in theory it should) but the CPU usage difference to -F 4 is around 10% so I consider it more like headroom for doing all the other stuff at the same time. The encoder uses a 24-thread pool, I also assign 24 threads to Avisynth processing. The Avisynth part takes about 20-30% of total CPU time when encoding.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 5th August 2020, 13:03   #7740  |  Link
LeXXuz
21 years and counting...
 
LeXXuz's Avatar
 
Join Date: Oct 2002
Location: Germany
Posts: 716
Quote:
Originally Posted by Boulder View Post
I also assign 24 threads to Avisynth processing. The Avisynth part takes about 20-30% of total CPU time when encoding.
Wait. You mean you set prefetch() to 24 in Avisynth?
Or are we talking about different things here?
LeXXuz is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:20.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.