Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 28th December 2019, 12:33   #1  |  Link
abolibibelot
Registered User
 
Join Date: Jun 2015
Location: France
Posts: 46
Avisynth, frame interpolation, ffmpeg x264, performance concerns

Hi,
I'm currently encoding a 1h23min video in 1280x720 resolution and 29.97 FPS framerate, using ffmpeg 32 bits + libx264 -crf 20 -preset "slower", through an Avisynth script which involves mainly interpolation commands to repair blurry frames (quite a lot – about 8000 out of 150000, using a combination of FrameSurgeon, Morpheus and Morph), from a losslessly rendered movie in Lagarith codec (about 120GB), with a computer based on an Intel i7 6700K CPU with 16GB of RAM. And the encoding rate is surprisingly low : 0.127x which is about 3.80 frames per second if I'm not mistaken (I started it at 02:51 and now at 11:22 it's about 75% completed). I remember having similar encoding rates with similar kind of footage with my former machine based on a 2009 low-end E5200 CPU (without frame interpolation but with at least as resource-intensive denoising filters). Is this to be expected in this case ? Are there ways to improve the performance ? Is it in part due to Avisynth running monothreaded ? (The ffmpeg process runs at 20-23% of available CPU power.) I've read a while ago that Avisynth could be run multithreaded but that it was considered experimental / potentially unreliable ; is it still the case, or has it improved since then ? Does this affect the outcome of the compression ? (I've read somewhere that multithreaded encoding tended to be a tad less efficient.) Also, would it improve matters to use Avisynth and ffmpeg in 64 bits ? I managed to run the Avisynth script in 64 bits, but then I noticed that some frame interpolations (among those made by the Morph function – actually all of them and apparently only those – I reported these observations in a dedicated thread) were wonky, I have no idea why, and it's unlikely to be solved (according to "StainleSS" from this thread, "Jenyok functions sometimes have 'undocumented features', and tend to be a bit big and cumbersome"), so I'll have to stick to 32 bits Avisynth for this task.
Thanks for any kind of clue.


EDIT : Actually, while it's true that the total number of interpolations I had to set for that video is in the vicinity of 8000 (about 5000 commands, some of which call for the interpolation of 2 or more consecutive frames), I already made the bulk of those on pre-processed intermediate files, and the number of interpolations in this final encoding script (either added or re-processed when it so happened that the ouput of the interpolation was improved on the exported video, after a stabilization treatment by Deshaker ; or re-processed with a different number of consecutive frames – for instance it sometimes gives a better result to interpolate three frames in a row rather than one or two, if the adjacent frames being interpolated from are "cleaner" and if the motion is moderate – ; or re-processed with Morpheus instead of FrameSurgeon if it performs better) is about 800 (for about 500 commands). But on 2018/05/12 I had encoded a first version of the same video with most of the FrameSurgeon commands (minus those added later on – about 5% I would say) executed at the encoding stage, with also a dozen Morph commands, plus a SMDegrain filtering on some parts (which I did not use in the current script) {wrong – see below}, and no Morpheus commands at the time (I'm not sure if it even existed then), and according to the created / modified timestamps it took less than an hour (started 21:11 / finished 22:08). I had made that encode with MeGUI, so I'm not sure what version of Avisynth was used, 32 bits or 64 bits. The x264 executable in the “tools” directory seems to be in 64 bits, but I don't know how the script could run in 64 bits since several plugins required for FrameSurgeon's operation were only available in 32 bits at the time.

The script used then was :
Code:
AVISource("I:\20131224\20151224.avi")
ConvertToYV12(matrix="Rec709")

FrameSurgeon(Cmd="G:\20131224 M2TS\Command.txt", Show=false)

morph(14698, 14701)
morph(14708, 14711)
morph(14728, 14731)
morph(17159, 17161)
morph(19871, 19873)
morph(22961, 22963)
morph(24163, 24165)
morph(60061, 60064)
morph(61017, 61021)
morph(61063, 61065)
morph(61065, 61069)
morph(61095, 61101)

Return(last)

S1 = Trim(0, 360)
S2 = Trim(361, 18883).SMDegrain(thSAD=200)
S3 = Trim(18884, 22819)
S4 = Trim(22820, 63800).SMDegrain(thSAD=200)
S5 = Trim(63801, 81930)
S6 = Trim(81931, 83390).SMDegrain(thSAD=200)
S7 = Trim(83391, 84190).SMDegrain(thSAD=400)
S8 = Trim(84190, 0).SMDegrain(thSAD=200)

S1 ++ S2 ++ S3 ++ S4 ++ S5 ++ S6 ++ S7 ++ S8
Well, actually SMDegrain was not used since I put a “Return(last)” command before.
The Command.txt file contained about 4800 commands. Sawbones was tremendously helpful in designing that list, but even with that I spent many hours on this strenuous task over the course of four months...

Last edited by abolibibelot; 30th December 2019 at 03:40. Reason: correction / precision about the number of interpolations
abolibibelot is offline   Reply With Quote
Old 28th December 2019, 14:43   #2  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,320
I can't comment on the functions/plugins you're using, but I've read comments in the past regarding ffmpeg being a bit slow. Can you use the CLI version of x264 instead?

I also can't offer much advice on multithreading as I tend to work around low CPU usage by running more than one encode at a time, or by encoding the video in sections simultaneously by making a copy or copies of the script and adding Trim() to the end of each to specify the sections to be encoded.
--stitchable in the command line will ensure the encoded sections of video can be appended, as long as the same encoder settings are used each time, at least for the CLI version of x264. I'm not sure about ffmpeg.

MP_Pipeline can also help.
hello_hello is offline   Reply With Quote
Old 28th December 2019, 15:04   #3  |  Link
richardpl
Registered User
 
Join Date: Jan 2012
Posts: 144
How can ffmpeg using ffmpeg encoding wrapper for x264 be bit slow? Please stop this nonsense spreading.
richardpl is offline   Reply With Quote
Old 28th December 2019, 15:38   #4  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,025
Quote:
Originally Posted by abolibibelot View Post
Hi,
I'm currently encoding a 1h23min video in 1280x720 resolution and 29.97 FPS framerate, using ffmpeg 32 bits + libx264 -crf 20 -preset "slower", through an Avisynth script which involves mainly interpolation commands to repair blurry frames (quite a lot – about 8000 out of 150000, using a combination of FrameSurgeon, Morpheus and Morph), from a losslessly rendered movie in Lagarith codec (about 120GB), with a computer based on an Intel i7 6700K CPU with 16GB of RAM. And the encoding rate is surprisingly low : 0.127x which is about 3.80 frames per second if I'm not mistaken (I started it at 02:51 and now at 11:22 it's about 75% completed). I remember having similar encoding rates with similar kind of footage with my former machine based on a 2009 low-end E5200 CPU (without frame interpolation but with at least as resource-intensive denoising filters). Is this to be expected in this case ? Are there ways to improve the performance ? Is it in part due to Avisynth running monothreaded ? (The ffmpeg process runs at 20-23% of available CPU power.) I've read a while ago that Avisynth could be run multithreaded but that it was considered experimental / potentially unreliable ; is it still the case, or has it improved since then ? Does this affect the outcome of the compression ? (I've read somewhere that multithreaded encoding tended to be a tad less efficient.) Also, would it improve matters to use Avisynth and ffmpeg in 64 bits ? I managed to run the Avisynth script in 64 bits, but then I noticed that some frame interpolations (among those made by the Morph function, not all of them, I may report these observations in a dedicated thread) were wonky, I have no idea why, and it's unlikely to be solved (according to "StainleSS" from this thread, "Jenyok functions sometimes have 'undocumented features', and tend to be a bit big and cumbersome"), so I'll have to stick to 32 bits Avisynth for this task.
Thanks for any kind of clue.
Post your entire script, we may be able to give you some pointers to accelerate things. Run the script with AVSMeter to verify that it is the bottleneck in your encoding chain (which seems likely).

Avisynth+ has come a long way and supports multi-threading most scripts/plugins without any major problems.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 28th December 2019, 16:58   #5  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,333
For the AVISynth part of things, multi-threading is the main way to provide speed increases. You can often get 2-5 times faster performance, but only with scripts which use functions that permit multi-threading.

As for newer computers being faster, that sadly just isn't very true anymore. Its been about fifteen years since clock speeds maxed out at around 3-4 GHz. That's the end of the line for copper-based interconnections. All the performance gains since then have from parallelism, of which multi-threading is one example. If the software being run can't take advantage of that, you won't get any performance improvement with a newer computer.

I'm still using the computer I bought eleven years ago. I'm about to upgrade the processor (I got the fastest available back then, but a couple of years later Intel came out with a better CPU that is compatible with my MB), but other than that, I see no reason to get a new computer.
johnmeyer is offline   Reply With Quote
Old 28th December 2019, 18:21   #6  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,320
Quote:
Originally Posted by richardpl View Post
How can ffmpeg using ffmpeg encoding wrapper for x264 be bit slow? Please stop this nonsense spreading.
Riddle me this....
The old PC I'm using at the moment is fairly slow on a good day and it's busy doing other stuff at the moment, but as a quick test I converted 10 flac files to mp3, two at a time using LAME. Here's the end of the foobar2000 log file:

Command line: "C:\Program Files\foobar2000\encoders\lame.exe" -S --noreplaygain -V 2 - "David Bowie - 10 - Ashes To Ashes.mp3"
Working folder: D:\
Encoder process still running, waiting...
Encoder process terminated cleanly.
Track converted successfully.
Encoder process still running, waiting...
Encoder process terminated cleanly.
Track converted successfully.
Total encoding time: 1:00.937, 36.43x realtime

ffmpeg's turn:

Command line: "C:\Program Files\foobar2000\encoders\ffmpeg.exe" -i - -ignore_length true -c:a libmp3lame -q:a 2 -id3v2_version 3 "David Bowie - 10 - Ashes To Ashes.mp3"
Working folder: D:\
Encoder process still running, waiting...
Encoder process terminated cleanly.
Track converted successfully.
Encoder process still running, waiting...
Encoder process terminated cleanly.
Track converted successfully.
Total encoding time: 1:14.969, 29.61x realtime

CBR MP3:

Command line: "C:\Program Files\foobar2000\encoders\lame.exe" -S --noreplaygain -b 128 -q 2 - "David Bowie - 10 - Ashes To Ashes.mp3"
Working folder: D:\
Encoder process still running, waiting...
Encoder process terminated cleanly.
Track converted successfully.
Encoder process still running, waiting...
Encoder process terminated cleanly.
Track converted successfully.
Total encoding time: 1:41.000, 21.97x realtime

ffmpeg's turn:

Command line: "C:\Program Files\foobar2000\encoders\ffmpeg.exe" -i - -ignore_length true -c:a libmp3lame -b:a 128k -compression_level 2 -id3v2_version 3 "David Bowie - 10 - Ashes To Ashes.mp3"
Working folder: D:\
Encoder process still running, waiting...
Encoder process terminated cleanly.
Track converted successfully.
Encoder process still running, waiting...
Encoder process terminated cleanly.
Track converted successfully.
Total encoding time: 2:00.953, 18.35x realtime

I'll confess I did the same for flac and ffmpeg was a tad faster than flac.exe and virtually the same as the standalone exe for opus, which surprised me a little. I've used ffmpeg builds in the past that were painfully slow compared to the standalone CLI encoders for most audio formats.

I tried an extremely unscientific x264 comparison, default settings, source resized to 720p in Avisynth and encoded with the CLI encoder. Source also opened directly with ffmpeg, resized to 720p and encoded with the default x264 settings.

Avisynth and x264 CLI managed 24.34 fps, while for ffmpeg it was 19 fps. That probably means nothing, but it was as close to a comparison as I could be bothered to make.

Last edited by hello_hello; 28th December 2019 at 18:28.
hello_hello is offline   Reply With Quote
Old 29th December 2019, 00:18   #7  |  Link
abolibibelot
Registered User
 
Join Date: Jun 2015
Location: France
Posts: 46
@ hello_hello
Quote:
I can't comment on the functions/plugins you're using, but I've read comments in the past regarding ffmpeg being a bit slow. Can you use the CLI version of x264 instead?
Apparently someone doesn't agree... Can x264 CLI use an Avisynth script as input and directly output as MP4 (with an already encoded M4A/AAC audio track), just like ffmpeg ?

Quote:
I also can't offer much advice on multithreading as I tend to work around low CPU usage by running more than one encode at a time, or by encoding the video in sections simultaneously by making a copy or copies of the script and adding Trim() to the end of each to specify the sections to be encoded.
I can try that indeed (since, as I expected, I noticed some residual defects in the encode I made today and which took about 11 hours to complete, which means that I'll have to do it again), normally it should be safe if trimming at fade-to-black points – but I've had so many SNAFUs along the way, involving for instance extra duplicated frames where there should not be (resulting either in audio desynchronization or, more recently, in a desynchronization of the numbers of the interpolated frames, the first batch having been identified on a partially rendered file with no audio, which for some reason had extra frames compared with the full length rendered file with audio... the horror, the horror!), that I'm wary of doing anything even remotely nonstandard.

@ Groucho2004
Quote:
Post your entire script, we may be able to give you some pointers to accelerate things. Run the script with AVSMeter to verify that it is the bottleneck in your encoding chain (which seems likely).
Well, here it is then (quite oddly, the file was refused with the .avs extension) :
20151224 pour encodage [20190730].avs.txt
And the associated FrameSurgeon command file (with a lot of comments – and lines beginning with “#” are commands I tried but which produced an unsatisfactory output) :
20151224 pour encodage [20190730] FrameSurgeon.txt

Quote:
Avisynth+ has come a long way and supports multi-threading most scripts/plugins without any major problems.
Again, there are so many moving parts and unexpected issues that it would have to run flawlessly to be worth the try at this point... (See for instance this other thread I just created, reporting some weird things I experienced with LWLibavVideoSource and some interpolation commands.) Is it guaranteed that the behaviour of every plugin used would be exactly the same with Avisynth+ in multi-threading mode ?

@ johnmeyer
Quote:
For the AVISynth part of things, multi-threading is the main way to provide speed increases. You can often get 2-5 times faster performance, but only with scripts which use functions that permit multi-threading.
Can this also be assessed with AVSMeter or some other tool ?
Could it also be related with the source being in Lagarith format ? When rendering from the NLE I've noticed massive discrepancies depending on whether the source files were mainly lossless (Lagarith typically) or lossy (MP4 or MTS typically). Yet the HDD throughput couldn't possibly be the bottleneck, as on any recent SATA drive it ought to be considerably larger than the data transfer rates required to max out the processing speed (in this case the Lagarith AVI is on a 8TB HDD and the output is on a 3TB HDD, the former can reach a reading speed of about 175MB/s, so it can parse the whole 120GB file in about 11.5min).

Quote:
As for newer computers being faster, that sadly just isn't very true anymore. Its been about fifteen years since clock speeds maxed out at around 3-4 GHz. That's the end of the line for copper-based interconnections. All the performance gains since then have from parallelism, of which multi-threading is one example. If the software being run can't take advantage of that, you won't get any performance improvement with a newer computer.

I'm still using the computer I bought eleven years ago. I'm about to upgrade the processor (I got the fastest available back then, but a couple of years later Intel came out with a better CPU that is compatible with my MB), but other than that, I see no reason to get a new computer.
Well, there must be a noticeable performance increase between the E5200 (2 cores, 2.5GHz) and the i7 6700K (4 cores / 8 threads, 4GHz), even for single-threaded processes, mustn't there ?

Last edited by abolibibelot; 29th December 2019 at 00:42.
abolibibelot is offline   Reply With Quote
Old 29th December 2019, 11:20   #8  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,025
Quote:
Originally Posted by abolibibelot View Post
@ Groucho2004

Well, here it is then (quite oddly, the file was refused with the .avs extension) :
Attachment 17104
And the associated FrameSurgeon command file (with a lot of comments – and lines beginning with “#” are commands I tried but which produced an unsatisfactory output) :
Attachment 17105


Again, there are so many moving parts and unexpected issues that it would have to run flawlessly to be worth the try at this point... (See for instance this other thread I just created, reporting some weird things I experienced with LWLibavVideoSource and some interpolation commands.) Is it guaranteed that the behaviour of every plugin used would be exactly the same with Avisynth+ in multi-threading mode ?
First of all, I'm not surprised that this script is slow. Again, confirm with AVSMeter (and also check memory usage). Running this multi-threaded with 32 bit Avisynth would most likely not work because of memory constraints. Use 64 bit AVS+, I can't think of any reason why it would not work.

I have no idea if this script would run properly multi-threaded. As you mentioned, there are many moving parts. Try it, use these guidelines to set it up.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 29th December 2019, 16:57   #9  |  Link
abolibibelot
Registered User
 
Join Date: Jun 2015
Location: France
Posts: 46
Quote:
First of all, I'm not surprised that this script is slow.
Could you elaborate ? What parts / functions or combinations thereof are slow in particular ?
StainleSS, author of Morpheus and FrameSurgeon, advised to run FrameSurgeon on an intermediate file pre-processed with the Morpheus commands – if I clearly understood what he meant ; what I sure don't understand is the reason why, as it certainly entails technical details about the way those two functions operate and process clips, which may not be optimized for them to work together. Apart from the fact that I currently don't have another 120GB readily available to store yet another lossless intermediate, would it be worth it, considering that it would require two steps, each several hours long anyway ? At least with the current setup (since, yes, I noticed a few residual defects on the encode I made yesterday so I'll have to start over) I can let it run overnight (or overmorning considering how messed up my biological cycles are at the moment).
But what is surprising is that the encoding rate stayed approximately the same through the end, even though there are large chunks with absolutely no interpolation commands or any other kind of processing.
Here is the ffmpeg log for what it's worth (not much I'm afraid, it's not even worth a paper clip so I won't be able to trade it up to get me a house... é_è) :
20151224 [20191228 ffmpeg log].7z.zip
(Since the limit for 7Z files is 200KB and the limit for ZIP files is 300KB and the file compressed in 7Z is 250KB and the file compressed in ZIP is 450KB, I had to put the 7Z in a ZIP for it to be accepted as an attachment.)
By the way, judging from these statistics at the end, are the encoding parameters well suited for the type of content (for instance I remember someone explaining that the when the last value in the "consecutive B-frames" field was high, and higher than the one before, it meant that there was room for further temporal compression), and is the resulting file likely to be played without trouble by a circa. 2011 LG BD550 Blu-ray disc player ?
Code:
video:2118184kB audio:114685kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.236024%
[libx264 @ 06f4b300] frame I:657   Avg QP:17.26  size: 95186
[libx264 @ 06f4b300] frame P:44802 Avg QP:20.63  size: 30845
[libx264 @ 06f4b300] frame B:103792 Avg QP:22.84  size:  6981
[libx264 @ 06f4b300] consecutive B-frames:  3.9%  3.2% 21.0% 71.9%
[libx264 @ 06f4b300] mb I  I16..4: 12.9% 65.6% 21.5%
[libx264 @ 06f4b300] mb P  I16..4:  1.6%  5.5%  0.9%  P16..4: 48.0% 15.1% 12.3%  0.3%  0.1%    skip:16.3%
[libx264 @ 06f4b300] mb B  I16..4:  0.2%  0.5%  0.1%  B16..8: 41.9%  4.6%  0.8%  direct: 5.6%  skip:46.1%  L0:43.5% L1:48.0% BI: 8.5%
[libx264 @ 06f4b300] 8x8 transform intra:68.0% inter:61.5%
[libx264 @ 06f4b300] direct mvs  spatial:99.9% temporal:0.1%
[libx264 @ 06f4b300] coded y,uvDC,uvAC intra: 61.4% 73.4% 29.0% inter: 18.5% 32.8% 1.8%
[libx264 @ 06f4b300] i16 v,h,dc,p: 26% 16% 10% 48%
[libx264 @ 06f4b300] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 11%  9% 12%  8% 12% 12% 13% 10% 12%
[libx264 @ 06f4b300] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 11%  5%  7% 12% 13% 13% 10% 13%
[libx264 @ 06f4b300] i8c dc,h,v,p: 43% 25% 19% 13%
[libx264 @ 06f4b300] Weighted P-Frames: Y:5.0% UV:2.8%
[libx264 @ 06f4b300] ref P L0: 48.2% 11.1% 17.3%  6.2%  5.5%  4.1%  4.3%  2.8%  0.3%  0.0%
[libx264 @ 06f4b300] ref B L0: 81.6%  9.4%  3.9%  1.9%  1.5%  1.1%  0.7%
[libx264 @ 06f4b300] ref B L1: 96.8%  3.2%
[libx264 @ 06f4b300] kb/s:3484.36
Quote:
Running this multi-threaded with 32 bit Avisynth would most likely not work because of memory constraints. Use 64 bit AVS+, I can't think of any reason why it would not work.
Again, I managed to make it work in Avisynth+ x64, but then the output of some Morph commands was abnormal.

Quote:
I have no idea if this script would run properly multi-threaded. As you mentioned, there are many moving parts.
Could you elaborate on the "moving parts" on the specific technical level of Avisynth operation ? ('cuz there are moving parts ev'rywhere, the universe is filled with void and moving parts – and for my part I'm constantly torn between the two... I think I'm a moving void ; therefore I should be the Master ; but I don't understand computers well enough yet.)

Last edited by abolibibelot; 29th December 2019 at 17:10. Reason: forgot to copy the statistics
abolibibelot is offline   Reply With Quote
Old 29th December 2019, 18:12   #10  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 8,456
Quote:
Apart from the fact that I currently don't have another 120GB readily available
I recently bought a 500GB SATA III 2.5" bare drive [external] for £5.00 [2nd user / ie used drive, ]:- @Cex/WeBuy:- https://www.google.com/search?source...4dUDCAc&uact=5
Specs on CNET:- https://www.cnet.com/products/seagat...gb-sata-6gb-s/

I also recently bought my 2nd hard drive dock,
1st one (about £20.00) :- https://www.morgancomputers.co.uk/pr...cking-Station/
2nd one(about £28.00) :- https://www.amazon.co.uk/gp/product/...?ie=UTF8&psc=1
(2nd dock supports both 2.5 & 3.5 inch drives, both SATA and IDE PATA capabilty, 3 drive bays, + one touch offline,[ie when no computer connected] drive to drive imaging function).
Can access two drives simultaneously via USB, ie appear as two external USB devices, (1)=SATA_2.5_or_3.5, (2)=SATA_2.5_or_3.5_or_IDE_2.5_or_3.5.
Both docks USB 3.0.

With a hard drive dock, you can easily swap and change drives, I got maybe a dozen bare drives that I can pop in whenever required.

both above docks are amongst the cheapest of their kind, but sufficient for my needs, and real handy for moving stuff from one machine to another,
assuming that you have a dock for each machine.

EDIT: 2nd dock currently unavailable on Amazon, but there are many similar items available.
Wish I had one or more of these doofahs/gizmos years ago.

EDIT: Both docks came with a disk of backup/restore software.

EDIT: Dock 1 [SATA 2.5 or 3.5]


Dock 2 [SATA + IDE PATA, 2.5 & 3.5 for both - 2 of the 3 possible drives simultaneously accessible (back one is the IDE bay)]
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 29th December 2019 at 20:58.
StainlessS is offline   Reply With Quote
Old 29th December 2019, 18:54   #11  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,025
Quote:
Originally Posted by abolibibelot View Post
Could you elaborate ? What parts / functions or combinations thereof are slow in particular ?
The sheer number of calls to FrameSurgeon and Morpheus.

Quote:
Originally Posted by abolibibelot View Post
Again, I managed to make it work in Avisynth+ x64, but then the output of some Morph commands was abnormal.
That's odd and I have to leave it to Master Stainless to figure that one out.

Quote:
Originally Posted by abolibibelot View Post
Could you elaborate on the "moving parts" on the specific technical level of Avisynth operation ?
As for 'many moving parts' - I just used your words, complexity would be a better choice.

Quote:
Originally Posted by abolibibelot View Post
By the way, judging from these statistics at the end, are the encoding parameters well suited for the type of content (for instance I remember someone explaining that the when the last value in the "consecutive B-frames" field was high, and higher than the one before, it meant that there was room for further temporal compression), and is the resulting file likely to be played without trouble by a circa. 2011 LG BD550 Blu-ray disc player ?
The statistics are useless to judge this. Read this post.

Have you run the script with AVSMeter yet? Any problems with that? I'm curious to see the results.
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 29th December 2019 at 19:05.
Groucho2004 is offline   Reply With Quote
Old 29th December 2019, 23:15   #12  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,320
Quote:
Originally Posted by abolibibelot View Post
@ hello_hello

Apparently someone doesn't agree... Can x264 CLI use an Avisynth script as input and directly output as MP4 (with an already encoded M4A/AAC audio track), just like ffmpeg ?
I'm pretty sure there's a build of x264 that supports writing the output directly to MP4. Probably not while re-encoding the audio. Encoding the audio separately and muxing it with the encoded video later on has ever bothered me though.

Quote:
Originally Posted by abolibibelot View Post
I can try that indeed (since, as I expected, I noticed some residual defects in the encode I made today and which took about 11 hours to complete, which means that I'll have to do it again), normally it should be safe if trimming at fade-to-black points – but I've had so many SNAFUs along the way, involving for instance extra duplicated frames where there should not be (resulting either in audio desynchronization or, more recently, in a desynchronization of the numbers of the interpolated frames, the first batch having been identified on a partially rendered file with no audio, which for some reason had extra frames compared with the full length rendered file with audio... the horror, the horror!), that I'm wary of doing anything even remotely nonstandard.
I've made copies of scripts and used Trim to encode sections of it simultaneously about a gazillion times. I couldn't say how it effects the audio side of things when it comes to ffmpeg, but sometime that's probably a reason to encode the audio and video separately, as you can split the video into sections without having to worry about any potential audio oddness.

For the situation you described above, I'd find the "bad" sections of video, make note of the closest keyframe numbers to the beginning and end of each bad section, open the encoded video in MKVToolNixGUI and tell it to split the output accordingly. That way you can use Trim to re-encode just the bad sections, and MKVToolNixGUI can append them to the rest of the video for you later on.
I realise you mentioned an MP4 output rather than MKV, but I know very little about MP4 muxers so I'm just explaining how it could be done for MKV. Being able to do that sort of thing easily with MKVToolNixGUI makes it almost mental to output MP4, unless of course you have hardware restrictions or you've found an MP4 muxer that doesn't give you suicidal thoughts ....

Anyway.... my point is.... I've encoded video in sections lots and lots of time....
.
I wouldn't bother putting the start and/or end frames for Trim where there's a fade to black. Ending one section on the last frame of a scene and beginning the next section with the following frame (and the first frame of a scene) is fine.
I still don't know if ffmpeg has a --stitchable argument for x264 though. Or maybe it doesn't need it. x264 didn't have a --stitchable option originally. It was introduced after a change broke the ability to append encodes.
https://gitlab.com/dozeo/x264/commit...0d074077909965
That's why I add --stitchable to every command line though, so if I later need to split out a section and re-encode it, I can. I learned my lesson the first time I had to repeat an entire movie encode because of one misspelt word in the hard-coded subtitles.

Last edited by hello_hello; 30th December 2019 at 01:43.
hello_hello is offline   Reply With Quote
Old 30th December 2019, 00:06   #13  |  Link
abolibibelot
Registered User
 
Join Date: Jun 2015
Location: France
Posts: 46
@ StainleSS
Quote:
I recently bought...
Yeah, thanks for the advice, but I already have a gazillion of HDDs lying around, waiting to be properly reorganised (major offender in terms of haphazard data hoarding – I plead guilty as charged), it's just that it's gonna be a humongous task in and of itself and I can't be bothered to do that until I'm finished (that's how I feel these days – finished I mean) with the currently discussed task... é_è
Also, I've become a bit wary of docking stations and external HDD enclosures, because their power supply is usually fragile which can potentially harm the connected drives in the long run (those devices are very sensitive to even minor electrical instabilities {*}). I used to use them but now do so only sparingly since I installed a hot-swap cage right inside the computer case (plugged through a UPS for added safety). Most of those are quite expensive (60 to 120€ – about the price of a decent full size computer case !), but after searching far and wide I found the Xigmatek 3-in-3 SATA HDD hot-swap cage, which was just under 30€ and is excellent overall. It's a less compact design than some other models (there are “4-in-3” design, meaning 4 3.5" HDD bays in 3 5.25" bays, or even “5-in-3” designs), but it's better for air flow, it has a 12cm fan (some similar sized and much more expensive models only have a 8cm fan), the trays are in plastic but sturdy enough and they can host 2.5" HDDs (they need to be screwed down, less convenient than a docking station for those smaller drives ; screws are not needed for 3.5" drives). The only slightly awkward element is the front door, which opens from top to bottom, it's less convenient than a door opening from one side and can be prone to accidentally breaking the hinge if one is not careful (at least it does have a front door). Other than that it's just what I was looking for. Too bad it's apparently no longer produced. I put it inside a Xigmatek Pantheon case, which I chose mainly for its two integrated hot-swap HDD bays accessible from the front pannel (a very rare feature at its mid-range price point, and a very rare feature period, last time I checked only a handful of high-end models had some), with a 12cm fan blowing right beside them, and the possibility to access an additional 4 HDD bays (without a dedicated SATA slot so it's not as convenient) from the front of the case.
If this video doesn't sell everyone on the virtues of internal HDD cages, nothing will !
https://www.youtube.com/watch?v=DA9AKLOVOKk

@ Groucho2004
Quote:
The sheer number of calls to FrameSurgeon and Morpheus.
Then why is the encoding rate about constant, despite the fact that some large parts of the video are compressed with no pre-processing aside from the colorspace conversion ?

Quote:
Have you run the script with AVSMeter yet? Any problems with that? I'm curious to see the results.
I just did : it gives an estimated time of 11h25min to parse the whole video, so the bottleneck seems to be there indeed.
Name:  AVSMeter 3.png
Views: 289
Size:  15.8 KB

As a further test I then ran it again on the same script, but with the already encoded MP4 video as source (loaded with FFVideoSource if that matters), and it's a tad faster, with a total time of 11h02min. And the “thread count” is at 9 against 4, although I don't know what that means here, or why there's such a discrepancy with everything else being equal.
Name:  AVSMeter 4.png
Views: 287
Size:  16.7 KB

Now for the x64 version of the script (I have to use AVSMeter x64 to analyse a 64 bits script, right ?) :
Name:  AVSMeter 5.png
Views: 285
Size:  16.2 KB
Oh sh*t... Now it says 48 minutes ! O_O And the CPU usage is barely higher (actually the CPU fan is less noisy so it must be running slower, which must mean that each core is running at a significantly lower percentage of its capacity – right ?), while the “thread count” is at 12. Member “johnmeyer” suggested that I could get a 2x to 5x improvement, it would seem like it's far more than that (I have yet to see how it translates when actually encoding to H.264). Is it consistent in this case ?

Also, I verified : in fact all the Morph commands produce an abnormal output in the 64 bits script. FrameSurgeon or Morpheus commands seem to produce a consistently consistent ouput. Since there are only about a dozen Morph calls for a total of about 20 processed frames, I could take screenshots and re-insert them with RemapFrames as I already did for other problematic frames, that way I could run that thing at full speed and hopefully get it over with by the end of the 2010 decade (which technically is a year from now).

Still, is it indeed preferred to use x264.exe rather than ffmpeg ? And can it process an Avisynth script just as smoothly and reliably ?




{*} Quote from the HDDGuru forum :
Quote:
Within these years I have had a series of HDD failures, probably 90% of these failures are from external HDD enclosures or 3.5" drives sitting in HDD Docking Stations and approx. 2 2.5" HDD failures in enclosures.
And I wrote this in that same thread :
Quote:
Most enclosures and all docking stations are not ventilated, so the HDDs connected through them operate at higher temperatures ; and they may be more prone to electrical instabilities as well (because of a poor quality power supply, or because the contacts can become faulty after some time, since there are more “moving parts” as opposed to a HDD connected directly to the motherboard with two cables that are never tinkered with). Now I tend to use internal hot-swap cages[*] more than external devices ; I've build my current computer with that purpose in mind : the enclosure is a Xigmatek Pantheon, which has all its HDD bays accessible from the front panel, including two hot-swappable ones (there's a 12cm fan blowing on them from the left side), and I added a Xigmatek 3-in-3 cage (which includes a 12cm fan, while the drives aren't too close as with some other models which allow to place 4 or even 5 drives in 3 bays but must run them hotter for that reason, plus it was cheap at 30 Euros, is compatible with 2.5" drives and does not require screws for 3.5" drives).
If you are in an area with high temperatures, adding fans could improve the HDD's longevity (regularly checking the HDDs' temperatures is a good practice, HD Sentinel is excellent for that purpose).
If you are in an area with electrical instabilities, using a UPS could improve the HDD's longevity (I have a Nitram PB 850 LCD).
abolibibelot is offline   Reply With Quote
Old 30th December 2019, 11:35   #14  |  Link
richardpl
Registered User
 
Join Date: Jan 2012
Posts: 144
Quote:
Originally Posted by hello_hello View Post
I'll confess I did the same for flac and ffmpeg was a tad faster than flac.exe and virtually the same as the standalone exe for opus, which surprised me a little. I've used ffmpeg builds in the past that were painfully slow compared to the standalone CLI encoders for most audio formats.

I tried an extremely unscientific x264 comparison, default settings, source resized to 720p in Avisynth and encoded with the CLI encoder. Source also opened directly with ffmpeg, resized to 720p and encoded with the default x264 settings.

Avisynth and x264 CLI managed 24.34 fps, while for ffmpeg it was 19 fps. That probably means nothing, but it was as close to a comparison as I could be bothered to make.
I can not reproduce your unscientific findings. Encoding mp3 with lame is marginally slower than ffmpeg here. Same applies to x264 cli. I think you are comparing wrong stuff and/or use very old ffmpeg version.
richardpl is offline   Reply With Quote
Old 30th December 2019, 11:47   #15  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,025
Quote:
Originally Posted by abolibibelot View Post
Oh sh*t... Now it says 48 minutes !
Can you please do this test with AVS+ 32 bit?
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 30th December 2019, 16:36   #16  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 8,456
abolibibelot,
Your mention of power supply problems with hard drive docks came just before I discovered that my new No2 dock has just failed,
looks like PSU problem, you must be psychic. Am returning to Amazon seller, and get another one [diff type as no longer available].
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 30th December 2019, 22:21   #17  |  Link
kuchikirukia
Registered User
 
Join Date: Oct 2014
Posts: 462
I think ffmpeg may have worse memory management for Avisynth than avs4x264mod. I'm running into problems using the latest MeGUI on heavy scripts that the older ones using avs4x264 don't have a problem with. (0.04 fps vs 3)
kuchikirukia is offline   Reply With Quote
Old 30th December 2019, 22:44   #18  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,025
Quote:
Originally Posted by kuchikirukia View Post
I think ffmpeg may have worse memory management for Avisynth than avs4x264mod. I'm running into problems using the latest MeGUI on heavy scripts that the older ones using avs4x264 don't have a problem with. (0.04 fps vs 3)
You're comparing apples and oranges (pipe tool vs. encoder) AND using some gui contraption which obfuscates how it interacts with the tools it's using.

By the way, both ffmpeg and avs4x264 use the Avisynth C interface. Memory management is done by Avisynth.
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 30th December 2019 at 22:58.
Groucho2004 is offline   Reply With Quote
Old 31st December 2019, 02:32   #19  |  Link
kuchikirukia
Registered User
 
Join Date: Oct 2014
Posts: 462
Quote:
Originally Posted by Groucho2004 View Post
You're comparing apples and oranges (pipe tool vs. encoder) AND using some gui contraption which obfuscates how it interacts with the tools it's using.
How is it comparing apples and oranges when it's obviously not the encoder that's the bottleneck so it's comparing ffmpeg as the pipe vs avs4x264?
Same script, ffmpeg chokes while avs4x264 runs fine, both running as pipes to x264.exe.

MeGUI doesn't obfuscate anything. The command line is right in the log.
kuchikirukia is offline   Reply With Quote
Old 31st December 2019, 04:46   #20  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,320
Quote:
Originally Posted by richardpl View Post
I can not reproduce your unscientific findings. Encoding mp3 with lame is marginally slower than ffmpeg here. Same applies to x264 cli. I think you are comparing wrong stuff and/or use very old ffmpeg version.
I think it's amusing you're certain ffmpeg can't be slower, but don't question how it can be faster, even if only marginally. Maybe you're comparing the wrong stuff.

I'm now using the XP compatible ffmpeg build dated 2019-08-30 from here.
https://rwijnsma.home.xs4all.nl/files/ffmpeg/
I was using the 2019-04-27 version, and the latest might be tad faster for mp3 encoding, but I discovered Lame.exe is much faster than Lame.exe today. I would've described it as slower, but you probably won't believe that.
I've been using Lame from the foobar2000 encoder pack. The version hosted by rarewares doesn't seem to run on XP. The one I tried today was downloaded from here.
https://tmkk.undo.jp/lame/index_e.html
It's possible I was using the AltiVec/SSE optimized version of Lame 3.99 without realising, or I've forgotten. It'd explain why I remember such a speed difference. Something like 30% faster for VBR encoding, based on my -V 2 encoding tests.

21 files. Encoded one at a time. Slowest to fastest below.

ffmpeg:
Total encoding time: 3:41.938
30.95x realtime

Lame 3.100.2 from the fb2k encoder pack.
Total encoding time: 3:28.719
32.91x realtime

AltiVec/SSE optimized version
Total encoding time: 2:54.140
39.44x realtime

abolibibelot, sorry for sidetracking the thread.

Last edited by hello_hello; 31st December 2019 at 05:06.
hello_hello is offline   Reply With Quote
Reply

Tags
avisynth, ffmpeg, framesurgeon, morpheus, x264

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:31.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.