Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 Encoder GUIs

Reply
 
Thread Tools Search this Thread Display Modes
Old 3rd February 2021, 00:07   #18901  |  Link
Dhry
Registered User
 
Dhry's Avatar
 
Join Date: Jan 2018
Posts: 14
Ripbot takes forever analyzing an MKV file I'm trying to transcode, and then throws this error into the JobsRejected.txt file:

LWLibavAudioSource: failed to open resampler.
(F:\Temp\RipBot264temp\job1\getinfo.avs, line 4)


Looks like this section of the avs file is:

4
5 #VideoSource
6 LoadPlugin("C:\Program Files (x86)\RipBot264\Tools\AviSynth plugins\lsmash\LSMASHSource.dll")
7 video=LWLibavVideoSource("<my filename>.mkv",cachefile="F:\Temp\RipBot264temp\job1\<my filename>.mkv.lwi",prefer_hw=3)

Any thoughts on how to fix?
Dhry is offline   Reply With Quote
Old 3rd February 2021, 17:36   #18902  |  Link
slalom
Registered User
 
slalom's Avatar
 
Join Date: Jan 2010
Posts: 417
Quote:
Originally Posted by Atak_Snajpera View Post
When file EncodingProgress.Pass1 or EncodingProgress.Pass2 in Chunks folder is missing then you lose whole progress. That file is being updated every time chunk gets encoded.
If someone can't avoid that, it would be nice to be able to select the remaining chunks (manually, like 64 to 95) for example right click, partial encode

and then run 2 cmds, CombineAllChunks & jobX_MuxFiles to get the job done

4K 10bit takes a lot of hours to complete!
__________________
E5 2697 v2 @ 3.0GHz on P9X79 Deluxe 24GB
Xeon E5-2680 v2 @ 3.1GHz 16GB
Sony Vaio VPC-F13Z1E/B
slalom is offline   Reply With Quote
Old 3rd February 2021, 21:19   #18903  |  Link
ReinerSchweinlin
Registered User
 
Join Date: Oct 2001
Posts: 367
Hi,

thanx once again for your wonderful peace of software

Any chance of supporting AMD Hardware-decoders?
ReinerSchweinlin is offline   Reply With Quote
Old 4th February 2021, 12:39   #18904  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 617
Quote:
Originally Posted by Atak_Snajpera View Post
When file EncodingProgress.Pass1 or EncodingProgress.Pass2 in Chunks folder is missing then you lose whole progress. That file is being updated every time chunk gets encoded. For your unstable PC I guess I would have to implement some backup system...

Regarding crashing ffmpeg.exe. I've just checked the code in EncodingServer.exe and crash window will always be automatically closed regardless of switches used. It would be much easier if you just manually executed /Chunks/1.cmd from shared folder and then expand details. Look for FAULTY MODULE: line
How come when ever thereís a problem with something to do with the RipBot process, you blame the userís computer ???

Iím not too impressed with you're suggesting that my PC in faulty.

The PC I'm using for this is a Ryzen 9 3950X, not over clocked, a descent amount of RAM, SSDís & Nvmeís for the main temp & processing drives for RB, I have to say that the ONLY thing faulty, is your software.

It happens on every PC I use, when encoding BIG x265 encodes.

I have been using RipBot264 for many, many years, and I have tried to help you sort out some problems along the way, and for most encoding, RipBot is pretty much the ďtop of the heapĒ, for anything up 1080p !!!

Once you start getting into movie length x265 encodes, with even basic filters, itís really not up to the task (in my opinion).

The allure of Distributed Encoding is exclusive to RB, but I believe there is a lot more than needs to be engineered into it, so it handles LONG x265 encodes, which in some case can be days in the process, and it HAS to be super reliable.

When you are not willing or able to let the DE process run itís course in one hit, due to power costís, etc, there HAS to be a very reliable resume function.

I have approached a some other RipBot userís, and they have said that the DE process does have problems with long encodes, and the resume function does not always work, and the encoding already done, is lost, and that can be many, many hourísÖfor nothing.

So itís not just my ďfaulty pcĒ, it must be everybodyís faulty pcís.

I sincerely challenge you to somehow get a movie length x265, HEVC, UHD (whatever you want to call it) file, and load it into what ever system youíve got, give it an Mdegrain1 filter, and DE, and see how you go, stressing that you have to stop (Abort) it randomlyÖI can almost guarantee, it WILL fail.

Another thing that is very miss leading is the ETA counter when doing x265 encodes, with DE.

If it displays that itís going to take, say 8 hours, then itís more like 24Öyou test it, youíll see what I mean. Itís fine with everything else, but not x265, so you really have no idea how long itís going to takeÖyou just have to let it go.

As far as I am aware, you are running Windows 7 on a single Xeon E5-2690 (not too sure), so that IS going to take a LONG, LONG time !!
Itís about the least you can do, to prove that itís not our faulty pcís.

So just quoting your reply to me :-

When file EncodingProgress.Pass1 or EncodingProgress.Pass2 in Chunks folder is missing then you lose whole progress. That file is being updated every time chunk gets encoded. For your unstable PC I guess I would have to implement some backup system...

So what causes those .pass files to go missing ??? Öoh wait, itís because RipBot didnít shutdown properly !!

Maybe YOU need to implement some redundancy, during the process.

Regarding crashing ffmpeg.exe. I've just checked the code in EncodingServer.exe and crash window will always be automatically closed regardless of switches used. It would be much easier if you just manually executed /Chunks/1.cmd from shared folder and then expand details. Look for FAULTY MODULE: line

And how do you test your code ?? I bet itís not on x265 files, or Windows 10 !!

So, Iím not trying to piss you off, I am trying to get you to understand that there are problems that need to be at least checked & tested thoroughly, due to the amount of time the process takes.

Us userís want to get the encodes done with as little drama as possible, not be chasing faults & time wasting re starts of the encoding process.
__________________
Not poorly done, just doin' it my way !!!
So much to do, and so little time :(
Pauly Dunne is offline   Reply With Quote
Old 4th February 2021, 15:18   #18905  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,599
If you are so confident that resume does not work most of the time then check if encodingprogress.pass1 is still present before and after you click abort. Your rants are meaningless for me if you do not provide any valuable feedback. (You didn't even bother to check why ffmpeg.exe crashed!)I'm not a fortune teller for god sake! (yet).

Last edited by Atak_Snajpera; 4th February 2021 at 15:20.
Atak_Snajpera is offline   Reply With Quote
Old 4th February 2021, 15:26   #18906  |  Link
Ryushin
Registered User
 
Join Date: Mar 2011
Posts: 318
Quote:
Originally Posted by Pauly Dunne View Post
Iím not too impressed with you're suggesting that my PC in faulty.

The PC I'm using for this is a Ryzen 9 3950X, not over clocked, a descent amount of RAM, SSDís & Nvmeís for the main temp & processing drives for RB, I have to say that the ONLY thing faulty, is your software.

It happens on every PC I use, when encoding BIG x265 encodes.

Once you start getting into movie length x265 encodes, with even basic filters, itís really not up to the task (in my opinion).

The allure of Distributed Encoding is exclusive to RB, but I believe there is a lot more than needs to be engineered into it, so it handles LONG x265 encodes, which in some case can be days in the process, and it HAS to be super reliable.

I sincerely challenge you to somehow get a movie length x265, HEVC, UHD (whatever you want to call it) file, and load it into what ever system youíve got, give it an Mdegrain1 filter, and DE, and see how you go, stressing that you have to stop (Abort) it randomlyÖI can almost guarantee, it WILL fail.

Another thing that is very miss leading is the ETA counter when doing x265 encodes, with DE.

If it displays that itís going to take, say 8 hours, then itís more like 24Öyou test it, youíll see what I mean. Itís fine with everything else, but not x265, so you really have no idea how long itís going to takeÖyou just have to let it go.
I don't think it's fair to blame Atak and Ripbot264 about x265. x265 is a whole new beast compared to x264. For me, with my settings, x265 takes four times longer to encode than x264. Add 4K to that and a 4K movie takes 16 times longer to encode than a HD using x264. I've encoded over 400 4K movies with great success. Though I'm not aborting my jobs and resuming for the most part. My jobs run straight through.

If you need jobs to complete within a certain time frame, you might have to add more CPU power to your mix or battery (LiFePO4) backup so the servers can keep running when the solar is not producing.

About reliability. My servers are rock solid. Run ECC memory, ZFS, etc. The Win10 VM that runs RipBot264 sits on top of Linux running KVM/LibvirtD. I don't have any issues except that occasional bug when click "Done" the job gets stuck and I have to abort RB. When I have to abort, about 1 in 10 times RB will have the job start over. I really don't have hardly any issues with RB and I'm very happy to have it.
Ryushin is offline   Reply With Quote
Old 4th February 2021, 16:55   #18907  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 617
Quote:
Originally Posted by Atak_Snajpera View Post
If you are so confident that resume does not work most of the time then check if encodingprogress.pass1 is still present before and after you click abort. Your rants are meaningless for me if you do not provide any valuable feedback. (You didn't even bother to check why ffmpeg.exe crashed!)I'm not a fortune teller for god sake! (yet).
I'll tell you why I didn't check ffmpeg...as soon as I saw that it was starting from scratch, I stopped it, and deleted the job !!!

What's the point, it'll probably happen again

When situations like this arise, most of us users have no idea what course of action to take, but you do, as you created it, so none of us, are fortune tellers, or mind readers !!

So it's hard to provide you with the "feedback" that you would like.
__________________
Not poorly done, just doin' it my way !!!
So much to do, and so little time :(

Last edited by Pauly Dunne; 5th February 2021 at 00:11.
Pauly Dunne is offline   Reply With Quote
Old 4th February 2021, 17:29   #18908  |  Link
Ryushin
Registered User
 
Join Date: Mar 2011
Posts: 318
I wanted to add that RB is the only program I know that can even resume an encode, at least in DE mode. I think in single threaded mode even RB will start over from scratch.

That being said, I do think resuming aborted jobs could work a bit better as that should work 100% of the time. For one of my clients, I deal with very large movie jobs, often larger than 1TB in size. Because this is commercial works, corruption could be hard to detect without using MD5 sums for the files.

Atak, perhaps when a chunk is finished, a MD5 sum is recorded in <chunk#>.md5 file. Upon recovering a job, if there are any .md5 files, they are compared with the chunk and if missing or incorrect, that chunk is encoded again. Then a master file has a list of the the number of chunks and that should not have to change. I'm more than willing to wait for a few cpu/storage cycles to resume a job.
Ryushin is offline   Reply With Quote
Old 4th February 2021, 20:24   #18909  |  Link
chainring
Registered User
 
chainring's Avatar
 
Join Date: Sep 2006
Posts: 163
FWIW, I've processed hundreds of 4K HDR movies with RipBot, all with x265 and the vast majority in DE mode. Failures are the exception, not the rule from what I've experienced. Four Windows 10 machines, all Dell and I make sure all updates are current before starting an encode party. When there is a failure; more like a stall, it's been older movies and generally ones where I'm taming grain with MDegrain. Those can sometimes be a royal pain since it'll stall the queue.
chainring is offline   Reply With Quote
Old 4th February 2021, 22:42   #18910  |  Link
slalom
Registered User
 
slalom's Avatar
 
Join Date: Jan 2010
Posts: 417
Quote:
Originally Posted by Ryushin View Post
I wanted to add that RB is the only program I know that can even resume an encode, at least in DE mode. I think in single threaded mode even RB will start over from scratch.

That being said, I do think resuming aborted jobs could work a bit better as that should work 100% of the time. For one of my clients, I deal with very large movie jobs, often larger than 1TB in size. Because this is commercial works, corruption could be hard to detect without using MD5 sums for the files.

Atak, perhaps when a chunk is finished, a MD5 sum is recorded in <chunk#>.md5 file. Upon recovering a job, if there are any .md5 files, they are compared with the chunk and if missing or incorrect, that chunk is encoded again. Then a master file has a list of the the number of chunks and that should not have to change. I'm more than willing to wait for a few cpu/storage cycles to resume a job.
I'll give you a scenario, abort a job, add 2 other jobs, edit them and restart encoding (the aborted one, doesn't have to be 4K/h265)

MD5 sums is a very good idea! Don't know much about that but I hear it

I do little 4K by choice, and I use mainly MDegrain1 or nothing so I don't loose much time to cry for
__________________
E5 2697 v2 @ 3.0GHz on P9X79 Deluxe 24GB
Xeon E5-2680 v2 @ 3.1GHz 16GB
Sony Vaio VPC-F13Z1E/B
slalom is offline   Reply With Quote
Old 5th February 2021, 00:15   #18911  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 617
Quote:
Originally Posted by Ryushin View Post
I don't think it's fair to blame Atak and Ripbot264 about x265. x265 is a whole new beast compared to x264. For me, with my settings, x265 takes four times longer to encode than x264. Add 4K to that and a 4K movie takes 16 times longer to encode than a HD using x264. I've encoded over 400 4K movies with great success. Though I'm not aborting my jobs and resuming for the most part. My jobs run straight through.

If you need jobs to complete within a certain time frame, you might have to add more CPU power to your mix or battery (LiFePO4) backup so the servers can keep running when the solar is not producing.

About reliability. My servers are rock solid. Run ECC memory, ZFS, etc. The Win10 VM that runs RipBot264 sits on top of Linux running KVM/LibvirtD. I don't have any issues except that occasional bug when click "Done" the job gets stuck and I have to abort RB. When I have to abort, about 1 in 10 times RB will have the job start over. I really don't have hardly any issues with RB and I'm very happy to have it.
It wasn't until I read your post that I realised that I had made a significant oversight in my "rant".

It seemed like I was wholey & soley blaming x265, when that wasn't my point...I know I should have written 4K encodes (as they are x265 by default, generally)..anyway, the main thing behind the whole post was DE's problems with LONG 4K encodes.
__________________
Not poorly done, just doin' it my way !!!
So much to do, and so little time :(
Pauly Dunne is offline   Reply With Quote
Old 5th February 2021, 00:25   #18912  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 617
What about a whole new version of Ripbot...

So, my post has sparked some interesting discussion, pro's & con's, which is all good.

Some possible very worthwhile suggestions have been made.

Here's another suggestion (for the future), what if the current RipBot264 was re-configured to ONLY process 4K (and upwards) movies !!!

It could have all the x264 & batch options ripped out, an error checking process on DE, the ETA meter synced, and that's it.

It could be called RipBot4K, or RipBotUHD...anyway you get my drift.
__________________
Not poorly done, just doin' it my way !!!
So much to do, and so little time :(
Pauly Dunne is offline   Reply With Quote
Old 5th February 2021, 05:01   #18913  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,599
No .
Atak_Snajpera is offline   Reply With Quote
Old 5th February 2021, 12:07   #18914  |  Link
ReinerSchweinlin
Registered User
 
Join Date: Oct 2001
Posts: 367
Quote:
Originally Posted by Atak_Snajpera View Post
Last digit indicates how many previous and next frames are taken for analysis. More means stronger denoising and more stable output.
MDegrain aims for fine detail retention while knlmeanscl is much more aggressive. My recommendation is as follows. Use MDegrain first and then knlmeanscl if you still need to remove any grain leftovers.
thanx for clarifying
Am I correct that older versions had "degrain 2" as option, so chossing degrain 2 gives the exact same results as older ones ? (I am asking because I started a Series of encodes which is around 40% finished and I would like to keep the settings..)

What happened to "adaptive" in the KNLMEANS Tab? (Just curious on this one)

I am just exploring some openCL Stuff on a VEGA64 and was wondering, if it would be possible to support the AMD Video Decoder in Ripbot.
ReinerSchweinlin is offline   Reply With Quote
Old 5th February 2021, 20:20   #18915  |  Link
whiskey
Drunken munky
 
Join Date: Mar 2005
Location: US, IL
Posts: 62
Anybody else still has issues removing lots of files at once from the batch window? Despite having all files selected it only removes one, then I have to de-select another one just to remove one more...
whiskey is offline   Reply With Quote
Old 6th February 2021, 01:30   #18916  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 617
Quote:
Originally Posted by Atak_Snajpera View Post
No .
I know you probably don't want to hear much more about this, but I'm hoping you didn't misunderstand what I was suggesting.

Due to your emphatic "No" !!

I was not suggesting for a second that you "gut" RipBot264, what I thought was a "twin" app, based on RipBot, but it ONLY did 4K x265 (and above) file encoding.
__________________
Not poorly done, just doin' it my way !!!
So much to do, and so little time :(
Pauly Dunne is offline   Reply With Quote
Old 6th February 2021, 13:23   #18917  |  Link
ReinerSchweinlin
Registered User
 
Join Date: Oct 2001
Posts: 367
Quote:
Originally Posted by whiskey View Post
Anybody else still has issues removing lots of files at once from the batch window? Despite having all files selected it only removes one, then I have to de-select another one just to remove one more...
I remember using the "del" key to quickly get rid of a lot of entries.
ReinerSchweinlin is offline   Reply With Quote
Old 6th February 2021, 17:40   #18918  |  Link
slalom
Registered User
 
slalom's Avatar
 
Join Date: Jan 2010
Posts: 417
Quote:
Originally Posted by ReinerSchweinlin View Post
thanx for clarifying
Am I correct that older versions had "degrain 2" as option, so chossing degrain 2 gives the exact same results as older ones ? (I am asking because I started a Series of encodes which is around 40% finished and I would like to keep the settings..)
It is exactly the same
Quote:
Originally Posted by whiskey View Post
Anybody else still has issues removing lots of files at once from the batch window? Despite having all files selected it only removes one, then I have to de-select another one just to remove one more...
No issues here
__________________
E5 2697 v2 @ 3.0GHz on P9X79 Deluxe 24GB
Xeon E5-2680 v2 @ 3.1GHz 16GB
Sony Vaio VPC-F13Z1E/B
slalom is offline   Reply With Quote
Old 6th February 2021, 22:50   #18919  |  Link
GZZ
Registered User
 
Join Date: Jan 2002
Posts: 569
Quote:
Originally Posted by Pauly Dunne View Post
It wasn't until I read your post that I realised that I had made a significant oversight in my "rant".

It seemed like I was wholey & soley blaming x265, when that wasn't my point...I know I should have written 4K encodes (as they are x265 by default, generally)..anyway, the main thing behind the whole post was DE's problems with LONG 4K encodes.
What is a long 4k encode ?

I have done close to 300 UHD 4K encoding to MKV with ripbots. I have done it with MDegrain 1, 2 and a few with 3. By long you mean in number of frames (duration) and for that I done The Deer Hunter and The Bridge On The River Kwai have taking looooong time because of MDegrain2 filter and I havent had a failure with encoding. Doing DE on 2 machines. Machine 1: Ryzen 3700x, Machine 2:Intel Core I7 8700K (none of the them is overclocked). All HDD is SSD.

I cant reconize a failure and I would like to know a title that fails for you, if the fail isnt repeatable and its random, then I would suspect hardware issues or network problems. Network problems is the thing that can give must issues and be hard to troubleshoot. I had a unstable USB network dongle once and it was a pain in the but, it could fail if I "overloaded" it with copy jobs.
GZZ is offline   Reply With Quote
Old 7th February 2021, 01:40   #18920  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 617
Quote:
Originally Posted by GZZ View Post
What is a long 4k encode ?

I have done close to 300 UHD 4K encoding to MKV with ripbots. I have done it with MDegrain 1, 2 and a few with 3. By long you mean in number of frames (duration) and for that I done The Deer Hunter and The Bridge On The River Kwai have taking looooong time because of MDegrain2 filter and I havent had a failure with encoding. Doing DE on 2 machines. Machine 1: Ryzen 3700x, Machine 2:Intel Core I7 8700K (none of the them is overclocked). All HDD is SSD.

I cant reconize a failure and I would like to know a title that fails for you, if the fail isnt repeatable and its random, then I would suspect hardware issues or network problems. Network problems is the thing that can give must issues and be hard to troubleshoot. I had a unstable USB network dongle once and it was a pain in the but, it could fail if I "overloaded" it with copy jobs.
Hi GZZ,

I would classify a long 4K encode anything approaching 3 hours, and over, more that 180 + chunks !!!

I was doing The Hobbit - An Unexpected Journey, extended version, using MDegrain1.(3 hours 4 minutes, I believe)

I had it setup on my Ryzen 3950X (as the client), and had a couple of servers helping along, a 3930K, Xeon E5-2697 v2, and dual Xeon E5-2690.

Now I have to say that I was NEVER intending on doing this in one "session", so I was going to have to rely on the resume function, that is sort of the DE process.

So I got thru approx 80 - 90 chunks the first day, Ripbot shutdown OK, next time, it resumed, so I did another 40 +/- chunks, using different "servers", but when it was time to shutdown for the day, I waited until the processing chunks were done, and then close them individually, then when I went to close down RipBot, it just froze, "not responding", and stayed that way for quite some time, until eventually I just had to kill it....I just knew that next time, there will be a resuming problem...

So next time, sure enough, it had "forgotten" where is was up to (approx 120 chunks had been done), and it started from chunk #1 !!!

Because it had started processing I thought there was no real course of action to be taken, so I just stopped it in disgust, and deleted the job.

Some would say that this might be my fault by not doing it in "one hit", but I refuse to do that due to electricity costs.

So for reliability of the resumption of DE, it needs to be some how backed up, or error checked during the process, so this doesn't happen.

Now I can't tell you what files I have problems with, as it's random !!

I would like to ask you about the ETA counter....do you find that it's "slow" when doing 4K encodes ??? I find that what ever it display's, you need to multiply that by about 3, (eg: a 6 hours displayed ETA, is closer to about 18), by the time the encode if done.

OK, so how long do your looooong encodes take, with your hardware setup ??

Do you do your encode in one hit ??

See, if no one else has to stop their encodes, they have no idea what I'm on about, and I am made out to be the "bad guy" !!!
__________________
Not poorly done, just doin' it my way !!!
So much to do, and so little time :(
Pauly Dunne is offline   Reply With Quote
Reply

Tags
264, 265, appletv, avchd, bluray, gui, iphone, ipod, ps3, psp, ripbot264, x264 2-pass, x264 gui, x264_64, x265, xbox360

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:04.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.