A custom linux distro made for doing encoding as fast as possible! [Archive]

View Full Version : A custom linux distro made for doing encoding as fast as possible!

Pages : [1] 2

morph166955

14th May 2007, 19:33

After two months of tinkering with my new 8-core system I've come some conclusions in terms of raw speed, encoding speed and operating systems

1) Because of the nature of a large distribution, it's not feasible to create a system thats optimized for everyone. Because of this, most of the kernels are compiled either statically that is much larger then it needs to be, or its compiled to use modules for everything which can be a pain also for other reasons. In addition, most of the base libraries are not compiled using any specific cpu optimizations or anything like that so that they are backward compatible with the oldest of systems. For obvious reasons, this can cause a slight slow down.

2) Most if not all of these distro's have a lot of superfluous stuff that can cause random slow downs or unwanted operations. What I was seeing on fedora was a lot of stuff running in the cron scripts (like beagle for example) that would slow down the machine while it was running. For a person who 95% of the time is doing audio/video stuff , we could really care less about things like that right!

The distribution that I'm currently compiling/testing is only going to have a limited amount of backward compatibility to allow for me to do optimization thats only compatible with newer processors (Prescott and up for now). In addition, I'll be creating several kernel/library versions that are more optimized for newer and more powerful systems. One thing that (initially) I'm going to limit everything to is EM64T support in the processor. Almost every processor that has been released in the past 12-18 months has had EM64T support (also known as x86_64 support in some parts of linux) so I don't think this is too outlandish at least in the beginning. I'll eventually compile a 32 bit version but I'd rather get the 64 bit version up and "stable" before trying to put a 32 bit version together. Just so everyone understands, the 64bit version is built with 32 bit support via multilib so don't think that the 64 bit version wont run 32 bit software, its just going to prefer the 64 bit stuff over the 32 bit if it has a choice.

Another major goal of this is to have "bleeding edge" versions of the different codecs/encoders/etc as well as the current system tools and kernels compiled, optimized and available for download as add-on packages that the user can download and install at their digression. By doing this we achieve the ability for the user to run what they want (and specific versions of each package such as gcc 4.1 vs 4.2 vs 4.3) and also only what they want instead of having to download a whole package of stuff that they don't want in order to get one thing they do want out of it. This distro is going to be designed for functionality and speed with little emphasis on the extra little "bells and whistles" that other distros have. Some would call this "form over functionality." One example of this, is X. While I will create a compile of X11, I doubt that I'll ever compile gnome, kde or any other software thats similar. You can run any gui software on X11 just as well as you can run it in kde and gnome for the simple reason that kde and gnome are just overlays on top of X11 to make it look nicer.

Ultimately, I want this to be something that people on here who have systems "dedicated" to encoding would choose to use over some of the other bigger distributions out there.

Right now I'm looking for 2 things.

First, I'm looking to create a list of what libraries, codecs, encoders, system tools, etc that everyone would want made available for this distribution when all is said and done.

The other thing is that I'm looking for people who would be willing to test this system. For now, the only requirements that I have is that the system has an Intel cpu thats EM64T capable and also compatible with either prescott, nocona, or core2 gcc optimizations. I WILL be making AMD compatible versions of this eventually, however seeing as I don't have an AMD to test on, this is going to be a secondary thing for me to do after I get a "stable" version for the Intels done. Also I should note that I'd prefer it if the volunteers have at least a decent working knowledge of linux and its configurations. The reason for this should be obvious. While I'll be around to assist people with problems they may encounter, I can't teach someone how to do every step of the most basic parts of this. Eventually I'll have it so that even the most basic user shouldn't have a problem with the setup or usage, but for now probably not.

For the alpha versions I'll write up a script that will install the software using an isolinux boot cd of some sort (it shouldn't matter which since tar does most of the work so knoppix, lfs, etc). I'll also include something in the script that will generate the appropriate configuration files that are needed for the system to boot properly (hopefully). There are a few small things that either can't be done by script or that I'm not comfortable writing a script to do (such as format the drives & install boot loaders). It's not that I can't write them, its that I don't want to screw up anyones drives if they are partitioned, or using multiple drives or anything like that by having a generic script. Id say that 99% of the install can be done by scripts though, the rest I'll writeup readme files for.

I'm running/testing this on both my core2 laptop (run both on xp via vmware6 and on a separate partition to run it natively) as well as my dual x5355 xeon system as its primary OS. Even though the XP system is 32 bit, I have intel's virtualization technology enabled to let me run the 64 bit kernel/softare.

My time frame is this. I'm finishing my finals this week and graduate on Sunday. I should have some time to work on it during my down time this week. After that, I have 2 weeks off in which time I hope to get my server up and running with in 48 hours of being home (cause I'm going crazy with it being down now). I've already compiled a base kernel and I can boot the system however it is very stripped down and as of now does not have support for some of my hardware or ssh or anything else loaded (its literally about as bare bones as you can get it to still boot). Assuming I can get something to work stably with in a few days, I'll make an alpha release around then which will include the base system, tools, etc as well as packages for all of the codecs and such as I compile them. From there on, I'll make releases as I fix problems or update things or what ever.

To finish this huge post up, I'm very open to suggestions as to ways to make this the most optimal solution for people like us who do a/v encoding or just general comments about this so please, if you have ideas, I want to know them!

Nematocyst

15th May 2007, 00:46

While I'm sure you aren't kidding about wanting to do encoding as fast as possible, that constraint doesn't fit for everyone. And that's why there are lots of distributions to choose from. Your idea of a list of various encoding related packages, though, does have merit. Such a thing is useful to anyone on any distribution interested in video/audio encoding.

It sounds to me like you don't need a custom encoding distribution though. I would think a source based distribution ought to allow you to both pick what runs, and optimize the heck out of it.

eousphoros

15th May 2007, 01:38

Gentoo. and I already have two boxes setup in a manner which is kinda like what you described above. Anyways you get an A for effort. :)

nfm

15th May 2007, 03:28

Please don't flame me, but I would recommend using other distros such as Ubuntu. Suse and Fedora are loaded with to much software. I tested amd64 Gentoo vs. Ubuntu and saw zero difference in encoding seed with x264.

What I do is install Ubuntu, compile my own K8 kernel with couple of other little tweaks and compile x264 or any other encoders with my cpu's flags. All of this can be done under 2 hours. Gentoo on the other hand, required me to put into hours and even days of hacking, editing configs to get started.

Thasp

15th May 2007, 07:11

Fast? I'd just be happy if it worked. Get filmdint to work on any highres source(not 720, but 1080) without green nonsense at the top, and I'll be happy, then get it to be fast. :)

Or at least fully multithreaded. I get 50% less usage on core 2 than with windows, which uses all of both cores with MT avisynth.

I don't think it's a distro thing as much as a software thing.

morph166955

15th May 2007, 08:22

well...lets just see if we cant make something that does everything we want and fast!

I'm already well along the way to having this distro compiled for my own system, so I'm going to finish it and release it like I have said above for anyone who wants to try it out to do. Once thats done, who knows what we can accomplish!

thasp, something that may give you something to play around with is an idea that I was using to do highspeed x264 encodes before i formated that setup. now granted, I have a few more cores to play with so that helped, but heres what I did. I ran 2 mencoder's with output of rawvideo in i420 format simultaniously, one outputed the even frames, the other did the odd ones (both to fifo pipes). I then wrote a very small C program to read the sources, resync the order of them, and dump out our original movie to another fifo pipe (this used very little cpu/memory btw). Finally i had x264 read from that pipe and do its encodes. When having only one mencoder thread i maxed out at 175fps (low res, very fast x264 settings) and x264 was not running anywhere near what it could. when i did it the second way, x264 was pulling close to 320fps and running much closer to full cpu. I found that more then 2 mencoder threads began to overrun what my hdd could spit out. I realize this solution is much less then optimal, but it may help you to get closer to full usage.

other then that, mencoder/mplayer sucks in terms of using threads and MT avisynth happens to do it really well. One of the packages I'm hoping to include is an avisynth 3 binary thats multithreaded (assuming theres a single thread version). hopefully that should bridge the gap that were seeing in terms of accessing our source on linux.

morph166955

18th May 2007, 06:28

so just to throw out an update here in case anyones watching this thread.

Now that my finals are done (thank god) I have been able to spend some quality time with my distro. I have a VERY basic version booted and running very smoothly in a vmware workstation. some versions of the software isnt the most recent/updated but since I was going by a how-to and its versions to get to this part i knew that to start. now that I've worked through it once end to end and I have more of an understanding of the process, im going to redo it using the newest versions of everything (that matters). Thats the plan for tomorrow! Assuming I can get that much done, I'll be doing a similar setup on my server once im home on Monday. I'll probably have a "working" version for anyone who wants it by the end of next week.

On that note, if you are someone who is interested in this, let me know!

Amnon82

21st May 2007, 09:40

x86-64 is a 64-bit microprocessor architecture and corresponding instruction set; it is a superset of the Intel x86 architecture, which it natively supports. It was designed by Advanced Micro Devices (AMD), who have since renamed it AMD64. This architecture has also been adopted by Intel under the name Intel 64 (formerly known as Yamhill, Clackamas Technology (CT), IA-32e, and most recently Extended Memory 64 Technology (EM64T)).This leads to the common use of the names x86-64 or x64 as more vendor-neutral terms to collectively refer to the two nearly identical implementations.

Well Intel uses AMD technology so your version should be also compatible with AMD64. I'm also in a developer-team of a distro (Paldo GNU/Linux (http://www.paldo.org)) so I know a "little" about linux. Tell me when the ISO is ready and I'll take a look.

morph166955

21st May 2007, 20:43

Awesome, thanks for the offer I'll let ya know!

What I was referring to was that if you look in the linux kernel config menu under the 64bit options there is Intel EM64T, AMD (i forget their name), and a hybrid one that works with both. I'm not 100% sure what the differences are (haven't had time to look into it much with finals, I will now that im done) but I have to assume there are at least some small differences otherwise that wouldn't be there. The software itself will all be x86_64 compatible so that should all work universally. Maybe I'll just make 3 different kernel versions, one for each of the above options, and let the user pick which one to install. I just got in from the last of the graduation stuff so I'm ending this post here so that I can go work on this toy finally and get something nice working!

Amnon82

24th May 2007, 23:37

... some News?

morph166955

25th May 2007, 01:50

I have/had a version that was up running no problems, except that openssl kept kicking back "invalid opcode" errors when doing the encryption tests during the make tests step. I believe I know why and I'm rerunning the install with a few settings tweaked and a different kernel compile (changed a few options in it that could have been causing a problem) so we shall see. I'm hopin to have something I can ssh into tonight. Otherwise everything else was working perfectly. I have/had network connectivity, all of my libraries compiled and tested fine (at least with their tests, I didn't test all of them manually) and all of the software that I loaded worked. I'll update you all once i get this openssl issue fixed cause its bugging the hell outa me right now.

morph166955

25th May 2007, 04:05

ok i just completely rebuilt the whole thing and im still getting the same issue. Heres what openssl is spitting out :

echo test normal x509v1 certificate
test normal x509v1 certificate
sh ./tx509 2>/dev/null
testing X509 conversions
p -> d
openssl[21986] trap invalid opcode rip:2ba27ca274d0 rsp:7fff2e27a5a8 error: 0
make[1]: *** [test_x509] Error 1
make[1]: Leaving directory `/packages/openssl-0.9.8e/test'
make: *** [tests] Error 2

its openssl 0.9.8e. this is the one thing holding me back right now from having a fully working setup so any help is appreciated!

EDIT: I had to update the quote because the openssl[21986]... line was for some reason missing from the log even when i passed stderr to it.

morph166955

25th May 2007, 04:51

probably should say that im doing an x86_64 multilib setup using linux 2.6.20.1, gcc 4.2.0 and glibc 2.5. if anyone wants any other versions let me know.

morph166955

26th May 2007, 17:37

So something else I've been wondering about. As of now I'm creating compiles using my target (and host in this case) to be "x86_64-unknown-linux-gnu" since thats what I was told to use and thats what all of the cpuid stuff spits back. Its obvious that some distro's modify the "unknown" to be their name. Does anyone know where do I do that? Most of the configure files in most of the software just uses "x86_64-*-linux-gnu" when its doing its checks so I don't think theres much to change on that side of things. Its more of, where do I actually change that. I'm assuming its somewhere in the kernel configuration or in the build of it but specifically where is it is what I want to know. I found the option in the kernel configmenu to append a string to the end of the version but nothing on replacing the part inside of it.

BTW in case anyones wondering, I have mostly decided on a name for this distro/project. "Encodix" coming from the phrase "The A/V Encoders Linux." I'd love to know what people think of the name since sometimes the names I think are cool are either really lame or they are being used elsewhere that I don't know about. If everyone (or at least most people) likes the name, I'll setup a sourceforge project for it so that we have an svn repository and somewhere to store packages and stuff thats easily accessible.

As it stands now, the only thing holding me back from releasing a prealpha version is this stupid openssl thing. If I can figure it out or find some way to create a library that works on here (maybe cross compile it?) then I should be able to release something. I'm going to make a post probably tomorrow with the versions of the libraries and software thats going to be in (or more exactly what I'm planning to have in) what I'm calling its "core" package as well as versions of the different add-ons (eg "x264 r656" or "xvid 1.2.-127"). Again remember, this isn't going to be for the faint of heart or the linux newbie. I'll make a script and a bootable iso with knoppix or isolinux or something that will ask a few simple questions and extract the appropriate packages and setup your locales and timezones and all that kinda stuff. The package list will also be far from complete or done. For example, avisynth 3 will probably not be listed since I haven't had any time to look at it yet, but it WILL (hopefully) be included in future revisions.

shevegen

27th May 2007, 05:13

That makes me kinda ... feel strange.

If people point out that they are using Gentoo, SuSE etc.. then they are losing the point that a distribution built by smart people focusing on one area - multimedia for that matter, or lets say video stuff- will beat these other distributions down-right hands off in that aspect, even if the general stuff will lend them to "greater usability". :)

Amnon82

27th May 2007, 12:46

@morph166955: maybe you look into UPKG (http://www.upkg.org/) and Paldo Distro (http://www.paldo.org) cos we do almost the same as you want to do. You can easily make a fork with your custom kernel. For the customiazion you can look on my ISOs: Paldo ISOs using Gnome, KDE, XFCE (http://www.paldo.org/index-section-packages-page-installercd.html).

Maybe this will help you with the package manager. Also new developer are always welcome to our project (first release was: 31 October 2004).

@shevegen: seems you still like my distro.

morph166955

27th May 2007, 17:55

@shevegen
What do you mean it makes you feel strange? What in particular are you referring too? I'm not really sure what you mean but I would like too so that you don't feel strange about this.

@Amnon
I have looked at Paldo and I must say I am very impressed with the work you guys have done on it. I have been contemplating how to do the packages and I have decided that for the time being (basically while this is in its (pre)alpha stage) that I'm simply going to upload bz2 files to sf.net that people can pick and choose from. Eventually I'll implement some more advanced form of package management that works with wget or something similar (possibly using upkg or something like that) so a person can just type a command and it installs or updates the software on the system and keeps them up to date.

The thing to remember about my distro which is where I see it skewing from Paldo (and please correct me if I'm wrong) is that my distro is intended on being compiled without very much hardware backwards compatibility in mind as well as the fact that I'm not going to be including things (as much as I can at least) that generate any additional CPU/HDD usage which could take away from the encoders available speed. Now before people panic by that statement, this is what I mean. Most of the distros out there have to (and understandably) be able to work on 99.95% of the systems out there including systems that go back to the original Pentium's or earlier. I see this as a HUGE hole in speed possibilities that optimization can bring in on things like the kernel and glibc and gcc that will in turn give us libraries and binaries that (hopefully) will be at least marginally faster then the big distros do but at the same time make it easy for a person who doesn't like to compile their own kernel to use. Yes, I realize that any distro can be customized to work quickly and recompiled with newer faster things but many people out there either don't have the knowledge, time, or patience to play around with it and do all of those little tweaks. I'll still be including versions for newer hardware but they are going to be tuned for that hardware. for example, I'm going to have 4 different kernel versions as of now. One for each of the 4 options available in the kernel config, AMD, Older xeons (nocona), core2 and newer xeons, and base x86_64. While the speed differences among them are probably negligible, I was always brought up with the thought that every little bit can eventually add up to something big.

In the end, the worst case scenario here for me is that I end up being the only one using it, in which case I still accomplished my ultimate goal of making a system for myself that is as fast as humanly possible. I'm just hoping that others will be able to benefit from my time and that it will be something fun to play around with.

Amnon82

27th May 2007, 18:52

Paldo is really flexible. UPKG uses tar.bz2 files and creates binaries out of it. So you can do Binary-Repositories or Source-Repositories.

If you want you can use our binary-packages or build them all from source on your pc. We have some meta-specs for creating an ISO-image. You can easily create your own. UPKG will build an ISO out of it. Paldo HowTos (http://www.paldo.org/index-section-support-page-howtos.html) - Paldo Wiki (german) (http://paldo.wikia.com/wiki/Hauptseite)

So if you want to use source-tarballs UPKG is the right choice to go with it. You can use source-tarballs but you have UPKG to take a look on the dependencies so you can delete them from the hdd without leaving unneeded files on the hdd.

You got PM so take a look on it. There is no must for using our packages but you can use them for your fork if you want to.

Shure you can start your own distro from scratch but why not use a basic flexible distro you can do what you want with it and add your kernel-optimations to it.

Paldo supports many hardware, but if you don't want it to do so, so don't use all packages.

It is always you how choose what gets installed on your pc. We only supply you with packages we think it is ok to use.

Paldo is flexible, easy to build own packages and to mod.

Take a look on my Installer CD images (http://www.paldo.org/index-section-packages-page-installercd.html) so you can see what I did with the live-cd-iso.

When I can do it, you can do it also. I only wanna help you to get an easy start for your distro. Take it or leave it :)

morph166955

28th May 2007, 17:42

IT WORKS! As of last night I have a functioning system that seems to be stable. I'm in the process of running a whole slew of tests on it and preparing some basic libraries/binaries (like gcc, glibc, emacs, samba, ssh/ssl, ntp, etc). In a few more days I'm going to try to package up the core and some basic software and stick it all in an installer and load it in a few vmware consoles on my laptop to see what that does. If it all works out though, I may have an ISO for you guys some time in the upcoming week or so.

Also just FYI for everyone, I'm moving this weekend and I start my new job Monday so I probably won't have internet for a few days and I probably will be unavailable on here until thats all calmed a bit but even though I don't have internet that doesn't mean I can't be playing around with the whole thing!

Inventive Software

30th May 2007, 00:23

Dumb question number 1... did you get openssl compiled right? ;)

Dumb question number 2: I'm assuming VLC or MPlayer or some other player would be used in future, for playback, so that it's HTPC-friendly?

morph166955

30th May 2007, 00:44

dumb answer 1...yes I got it. figured out that it was cause i was using gcc 4.2.0. I used gcc 4.1.2 and it worked perfectly.

dumb answer 2... mplayer is going to be used initially because thats what I'm familiar with. I'll look into vlc once I get things more calmed down. I'm running compiles and tests on glibc & gcc to make sure that they are 1000% stable. Once I'm sure about them I'll finish up my optimized builds for the rest of the libraries against them and then release something for testing.

Inventive Software

30th May 2007, 00:48

Cool! Look forward to it. This could be my summer project! :D

morph166955

30th May 2007, 16:06

now thats the spirit! I'll try to get something out to you guys some time next week/weekend. With my upcoming move and me rushing to finish current projects work on this is slowing down to a crawl probably until around next Tuesday. I'll still be working on it and making builds but no where near as fast as I've been in the past 2 weeks.

At this point I think its safe to ask if anyone has any particular software or version of any software they specifically want to try out? If it's something thats more obscure to find, posting a link to where I can get the source/docs would be awesome!

Inventive Software

30th May 2007, 17:23

I have one small request. I'm looking to refresh the Linux distro on my current desktop, possibly with this "Encodix" you're making. However, the kickback is the limitation of the processor... it's a Celeron 800 MHz with crucially MMX and SSE only. :( No SSE2 or SSE3 or SSSE3. How difficult would it be to adjust the compile flags so that it doesn't use these "more advanced for my processor" optimisations, so that the Kernel and associated software doesn't throw a paddy when compiling / running?

morph166955

31st May 2007, 02:10

it actually would require a full rebuild of everything from the first step sadly. Nothing would be compatible for one simple reason. Its not because of the SSE2+ stuff...its because of the 32 vs 64 bit part. This system uses whats known as x86_64 multilib for its library setup. It primarily uses 64 bit and only falls back onto its 32 bit stuff if it absolutely has too (things like mplayer need 32 bit for some things and wont work on 64 bit). For me to make a 32 only bit install would require me starting all the way at the beginning. For a system like that, I'd say go with ubuntu or something similar since they should be about as fast as your system can handle and already exist. You won't see any gains out of my build with a system of that age simply because the stuff out there is already optimized for systems like that where mine is designed to use things like 64-bit and SSE3 when it can.

Amnon82

31st May 2007, 14:17

@Inventive Software: I just finished my newest Build of Paldo GNU/Linux Installer CD build (http://www.paldo.org/%7Eamnon/paldo-installer-cd-testing-2007-05-29-1715.iso). It is optimized for your PC. Also I've a build for 64bit CPUs on my download page (http://www.paldo.org/index-section-packages-page-installercd.html) to try out. Also using different desktops if you don't like gnome at all.

"Encodix" is for PCs from AMD64 /P4EXT up and not for your old CPU. Sorry.

Inventive Software

31st May 2007, 14:44

Thank you Amnon. I shall download it later and have a play. @morph166955: That's a shame, so perhaps I can use yours for my new PC around the Autumn, depending on how much money I have. ;(

Amnon82

31st May 2007, 15:29

@Inventive Software: When morph166955 has his first iso online I'll take a look on his ISO and tell you if it faster than my kernel ...

morph166955

1st June 2007, 14:26

Well I think faster is going to be a relative thing.

Will my kernel be faster on a brand new high speed box? Yea probably even if its a small amount

Will my kernel be faster on an older box? Probably not by much although I'm hoping that its at least marginally faster.

For obvious reasons with this project, one kernel build will just not be enough. I'm planning on doing at least 4 kernels right now, probably several more as time goes on. One will be a generic x86_64 kernel with only standard kernel optimizations in it. This will serve as a base kernel (or an emergency boot kernel in case one of the others has unforeseen issues) as well as the one I'll probably load up on the boot cd. I'll then be making one with what they are calling "Older Xeon" optimizations in the kernel menuconfig. I'm assuming this is a Nocona build (or something similar since I think it does its own assembly optimization as opposed to asking gcc to do it) since I can't really think of anything else it would be referring too since they are all 64-bit options. I'll also be doing a "Core2/Newer Xeon" build for the cutting edge machines (I'll probably be running this kernel as my "daily driver" of sorts). The fourth kernel will be using the AMD option that is in the config. I've got to assume that the AMD build will be using a newer 64-bit instruction set that the intel's aren't compatible with otherwise they would just include that with the x86_64 generic build. After I have a stable working platform, I'm considering building gcc 4.3 from its svn and trying to make some even more tweaked kernel builds with its core2/ssse3 optimizations if thats even possible. I've got to do some more research and look into the Makefiles and all to see how its actually doing the build and what is safe and/or possible in terms of using the compliers optimizations vs what is in the kernel itself already.

@Inventive:
I've done some contemplation on your request, and I'll give you this offer if you want it. If you can wait a little for me to get my 64-bit build off the ground and figure out what versions of what software play nicely and how to compile them, I'll attempt to create a 32-bit version. It was always my intentions to make a 32-bit version of this eventually, it just wasn't on the forefront of what I needed for my own setup since I wanted to be on a 64-bit platform. Hopefully by that point I'll have it down to a science where I can just let it run a very well tuned script that I can leave running while I go to work or something (as it stands now its about 10 scripts to do the build in stages). I will say this though, until I am marginally confident with the stability of the 64-bit platform I won't be porting over to 32-bit for obvious reasons. If its not stable in some areas on one platform, who's to say it will be on another. A 32-bit build is simpler since there is no need to worry about having two sets of libraries and binaries and everything else (one for 64-bit and one for 32-bit). The software versions will remain the same for simplicity, they will just be compiled differently. To give you a hint on when this may be (or more exactly when you might want to remind me I made this post), the versioning system I'm using now for my tracking is on 0.0.2 (aka the prealpha version). The first alpha will be 0.1(.x) (which is the first one that I'll be uploading) and that will go to 0.4(.x). The first beta will be 0.5(.x) and that will go to 0.8(.x). My RC builds will be 0.9(.x) and the full release versions will be 1(.x). When you see me get into the beta builds, remind me of this post since by then I should have the versions of the software and most of the big issues handled by that point and I'll be dealing with the little annoying bugs instead.

Now for my disclaimer. I'm staring a new and very intensive job this upcoming Monday (June 4th). I have no idea when I'm going to have free time to play around for the first bit minus weekends which knowing what I'll be doing will be filled with studying up for my certification tests. This could be a while down the line until I get to a 32-bit version. If I see that a month from now I'm not where I want to be, I'll consider doing a build then if you can wait that long. One thing you can do until then is what I currently have running on my box. I'm dual-booting with Ubuntu 7.0.4 server so that I have something that can function to do something should the need for it arise. If you partition your disc now and just leave the second half free, you could do this for once I have the 32-bit builds built. Load it onto the second partition and just tell grub to boot from that as a default. Once your happy with it, then I'll work with you too move things around so you can either have one partition running my build, or have ubuntu on the second partition and mine on the primary, or what ever else you want. Moving my distro from one partition to another should be relatively easy (I've done it once already myself and it took all of about 5 minutes). Hope this works a little better for you!

Inventive Software

1st June 2007, 15:29

@Inventive:
I've done some contemplation on your request, and I'll give you this offer if you want it. If you can wait a little for me to get my 64-bit build off the ground and figure out what versions of what software play nicely and how to compile them, I'll attempt to create a 32-bit version. It was always my intentions to make a 32-bit version of this eventually, it just wasn't on the forefront of what I needed for my own setup since I wanted to be on a 64-bit platform. Hopefully by that point I'll have it down to a science where I can just let it run a very well tuned script that I can leave running while I go to work or something (as it stands now its about 10 scripts to do the build in stages). I will say this though, until I am marginally confident with the stability of the 64-bit platform I won't be porting over to 32-bit for obvious reasons. If its not stable in some areas on one platform, who's to say it will be on another. A 32-bit build is simpler since there is no need to worry about having two sets of libraries and binaries and everything else (one for 64-bit and one for 32-bit). The software versions will remain the same for simplicity, they will just be compiled differently. To give you a hint on when this may be (or more exactly when you might want to remind me I made this post), the versioning system I'm using now for my tracking is on 0.0.2 (aka the prealpha version). The first alpha will be 0.1(.x) (which is the first one that I'll be uploading) and that will go to 0.4(.x). The first beta will be 0.5(.x) and that will go to 0.8(.x). My RC builds will be 0.9(.x) and the full release versions will be 1(.x). When you see me get into the beta builds, remind me of this post since by then I should have the versions of the software and most of the big issues handled by that point and I'll be dealing with the little annoying bugs instead.

Now for my disclaimer. I'm staring a new and very intensive job this upcoming Monday (June 4th). I have no idea when I'm going to have free time to play around for the first bit minus weekends which knowing what I'll be doing will be filled with studying up for my certification tests. This could be a while down the line until I get to a 32-bit version. If I see that a month from now I'm not where I want to be, I'll consider doing a build then if you can wait that long. One thing you can do until then is what I currently have running on my box. I'm dual-booting with Ubuntu 7.0.4 server so that I have something that can function to do something should the need for it arise. If you partition your disc now and just leave the second half free, you could do this for once I have the 32-bit builds built. Load it onto the second partition and just tell grub to boot from that as a default. Once your happy with it, then I'll work with you too move things around so you can either have one partition running my build, or have ubuntu on the second partition and mine on the primary, or what ever else you want. Moving my distro from one partition to another should be relatively easy (I've done it once already myself and it took all of about 5 minutes). Hope this works a little better for you!

That's fine by me. I'm replacing Fedora 6 this afternoon with the Paldo ISO that Amnon linked to earlier, so that'll be an adventure I'm sure! :) I have no problem with waiting, I'm relatively patient. ;)

morph166955

1st June 2007, 16:16

FYI, Fedora 7 released today (or late yesterday).

Inventive Software

1st June 2007, 19:28

I was aware, thank you. ;) That'll be my next download target for my university's monstrous bandwidth! :D

shevegen

2nd June 2007, 00:51

I think what is also needed is that smart people help out the not so smart people :)
What i mean like is get together to exchange information and squeeze the best out of what's possible, including the underyling OS but also specific apps via IRC (maybe), wikis (hopefully), forum (very hopefully) :)

Amnon82

2nd June 2007, 12:28

@Inventive Software: Paldo uses upkg (English Doc (http://paldo.wikia.com/wiki/UPKG_Dokumentation_%28Englisch%29)) as package manager. If you want to install for example vlc just open a terminal and write

sudo upkg-install vlc

Other packages you can find here: Paldo Packages (http://www.paldo.org/index-section-packages.html)

@shevegen: A wiki or irc-chan is not a problem. It can be easy to be done. Seems you wanna help ...

@morph166955: As soon as you have a working ISO I'll take a look on it. I'm very interested on your install-script.

morph166955

11th June 2007, 22:18

Hey all. Sorry for the huge delay in me replying. My internet just got setup Saturday night so its been a little hard to get online since I've been down here up until now. Just thought you all would be curious to know a little update. I'm virtually ready to make an iso for this. I did an encode with mencoder piped into x264 last night of some 720p stuff, I'm going to watch it in a few minutes to make sure it all worked properly but as far as I could tell it worked flawlessly. Assuming all that works, I'm going to build a package that I'll load up on my laptop just to make sure that it all works on at least one other system before sending it out (which I have no doubt that it will...but it can't hurt to verify). Once thats all good, ill make some sort of basic install script that will work for now, stick it in an iso and upload it somewhere for the people who want it.

Inventive Software

12th June 2007, 22:58

I did an encode with mencoder piped into x264 last night of some 720p stuff
How fast was it? :D

morph166955

12th June 2007, 23:44

well i had it running with basically the same settings as megui's insane profile and it ran close to about 8fps. This was more of a sustained high load type test that I was doing. I'm going to kick it down a bit and see how long it takes on "normal" settings.

RyosukeFC

19th June 2007, 21:54

morph166955

19th June 2007, 22:41

I generally do this in 2 ssh windows or using screen to make the output easier to read

SCREEN 1:

mkfifo istream.yuv

mencoder myvideo.avi -ovc raw -of rawvideo -nosound -o istream.yuv -vf harddup,format=i420 -mc 0 -noskip

(I place any/all scale, crop, etc after harddup and before format, it seems to work the best. -nosound is possibly unnecessary and can also screw up audio sync so use at your disgresion)

SCREEN 2:
x264 --myx264options -o myoutput.264 istream.yuv 1280x720

(replace 1280x720 with your final output resolution from mencoder)

ENJOY!

artvandelay

20th June 2007, 00:21

What I want to know is how to pipe mencoder into x264. I've been fighting with Linux encoding for a long time now, knowing that if I could somehow replace my tried+true DVDdecrypter->DGIndex->Gknot(for .avs generation)->MeGui+x264->MKVtoolnix chain with something that works on Linux, I'd be so much less dependent on Windows. It really sucks having to constantly reboot every time I want to do some encoding, all the while feeling guilty not getting "real work" done (in Linux)...

If someone could just point me in the direction of some information about arguments to mencoder/mplayer that would make output appropriate for x264's stdin handler, that would be brilliant.

i feel the same way. the _only_ thing i use windows for right now is megui. i like to do dvd rips with nero aac and x264 along with avisynth. right now there just isn't a good linux solution it seems. i know nero aac runs perfect in wine and mp4box is available but i really need something to replace megui's auto-detect and avisynth script creator to do my video encoding.

morph166955

20th June 2007, 00:26

i feel the same way. the _only_ thing i use windows for right now is megui. i like to do dvd rips with nero aac and x264 along with avisynth. right now there just isn't a good linux solution it seems. i know nero aac runs perfect in wine and mp4box is available but i really need something to replace megui's auto-detect and avisynth script creator to do my video encoding.

I assume you are completely ignoring my post right above yours which says EXACTLY how to do it

artvandelay

20th June 2007, 00:38

I assume you are completely ignoring my post right above yours which says EXACTLY how to do it

i'm not ignoring it. but i'm saying i use megui to autodetect what kind of deinterlacing i need to use. i also don't know how to rip a dvd to a yuv file. i don't even know what a yuv file is.

morph166955

20th June 2007, 00:43

ok...deinterlacing...mencoder has one, maybe two filters to does that for you "automagically". take a look at pp=lb or pp=fd and pullup/softskip...they generally work perfectly for normal content.

learning mencoder will take all of about an hour of playing around with it and reading the manual a little. you can find the manual online at http://www.mplayerhq.hu/DOCS/man/en/mplayer.1.html

as for the yuv file, your not actually writing a yuv file. a fifo (which is created by the mkfifo command above mencoder) is in its simplest form a pipe (as seen by a "|" in a command) that instead of being sent to stdin the system directs it through a "fake" file. yuv is what x264 understands to be a raw YUV 4:2:0 input.

and to handle the dvds, just replace myvideo.avi with "dvd://(chapter) -dvd-device /dev/mycddrive" and it works just the same as any other video file would.

artvandelay

20th June 2007, 00:55

morph166955

20th June 2007, 00:59

RyosukeFC

20th June 2007, 04:14

Me too. Vanilla SVN all the way. ;)

But I've noticed, at least a while back, that I had to manually alter the Makefile of x264, because it had SSE2 hardcoded into the gcc line (-msse2 and -DHAVE_SSE2 or something similar), and on friends' Athlon-XP machines I had to take care of it for them. Ah yes, the good old days...

Anyway...

@morph: Thanks for the bit about using a fifo. It's a pretty obvious solution that simply never occurred to me (slaps forehead).

So what I think I'm going to do is write a quick and dirty MEcli using just straight C/C++, and I'll specifically target x264. Mencoder is so feature-packed, people don't realize how much of MEgui's functionality it ISN'T necessary to duplicate under Linux (AVISynth, DGDecode.dll, etc.) since mencoder will accept raw vobs as input.

None of this is to knock Sharktooth's or Doom9's hard work with MEgui (and/or whoever else helps with that project), they just seem really unmotivated to help us Linux enthusiasts by at least offering a Linux version of the parts of MEgui that *would* work without AVISynth or DGDecode.

Summary: (1) Much respect all around and (2) let's us Linuxy people keep at it, and see what we can come up with.

artvandelay

20th June 2007, 22:58

now theres a good question...and heres a good and simple answer...it does it for you! the wonderful developers who have made us this amazing piece of software already incorporate virtually every important speed increase that they can while simultaneously having their configure script figure out what the local system can and can not run properly. If you compiled x264 on your system, its as fast as it can get it. As for the output of x264, well that is the H.264 standard so thats not going to really change.

The only option that you could say might help is the Adaptive Quantization patch thats floating around on here (look for sharktooths patches in the sticky at the top of the AVC forum it should be in there). For more info on that, id recommend doing a search since i've never really used it. I'm more of a vanilla svn type of person myself.

i downloaded the svn code and checked it out for myself. they do seem to put in all the optimizations when you run the configure script, but coming from my years of gentoo i know the stereotypical fastest settings aren't always the best for every architecture. on your dual quad xeons and even my dual core opteron they probably are, but for older cpus they may not be. unless the script scales back on older cpus, i haven't really bothered on my slower machines.

which brings me to my next point. i haven't read the whole thread yet, but what do you think you can do on the distribution-level to enhance encoding? i'd love to help out any way i can, but it seems that maybe just a guide would work fine for most people to use with their already setup linux system.

artvandelay

21st June 2007, 03:10

what am i doing wrong?

mkfifo istream.yuv
mencoder VTS_04_1.demuxed.m2v -ovc raw -of rawvideo -o istream.yuv -vf harddup,format=i420 -mc 0 -noskip
x264 --pass 2 --bitrate 2048 --stats ".stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads auto --thread-input --no-dct-decimate --no-psnr --no-ssim --progress --output video.h264 istream.yuv 720x480

results with:
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [error]: ratecontrol_init: can't open stats file
x264 [error]: x264_encoder_open failed

morph166955

21st June 2007, 04:03

well its a 2 pass encode...so you have to run x264 with "--pass 1" before "--pass 2"

and dont forget, you have to fire up mencoder again for the second pass...

RyosukeFC

21st June 2007, 05:05

It's a bit OT, I know, but...

I'm curious how the whole fifo thing works at the kernel memory allocation level. Obviously, there's a dynamic memory buffer somewhere, but in the context of x264 and mencoder being on either end of the pipe, the bottleneck is x264, not mencoder. So do mencoder's write() operations just take longer than usual? Any other way of handling it seems like it would run out of physical memory if the source file was big enough. So the kernel must just be artificially limiting the speed of mencoder's write() calls.

Hmm. Anybody know?