X264 encoding for 50,000 shortform videos

[blanka33]

Hi

I currently have been tasked with the job of analysing the production required to reformat up to 50k short form videos (between 2 and 15 mins) into a super sexy new streaming (progressive download) formats for the service that we will be supplying them to.

The sources are mainly mpeg2 (some uncompressed avi) but range in aspect ratio's and bitrates wildly from 8Mbps PAL to 50i 1080p.

The output we have decided on is 3 "SD" and one "HD" formats and have developed the x264 command lines to suit encoding as much quality for the lowest possible bitrate (the margins between us making money and crashing due to bandwidth costs is very fine)

My issue is, up until now we have been using Rhozet Carbon for our encoding, which is great in way of management of huge amounts of files of huge amounts of different formats (including pre-processing like normalisation and cropping) but this won't (without a lot of work developing a directshow plugin) encode video using x264 and insist on using the frankly quite rubbish quicktime and mainconcept encoders.

So, basically, instead of killing myself writing batch commands for every possibility of input, encoded into 4 different outputs, then muxed together with audio and relevant hint tracks etc. Or trying to use megui or similar. Has anyone got a solution that will make our lives any easier? Or do I have to make the decision between crappy video quality or killing myself with batch files.

Sorry, post was a bit longer than expected, hope someone has a few good ideas.

[CruNcher]

Hmm strange other Porn sites also get this done ;)


So, basically, instead of killing myself writing batch commands for every possibility of input, encoded into 4 different outputs, then muxed together with audio and relevant hint tracks etc. Or trying to use megui or similar. Has anyone got a solution that will make our lives any easier? Or do I have to make the decision between crappy video quality or killing myself with batch files.

Also where is the problem you learn something that way, and if you want to advance in what you currently do then you have to learn nobody here will do this work for you (and if you really just to lazy then sorry for you, it's your job you get paid for it), finding commercial solutions that fit your tasks and maybe be more profitable for you are also not that hard to find really and someone who throws around with Rhozet Carbon Encoder i guess knows about what he's talking.

Instead you come here and want everything on the silver table that makes me sick, sorry.

[smok3]

make it a two step process then

a.
(carbon will take preprocessing and dump the thingy into some tmp uncompressed file), then feed the x264 with that < my experience is that this step takes a lot of manual work, so if you really already have a nice automagic way, keep it.
b.
for x264 you could make a folder-watch type of thing batch files (two folders, two scripts to watch/encode), or possibly use a gui that already has that capability.

[CruNcher]

smok3 if i guess from which service blanka33 is and in which position and he asks such questions then i dunno what to think about it, nothing personal Blanka33 can't really support a closed platform that's only available for a small group of users additionally in a special context, maybe im to harsh i know but that's how i am ;)
Though of course i find it a good thing that you realized that you can have both good quality at lower cost and are willing to enhance your service and im sure others like smok3 gonna help you in achieving that for you and your targeted users :)

[smok3]

CruNcher: honestly i don't know what you are talking about? Certainly, personally i'am not interested in giving "porn site a silver table" - which seems to be your assumption, but my assumption is that nobody is guilty until proven.

[benwaggoner]

Carbon 3.20 (just out) includes some nice improvements in its H.264 encoder, and new H.264 presets I made for them between my leaving Microsoft and joining Amazon. You might want to give it a run to see if it's good enough for what you want to do.

FWIW, I've found a little Y and Cr denoise can go a long way in improving encoded quality, roughly equivalent to the x264 denoise mode.

[Blue_MiSfit]

Cool tip, Ben! I will check out your presets - I've been meaning to spin up a VM and try 3.20.

Carbon does a good job of conforming a pile of disparate assets into a uniform output - mostly.

Derek

[blanka33]

Hmm strange other Porn sites also get this done ;)



Also where is the problem you learn something that way, and if you want to advance in what you currently do then you have to learn nobody here will do this work for you (and if you really just to lazy then sorry for you, it's your job you get paid for it), finding commercial solutions that fit your tasks and maybe be more profitable for you are also not that hard to find really and someone who throws around with Rhozet Carbon Encoder i guess knows about what he's talking.

Instead you come here and want everything on the silver table that makes me sick, sorry.

Hey Cruncher

First of all... good guess but it's not porn i'm afraid, think music videos (although the line between music videos and porn are starting to blur these days), The company I work for have a service that allows you to stream 50k of them for free (Ad funded of course). Feel free to guess what service it is!

Secondly, I assure you i'm not asking for anything on a silver table, just using every tool at my disposal to make the right choice for the company. I have many lines of enquiry of which Doom9 is one, I would much rather be pointed in the right direction to software that genuinely encoding experts use for a solution rather than trawling through meeting after meeting of sales people hawking their overpriced encoders. The best case would be if I could use our current infrastructure (which I am generally happy with) in conjunction with a x264 solution that genuinely works therefore saving my company £xxxxx's on software and support contracts.

I hope you understand where I am coming from and I thank you all for your suggestions.

[CruNcher]

The company I work for have a service that allows you to stream 50k of them for free (Ad funded of course).

yes i know see my 2nd post the porn thing was just a joke btw ;)

[kieranrk]

Hey Cruncher

First of all... good guess but it's not porn i'm afraid, think music videos (although the line between music videos and porn are starting to blur these days), The company I work for have a service that allows you to stream 50k of them for free (Ad funded of course). Feel free to guess what service it is!

Secondly, I assure you i'm not asking for anything on a silver table, just using every tool at my disposal to make the right choice for the company. I have many lines of enquiry of which Doom9 is one, I would much rather be pointed in the right direction to software that genuinely encoding experts use for a solution rather than trawling through meeting after meeting of sales people hawking their overpriced encoders. The best case would be if I could use our current infrastructure (which I am generally happy with) in conjunction with a x264 solution that genuinely works therefore saving my company £xxxxx's on software and support contracts.

I hope you understand where I am coming from and I thank you all for your suggestions.

Ignore everything Cruncher says, seriously. He is a veritable cornucopia of stupidity.

There isn't a particularly good project for doing something like you suggest - the closest I've seen was a person writing something called FreeFactory for his internal use at a TV station and then open sourcing it.

[ajp_anton]

Maybe you already got it working, but I would make an Avisynth script that handles the different input formats and outputs the resolution you want, maybe with some processing added.
Then loop through each input file, generate an .avs file with the whole script with the only differences being the input filename, and encode from there.

[StainlessS]

Purely to generate AVS files (for Mpeg2 source) from a template (as suggested by ajp_anton), you might look at
DGIndex Batcher, it's not ideal for your purpose but might help (perhaps, command line solution would be better/quicker).
http://forum.doom9.org/showthread.php?t=163369

EDIT: DGIndex Batcher is a "user proxy" for the GUI of DGIndex, ie manipulates the mouse and
keyboard. It does so using the mouse at a realistic-ish human speed, but if you wanted to
speed this up a bit, you only have to alter a value at the beginning of the source and
re-compile (it's written in Freeware AutoIt language [not too much different to Basic]).

Tried unsuccessfully to find "FreeFactory", anyone got a link?

[blanka33]

Maybe you already got it working, but I would make an Avisynth script that handles the different input formats and outputs the resolution you want, maybe with some processing added.
Then loop through each input file, generate an .avs file with the whole script with the only differences being the input filename, and encode from there.

This had crossed my mind, I was thinking I could automate most of the pixel aspect and cropping issues by pulling the info out from media info or similar and then using the info to auto generate an avs that will format the mpeg correctly ready to use in a batch x264 command line I have written. Seems a little convoluted though and if I can get Carbon to do it for me then that would be preferable so I will definitely try both benwaggoner and smok3 ideas to see if I can get our money's worth out of Carbon.

The alternative is to get our dev team to spend some time trying to build a x264 plugin for carbon, I know this is possible and works well as another company we work with has this already (but annoyingly won't give me any more details) Does anyone have any knowledge of doing this or anyone that does have this background. Our dev team are really good but it will take them some time to learn and develop this so it would be good to have someone that has gone through the process to ask a few questions to to get us started.

Thanks everyone for all their help so far, your insight has been very helpful.

[blanka33]

Purely to generate AVS files (for Mpeg2 source) from a template (as suggested by ajp_anton), you might look at
DGIndex Batcher, it's not ideal for your purpose but might help (perhaps, command line solution would be better/quicker).
http://forum.doom9.org/showthread.php?t=163369


Will also look into this too. thanks StainlessS

[smok3]

a 2 minute glance over carbon features briefly mentions command line events, so why not install demo on your local machine and try that?

[StainlessS]

blanka33,

Below something I posted on AutoIt forums about both DGIndex and MEGUI Batchers, just after last
version was released, have not yet put this on Doom9.

As an afterthought, next version I shall remove the GPL licence stuff and replace
with Freeware licence so as not to place any restrictions on AutoIt users.
EDIT: You can take it as applying already.


AutoIt is a very powerfull language with a lot of system type stuff written in it, for eg
Setup/Sysprep and the like. You may be able to coax it into modifying each AVS file
based on info contained in the created d2v/log file (or MediaInfo) [eg Aspect Ratio], and to
perhaps create a batch command file.

EDIT: There is an example in the MediaInfo.dll source for using the dll from AutoIt.

EDIT: Also see here to see how MeGUI deals with Aspect Ratio signalling in AVS files.
http://mewiki.project357.com/wiki/MeGUI/Aspect_Ratio_Signalling_in_AviSynth_Scripts

EDIT: perhaps source from here might provide ideas.
http://forum.doom9.org/showthread.php?t=164449&highlight=avs2yuv

[blanka33]

Carbon 3.20 (just out) includes some nice improvements in its H.264 encoder, and new H.264 presets I made for them between my leaving Microsoft and joining Amazon. You might want to give it a run to see if it's good enough for what you want to do.

FWIW, I've found a little Y and Cr denoise can go a long way in improving encoded quality, roughly equivalent to the x264 denoise mode.
Hi Ben

I know this is a while after I posted this but I was just wondering how you got a copy of Carbon 2.0, I can only find 3.19 available to me and I'm not getting an answer from my Rhozet contact?

I have come up with a x264 solution but it's still a bit convoluted so wouldn't mind trying the improved h264 in Carbon.