Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 11th July 2019, 19:11   #1  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
FFAStrans - FFMpeg Avisynth Transcoder

Hi,
I've been using Avisynth for years, but ever since 2016 we needed to automatize our workflows a lot as we had to encode too many files and we didn't have enough encoders, so manually encoding each and every raw file wasn't an option anymore.
Since then, I've started creating a few batch scripts and Avisynth scripts to automatize some common things based on the characteristics of every file, but then I came across a project that Steinar Apalnes was working on called FFAStrans and I switched to it.
Today I'm gonna talk about it and the power of open source, as I think that many broadcasters and production houses are paying a lot of money for softwares like Telestream Vantage, Harmonics ProCoder, Adobe Media Encoder, Selenio, AWS Transcoder and similar while Avisynth, ffmpeg and x264 can deliver better results and are also free.


User Interface:




This is how the user interface of one of my main workflows looks like. It might seem a bit messy, but it's actually pretty clear.
Let's take a look at the Avisynth_in workflow which is meant to encode files in MPEG-2 XDCAM with the following specifications:

Video Codec: MPEG-2
Resolution: 1920x1080
Bitrate: 50 Mbit/s
Sampling: 4:2:2 yv16 planar
Framerate: 25i
Type: Interlaced TFF
Bit depth: 8bit

Audio Codec: PCM
Bit Depth: 24bit
Audio Channels: 8ch mono

The workflow starts with a watchfolder, then it starts indexing audio and video using FFMpegSource2, then it checks for the number of audio channels and basically always makes 8 channels (i.e if the original file has 1 ch, it duplicates it to all the 8 channels, if it has 2 audio channels, it duplicates left and right to the 8 channels and so on). Then, it applies loudness correction by making sure that it's 48'000Hz, 24bit and that the loudness is -24LUFS, then it checks for interlacing and it decides whether it has to deinterlace or not and eventually use bob-deinterlacing using yadif and so on. After that, it cheks the resulting framerate and it decides whether it has to apply a speed-up + pitch adjustment (23.976fps -> 25), or if it has to leave it as it is 'cause it's 25fps progressive or if it has to blend everything to 50fps progressive and then divide in fields or just divide everything in fields 'cause it's already 50fps progressive. Last but not least, it converts everything to yv16 and it applies clipping in order to make sure that Luma and Chroma are in limited tv range (16-235) and then Avisynth gives the resulting uncompressed A/V stream to ffmpeg which encodes it in XDCAM and then remuxes it using BBC BMX Transwrap. Finally, it's gonna deliver the file to a shared storage and it's gonna delete the original file which has already been encoded (if it has been encoded successfully).


By default there are quite a few blocks, but it's very easy to create new blocks and entirely customize them. For instance, this is what my block called "Limiter Tv Range" looks like:



Let's zoom in:



As you can see, the user interface is just a visual representation of what Avisynth is doing, therefore it's possible to do pretty much everything you generally do with Avisynth but in automatic and since automatize workflows is something companies like... a lot, I thought that this was the right way forward.
Here's another example; this time, the block is the one in charge of the Loudness Correction:



See? Each and every block is just like a normal Avisynth Script that you generally write down on a daily basis whenever you have to encode something.
For instance, Limiter Tv Range is basically just clipping:

Code:
Limiter(min_luma=16, max_luma=235, min_chroma=16, max_chroma=240)
Loudness Correction is basically just:

Code:
ResampleAudio(48000)
Normalize(0.22)
ConvertAudioTo24bit()
Blending and interlacing to 25i is basically just:

Code:
ConvertFPS(50)
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()
The channels duplicator for stereo files is basically just:

Code:
video=FFVideoSource("file.mov")
audiook=FFAudioSource("file.mov")
audio=MergeChannels(audiook, audiook, audiook, audiook)
AudioDub(video, audio)
and so on...
Oh, just to be precise, I said that it's based on Avisynth, but it's actually based on Avisynth+ (both x86 and x64), so it can use all the additional features of Avisynth+ like regular high bit depth, dealing with different color curves, HDR and so on; for instance, this is what converting a C-Log3 to BT709 linear looks like:



Let's zoom in once again:



As you can see, it basically takes the input as it is (either 8bit, 10bit, 12bit etc), it brings everything to 16bit planar, then it converts it to RGB, applies one of my LUTs and then it dithers everything down to 8bit with the Floyd-Steinberg error diffusion.

You can basically create many different workflows according to your needs, 'cause the potential of Avisynth is huge; those are some of the watchfolders I made:



As you can basically see, it pretty much encodes files for either archival or as mezzanine files to be used later or as files to be delivered to users via whatever service and so on.

Alright, let's crack on and see how encoding looks like, shall we?



The monitor looks essential and perfectly productive; you can basically see which workflow is doing what and how long it's gonna take before the encode finishes based on how long a file is and the fps of the encode. It can deliver the resulting file to a folder or to an FTP, according to your needs. In my case, I just set it to deliver the file to a shared folder in a shared storage.
As to the CPU efficiency, it's as efficient as Avisynth and ffmpeg/x262/x264 etc would be on your CPU.
In my case, since I still do many encodes in MPEG-2 and the encoder is definitely not optimized for multithreading, the power of Avisynth comes in parallel encodes rather than single encodes, which can saturate all the cores/threads of my Dual Xeon:





If you work in a company and need to encode many different files on a daily basis, I strongly recommend you to choose FFAStrans as it's free to use, based on open source programmes like Avisynth+, ffmpeg and so on and it's completely customizable.

This is supposed to be an introduction to this awesome software which I've been using ever since 2016-2017 and that I still use on a daily basis whenever I don't have time to manually write a script and manually encode a masterfile.
Of course, not everything can be done automatically and a manual encode should always be preferred to an automatic one, but let's face it: sometimes having something implemented for you that helps you finishing your work is actually very useful. If you want a complete guide on how to use it with many more screenshots, let me know. In the meantime, here's my personal workflows package which comes pre-loaded with scripts, plugins, and makes use of my LUT collection:

FranceBB Custom Workflows Package: Link

FFAStrans Stable version: Link

Please note that the legacy 32bit is legacy 'cause it's a very old version with some components modified with XP compatible ones, but you shouldn't really be using it.


Useful Links:

Official FFAStrans website by Steinar Apalnes: https://ffastrans.com/wp/
FFASTrans Wiki: http://www.ffastrans.com/wiki/doku.php

Developer: Steinar Apalnes
Maintainers: emcodem, momocampo, FranceBB

Last edited by FranceBB; 20th April 2021 at 22:00.
FranceBB is offline   Reply With Quote
Old 11th July 2019, 20:55   #2  |  Link
Gser
Registered User
 
Join Date: Apr 2008
Posts: 418
Huh so people really use x262?
Gser is offline   Reply With Quote
Old 11th July 2019, 21:55   #3  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
The images with the "graphs" are not really readable, could you post a higher res or a version with a zoomed in area?
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline   Reply With Quote
Old 12th July 2019, 01:19   #4  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Quote:
Originally Posted by Gser View Post
Huh so people really use x262?
MPEG-2 in 2019? Yep. Either x262 or Lavc58.19.102 mpeg2video encoder. Anyway, according to PSNR and SSIM values I've got in many tests their performances are not very good. Still, they are free and open source, so this is what I use. Besides, old playout ports force me to encode in MPEG-2 still today so... this is pretty much it.

Quote:
Originally Posted by ChaosKing View Post
The images with the "graphs" are not really readable, could you post a higher res or a version with a zoomed in area?
Sure, I'm gonna zoom them tomorrow and upload a few workflows zoomed in so that they are readable.
FranceBB is offline   Reply With Quote
Old 13th July 2019, 17:58   #5  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Alright, as requested, here are the zoomed in images.

Let's start with Avisynth_in:



Let's zoom in:



As you can see, once it indexes the audio and video file, HuffYUV is called and it creates a mezzanine lossless video file.
Then, such a video is indexed again and channels are duplicated/ordered accordingly.
Then, Loudness correction to 48'000Hz, 24bit and -24LUFS with Normalize(0.22) is applied.
The next step is yadif which decides whether it's gonna deinterlace it, bob-deinterlace it or leave it as it is if it's progressive.
Of course, this doesn't work with telecined material, which has to be handled differently but once again, as I said before, a manual encode should be preferred over an automatic one and in case of telecined material, you could just make the effort to write

Code:
tfm(mode=1,pp=5,slow=2,micmatching=2,clip2=tdeint(mode=2,type=3)) 
tdecimate()
Anyway, let's keep going with the second zoomed part of the Avisynth_in workflow.



As you can see, blending is applied according to the framerate. The idea is to blend everything to 50fps progressive and then divide into fields upon encoding. If it's over 70fps, then it just drops frames 'till it gets to 50fps. This workflow is mainly used for our news channel, therefore blending is more than enough. For the other channels, I created another conditional which applies speed-up with pitch adjustment if the framerate is 23.976fps, or leave it as it is if it's 25fps progressive and makes a progressive flagged as interlaced.
Then clipping is applied in the "Limiter Tv Range" block and the resulting uncompressed A/V stream is encoded in MPEG-2.
Lastly, the resulting file is remuxed by BBC BMX Transwrap, delivered to a folder and the original file is deleted.
"Pulizia Cache" means "Cache deletion", which basically removes the mezzanine file, the Avisynth Script created and the original file from the temp folder.


Let's take a look at the "Repair_in" workflow:



As you can see "Repair_in" is just a fancy name for "Linear Interpolation" performed by mvtools. It basically consists in Indexing, deinterlacing eventually interlaced materials, Resize using Spline64Resize, Linear Interpolation with mvtools, clipping 16-235 limited tv range, encoding and delivering the file, then deleting the temp files/cache.
Sure, linear interpolation ain't perfect and it might introduce artifacts, however it can be useful in some cases like whenever a normal static pan has been recorded by a camera and we want it to be as smooth as possible. MVTools in this case always tries to apply linear interpolation and create a real 50fps progressive file which is then divided into fields to make a real 25i file.

Speaking of MVTools, let's take a look at the workflow "Slowmo_any":



Let's zoom in:



I gotta say that whenever we need to make a slowmotion, we record it with a very high framerate, like at 100, 200, 300, 500, and even 700fps, however we might not have cameras capable of recording such an high framerate at an high enough resolution; for instance, 700fps can be recorded by our cameras in HD only (1280x720). As a matter of fact, we generally upscale them to FULL HD and 4K. So, whenever we need to slow down those files, we simply use "AssumeFPS(50)".



After AssumeFPS(50), we apply clipping in order to have luma and chroma in Limited Tv Range and then we divide in fields to get a 25i and encode it. If and only if we need to make a slowmotion of something that we didn't record ourselves at high framerate, then we use MVTools and we cross our fingers.

Let's now take a look at this x264 workflow:



Let's zoom in:



As you can see, this workflow is made to produce low-res, low-bitrate proxy files to be viewed by journalists, reporters and so on out of our MPEG-2 50Mbit/s yv16 mezzanine files. In the x264_proxy workflow, the duration of the file is analysed and bitrate is calculated accordingly in order to be as large as 2GB maximum. If it's only few seconds, it would get a too high bitrate, so it's capped at 2500 kbit/s. x264 What'sapp has a similar constrain.



x264 and x264 "Archivio Esterne" means "Archive for External Productions" and in this case, x264 can go up to 25 Mbit/s level 4.1, which is a good enough bitrate for a video to be used for productions that are sent on site.

All these workflows make use of custom avisynth scripts and you can create them as well by right-clicking on the UI and either write your own script from scratch, use some of my personalized blocks (Scripts) or use some of the built-in blocks/scripts that Steinar wrote:










I hope it makes things a bit more clear.
I'll dive into the creation of workflows in a more accurate way if you want.

Cheers,
Frank.

Last edited by FranceBB; 13th July 2019 at 18:06.
FranceBB is offline   Reply With Quote
Old 16th July 2019, 12:19   #6  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,531
Now that's awesome !!!
Deep respect and many thanks !
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Old 23rd July 2019, 20:21   #7  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,986
That's super cool!

Any support for CC / subtitles, or any other embedded VANC?
Blue_MiSfit is offline   Reply With Quote
Old 25th July 2019, 05:00   #8  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Quote:
Originally Posted by Blue_MiSfit View Post
That's super cool!

Any support for CC / subtitles, or any other embedded VANC?
Steinar didn't make any blocks to include subtitles and I didn't create a custom block 'cause I personally use FAB/Cavena to mux .stl and .890 subtitles in .mxf files as other open source alternatives don't really work well with anything other than .srt/.ssa/.ass. Anyway, since FFAStrans is based on Avisynth and FFMpeg, as long as there is a way to handle things with them, you can create a custom block. For instance, I know that with

Code:
ffmpeg -f lavfi -i movie=input.ts[out+subcc] -map 0:1  output.srt
you can extract subtitles from a transport stream, however I have no idea whether it's gonna work with an mxf or not.


By the way, I'm gonna re-upload my pre-loaded version to Mega as the link expired.
FranceBB is offline   Reply With Quote
Old 25th July 2019, 09:31   #9  |  Link
mariush
Registered User
 
Join Date: Dec 2008
Posts: 589
Isn't HuffYUV kind of slow? I started using MagicYUV years ago and it was much faster and better multithreaded than HuffYUV and there's a free version available: https://www.magicyuv.com/
mariush is offline   Reply With Quote
Old 25th July 2019, 10:37   #10  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Quote:
Originally Posted by mariush View Post
Isn't HuffYUV kind of slow? I started using MagicYUV years ago and it was much faster and better multithreaded than HuffYUV and there's a free version available: https://www.magicyuv.com/
Well, I've been using HuffYUV ever since the very beginning (years ago), however that's something I made myself as a custom block, 'cause in the Steinar implementation (which is the default one) you can use x264 lossless to encode your temporary file which is heavily parallelised.
FranceBB is offline   Reply With Quote
Old 30th July 2019, 23:53   #11  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Home version of Telestream Vantage (well- not as many features, but priced nicely instead)

Last edited by kolak; 30th July 2019 at 23:59.
kolak is offline   Reply With Quote
Old 23rd February 2020, 10:51   #12  |  Link
reisub
Registered User
 
Join Date: Feb 2020
Posts: 5
Quote:
Originally Posted by FranceBB View Post
Steinar didn't make any blocks to include subtitles and I didn't create a custom block 'cause I personally use FAB/Cavena to mux .stl and .890 subtitles in .mxf files as other open source alternatives don't really work well with anything other than .srt/.ssa/.ass. Anyway, since FFAStrans is based on Avisynth and FFMpeg, as long as there is a way to handle things with them, you can create a custom block. For instance, I know that with

Code:
ffmpeg -f lavfi -i movie=input.ts[out+subcc] -map 0:1  output.srt
you can extract subtitles from a transport stream, however I have no idea whether it's gonna work with an mxf or not.


By the way, I'm gonna re-upload my pre-loaded version to Mega as the link expired.
FranceBB, the link is dead. Please upload the "FFAStrans FranceBB pre-loaded version" again.

Last edited by reisub; 23rd February 2020 at 10:54.
reisub is offline   Reply With Quote
Old 25th February 2020, 19:59   #13  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Quote:
Originally Posted by reisub View Post
FranceBB, the link is dead. Please upload the "FFAStrans FranceBB pre-loaded version" again.
I wanted to but I'm stuck at home due to the COVID-19. (don't worry, I didn't get it, I'm a healthy young guy xD).
Anyway, everybody but a limited number of people are working from home; they call it "smart working".
They're limiting the access to everybody and I can't get the whole package from the server 'cause my VPN doesn't let me access it directly for security reasons. When this whole mess will end I will, I promise.
As you can figure out yourself, the server I'm talking about with FFASTrans is in Italy.
FranceBB is offline   Reply With Quote
Old 26th February 2020, 21:54   #14  |  Link
reisub
Registered User
 
Join Date: Feb 2020
Posts: 5
Quote:
Originally Posted by FranceBB View Post
I wanted to but I'm stuck at home due to the COVID-19. (don't worry, I didn't get it, I'm a healthy young guy xD).
Anyway, everybody but a limited number of people are working from home; they call it "smart working".
They're limiting the access to everybody and I can't get the whole package from the server 'cause my VPN doesn't let me access it directly for security reasons. When this whole mess will end I will, I promise.
As you can figure out yourself, the server I'm talking about with FFASTrans is in Italy.
Ok. Please upload the program when it will be possible for you.
reisub is offline   Reply With Quote
Old 26th April 2020, 16:36   #15  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Sorry for the delay, but I finally managed to connect remotely to all my servers and after fixing a few issues and making sure everything was running as it should, I made a new backup of the current FranceBB pre-loaded build.
Please note that it's a custom-made dual build of the original Steinar Apalnes implementation, so it's both x86 and x64, but it's supposed to be running on a x64 version of Windows 10.
The fact that it's dual is NOT to make it run on x86 systems, but to make Avisynth x86 plugins run with ffmpeg x86 for backward compatibility reasons whenever it's not possible to have a full x64 workflow.

Processed 1 file, writing output:

Output: "Z:\FFASTrans_v1.0.0.4_FranceBB_preloaded_build.exe"
Install: 2 pages (128 bytes), 1 section (2072 bytes), 12848 instructions (359744 bytes), 84352 strings (807370 bytes), 1 language table (262 bytes).
Datablock optimizer saved 47942 KiB (~16.0%).

Using lzma compression.

EXE header size: 50688 / 37888 bytes
Install code: 129931 / 1170040 bytes
Install data: 256678773 / 1403460488 bytes
CRC (0xA8FF5255): 4 / 4 bytes

Total size: 256859396 / 1404668420 bytes (18.2%)
(source size was 403200292 bytes)

Download: FFASTrans_v1.0.0.4_FranceBB_preloaded_build.exe

Note:
Please note that you MUST change the path to make sure that workflows are running properly on your system; remember that this is an ad hoc version made for the company I work for.
Workflows are supposed to be used as an example to make your own workflows.
Several different workflows have been included.
Several matrices of linear transformation have been included.

**from time to time you might find notes written in non-English language as an added bonus xD**
FranceBB is offline   Reply With Quote
Old 28th April 2020, 14:22   #16  |  Link
reisub
Registered User
 
Join Date: Feb 2020
Posts: 5
Quote:
Originally Posted by FranceBB View Post
Sorry for the delay, but I finally managed to connect remotely to all my servers and after fixing a few issues and making sure everything was running as it should, I made a new backup of the current FranceBB pre-loaded build.
Please note that it's a custom-made dual build of the original Steinar Apalnes implementation, so it's both x86 and x64, but it's supposed to be running on a x64 version of Windows 10.
The fact that it's dual is NOT to make it run on x86 systems, but to make Avisynth x86 plugins run with ffmpeg x86 for backward compatibility reasons whenever it's not possible to have a full x64 workflow.

Processed 1 file, writing output:

Output: "Z:\FFASTrans_v1.0.0.4_FranceBB_preloaded_build.exe"
Install: 2 pages (128 bytes), 1 section (2072 bytes), 12848 instructions (359744 bytes), 84352 strings (807370 bytes), 1 language table (262 bytes).
Datablock optimizer saved 47942 KiB (~16.0%).

Using lzma compression.

EXE header size: 50688 / 37888 bytes
Install code: 129931 / 1170040 bytes
Install data: 256678773 / 1403460488 bytes
CRC (0xA8FF5255): 4 / 4 bytes

Total size: 256859396 / 1404668420 bytes (18.2%)
(source size was 403200292 bytes)

Download: FFASTrans_v1.0.0.4_FranceBB_preloaded_build.exe

Note:
Please note that you MUST change the path to make sure that workflows are running properly on your system; remember that this is an ad hoc version made for the company I work for.
Workflows are supposed to be used as an example to make your own workflows.
Several different workflows have been included.
Several matrices of linear transformation have been included.

**from time to time you might find notes written in non-English language as an added bonus xD**
Thanks for the build. What is the password to change workflow?
reisub is offline   Reply With Quote
Old 28th April 2020, 18:45   #17  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Quote:
Originally Posted by reisub View Post
Thanks for the build. What is the password to change workflow?
Modified without password.

Last edited by FranceBB; 30th April 2020 at 23:26.
FranceBB is offline   Reply With Quote
Old 30th May 2020, 21:19   #18  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Updated build to v1.0.0.5

Changes:

- Improved conditional logic in some workflows.
- Ability to re-create the exact folder structure in x262_in and Avisynth_in
- Introduced x265 lossless mode
- Introduced an anti-aliasing block
- Introduced a post-processing block for old BetaCAM sources with NNEDI3 + Spline64 upscale, automatic white balancing, automatic luma/chroma adjustments and basic sharpening.
- General code cleanup in customized workflows
- Temporarily removed FrostyBorders

New Build Download

This is a new updated build which contains some of the workflows I made that are meant to be used in the broadcast company I work for.
Unfortunately, some of them have not been included, but the reason is that those I removed were using FTP or had other critical credentials to access our internal server inside our .bat or other blocks and therefore I couldn't safely share them.
Anyway, those inside the build are pretty much the most used one, so they should cover a broad range of cases.
They're mainly focused on MPEG-2 XDCAM50 encoding, however, they can be easily modified and used to encode and deliver other formats.
The only "catch" is that they're really meant for PAL regions, so I don't know how useful can be for those living in NTSC countries.
Please note that we have three platforms:
TG24 which stands for "Television Journal 24h" in other words a round-the-clock news channel. It's basically the non-English version of Sky News.
SPORT which is basically dedicated to sports channels.
PCE which stands for "Production Cinema Entertainment" which is dedicated to our cinema channels, our original productions, our general purpose channels and so on, but only for "promo" i.e "commercial ads", in fact the real programs that fall in this category are encoded with manually written scripts and 16bit planar precision post-processing to deliver the best possible quality all the time and those tasks couldn't be automatized (why would I even care to go to work if they could? xD).
Anyway, those three platforms have three different RAID-6 storages in which there are different input folders, namely "Avisynth Server 1" -> TG24 / SPORT / PCE -> "Node 1 / 2 / 3 / 4" -> OUTPUT.
There's a conditional logic that outputs the file to the correct folder according to which platform is being used (i.e in which server the input folder is) and which node has been used.
There are also some workflows that deliver to either Vantage or to MediaDirector so that they can check-in the file inside AVID Interplay Access automatically.
Alright, let's get started and dig into our workflows.

01 - AppleProRes_in

Image: Link

This is a very simple workflow.
It's not really used much as it's just a quick and dirt AppleProRes encode using iCodecPro from ffmpeg.
The only reason it's there is that sometimes, when we have to deliver files (that have already been post-processed and encoded and that are ready to air) to other companies they ask us to send them in AppleProRes, so I needed something to automatize it.

02 - Avisynth_in & ANY

Image: Link

This is more like it.
This workflow is one of the most used.
It basically uses Avisynth x64 to index the source file, it checks the audio channels and make sure to always have 8 mono channels (if you have like a stereo file with just FL FR, it then maps CH1 FL CH2 FR, CH3 FL CH4 FR CH5 FL CH6 FR CH7 FL CH8 FR) and it performs a Loudness Correction to -24 LUFS, then it bob-deinterlaces everything if needed, and it decides what to do:
if it's 23.976fps or 24fps it performs a 4% speed-up + pitch adjustment to 25p and encodes it as progressive flagged as interlaced
if it's 25p it leaves it alone and encodes it as progressive flagged as interlaced
if it's 25i it bob-deinterlaces it to 50p and then re-interlaces it correctly to avoid any interlacing artifacts
if it's below 23p then it blends it to 50p, divides it in fields and encodes it to 25i
if it's above 24p but below 50 it blends it to 50p, divides it in fields and encodes it to 25i
if it's 29.970p then it blends it to 50p, divides it in fields and encodes it to 25i
if it's 29.970i then it bob-deinterlaces it to 60p, blends it to 50p, divides it in fields and encodes it to 25i
if it's 50p then it divides it in fields and encodes it to 25i
if it's 60p then it blends it to 50p, divides it in fields and encodes it to 25i
if it's above 60p but below 70p, it blends to 50p, divides it in fields and encodes it to 25i
if it's above 70p then it just drops frames to 50p, divides it in fields and encodes it to 25i

Then it brings luma and chroma in Limited TV Range 8bit planar 0.7mV 16-235 by clipping, lastly it encodes and delivers the file.

Of course, if you have a structure like:

Folder: Shooting\Day1\CameraA\000454.mov

it creates the very same path in the output:

Folder: Shooting\Day1\CameraA\000454.mxf

so if you have several folders and sub-folders with many different files inside, you can copy all of them in the input watchfolder and it will create them back in the output.

Lastly it will perform "Cache Cleanup" just like any other workflow which means that the input file is gonna be deleted along with all the temporary files created in-between.

03 - Avisynth_in Antialiasing SLOWMO

Image: Link

It performs a slow-motion of the input files using AssumeFPS, then it performs anti-aliasing and brings luma and chroma to limited TV range and encodes.
Audio is preserved and slowed down as well when possible (i.e unless the clip is slowed down too much).


04 - Avisynth_in P2

Image: Link

This takes a Panasonic P2 folder structure which has 1 video and 4 different mono PCM audio channels NOT Embedded (i.e not muxed together).
It uses the built-in function of FFASTrans which reads the .xml metadata and uses them to detect video and audio files and put them together using Avisynth.
After that the very same logic is applied, namely Loudness Correction to -24 LUFS and Luma and Chroma in Limited TV Range.
The user can choose whether the files within the folder have to be appended together to make a single final Masterclip or whether they should be encoded one at a time.

05 - Avisynth_in SD

Image: Link

This is rarely used nowadays, but it takes the file in input and encodes in IMX50.
The same logic is applied: 8 audio channels, Limiter TV Range and Loudness Correction.

06 - Avisynth Remaster

Image: Link

This is still at a very beta stage and it hasn't been used yet other than for some tests.
Basically, what it does, is that it receives an old SD file and it upscales it to FULL HD in the least painful way (NNEDI3 + Spline64 with 16bit planar precision) and it also performs automatic white-balancing and it attempts to fix color-dominance if detected (i.e blue tint etc). Lastly, it performs a quick and dirt sharpening, clips everything to 16-235 and performs loudness correction.
It's still a beta 'cause I still have to decide which Sharpening filter I'm gonna use (LSFMod, SeeSaw etc) and whether I'm gonna use GamMac in a sort of automatic fashion or not.
Besides, I still haven't made up my mind about which option I'm gonna use to go from BT601 to BT709: the old ColorMatrix which works in 8bit planar only? Matrix from HDRTools which works in 16bit interleaved? I still have to make a few tests.
Last but not least, I don't know whether I'm gonna go on with it or if I'm gonna include the whole Fred Van de Putte series of filters instead.
Again, I gotta test.

07 - DNxHD

Image: Link

Quick and dirt workflow that does exactly what the AppleProRes one does, but for DNxHD.

08 - HuffYUV_in

Image: Link

This is for lossless mezzanine file only that have to be encoded as it encodes the input in HuffYUV lossless.

09 - Linear Interpolation

Image: Link

The workflow speaks for itself as it uses MVTools and its forward and backward vectors to perform linear interpolation to 50p in case it's needed.
This workflow does not maintain AR (contrary to the other workflows) and it's only used when it's really necessary, like if we have a mixed content show and we want everything to be at true 25i and smooth, however it's only used if the director specifically tells us to do so as he becomes responsible for eventual artifacts created every now and then by linear interpolation and of course if there are too many or if he thinks that it's not the result he wanted, he can definitely tell us to do it again without linear interpolation and have an hybrid show instead (progressive + interlaced).

10 - Linear Transformations 1, 2, 3

Image: Link 1 - Link 2 - Link 3

These 3 workflows apply one of my matrices of linear transformation to map each and every point of a curve into another with 16bit precision.
All calculations are done in PlanarRGB 16bit and the result is dithered down to 8bit planar using the Floyd-Steinberg error diffusion.
The resulting output is then limited to 8bit TV Range 16-235, loudness corrected to -24 LUFS and encoded.
It performs the following linear transformation:

HLG BT2020 -> Linear BT709
PQ BT2100 -> Linear BT709
S-Log3 -> Linear BT709
C-Log3 -> Linear BT709
Log-C -> Linear BT709
D-Log -> Linear BT709

Of course it can be replicated and used with any of my LUTs to perform whatever linear transformation you want.
As for HLG and PQ, I've been thinking for a while about using a tone-mapping algorithm instead, namely Reinhard, using the plugin from Jean-Philippe Scotto di Rinaldi (although the one from Donald Graft is also a perfectly valid alternative, so I don't which one I'm gonna use or if I'm gonna keep a custom LUT instead).

11 - Slowmo ANY

Image: Link

It performs slow-motion using AssumeFPS() to anything unless it's 25p. In that case it leaves it as it is.
It's the same as Slowmo-Antialiasing, but it doesn't do any antialiasing.
As always, the output is brought to 8bit Limited TV Range 16-235.

12 - x262_in

Link: Image

It's a conversion to XDCAM 25i without using any fancy conditional logic or Avisynth.
It started with x262, but I then changed it to ffmbc and now it's using the MPEG-2 encoder from ffmpeg with the built-in logic of the XDCAM block made by Steinar Apalnes.

13 - x262 DVD in

Link: Image

Just like the one above, it started with x262 but now it's using the MPEG-2 encoder inside ffmpeg and it produces an ISO file that it's ready to be burned in a PAL DVD.
The reason why there's no manually-written conditional logic at all other than the one written by Steinar for this particular block is that if we make a DVD it's from one of our final file that is already ready to go on air, therefore nothing fancy has to be done other than interlace-aware downscaling and BT709 to BT601 conversion.
As a matter of fact, this replaced the old method of using physical DVD recorders to literally make a DVD from the hardware-downscaled FULL HD SDI input.

14 - x264

Link: Image

First of all, we don't internally encode to H.264 straight away and it's generally used as something that we send to either users or external productions that need it as a backup.
There are quite a few watchfolders, some of them encode in FULL HD with a very low bitrate, some other downscale to a ridiculous resolution and encode it with a file-size that is just right for WhatsApp, some other make a crappy proxy that is used... you know... as a proxy to just know what that file is about and one is used to encode the input files to 50p FULL HD BT709 4:2:2 8bit planar 25 Mbit/s and that's the file that is sent as a very last resort backup to external on-field productions if everything else fails.
For instance, if they're in an OBI-Van and the masterfile of the intros can't be used and their second HDD backup fails and they can't connect to our storage and their disk backup fails as well then they're gonna use this 25Mbit/s H.264 file for the opening before the program begins. Opening means just a small intro or a logo, a bumper etc.


15 - x265

Link: Image

It can be used to either encode a file that is already ready to go but in H.265 with a very low crf to be delivered to users via internet (nothing linear) or as an alternative to HuffYUV for lossless mezzanine file if the x265_lossless fodler is selected instead.

16 - Avisynth Metadata

Link: Image

The name is misleading since it doesn't have to do anything with metadata, but it's the name of the department who uses it "Metadata Management".
Basically those are cheap low-res proxy that are generally sent away to celebrities that have been interviewed and wanna have a copy before it goes on air.
It's 25p, low quality, H.264, with a logo and a timecode to avoid leaks since we don't really have any control of the file once it has been sent out to the celebrity.
As a matter of fact, it's done by another department and not the one I work in 'cause they are aware of legal rights and other stuff about what can and cannot be done (I don't know any of that so I don't really mess up with that).


Alright, that's pretty much it.

Have a good week-end,
bye!
FranceBB is offline   Reply With Quote
Old 17th July 2020, 10:26   #19  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Every time I get my hands on a workflow to fulfill a new requirement, I get the feeling that things are getting a bit out of hands (i.e they're getting enormous xD)

FranceBB is offline   Reply With Quote
Old 15th October 2020, 13:24   #20  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Updated workflows and moved them to Github: https://github.com/FranceBB/FranceBB...ans%20Workflow
See first post.
FranceBB is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:26.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.