FFAStrans - FFMpeg Avisynth Transcoder

[kolak]

Originally Posted by FranceBB

Every time I get my hands on a workflow to fulfill a new requirement, I get the feeling that things are getting a bit out of hands (i.e they're getting enormous xD)

Maybe introduce "Forward/Receive" action which is what Vantage does.
Then 1 big workflow can be split into few smaller ones and output of each gets passed to next one. Easier to organise and manage.

[FranceBB]

I was writing a Wikipedia Page about FFASTrans but I'm struggling to get it approved 'cause they say that it lacks official references...

Here's the page: https://en.wikipedia.org/wiki/Draft:...nth_Transcoder

Any suggestions? Have you ever managed to get something approved on Wikipedia?

[patul]

He was complaining about COI, conflict of interest, as if you "advertise" the software. I don't know how to resolve it though, probably a further read on this page will help, https://en.wikipedia.org/wiki/Wikipe...ct_of_interest. A suggestion on writing the content, probably you need to cite the references within the content, rather than just put up reference list.

[FranceBB]

Originally Posted by patul

He was complaining about COI, conflict of interest, as if you "advertise" the software. I don't know how to resolve it though, probably a further read on this page will help, https://en.wikipedia.org/wiki/Wikipe...ct_of_interest. A suggestion on writing the content, probably you need to cite the references within the content, rather than just put up reference list.

The COI wasn't originally there, it came after I wrote him a message saying him that I wanted to talk to him to discuss what I could improve and modify to get it approved, but rather than replying to me via message, he flagged it as COI... Not the reaction I expected, given that I wanted to ask the reviewer what I could do better to improve the page and get it accepted...
Anyway, about the references, I tried to add as many as I could, but it still didn't work...

Originally Posted by kolak

Maybe introduce "Forward/Receive" action which is what Vantage does.
Then 1 big workflow can be split into few smaller ones and output of each gets passed to next one. Easier to organise and manage.

Well, I decided to split my workflow and it worked.
The way I did it was by calling other workflows using the REST-API, so what you suggested is exactly what I did.
The way it's implemented right now, though, requires a bit of knowledge and scripting of the REST-API, but it can be done.

[patul]

I meant for inline citation, not just reference. Please see in the picture below. I'm not really good in English, sorry.

[FranceBB]

Ah!! I see!
Yeah, it makes sense. I'll try adding that.

[FranceBB]

Version 1.2.0 Stable Released!!

Changelog:

- New filter: Safe Color Limiter makes sure you video stays within legal TV ranges.

- New filter: Linear Transformation for linear conversion between different color formats like HDR <-> SDR.

- Both conditional and populate nodes now supports dynamic number of fields and resizes the GUI accordingly.

- Added new operator mode to support regular expressions in the conditional node.

- Added the ability to send attachments and a "Reply to" field to the "Send E-mail" node.

- Added HD profiles to the "DV/DVCPRO" encoder.

- Added lasso functionality for selecting several nodes in the editor.

- Added new function $regreplace() for replacing strings using PCRE regex patterns.

- Added new function $jsonput() for adding JSON data to an existing or create new JSON formatted string.

- Added new variable %s_wf_folder% for getting folder of the running workflow.

- Added new variable %i_job_priority% for getting priority of the running job instance.

- Added new variable %s_split_id% for getting split id of the running job instance.

- Added file count to the "Hold" node wait for file option.

- Added option in "Custom AviSynth" filter to attempt adapting colors pace metadata to video properties.

- Copied nodes with connection within the group is now kept.

- Selected nodes can now be moved simultaneously while keeping their relative position.

- The "YouTube"-decoder will no longer try to download complete lists.

- Improved start TC detection on some formats.

- The "H.264" encoder can now produce MXF files and uncompressed PCM audio in the MOV, MKV and MXF wrappers.

- The "A/V Media" decoder now dither if source video bit depth is higher than specified in the node.

- Increased the number of max active jobs the inbuilt monitor would display.

- The rest service now returns 503 if database is not available. Mostly a problem when running FFAStrans from share.

- Aborting jobs will now try more graceful shutdown of child processes before force killing.

- General improvements to file monitoring and job initiation reliability.

- Addressed several stability issues when running FFAStrans from a share.

- Fixed issue where running FFAStrans would not properly close on shutdown.

- Submitting files to other workflows than the current in editor did not work.

- Video with too short audio could result in corrupt endings.

- Results from PSNR and SSIM in "Video quality" analyzer was switched.

- Fixed issues with catching all STDOUT data from "Command Executor".

- Fixed GUI issues when controlling manager through TeamViewer.

- Fixed several reliability issues with the camera monitors.

- Fixed crash when attempting to use predefined levels in "Colorspace" filter.

- The workflow counter in FFAStrans Workflow Manager lower status bar did not work.

- Fixed issue with the FFAStrans Workflow Manager causing graphic issues and instability when working with large workflows.

- Custom nodes would crash if trying to open a help section that did not exist.

- Variables within variables was not properly expanded.

- The web interface would sometimes show hanging node even though it had finished.

- When submitting file through the API, the job id was initially written to a wrong key.

- Fixed crash that sometimes occurred when enumerating child processes.

- Variables or statics with data containing the pipe "|" character was not displayed in the variables list.

- Numerous other minor bug fixes and improvements.

Download: Link

The website has now been re-written from scratch: http://ffastrans.com/wp/

Cheers,
Frank

[FranceBB]

Version 1.3.0 Stable Released!!

Finally a new version that doesn't just focus on fixes and improvements but that also has a lot of new nice features.

Changelog:

New workflow features:
- Sub splits: This feature enables sub connected splits (brown) to always finish before the main splits (black).
- Workflow connection: Enables you to forward a job to another workflow.

New functions:
- $lookup() and $lookuprep() for table lookup and lookup replacement.
- $sort() for sorting an array or a string.
- $split() for splitting a string into an array.
- $count() for counting occurrences of a string or single character.
- $foreach() for dynamically executing operations on an array
- $string() for formatting strings like printf in C.
- $xxhash64() for 64 bit version of xxhash.

New node:
- “For each”: Will loop through an array of values and dynamically create a new split/branch for each index.


New variables:
- %s_start_tc_alt% for alternative start TC taken from creation date of media.
- %s_gamma% for gamma curve of media
- %s_chroma_loc% for chroma location of media.
- %i_gmt_bias% for the local GMT bias.
- %s_version% for current running FFAStrans version.
- %s_last_error% for retrieving last viable error message.

New API methods:
- GET /workflows/status for getting current status
- POST /workflows for importing FFAStrans exported workflows.
- POST /presets for importing FFAStrans exported presets.
- POST /user_variables for importing FFAStrans exported user variables.
- GET /history now supports ?start and ?count for getting custom range.

Others:
- All encoders now support applying custom or inbuilt LUTs.
- All encoders now support setting and converting full/limited video range.
- Huge speed improvement when muxing to BMX MXF as FFAStrans no longer creates an intermediate file before remuxing.
- Added search function in the variables selector.
- Added free text search function for workflows and nodes.
- The XAVC node now support class 300 and 480 in UHD.
- The AVC-Intra node now support class 2K and 4K.
- Added option to use user variables as framerate in “ProRes” and “DNxHR” encoder nodes.
- Added user selectable quality tunings to the the “XDCAM-HD” node.
- Added “Writing library” mode to the “ProRes” encoder node for improved compatibility in some scenarios.
- Added HLG and PQ signaling to the “H.264” encoder.
- Added option to save complete VMAF json to a variable in the “Video Quality” node.
- Added access to the user variables from main manager menu.
- Added user selectable zero padding to unique output file names in the delivery nodes.
- The header section in “HTTP communicate” node now supports using variables.
- The “%s_original_path~n%” variable now supports minus notation for retrieving path names from the right side.
- The “%s_recursed_path%” variable now support the same notation type as %s_original_path~n% with %s_recursed_path~n%.
- Added option to export job logs from inbuilt status monitor.
- The “Hold” node now supports arrays as list of files.
- Updated to use the new maintained yt-dlp instead of youtube-dl.
- Changing a variables/static name now automatically updates all applicable workflows.
- Expanded the $exists() function to support path recursing and returning a file list in array.
- Added support for adding data to an existing array using the $jsonput() function.
- Fixed issue where the “H.264” encoder produced corrupted MXF files.
- Fixed various bit depth issues in the “H.264/5” encoders.
- Fixed “Generic MPEG” was forced to 16:9 aspect.
- Lots of minor changes and improvements.

Download: Link

[FranceBB]

Version 1.3.0.1 Stable Released!!

Changelog:

- Fixed potential endless loop in “A/V Decoder” node.
- Fixed a behaviour that would make the XDCAM-HD encoder would sometimes stop encoding on very complex content.
- Fixed an issue where sources with yuvj420p pixel format would sometimes lead to failed encoding.
- Fixed an issue that caused bad frame rate conversion on small fps difference between source and output.
- Fixed an issue where host settings was not properly saved.
- Fixed error message on multiple immediate retries from failed node.

Download: Link

[FranceBB]

Version 1.3.0.2 Stable Released!!

Changelog:

- Fixed progressive closed GOP encoding in XDCAM-HD.
- Fixed a bug in the “Video quality” node which would sometimes return a bogus “inf000000” value.
- Fixed a bug in which the “Video quality” node would fail if source or ref was IMX-D10 content.
- Fixed an issue which caused the “%s_cache_record%” variable to not be correctly populated.
- Fixed the “Sequence” monitor disregarding the “Recurse” setting.
- Fixed a bug in which the “Sequence” monitor would sometimes not work due to an invalid input option being set.
- Fixed a bug which caused two succeeding encodings nodes on the same branch to fail when using the inbuilt encoders.
- Fixed minor issues with properly recognizing some pixel and color formats when processing.
- Fixed an issue which caused the loudness table in “Audio extract” encoder gui to not set a value.
- A few other minor edge case issues addressed.

Download: Link

[FranceBB]

Version 1.3.1 Stable Released!!

Changelog:

- Added 4:4:4 profile (XQ) to the ProResKS encoder.
- Added "MXF (BMX)" wrapper option to the Panasonic P2 monitor.
- Added option for removing farming settings when exporting workflows
- Added PUT method for updating existing workflow through API
- Added user option to read from STDERR and/or STDOUT in the "Command executor" node.
- Improved general robustness when executing many jobs at the same time.
- Reduced general priority on sub processes in order to not halt system.
- Fixed some special case FTP monitor issue that caused files to not be downloaded properly.
- Fixed framerate bug when encoding to BMX (MXF)
- Fixed an issue where setting %s_source% to "" (nothing) resulted in some erratic behavior.
- Fixed some job retry issues.
- Fixed audio config crash in the DV encoder.
- Fixed rare memory leak issue.
- Fixed MOV bug in the DNxHD encoder
- Fixed a problem with the A/V Media decoder created zero-length clips.
- Fixed an issue that caused some image sequences to not be properly parsed
- Fixed a problem where plugin processors GUI got stuck behind manager window.
- Fixed a bug where the "Fade" filter ignored audio/video enable/disable
- Fixed an issue that caused the "Custom FFmpeg" encoder to not work properly when using custom ffmpeg mapping.
- Fixed an API issue where history was not always correctly returned.
- Fixed an issue causing some AviSynth based video filters to not work properly on older CPUs
- Fixed a flaw with priority connections in conjunction with executing on error.
- Several other minor improvements and fixes

[FranceBB]

FFAStrans 1.4.0.1 Released!!

Changelog:

- Hosts group: Group hosts into different named groups and assign nodes or/and workflows to them.

- Job slots per node: Assign a higher number of job slots to a node in order to better "allocate" resources on the executing host.

- Cron scheduling support for monitors

New functions:
$dateweek(): Calculate week numbers
$timecalc(): Time calculator
$waccess(): For testing write access
$owner(): For getting owner of a file
$asplit(): Split/format array into a string
$ffconcat(): For creating a "ffconcat version 1.0" string
$shortcut(): For parsing Windows shortcuts and macOS alias files

- New variables:
%i_v_precharge%: Get precharge value from MXF files
%s_options%: System variable for advanced altering of job behavior
%f_pix_aspect%: Pixel aspect ratio as float
%s_frame_rate%: Represent frame rate as num/denum string

- New API method:
GET /jobvars/ for getting final value of user variables after job end

- Improved processing speed, quality and color precision for all encoders.
- Huge database I/O optimizations for those with larger farms.
- Dolby E decoding in the "A/V Decoder".
- "Forget missing files" are now supported by all applicable monitor nodes.
- Support the new frame properties in AviSynth+.
- Support Mac alias in the "Folder" monitor and improved Windows shortcut support.
- Improved support for encoding files with cover art.
- Properly support audio buffer files > 4GB in the "A/V Decoder".
- Support reading stdout/err in command executor as unicode.
- Support for XDCAM-EX 35 720p in the "XDCAM-HD" encoder.
- Support multiple selection in the status monitor.
- Inbuilt status monitor now shows history pages instead of one huge list.
- Improved "Color conversion" filter.
- Improved navigation in the status monitor using arrow keys.
- Improved up-time reliability.
- Native support for "ffconcat version 1.0" input sources.
- Native support for reading IMF packages.
- Support writing 64 bit audio files in the "Extract audio" node.
- Support UTF-8 formatted mail bodies in "Send E-Mail" node.
- Playlist support in "YouTube" decoder and save list to variable.
- Clear complete work cache dir from FFAStrans Manager.
- Support SinPowResize kernel in "Resize" filter.
- Improved automatic cleaning of log and work files.
- FFAStrans will now archive job logs and keep them for an extended period.
- Fixed several long path issues in monitors and FTP delivery.
- Fixed slow opening of long workflows.
- Fixed comparing negative number not working in the "Conditional" node.
- Fixed graphical artifact when moving too far out on a large workflow.
- Fixed some number like strings being strictly regarded as number.
- Lots other minor fixes and improvements.

[FranceBB]

FFAStrans 1.4.0.5 Released!!

Changelog:

- Fixed SAR was not set when auto cropping IMX source in the A/V Decoder
- Fixed Interlace analyze node reported bogus results
- Fixed wrong progress on video analyze node
- Fixed rare issue where priority branches did not end properly
- Fixed issue that in some conditions caused a job queue to build up
- Fixed A/V Decoder not properly decoding some broken MPEG2
- Removed invalid 4:4:4 AVI option in the Uncompressed encoder node
- Removed too strict variable check when submitting jobs via API
- Fixed continuously growing files being picked up too early
- Fixed multiple hosts picking up the same growing file
- Fixed loudness analyser not returning correct channel layout
- Fixed error in priority splits
- Fixed stills decoder having malformated color conversions.
- Fixed rare crash on badly formated time code
- Fixed a file locking anomalies

[FranceBB]

Well, those were from 4 years ago and although they have been working fine up until a while ago, they're not gonna work with 1.4.x 'cause starting from 1.4 we have the Avisynth frame properties.
Before frame properties either you had an input, you submitted the file to an encoder node and let the filter_builder take care of creating a command line for you to encode the file properly by performing all the necessary conversions like deinterlacing, converting the primaries, transfer and matrix etc or you could just write your own AVS Script and then basically tell the filter_builder "hey, don't do anything, I've done everything, just encode the file".
In other words, there was no in between scenario.
Now, with the new frame properties, we actually have a way to interact with the filter_builder.a3x which means that we can have scenarios in which you're doing some things within your Avisynth Script but you don't wanna take care of some other things and you want the filter_builder to do it for you. This is where frame properties come in handy.
I'll update the workflow collection on Monday, first thing in the morning. You're gonna be able to see where frame properties are used and how.

[FranceBB]

FFAStrans 1.4.0.6 Released!!

Changelog:

- Fixed some illegal characters when parsing mac aliases.
- Fixed occasional double registration of new monitored files.
- Fixed loudness analyser not storing correct real peaks data.
- Fixed rare error that sometimes caused retried jobs to not start.
- Fixed audio files getting double length when converting to video formats.
- Updated BMX to include fix for wrongly flagged aspect ratio on some MPEG-2 SD formats.
- Fixed rare bug that sometimes caused FFAStrans to not continue with jobs containing priority splits.

[FranceBB]

FFAStrans 1.4.0.7 Released!!

Changelog:

- Fixed a bug occurring when monitoring growing files.
- Fixed a stills encoder bug occurring when encoding a single image.
- Fixed a potential race condition bug occurring when several nodes are parsing the same media source.
- Fixed a rare API bug occurring when submitting huge JSON bodies.
- Fixed a color scheme bug in the workflow manager affecting the representation of bypassed nodes.
- Fixed a bug that caused all running workflows to restart when just one workflow was being stopped.
- Fixed a bug that could cause FFAStrans to not clean up old jobs/logs properly.
- Fixed some aggressive enumeration of history records.
- Several other minor bug fixes and improvements.

[anton_foy]

Awesome! I like to start to use it but I have not found if (probably the answer is no) it supports encoding in >10-bit for dnxhr/prores or if you can recommend another codec for the use in post production? Thanks!

[FranceBB]

Unfortunately you're right, but there's nothing I can do about it. The problem with DNX and ProRes is that both implementations rely on the Lavc encoder which is built in ffmpeg and nobody ever developed it to make it go past 10bit planar, so we can't just add an encoding profile that doesn't exist.
If someone is gonna develop a new high bit depth version of the DNX HR and AppleProRes encoder that supports 12bit, we'll definitely add support for it, but until then I'm afraid we don't have much choice.

[anton_foy]

Originally Posted by FranceBB

Unfortunately you're right, but there's nothing I can do about it. The problem with DNX and ProRes is that both implementations rely on the Lavc encoder which is built in ffmpeg and nobody ever developed it to make it go past 10bit planar, so we can't just add an encoding profile that doesn't exist.
If someone is gonna develop a new high bit depth version of the DNX HR and AppleProRes encoder that supports 12bit, we'll definitely add support for it, but until then I'm afraid we don't have much choice.

I understand thanks. But in virtualdub2 I can go 16-bit with some lossless codecs. A tad too big files when in 4k. If not dnxhr or prores then any other codec that can do >10-bit in ffastrans?

[FranceBB]

Originally Posted by anton_foy

I understand thanks. But in virtualdub2 I can go 16-bit with some lossless codecs. A tad too big files when in 4k. If not dnxhr or prores then any other codec that can do >10-bit in ffastrans?

Yeah when using HuffYUV and UTVideo files can get really big really fast. Currently there isn't a native encoding node for those, but I can add it to the lossless encoder which is currently v210 only. Of course one could just use the custom ffmpeg script and encode in any codec supported by ffmpeg like ffvhuff in this case, however that would skip the filter_builder entirely. Basically when an encoding node is used the filter_builder creates the right filter chain for you to prepare the input stream and make it ready to be encoded in whatever target you want. The filter_builder.au3 tackles each "problem" one step at a time, so for instance if your source is an Slog3 and your target is BT709, it knows that it has to go to RGB 16bit full range first, apply the appropriate LUT and go back to YUV. If your source is UHD and your target is FULL HD it knows that it has to downscale (it allows you to crop or add borders or stretch) and so on and so forth. It knows about interlacing, about chroma placement, about matrix, transfer and primaries etc. it even knows that when going from a progressive input at an higher resolution to an interlaced output at a lower resolution (let's say from UHD 50p to FULL HD 25i while dividing in fields) it needs to preblur vertically to avoid aliasing. I mean, the filter_builder is there to help but there's a big "but": when using a custom ffmpeg script (i.e encoder) it won't do anything as it doesn't know what you wanna do, which means that if you use ffvhuff in a custom ffmpeg script you'll have to tackle everything yourself. Of course, you can still use an indexer (i.e an A/V Decoder node) so that you can write your own logic in Avisynth and prepare the input stream to be encoded based on a bunch of conditionals and/or custom Avisynth scripts. In other words, if you need to encode in FULL HD 4:2:2 Slog3 25p 12bit you can output exactly that from Avisynth so that the encoding node won't have to do anything weird and it will only have to encode. That being said, you can also mix and match so that if you use one of the encoding nodes but you don't want the filter_builder to take care of everything, you can still do that in Avisynth provided that you then set the appropriate frame properties via PropSet() so that the filter_builder is gonna know that you already took care of that yourself. Speaking of encoders with high bit depth support (higher than 10bits) the only one that pops to my mind is the H.265 encoding node which is obviously calling x265, so it supports 12bit out of the box, but if you're interested in things like HuffYUV and UTVideo you're gonna have to go with a custom ffmpeg script inside FFAStrans I'm afraid. But please let me know and in case I'll add them to the lossless encoder.