Thanks for the reply, I understand that the mfd->gvalue etc are type int with mixed values for multiple channels.
I already had to modify part of the code because vaporsynth getReadPtr does not return RGB combined.
I have to disagree that it's impossible for me to understand how the lookup tables are filled vs how they are used.
While I will agree that I am diving into the deep end working on new things, i'm quickly learning a lot, and i'm starting to feel like i'm getting past a lot of the initial learning curve.
With enough debugging and testing I could figure out the reason they have the extra step that appears to have no effect on the output of the frame.
but for now, I am doing exactly that, just porting the filter and feeding input and grabbing output.
if I ever want to do more with it, like supporting greater bit depth then I will obviously need to delve deeper.
I have finished converting this to use integer input instead, and updated the github:
https://github.com/xekon/GradCurve