ChatGPT vs. Claude on ColorMatrix conversion - which answer is correct?

[orion44]

I've asked GPT-4o and Claude 3.5 Sonnet the same question, but they gave different answers.

Which answer do you think is the correct one?

Question: "When resizing 1080p video to SD resolution (640x360), should I apply ColorMatrix ('ColorMatrix(mode="Rec.709->Rec.601")')
before or after resizing the video, to convert from Rec.709 to Rec.601?"

GPT-4o: When converting a 1080p video to SD resolution, you should apply the `ColorMatrix("Rec.709->Rec.601")` conversion **before resizing** the video. Here’s why:

1. **Color Primaries and Matrix**:
- HD content (like 1080p) is typically encoded with Rec.709 color space, while SD content (such as 640x360) is usually in Rec.601 color space. The color matrix conversion from Rec.709 to Rec.601 ensures that the colors are accurate for the SD standard.

2. **Avoiding Interpolation Artifacts**:
- Applying `ColorMatrix` before resizing preserves color accuracy by converting colors in their original, unscaled form. Resizing the video first and then applying the color conversion can lead to minor interpolation artifacts, as the color matrix adjustment may not apply as cleanly after scaling.

3. **Performance**:
- Applying `ColorMatrix` before resizing is generally faster because it processes fewer pixels in the original HD frame than it would after the image is scaled down.

### Correct Usage Example

```avs
# Convert Rec.709 to Rec.601 for SD resolution
ColorMatrix(mode="Rec.709->Rec.601")

# Resize to SD resolution
BicubicResize(640, 360)
```

By doing it in this order, you ensure that the color conversion is done on the full HD image, preserving color fidelity and achieving accurate color reproduction in your SD output.

Claude 3.5 Sonnet: The ColorMatrix conversion from Rec.709 to Rec.601 should be applied AFTER resizing. Here's why:

1. The resize operation should be done in the original colorspace (Rec.709 for HD content) to maintain the best quality during the scaling process
2. Only after you've resized to SD resolution should you convert to Rec.601, since that's the standard colorspace for SD content

Correct order:
```
video = LSMASHSource("your_video.mp4")
video = Spline36Resize(video, 640, 360)
video = ColorMatrix(video, "Rec.709->Rec.601")
```

Incorrect order:
```
video = LSMASHSource("your_video.mp4")
video = ColorMatrix(video, "Rec.709->Rec.601")
video = Spline36Resize(video, 640, 360) # Don't do this
```

This order ensures:
1. The resize operation works with the full color information in the original colorspace
2. The colorspace conversion happens only once, at the final resolution
3. Better preservation of color accuracy during the entire process

[Z2697]

Applying `ColorMatrix` before resizing is generally faster because it processes fewer pixels in the original HD frame than it would after the image is scaled down.
Really??

[Z2697]

I'm convinced by the second point by ChatGPT

[FranceBB]

Please don't use AI for encoding related questions as they generally hallucinate a lot and especially in Avisynth they even tend to give non working outputs.
I'd like not to see them anywhere on Doom9 as it's literally one of the last few places that is preserving human knowledge without being polluted by artificially generated stuff
Ask directly on Doom9 after looking at other threads with similar questions.
Now, doing it before or after will not realistically have an impact on the overall quality and this is because of how Colormatrix() works, hence why both AI are hallucinating here.


FFMpegSource2("whatever.mxf", atrack=-1)

ColorMatrix(mode="Rec.709->Rec.601")

SinPowerResize(640, 360)


Now the AI is clearly thinking "oh, this way Colormatrix() is gonna work on the full resolution of the video so that it first upscales the chroma from 4:2:0 planar to RGB, then it applies the matrix conversion and then it scales back to 4:2:0, so we're actually performing the conversion at a higher quality 'cause we're preserving the chroma before we resize."


FFMpegSource2("whatever.mxf", atrack=-1)

SinPowerResize(640, 360)

ColorMatrix(mode="Rec.709->Rec.601")


Here the AI will probably think "see, if we perform the colormatrix conversion after the downscale, then the chroma, which in the 1920x1080 resolution was 960x540, has been further downscaled to 320x180 'cause we're starting from 640x360 in 4:2:0, so when we upscale the chroma to match the luma resolution in RGB and apply the colormatrix conversion we're working with fewer pixels and therefore the result is worse".

Now, all of this would make sense if we were using avsresize, in other words:


z_ConvertFormat(colorspace_op="709:709:709=>470bg:601:470bg")


This is because avsresize is based on zlib and such a library will ALWAYS internally convert to RGB when it performs the matrix conversion (in this case BT709 to BT601 PAL), so by doing it in 1920x1080 4:2:0 it would be upscaling the chroma from 960x540 to 1920x1080 to go to RGB, then perform the conversion, then downscale it back to 960x540 to get a nice 1920x1080 4:2:0 and then you would call SinPowerResize to finally go to 640x360. That's how it's supposed to be done to preserve quality and achieve the best conversion, everybody is happy, job done.

BUT... we're using Colormatrix() here and Colormatrix() DOES NOT internally convert to RGB as all the operations are done in YUV.
So... does it make a difference to perform the conversion before downscaling with Colormatrix()? Maybe.
Is it noticeable at all? No.
Can you do it once you've downscaled to save CPU cycles? Absolutely.

[lollo2]

Please don't use AI for encoding related questions as they generally hallucinate a lot and especially in Avisynth they even tend to give non working outputs.
I'd like not to see them anywhere on Doom9 as it's literally one of the last few places that is preserving human knowledge without being polluted by artificially generated stuff


+ 1

Well said!

[kolak]

ChatGPT doesn't know things or create them.
It repeats what have been already said when it comes to "knowledge".
Its answers will be based also on doom9 :D

[orion44]

ChatGPT doesn't know things or create them.
It repeats what have been already said when it comes to "knowledge".
Its answers will be based also on doom9 :D

I've noticed this also.

Newer versions of ChatGPT seem to have the entirety of doom9 forum in its database.

[VoodooFX]

ChatGPT doesn't know things or create them.
It repeats what have been already said when it comes to "knowledge".

That's not quite true. Same you could say about humans.

[kolak]

No. Human can do own tests and come to new conclusions. ChatGPT knowledge is purely based on internet (what already been said/tested/proven etc.).
It's more advanced only in language related tasks.

[GeoffreyA]

No. Human can do own tests and come to new conclusions. ChatGPT knowledge is purely based on internet (what already been said/tested/proven etc.).
It's more advanced only in language related tasks.

I think these are early versions of part of the human mind, perhaps the language faculty, lacking the sense modalities, persistent state, perpetual training in real-time, and using a lot of power.

Though by Microsoft researchers, this paper had some interesting findings:

https://arxiv.org/abs/2303.12712

[Z2697]

I think these are early versions of part of the human mind, perhaps the language faculty, lacking the sense modalities, persistent state, perpetual training in real-time, and using a lot of power.

Though by Microsoft researchers, this paper had some interesting findings:

https://arxiv.org/abs/2303.12712

Oh yeah, research by the big money behind the AI model being studied, I totally belive it.

[GeoffreyA]

Microsoft is a joke, no doubt about that. In general, though, I don't see why neural networks can't match the human brain. Right now, they're missing a lot of parts, and power is a barrier. Perhaps the turning point will be biological circuitry. As for the hard problem of consciousness, we haven't even explained how it works in the brain, but it is tied to certain parts (the anterior cingulate cortex, I believe), so conceivably, might be some trick of recursive circuitry and a working temporal window. Having said all that, I'm actually one who's job is being threatened by AI, and am not cheering the laughable bandwagon at all, where every company is hungry to plunge their hand into that pot of dollars. My interest is more from a science-fiction, philosophical point of view :)

[VoodooFX]

No. Human can do own tests and come to new conclusions.
Take out all preexisting knowledge and your most advanced tests will be counting the fingers on a left hand. ;)

[kolak]

I didn't say you want to abandon existing knowledge, but to come up with something new based on it.
Ask ChatGPT to prove his method is better than others.
All what it does when it comes to "avisynth" is search and repeat what others said.
It's just a fancy google search which better understands your questions and can give nicer answers (due to its ability to understand language very well).

[VoodooFX]

...repeat what others said.


No, it can synthesize of what it knows, so it can come up with answers never said in previous (training) knowledge.

[kolak]

This is still rather limited.
Typically it won't tell you anything more than what you can find on the net.
Beauty is that ChatGPT will do it quickly and present it in nice form ready to digest :)

[Z2697]

Let's go back to original track a bit.
BUT... we're using Colormatrix() here and Colormatrix() DOES NOT internally convert to RGB as all the operations are done in YUV.
But even we're not going a roundtrip to RGB, that still means it needs to somehow make the subsampled planes "unsubsampled", right?
I don't know how this filter works, but I assume the simplest way is to use the equivalent of PointResize.
Either it creates a upsampled copy in memory, or just reference the same pixel 4 times, the effect should be identical, right?

Now here comes the second question: when you do this roundtrip to RGB or 4:4:4, what resampler to use?
I again assume PointResize might be a good choice because you avoid "actual" interpolation, if not doing any other operation, PointResize shoule be just perfectly reversible by itself.
But is the "blocky" chroma detrimental to the matrix conversion?

Since we are talking about AI: most image / video processing NNs nowadays take RGB as input. For the roundtrip, what resampler to use then? (we probably should take different types into account: super resolution, restoration, frame interpolation, etc.)


EDIT:
(I still can help but go off track)
No, it can synthesize of what it knows, so it can come up with answers never said in previous (training) knowledge.
Beauty is that ChatGPT will do it quickly and present it in nice form ready to digest
Also don't forget it sometimes lies. It generates wrong answer with unwavering tone, not the actual lying of course, that's a human (or post-human skynet "AGI") thing.
It's generating. It's not answering. It generates something resembles an answer.
It's lying, not because it want to lie. The machine and data determines it will lie.

You can still argue that we humans will give wrong answers, we will lie, we don't even know if it's really us that "want" to lie.
We don't know what Intelligence is. We don't understand conciousness.
But this is a bottomless rabbit hole as of now.

[GeoffreyA]

Indeed, till we crack the riddle of consciousness, we can't answer some of these questions. My feeling, though, purely speculative, is that this is the primitive version of the technology in our brains. If the result is eventually the same, is there really a difference? As time goes by, people are going to deny that these things match the human mind, or when sentience is implemented (or emerges, which I doubt; simple creatures have awareness; thus, it is likely some circuit or architecture), deny that too because of feeling threatened or human pretensions to superiority. But this is rather for the realm of machine ethics.

Back to the topic a bit, another point left out by the models' answers is downscaling in the linear domain, though Claude seems to be vaguely scraping the surface of that.

[Frank62]

Why does the single-celled organism move forward? It has no stomach that growls, no brain that can think about the fact that it might find something to eat after moving forward, or at least that the probability of doing so increases. A simple mathematical odyssey? Hardly. God? Possibly at the very end (I abstain), but the path before that is still completely unclear!

My opinion, translated with DeepL (to write good English once).

[GeoffreyA]

1. why does the single-celled organism move forward? It has no stomach that growls, no brain that can think about the fact that it might find something to eat after moving forward, or at least that the probability of doing so increases. A simple mathematical odyssey? Hardly. God? Possibly at the very end (I abstain), but the path before that is still completely unclear!

I think it could be some low-level mechanism based on chemicals that drives a cell to X, Y, Z. Without it, life would flounder.

[Frank62]

There are low-level mechanisms for everything anything living does. But this does not answer the question.

[GeoffreyA]

You're right: I answered how, but not why. I admit it's a challenging one. If we deny that the cell has any rudimentary agency, it boils down to the mechanism; there is a mechanism and it moves. We then ask, where does the mechanism come from? And this takes us to the ultimate riddle on the origin of life.

On the other hand, if we imagine there is some form of "agency"---and as we see elsewhere in our bodies, various cells have scripted behaviour quite apart from "us"---then we could say that it "knows" it must get the food and does so, according to some script. But this leads us back to the earlier question on the origin.

[microchip8]

This thread has gotten wayyyy off-track!

[filler56789]

This thread has gotten wayyyy off-track!

Yes, but at least it became far more interesting than the A.I. chats themselves. :D

[Frank62]

...
then we could say that it "knows" it must get the food and does so, according to some script.
...

If it were a script, you could make predictions. But you can't do that with living organisms. Not even the smallest movement of an organism can be predicted. There are theories according to which the fruit fly flies to the food source and is possibly diverted from it by air currents or an obstacle, and so on...
I am of the opinion that even with the most precise knowledge of the circumstances, it is NOT possible to predict exactly where the next movement of a living being will be directed, and this - I believe - has nothing to do with quantum phenomena (although on a smaller scale all life has a lot to do with it), but with something that still awaits a more precise explanation.
This difference to non-living material makes a lot of difference, but at first glance it doesn't have that much to do with the difference between biologically based intelligence and A. I. But ultimately it does (I think), but this is really twice off topic, sorry.
(Translated with DeepL)

[Z2697]

If it were a script, you could make predictions. But you can't do that with living organisms. Not even the smallest movement of an organism can be predicted. There are theories according to which the fruit fly flies to the food source and is possibly diverted from it by air currents or an obstacle, and so on...
I am of the opinion that even with the most precise knowledge of the circumstances, it is NOT possible to predict exactly where the next movement of a living being will be directed, and this - I believe - has nothing to do with quantum phenomena (although on a smaller scale all life has a lot to do with it), but with something that still awaits a more precise explanation.
This difference to non-living material makes a lot of difference, but at first glance it doesn't have that much to do with the difference between biologically based intelligence and A. I. But ultimately it does (I think), but this is really twice off topic, sorry.
(Translated with DeepL)

You can't even reliably predict a real world double-rod pendulum. It's just how current science works.

[GeoffreyA]

If it were a script, you could make predictions. But you can't do that with living organisms. Not even the smallest movement of an organism can be predicted. There are theories according to which the fruit fly flies to the food source and is possibly diverted from it by air currents or an obstacle, and so on...
I am of the opinion that even with the most precise knowledge of the circumstances, it is NOT possible to predict exactly where the next movement of a living being will be directed, and this - I believe - has nothing to do with quantum phenomena (although on a smaller scale all life has a lot to do with it), but with something that still awaits a more precise explanation.
This difference to non-living material makes a lot of difference, but at first glance it doesn't have that much to do with the difference between biologically based intelligence and A. I. But ultimately it does (I think), but this is really twice off topic, sorry.
(Translated with DeepL)

At the risk of going further off-topic, I must respond to this thought-provoking comment. Your argument is that, in principle, and with full knowledge of an organism and its environment, it still can't be predicted and this is due to some quality other than quantum uncertainty. As Penrose pointed out, it is possible for a system to be deterministic and non-computable. Then, if we were to ascribe it to quantum processes, which I don't think it is either, the Schrödinger-equation side of things is deterministic, and it is only when collapse happens, during measurement, that there is probabilistic jumping to a classical configuration. Some piece is missing.

In the end, it could simply be that the overwhelming complexity of an organism and its environment renders prediction impractical. But if impossible in principle, it would be good to ask, where exactly does the impossibility come in? Is it non-computable; quantum probability; or something outside the universe, inaccessible from within? (Though the latter goes into the realms of metaphysics.)

[Frank62]

I am sorry too to keep somehow off-topic, but hope we will find mercy with our little discussion.

You can't even reliably predict a real world double-rod pendulum. It's just how current science works.

The unpredictability of a double-rod pendulum is due to the lack of accuracy with which the conditions can be known, because the slightest deviation due to the slightest influences leads to increasing changes.
It is a supposedly simple system, but due to the above property it is actually inaccurate, this is the chaos-theory-trick with it. In principle, however, the (inanimate) universe remains deterministic, even if there are double-bar pendulums in it.
The WILL of a living being is something completely different, located at a much higher level of complexity.
It is unpredictable for completely different reasons.


Some piece is missing.
Yes, exactly.

In the end, it could simply be that the overwhelming complexity of an organism and its environment renders prediction impractical.
That's, what I don't think, but of course possible.

But if impossible in principle, it would be good to ask, where exactly does the impossibility come in? Is it non-computable; quantum probability; or something outside the universe, inaccessible from within? (Though the latter goes into the realms of metaphysics.)

I believe it is
- not computable
- not a phenomenon based on quantum mechanics, as it is located far above the level of any quantum decoherence
- nor is it something that requires metaphysics or even God

I simply believe that we are dealing here with something truly unexplored, but which is very much part of this world and can very well be explained in the future (although "explain" is a very difficult word...), and what in the end makes a huge difference between our biological kind of intelligence and A. I.

[Z2697]

Whatever WILL is, my point is actually not related to WILL itself.
You made it sounds like "being unpredictable == having will", I disgree with that.
I think single cell living is unpredictable because of the chaos theory, although it's probably trillions times more complex than a double-rod pendulum.
(OR DOES IT? Does double-rod pendulum have WILL? Can we even prove it not?)

Oh guys, please stop arguing around the (currently) UNKNOWABLE.

[Frank62]

Whatever WILL is, my point is actually not related to WILL itself.
You made it sounds like "being unpredictable == having will", I disgree with that.
I did not mean to "make anything sound like".
And I did really not say or mean: "being unpredictable == having will", rather
"having will == being unpredictable", but I didn't say that either.
I had said: A biological organism is unpredictable even if all known variables are known. And neither because of chaos theory nor for quantum mechanical reasons. I later used the term "will" as a possible reason for this. No more and no less.

I just wanted to give my opinion on a possible difference why A. I. might differ from biologically evolved intelligence, and what reasons there might be for this.

I think single cell living is unpredictable because of the chaos theory, although it's probably trillions times more complex than a double-rod pendulum.
I don't. I understand you well, but we are just of different opinion here.

(OR DOES IT? Does double-rod pendulum have WILL? Can we even prove it not?)
That would be philosophizing about something that is very likely nonsense. Fun for (some) philosophers, but (most likely) a waste of time for others who are interested in finding out what is really happening. I'm not interested in that kind of discussion any more than you are.
I just found Geoffrey's points very interesting and have tried to post a small part of mine here.

Oh guys, please stop arguing around the (currently) UNKNOWABLE.

We hadn't even started arguing yet. Just listed a few potentially interesting points.
And: why? A: I. is one of the most fascinating topics of our time and influences so many things also in the video sector that the discussion doesn't even seem very off-topic to me.

[VoodooFX]

The WILL of a living being is something completely different, located at a much higher level of complexity.
It is unpredictable for completely different reasons.

It's unpredictable for the same fundamental reasons - too many influences.
Your "WILL" is just various chemical reactions in your brain.

I had said: A biological organism is unpredictable even if all known variables are known.

If all variables are known it would be predictable like a clock.

[filler56789]

Your "WILL" is just various chemical reactions in your brain.

That's an assumption that shoud have been properly-killed in the 19th century.
Sadly it's still alive and well.

[Frank62]

It's unpredictable for the same fundamental reasons - too many influences.
Your "WILL" is just various chemical reactions in your brain.
Not in any way.
But I think that's really going too far here.

If all variables are known it would be predictable like a clock.
The question is which variables. I said if all KNOWN variables were known.
But there are - quite obviously to me - variables in living organisms that are NOT limited to (anyway Heisenberg fuzzy) particles and their movements, and these remain UNKNOWN as long as ... well... nothing exact is known about them.

[GeoffreyA]

Roger Penrose has maintained that there is a non-algorithmic or non-computable basis to consciousness. This seems similar, Frank, to what you are terming "will." If we were to boil it down to its essentials, this would mean that will causes the movement of particles that cannot be predicted from the previous state (but is, possibly, determined by it and leaving out quantum reasons). If this were true, especially the determinism part, it would mean that our will is an illusion, is determined, but cannot be predicted.

I am of the opinion that consciousness is mechanistic (or "materialistic") and doesn't require the exotic qualities we are describing here. However, we know so little and I am quite open to there being more than meets the eye. For one, consciousness seems intimately tied to time, the nature of which is still a mystery in physics. Understanding time may well resolve many of these questions.

It could well be that the missing, non-quantum, non-computable physics under discussion in this thread resolves the tension between quantum mechanics and general relativity.

But there are - quite obviously to me - variables in living organisms that are NOT limited to (anyway Heisenberg fuzzy) particles and their movements, and these remain UNKNOWN as long as ... well... nothing exact is known about them.

I think particles and waves are mere approximations, from the space-time point of view, of deeper entities. QM has partly hit on their rules but the picture is not complete. (The missing theory would solve the measurement problem and account for the domain where the crossing over from quantum to classical happens.)

[Frank62]

...
If this were true, especially the determinism part, it would mean that our will is an illusion
...
I do not understand this part. Can you please explain?

...
Understanding time may well resolve many of these questions.

It could well be that the missing, non-quantum, non-computable physics under discussion in this thread resolves the tension between quantum mechanics and general relativity.
...


That would be of course wonderful and fascinating.
I think that if we really want to fundamentally broaden our horizons, we need a completely different view of what is, i.e. a different perspective, a different model.
Above all, this has to do with INFORMATION and an informational view and approach.

[GeoffreyA]

I do not understand this part. Can you please explain?

If everything is deterministic, free will is illusory. We are set on a course but have the perception that we choose. Similarly, if there is non-determinism because of a random quality in Nature, we still do not choose.

I think that if we really want to fundamentally broaden our horizons, we need a completely different view of what is, i.e. a different perspective, a different model. [...] Above all, this has to do with INFORMATION and an informational view and approach.

I agree with you. For a long time I've believed that reality, at some low level, is a non-spatial, non-temporal domain, the elements of which are perhaps information or something else. Wherever we look, information seems to hit us. It appears fundamental. I think what we see in our world is a high-level "rendering" from low-level data, according to various transformations, like that in a computer game or simulation. At an intermediate level, we've got the quantum regime where things are being calculated in the complex space according to the rules of QM. Lastly, it is mapped, as an approximation, to the space-time domain where other rules hold sway; this mapping isn't perfect, so we see strange quantum "oddities" at times (entanglement,* quantum eraser, etc.). We're going from BT.2020 to BT.709, a lossy operation!

* (The EPR phenomenon demonstrates, to me, that distance, or space, is not fundamental. Those entangled particles have their existence in a pre-spatial realm.)

[Frank62]

A pleasure to read this. :)

[VoodooFX]

A biological organism is unpredictable even if all known variables are known.

That sounds oxymoronic, no need to use "known" twice, from the first "known" we already know that they are known.

Here your sentence is un-oxymoroned:

A biological organism is unpredictable even if some variables are known.

[Frank62]

Could it be that you meant "pleonastic"? Oxymoronic requires opposites rather than overlapping doublings.
I had all this translated by DeepL (now again) so as not to be misunderstood once in this forum - unfortunately it didn't help. I should check the translation next time so that readers as accurate as you don't have any problems reading it.

The second "unknown" originally meant (in German) the value of the variable, the first "unknown" meant the variable itself.
In connection with "unknown", the "unknown value of variables" is often incorrectly (if you take it as accurately as you do) expressed as "unknown variables". So my point here was that:

1. not everything is known that could be variable in the context
2. of the variables that are known (researched), their value is also known
In context: Even if you know all THE conditions that ARE PHYSICALLY KNOWN AND RESEARCHED at a point in time before a thought, it is possible that other variables that ARE NOT KNOWN will throw a spanner in the works, and you can not predict.

However, since your friendly comment was meant purely destructive anyway, probably intended to express contempt, you are in a way right, in that your intention was successful. Thank you. These things always hurt exactly then, when you know that they come from intelligent people who really know something.

Translated with DeepL.com (free version)

[GeoffreyA]

A pleasure to read this. :)

Same. It's been a challenging and delightful conversation that got me thinking again about ideas that haven't been in my head much these days. Thanks, Frank.