[FranceBB]

Before regular high bit depth was introduced, there were two high bit depth version: stacked and interleaved. Stacked was used by Dither Tools and Interleaved was used by HDRTools, however the most used was stacked by dither tools.
Since Avisynth wasn't actually able to have 16bit internally, but only 8bit, in order to be able to work with 16bit precision, the image was divided in two parts: MSB (most significant bits) and LSB (less significant bits).
With 16bit stacked, the image has double height as MSB and LSB are "stacked" one on the top of the other. This is what Dither_Tools uses.
With 16bit interleaved, the image has double weight as MSB and LSB are "interleaved" one after the other. This is what HDRTools uses.
In you case, this is your code:

LSMASHVideoSource("C:\Users\frank\Daten\Samples\test.mp4")
dither_convert_8_to_16()
dither_quantize(bitdepth=10, reducerange=true, mode=0)
dither_out()
BicubicResize(1280, 720)
Trim(1264, 1563)

Alright, let's take a look at it:

LSMASHVideoSource("C:\Users\frank\Daten\Samples\test.mp4")

Now you are indexing your 8bit source, which is fine.

dither_convert_8_to_16()

Now you have just brought everything from 8bit planar to 16bit stacked.

dither_quantize(bitdepth=10, reducerange=true, mode=0)

Now you are dithering it down to 10bit stacked.

dither_out()

Finally, you are outputting 10bit stacked.

BicubicResize(1280, 720)
Trim(1264, 1563)

Now, this is something you can't do: you can't resize using an 8bit resizer which is not high bit depth aware, 'cause this poor resizer thinks that it's dealing with 8bit data at a "weird" resolution and it just resizes it, screwing up MSB and LSB.

For instance, this is ColorBars 848x480 8bit planar:



This is colorbars 16bit stacked (848x960) 'cause MSB and LSB are stacked one on the top of the other:



Once you apply reduce range, you get 10bit stacked 848x960, 'cause MSB and LSB are still stacked one on the top of the other, but with 10bit precision rather than 16bit.


If you are using a pipe to x264 or x265, you can definitely output 10bit stacked and let it encode it in 10bit planar, however, unless you are using a particular dithering method and you wanna stick with that for any particular reason, then I suggest you to output 16bit stacked to x264/x265 and then let x264/x265 dither it down and encode to 12bit/10bit/8bit planar.
Please note that every time you are using high bit depth (16bit stacked or interleaved) and you wanna filter in Avisynth, the filter you are using MUST be high bit depth aware and you MUST set it to work with the correct version of high bit depth, 'cause otherwise you are gonna screw up MSB and LSB.

I've been stuck with Avisynth 2.6.1 for years after 2016, but in the end I moved to Avisynth+ and I gotta say that it made my life so much easier as I can work with regular 10/12/14/16bit planar, which is great as I don't have to bother about MSB and LSB anymore. Sure, there are some "legacy" filters that have never been updated to work with planar 16bit, so I still have to use 16bit stacked or 16bit interleaved, but in the end I can go from one to the other and always go back to regular 16bit planar and output it to my target application/encoder and be sure that it's gonna read it, no matter what. For instance, with Avisynth+ pinterf mod:

#Indexing 10bit source
video=FFVideoSource("test.mxf")
audio=FFAudioSource("test.mxf")
AudioDub(video, audio)

#From 10bit planar to 16bit planar
ConvertBits(bits=16)

#From 16bit planar to 16bit stacked
ConvertToStacked()

#Denoise with 16bit precision
dfttest(sigma=64, tbsize=1, lsb_in=true, lsb=true, Y=true, U=true, V=true, opt=3, dither=0)

#From 16bit stacked to 16bit planar
ConvertFromStacked()

#Resize with 16bit planar precision
Spline64Resize(1280, 720)

#Dithering down from 16bit planar to 10bit with the Floyd-Steinber error diffusion
ConvertBits(bits=10, dither=1)

#Clipping for limited TV Range 10bit
Limiter(min_luma=64, max_luma=940, min_chroma=64, max_chroma=940)

As you can see, I can go from an high bit depth version to another.
If I want to, I could go from Planar to Interleaved and vice-versa as well.
If I was in Avisynth 2.6.1, I would have done something like:

#Indexing 10bit source as 16bit stacked
video=LWLibavVideoSource("test.mxf", stacked=true)
audio=LWLibavAudioSource("test.mxf")
AudioDub(video, audio)

#Denoise with 16bit stacked precision
dfttest(sigma=64, tbsize=1, lsb_in=true, lsb=true, Y=true, U=true, V=true, opt=3, dither=0)

#Resize with 16bit stacked precision
Dither_resize16nr(1280, 720, kernel="spline64")

#Dithering down to 10bit stacked with the Floyd-Steinberg error diffusion
Dither_quantize(bitdepth=10, reducerange=true, mode=6)

And then i had to apply clipping to make sure that luma and chroma are in limited tv range when I encode with the encoder, rather than in Avisynth.

Anyway, I hope this helps and clarifies your doubts about stacked, interleaved and planar.
Please also note that Avisynth 2.6.1 is no longer maintained and that new filters won't work in stacked/interleaved but only in planar. For instance, DGHDRtoSDR by Donald Graft works with 16bit planar only, ResizeMT by Jean Philippe works in planar only, so with Avisynth 2.6.1 it will be limited to 8bit precision, Hable, Mobius and Reinhard tone-mapping algorithms will work in 16bit planar only so you won't be able to use them in Avisynth 2.6.1, Cube will work in planar RGB 16bit only with a good enough precision, 'cause in 8bit planar RGB it would be too approximated and so on.