Proposal for AviSynth 2.6

[IanB]

The 128k table will play havoc with the cache. (Ian pointed this out to me some time ago.) :cool: I seem to remember the elements were int's and there were multiple tables, such that the numbers came out about the size of a small L2 cache. I implied it might be borderline and need severe testing.

[IanB]

IanB,
1) In my opinion, we may use CS_METADATA=CS_UNKNOWN=0 for metadata clips. Almost all filter will not process such clips (it is not planar, not interleaved, not YV12, not RGB, not YUY2, etc). ... Is it possible?Unfortunately not all programmer do the right thing. For example slack implementation of vertical filters can just smack the Y, U & V planes, knowing columns are independant and that on non-planar formats the size of the U & V planes are zero and that PLANAR_Y is the same as the "only" plane.

I suppose we could add it to the API to uncourage people out of bad habits but we can't fix it for the legacy world.

:EDIT: If munging of metadata stored in a clip is a problem then check summing it might help. Throw an exception when it's broke.

2)The second question is about MVTools hack with audiochannels number. ... Any comments?It's a hack. As I understand it there is a higher level problem to addressing this properly. As the language stands filters can only return 1 thing, basicly an AVSvalue that is not an Array. Well a filter could return an array but you would have dificulties handling it within the script language, deep cloning of AVSValue arrays is not currently supported, so elements would either get prematurely released or never released i.e. leak.

In this MVTools case adding tagging to PClips would probably be sufficient but it would not be a total solution.

[squid_80]

squid_80 : it's not like I had the choice... I must put the data into the videoinfo, and it is made of int only, so...Fair enough, I do concede that you had no choice. Just wanted to get the word out that it's not something that should be done lightly. Using num_audio_samples instead of nchannels is a quick fix for 64-bitness anyway.

(I think I've turned bitter after wading through the resizing code and the Decomb source code. And the fact that I nearly failed a programming project at university three years ago for trying to be clever and doing the same thing. How was I supposed to know the lecturer would use an UltraSPARC for marking??)

[mg262]

Well a filter could return an array but you would have dificulties handling it within the script language,Do we need to handle such material in the script language? As long as the AVSValue is correctly passed into into the client filter, I would have said that's fine... after all, we're discussing how to stop filters from processing this data anyway!

CS_METADATA: While it would certainly be cleaner to stop most filters from processing metadata, I think that the proper solution is to type all data (as I believe is possible in 3.0). And for what it's worth, in Motion I very deliberately chose a format that could be played with by script authors (for example to binary-mask-merge together two sets of motion vectors); in fact it is precisely a 64-bit single-planar colourspace, and I want to allow any operations that are sensible on this colourspace.

Manao,
the behaviour of disguised pointers is specifically stated to be implementation dependent (page 8?? of 3e C++ book... can't find it at the moment). That wasn't meant to be a criticism, though. squid_80 has just given some examples where it would fail; also garbage collection and maybe distributed processing.

[Manao]

mg262 : I didn't take the remark as a critic, I rather wanted to know what could go wrong if I was doing that. I didn't thought of 64 bits compatibility. But int on 64 bits platform are 64 bits, so it shouldn't matter.

Garbage collector can't work with C++, or am I missing something ?

I know that what I done isn't proper C++ coding. Since I already use a global object, I should have used it to transmit data back and forth between filters, but I initially thought that filters would work outside MVTools dll ( silly dream ).

Anyway, that could be made cleaner.

[mg262]

Garbage collector can't work with C++,I don't know whether anyone has actually written one, but the language states that it's perfectly legitimate.

(I hadn't thought of 64-bits either! But IMO the thing about these mechanisms that look almost-safe... disguised pointers, memory leaks, global data... is that they cause problems in some weird case that is hard to think of.)

[Wilbert]

@Sh0dan,
@Wilbert: You might want to re-test your "interp" code. It wasn't activated in the code, as it was missing in the filter definition - this was also why the isse switch you had at the end was acting up.
I can't believe i missed that :) I will retest the interpolation tomorrow.

@IanB,
I didn't test the speed. I know it would be a bit slower, but i think not that much when processing *all* pixels. Anyway, i hope it's good now with Sh0dan's speed improvements.

[squid_80]

mg262 : I didn't take the remark as a critic, I rather wanted to know what could go wrong if I was doing that. I didn't thought of 64 bits compatibility. But int on 64 bits platform are 64 bits, so it shouldn't matter.No they're not. Ints remain 32 bits wide in both windows and linux. Longs aren't an option since they're 64-bits in linux, 32 in windows. That's why I said a quick fix is to use num_audio_samples(which is int64) instead of nchannels.

[Leak]

Garbage collector can't work with C++, or am I missing something ?
Oh, it can... (http://www.hpl.hp.com/personal/Hans_Boehm/gc/)

It would work better if it actually knew about the datastructures being used, but treating everything as a pointer works too as long as you know all allocated blocks - it just might happen that a bogus pointer (that really is some int value) prevents a block from being deallocated, but that shouldn't happen too often.

np: Bring Back The Buffalo - Breakfast At Sophia's (Bring Back The Buffalo)

[Fizick]

MVtools is for example of this ploblem only.

Probably in MVtools we can simply add some parameters to MVAnalyse, which must be set by user (what additional data will be needed in compensation function).
Or some more info is transmitted?
Probably it is offtopic here. sorry.

About CS_METADATA. May be we can not add it (as alias), but simply use CS_UNKNOWN. Can its BitPerPixels value be changed to 8 in AVisynth 2.5.7 ? :)

[Wilbert]

update 22-11:

bug in AviSource (latest CVS):

AviSource("D:\Temp\andreSample.avi",fourCC="dvsd", pixel_type="yuy2")

reports 'decompressor couldn't produce YV12', but the DV codec can't output YV12 (only YUY2).


if (bOpen && fYV12 && ICERR_OK == ICDecompressQuery(hic, pbiSrc, &biDst)) {
vi.pixel_type = VideoInfo::CS_YV12;
_RPT0(0,"AVISource: Opening as YV12.\n");
bOpen = false; // Skip further attempts
} else if (forcedType) {
env->ThrowError("AVISource: the video decompressor couldn't produce YV12 output");
}

forcedType is set to true (since a pixel_type is specified) hence the error.

It also doesn't work if pixel_type is not specified, since fYV12 is set to true, but it can't be opened as YV12. Since bOpen is set to false, it doesn't check YV411, and so on.

I'm not sure how to change this without using the "old" nested 'if else' statements.

[Wilbert]

bug in Overlay (latest CVS):


c = ImageSource("F:\Pictures\original.jpg").AddBorders(1,0,1,0).ConvertToYUY2
Overlay(c, c, mode="HardLight", opacity=1.0)

The luma (chroma idem) range is [0,255] and not scaled to [16,235].

I checked it for YV24 and YUY2. I didn't check other modes.

[sh0dan]

@Wilbert: This is by design. No coring is applied by itself, if luma (or chroma) are resulting in larger Y-values. The formula for hardlight is:

outY = baseY[x] + overlayY[x]*2 - 256;

The only place where Overlay cares about pc_range/tv_range, is when RGB input and output are converted. Otherwise it doesn't really care about the coring value.

[Wilbert]

@Wilbert: This is by design. No coring is applied by itself, if luma (or chroma) are resulting in larger Y-values.
Ok, i know. But, i assume best is to scale (or clamp ?) to CCIR-601 values when using those modes?

[IanB]

While reviewing some code with Arda, I had pause to think a little harder than usual about this coring issue. I think we are doing ourselves a very great disservice by continually clamping values in YUV processing.

Some background:- The CCIR-601 black=16, white=235 concept dates back to the transition from analogue to digital processing in the TV industry. In the analogue domain particularly when processing video signals with a colour sub carrier present there had to be provision for an amount of "head room" to prevent clipping when instantaneous signal value could exceed the nominal peak white or peak black levels. This often happened when mixing video for special effects or overlaying text titles. The visual results from overloading a video amplifier was quite horrid. The normal solution was to build head room into the video processor and, if required, finally attenuate the signal at the final output to conform to standard levels.

To allow this head room to be carried through a video processing chain including a new digital video processor the engineers of the day cleverly mapped the accepted overload levels for peak super-black=0 and peak super-white=255, which gave standard black=16 and standard white=235.

I propose we now follow this model with the coring option.

i.e. With coring=true we treat black=16 and white=235 BUT we do NOT clip at these levels. Further all 2.6 YUV filer are expected to have stable (documentable) behaviour when processing head room values. If people want clipping then they need to explicitly use the Limiter filter.

For coring=false we simply continue to use PC levels black=0 and white=255 as in previous versions.

Thoughts!

[Wilbert]

i.e. With coring=true we treat black=16 and white=235 BUT we do NOT clip at these levels. Further all 2.6 YUV filer are expected to have stable (documentable) behaviour when processing head room values. If people want clipping then they need to explicitly use the Limiter filter.

For coring=false we simply continue to use PC levels black=0 and white=255 as in previous versions.
Could you give a concrete filter example where this problem exists (ie luma/chroma are clipped while it needn't be)?

For me, when head room values are allowed (and kept), you need to treat it as a [0,255] YUV clip.

[IanB]

@Wilbert,

A [16..235] clip with over-range values is not always the same as a [0..255] clip.

Your ColorMatrix() filter needs to know the black level to work correctly, it does an env->Invoke("Limiter", ...) on the output. If a user would rather use Tweak(Sat=0.98765, Cont=0.94321) to more gracefully scale the possible over-range values they cannot.

It is not really a problem per se, rather more a philosophy that allows greater scope to control values. Clipping is a crude way to protect against over-range values, doing it should be a choice. For some cases scaling is a better choice, a choice users do not have if clipping is built into filters. Other more elabourate options for dealing with over-range values might be some form of squishing i.e. [233..235]->233, [236..245]->234, [246..255]->235, whatever it is it should be a choice.

A more hypothetical case :- Filter A does some image processing that can result in slight over-range values. The algorithm needs to know what the black and white levels are and author of the filter implements a coring option and because he knows he might generate over-range values he does standard output clipping. Filter B does some advanced edge detection, it is level agnostic. A user wants to chain filters A and B because he knows the processing filter A does will help filter B better recognise the edges he is interested in. His input data is standard TV range, so to get the effect he needs he enables the coring option. However the clipping filter A does causes a sharp change in the 1st derivative of the output, this spoils the results from filter B. The user is screwed, with coring off filter A doesn't process correctly, with corring on filter B trips over the abupt delta caused by the clipping.

[sh0dan]

Overlay actually applies an "advanced", though very unscientific method for handling cases, where Y>255 or <0. What I do is, that I gradually fade out chroma, thus creating pure white or pure black. This works surprisingly well visually. Here is the actual algorithm.

if (Y>255) { // Apply overbrightness to UV
int multiplier = max(0,288-Y); // 0 to 32
U = ((U*multiplier) + (128*(32-multiplier)))>>5;
V = ((V*multiplier) + (128*(32-multiplier)))>>5;
Y = 255;
}

Therefore, if Y is 288 or more, U&V will be 128 (neutral).

A similar algorithm could also be applied for coring, thereby neutralizing chroma, if Y is 255. 254 will leave a little of the original chroma, 253 a bit more and so on.

[IanB]

@Sh0dan,

Precisely another example for not arbitarily clipping inside filters, there are just so many ways to deal with over-range values, just given the chance.

We probably should crack a new thread on ways to manage over-range yuv data.

[Fizick]

Sorry for old thread post, but i see it as most appropriate.

I am interested in not-subsampled YUV color format, but not planar (3 planes), but interleaved.
one plane YUV24 like RGB24: YUV YUV YUV ... (or VUY... in memory)
or YUV32 like RGB32 (RGBA): YUVA YUVA (or VUYA ... in memory)

This format is convinient for:
- motion estimation (SAD) (luma with chroma),
- block (pixel) replacement,
- various filters (denoisers, etc) which take into account the color similarity of all pixel color component.

Implementation should be very similar (read "absolutely equal") to current RGB code in filters (besides color conversion).

Comments?

[MfA]

You can get cheap GPUs with multi GB/s transfer speed now. IMO the time is about right to put GPU support in Avisynth (ie. the ability to pass GPU memory reference to frames rather than system memory pointers, and potentially some GPU versions of the internal functions).

[sh0dan]

@MfA, Fizick: 2.6 is already way too feature-creeped, thus we still don't even have an alpha.

@Fiz: Yes, it would be nice, but more colorspaces means more complexity - and since it brings nothing to the end user (since we already have planar 4:4:4) - I cannot support yet another cs.

@MfA: That would IMO belong in a fork. GPU processing is very interesting - but it would be much better to start with a clean slate, and design it entirely for the GPU.