vsMidas PyTorch version

[Dogway]

I'm getting this error when calling the vsmidas filter. It wasn't bundled in Hybrid but it had all the requirements fulfilled already:

File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsmidas\__init__.py", line 108, in midas
module.load_state_dict(parameters)
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DPTDepthModel:
Missing key(s) in state_dict: "pretrained.model.layers.3.downsample.reduction.weight", "pretrained.model.layers.3.downsample.norm.weight", "pretrained.model.layers.3.downsample.norm.bias", "pretrained.model.head.fc.weight", "pretrained.model.head.fc.bias". 
Unexpected key(s) in state_dict: "pretrained.model.layers.0.downsample.reduction.weight", "pretrained.model.layers.0.downsample.norm.weight", "pretrained.model.layers.0.downsample.norm.bias", "pretrained.model.layers.0.blocks.1.attn_mask", "pretrained.model.layers.1.blocks.1.attn_mask", "pretrained.model.head.weight", "pretrained.model.head.bias". 
size mismatch for pretrained.model.layers.1.downsample.reduction.weight: copying a param with shape torch.Size([768, 1536]) from checkpoint, the shape in current model is torch.Size([384, 768]).
size mismatch for pretrained.model.layers.1.downsample.norm.weight: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([384]).
size mismatch for pretrained.model.layers.1.downsample.norm.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([384]).
size mismatch for pretrained.model.layers.2.downsample.reduction.weight: copying a param with shape torch.Size([1536, 3072]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
size mismatch for pretrained.model.layers.2.downsample.norm.weight: copying a param with shape torch.Size([1536]) from checkpoint, the shape in current model is torch.Size([768]).
size mismatch for pretrained.model.layers.2.downsample.norm.bias: copying a param with shape torch.Size([1536]) from checkpoint, the shape in current model is torch.Size([768]).

So in VapourSynth my installed PyTorch is this:

torch                  2.0.1+cu118
torch-tensorrt-fx-only 1.5.0.dev0
torchvision            0.15.2+cu118

But it seems MiDaS requires v1.13.1, how do you go in this situation?


PD: Also, do you know of a good AI segmentation filter? Maybe with granularity option. I know OpenCV can do some but looking for some novel AI approach.

[Selur]

I can confirm that this doesn't work, so I would recommend opening a thread over at https://github.com/HolyWu/vs-midas/issues, since downgrading torch to 1.13.1 would definitely break and deinstall lots other stuff.

Also, do you know of a good AI segmentation filter? Maybe with granularity option. I know OpenCV can do some but looking for some novel AI approach.

No, and 'granularity option' and 'AI approach' are usually incompatible requirements.

Cu Selur

[Dogway]

Originally Posted by Selur

No, and 'granularity option' and 'AI approach' are usually incompatible requirements.

True, true. I guess there might be room to segment every object or only foreground/background. At worst I can paint fill.

I will try to install the original version under an env. Strangely it asks for Python 3.10.8 where I use 3.10.6 for Stable Diffusion so let's see.

[Selur]

I created an issue entry over at github https://github.com/HolyWu/vs-midas/issues/2.

Cu Selur

[_Al_]

offtopic, Selur , in your scripts:

clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)

that red part is always False, does not matter what, so that line is just effectively always setting a prop to 5

[Selur]

@_AI_: Thanks for the info.

[Selur]

@Dogway: HolyWu updates vsmidas (sadly I still can't use it since its timm 0.6.13 dependency conflicts with vs-ddcolors timm dependency), maybe that is of help to you.

Cu Selur