Completely split into lines

[jmac698]

Hi,
This is something I've wanted to do a for along time. I want to split a video in every one of it's lines, and I'm not gonna write 480 crop statements ;) I've used recursion in the past to generate a frequency sweep and found it really slow (though now I can just use lutspa).
So I thought of using separatefields a few times. Then I got stuck on how to split by the other prime factors, ie 480=2^5*3*5. I can't just split into 3 sections you understand I need to split up every 3rd line. Luckily in this case the lowest factor is 15 so I could do that individually.
Anyhow I came up with a compromise, just pad to nearest pwr2 and split all lines.
I find this pretty tough, any ideas?

#Testing splitting video by lines, then motion compensation
#576=2^6*3*3
#480=2^5*3*5
#Outline: quadruple height, addborders to the next power of 2 of original height
#unweave all the way
#motion compensate pair of clips
#weave back
#you can use conditionalreader to avoid mc extra lines in specific heights
#this reminds me of fft butterfly, perhaps you can find lines 480-511 by reversing bits of index (to avoid processing)
w=32
h=32
BlankClip(width=w,height=h, pixel_type="YUY2",color_yuv=$40B080)
converttoyv12
mt_lutspa(mode="absolute",yexpr="y",uexpr="128",vexpr="128",chroma="process")
converttoyuy2
#return last
global splith=height
nearestpwr2=int(pow(2,int(log(splith)/log(2)+1)))
padh=nearestpwr2-splith
addborders(0,0,0,padh)
unweavestep
unweavestep
unweavestep
unweavestep
unweavestep
unweavestep

function unweave(clip v) {
v.assumeframebased.separatefields
}

function unweavestep(clip v) {
v
t=splith%2
global splith=t==0?splith/2:splith
t==0?unweave:last
}


So I need to process separately on the (equivalent of) each pair of video lines from two clips eventually.
Oh ya, could also use a gcd(a,b) and factorn(int n, int index) function

[Gavino]

Not sure why you would want to do this, but anyway...nearestpwr2=int(pow(2,int(log(splith)/log(2)+1)))
Shouldn't that be int(pow(2,ceil(log(splith)/log(2))))?
Otherwise if the height is a power if 2, you will go to the next higher one instead.

Also, the number of unweave steps should depend on the exponent, ie ceil(log(splith)/log(2)), not hardcoded at 6.
A job for a GScript loop? ;)

[jmac698]

unweave steps uses splith to determine if unweave should be applied. It is called 6 times manually to avoid recursion.

[Gavino]

Yes, I know, but that puts an arbitrary upper limit on the number of splits.
Seems a natural candidate for a GScript 'for' or 'while' loop. ;)

[TheSkiller]

Actually I always wanted to do that too, but in addition I wanted to put the lines back together in a new order for each frame. The result is the original frame but with it's lines swapped all over the place. So for example line 384 gets swapped with line 197, line 412 gets swapped with 59 and so on - "line shuffling".
This is actually just the way pay TV channels were scrambled with "Syster" (in Germany) back in the days.
I thought it would be pretty cool to do that, especially if the way the lines get swapped is "predictable" so that it can be completely undone (descrambled so to say). :cool:

Too bad I don't even know where to start when it comes to rearranging the lines, I mean it could be done with StackVertical(), but... :rolleyes:

[Gavino]

I want to split a video in every one of it's lines, and I'm not gonna write 480 crop statements ;)
Actually, it's very simple.
A single Crop is enough if you put it in the right place. :)
# Split a clip into strips of 'n' lines.
# Individual lines can be obtained by setting n=1, but note n must be even for YV12
# Requires GRunT
function SplitLines(clip c, int n) {
Assert(c.height%n == 0, "Clip height not a multiple of 'n'")
Assert(!(c.IsYV12() && n%2==1), "'n' must be even for YV12 clips")
nStrips = c.height/n
c = c.ChangeFPS(nStrips*Framerate(c)).AssumeFPS(c) # Repeat each frame 'nStrips' times
BlankClip(c, height=n) # template for ScriptClip result
GScriptClip("c.Crop(0, (current_frame%nStrips)*n, 0, n)", args="c, nStrips, n")
}

It could also be done without GRunT (ie with the standard ScriptClip) by using global variables, but that stops the function being used more than once in a script.

[jmac698]

That's awesome, thanks!
Now put it back together again :)

[jmac698]

ok, here it is:

# Split a clip into strips of 'n' lines, you can do some processing then, and merge back together
# Individual lines can be obtained by setting n=1, but note n must be even for YV12
# Requires GRunT,GScript
v=colorbars.trim(0,3)
h=v.height
splitlines(v,2)#split to lines of height 2
mergelines(h)#merge until height=h
function SplitLines(clip c, int n) {
Assert(c.height%n == 0, "Clip height not a multiple of 'n'")
Assert(!(c.IsYV12() && n%2==1), "'n' must be even for YV12 clips")
nStrips = c.height/n
c = c.ChangeFPS(nStrips*Framerate(c)).AssumeFPS(c) # Repeat each frame 'nStrips' times
BlankClip(c, height=n) # template for ScriptClip result
GScriptClip("c.Crop(0, (current_frame%nStrips)*n, 0, n)", args="c, nStrips, n")
}

function MergeLines(clip c, int h) {
l=c.framecount
n=h/c.height#n is number of frames to merge, e.g. 480 split to 2line has n=240
GScript("""
for (j=0,l/n-1) {
c.trim(j*n,j==0?-1:j*n)
for (i=1, n-1) {
stackvertical(c.trim(i+j*n,i+j*n))
}
b=(j==0)?last:b+last
}
""")
b
}

Is it gonna run out of memory or anything?
It's slow, is there any other ways?

[hanfrunz]

mmmh i think this whole task should be done inside a filter. That would be a lot faster. I do not have the time to do it right now, maybe somebody is quicker than me :)

hanfrunz

[jmac698]

I found a perfectly fast way to do it already :)
But I found a strange bug, with version 2.

# Split a clip into strips of 'n' lines, you can do some processing then, and merge back together
# Individual lines can be obtained by setting n=1, but note n must be even for YV12
# Requires GRunT,GScript
v=colorbars.trim(0,10)
h=v.height
splitlines(v,2)#split to lines of height 2
mergelines2(h,blankclip(height=480))#merge until height=h, note why do I need blankclip(height=480)?
function SplitLines(clip c, int n) {
Assert(c.height%n == 0, "Clip height not a multiple of 'n'")
Assert(!(c.IsYV12() && n%2==1), "'n' must be even for YV12 clips")
nStrips = c.height/n
c = c.ChangeFPS(nStrips*Framerate(c)).AssumeFPS(c) # Repeat each frame 'nStrips' times
BlankClip(c, height=n) # template for ScriptClip result
GScriptClip("c.Crop(0, (current_frame%nStrips)*n, 0, n)", args="c, nStrips, n")
}

function MergeLines(clip c, int h) {
l=c.framecount
n=h/c.height#n is number of frames to merge, e.g. 480 split to 2line has n=240
GScript("""
for (j=0,l/n-1) {
c.trim(j*n,j==0?-1:j*n)
for (i=1, n-1) {
stackvertical(c.trim(i+j*n,i+j*n))
}
b=(j==0)?last:b+last
}
""")
b
}

#Note: this version gives a yellow tinge which I can't explain - bug somewhere? Also slow to start and per frame
function MergeLines2(clip c, int h, clip template) {#on a guess that stackvertical is too slow
l=c.framecount
h2=c.height
n=h/c.height#n is number of frames to merge, e.g. 480 split to 2line has n=240
template=template.trim(0,-1)#use just 1 frame at a time
GScript("""
for (j=0,l/n-1) {
template
for (i=0, n-1) {
overlay(last,c.trim(i+j*n,i+j*n),y=i*h2)
}
b=(j==0)?last:b+last
}
""")
b
}

#this version is fast per frame
function MergeLines3(clip c, int h, clip template) {#on a guess that stackvertical is too slow
l=c.framecount
h2=c.height
n=h/c.height#n is number of frames to merge, e.g. 480 split to 2line has n=240
template=template.trim(0,-1)#use just 1 frame at a time
GScript("""
for (j=0,l/n-1) {
template
for (i=0, n-1) {
layer(last,c.trim(i+j*n,i+j*n).isrgb32?resetmask(c.trim(i+j*n,i+j*n)):c.trim(i+j*n,i+j*n),y=i*h2,level=257)
}
b=(j==0)?last:b+last
}
""")
b
}


I'm having some very odd bugs here. G, I think your plugins are buggy :(
First, merge2 gives an odd color. Next, merge3 crashes sometimes as I step back and forth through it. Next, why does blankclip get corrupted?

[Gavino]

I'm having some very odd bugs here. G, I think your plugins are buggy :(
It's always a possibility, but I really don't think so in this case.
First, merge2 gives an odd color. Next, merge3 crashes sometimes as I step back and forth through it. Next, why does blankclip get corrupted?
I think the odd color is probably due to a bug in Overlay. I need to investigate more, but it only seems to affect RGB and with SplitLines(n) where n<=8.

merge3 crashing over repeated stepping could be running out of memory. Try with Task Manager open showing memory usage.

You have a bug in merge3 with RGB32 clips (which is probably what is causing your blankclip problem). Layer uses the alpha channel in RGB32 for transparency and ColorBars has a blank (ie transparent) alpha channel, so all the added layers (derived from the original ColorBars) are transparent. At the start of MergeLines3, you need to add
c = c.IsRGB32 ? c.ResetMask() : c # ensure RGB32 opaque

[Gavino]

After a bit of experimentation, I've tracked down the source of the yellow tinge - it's Overlay.
Since Overlay works internally in YUV, an RGB input clip undergoes a RGB->YUV->RGB conversion.
This conversion is not lossless and, according to the documentation, the one done in Overlay is of "slightly worse quality".
Since MergeLines2 calls Overlay repeatedly to construct each frame, color degradation builds up, particularly noticeable with initially grey sections.
The effect gets stronger with more repetitions, which is why it is most noticable with SplitLines(2) as input.
Moral of the story - Overlay is best avoided for RGB, especially if used repeatedly (use Layer instead).

Performance and memory use in all the MergeLines functions can be improved by building each frame in parallel, replacing Trim and Splice by SelectEvery(). Here's a revised MergeLines():
function MergeLines(clip c, int h) {
n=h/c.height #n is number of frames to merge, e.g. 480 split to 2line has n=240
c.SelectEvery(n)
GScript("""
for (i=1, n-1) {
stackvertical(c.SelectEvery(n, i))
}
""")
}
The same change could be made to the Overlay and Layer variants, but I think using StackVertical should be more efficient and works for all colorspaces.

[jmac698]

I'm getting really inconsistent results. I think once I trigger the bug, everything else is going to act weird and unrepeatable.
Your tip helped, but still the version with overlay uses increasing amounts of memory as you step through each frame. The same code with layer uses the same, small amount of memory.

[jmac698]

Great detective work, however I'll call that a bug - overlay should mix in rgb if both inputs are rgb, and I just proved it can be an issue.
Trying to make a simple demo, I ran into another bug:

#crop bug
exposebug=false#set to true to crash
n=exposebug?241:240
colorbars
crop(0,n,0,n)
#~ File "pyavs.pyo", line 310, in _GetFrame
#~ File "avisynth.pyo", line 139, in BitBlt
#~ WindowsError: exception: access violation reading 0x02DDFFE0

[Gavino]

Your tip helped, but still the version with overlay uses increasing amounts of memory as you step through each frame. The same code with layer uses the same, small amount of memory.
Overlay is a known memory hog, so I think the StackVertical version should be best for overall performance.
Great detective work, however I'll call that a bug - overlay should mix in rgb if both inputs are rgb, and I just proved it can be an issue.
Not really a bug, more a limitation, since it was designed to work that way. If there is a bug, it is that the RGB<->YUV conversion is sub-optimal.
Trying to make a simple demo, I ran into another bug:

#crop bug
exposebug=false#set to true to crash
n=exposebug?241:240
colorbars
crop(0,n,0,n)
Yes, it should give an error instead of crashing.
Does the right thing for YUV input.

[jmac698]

This is driving me nuts. I tried your version, but I can't say for sure if it's fast, too many weird things going on. I'm getting "Not a clip" which makes no sense:

v=colorbars.trim(0,10)
h=v.height
splitlines(v,2)#split to lines of height 2
MergeLines(v,h)#merge until height=h, note why do I need blankclip(height=480)?

function SplitLines(clip c, int n) {
Assert(c.height%n == 0, "Clip height not a multiple of 'n'")
Assert(!(c.IsYV12() && n%2==1), "'n' must be even for YV12 clips")
nStrips = c.height/n
c = c.ChangeFPS(nStrips*Framerate(c)).AssumeFPS(c) # Repeat each frame 'nStrips' times
BlankClip(c, height=n) # template for ScriptClip result
GScriptClip("c.Crop(0, (current_frame%nStrips)*n, 0, n)", args="c, nStrips, n")
}

function MergeLines(clip c, int h) {
n=h/c.height #n is number of frames to merge, e.g. 480 split to 2line has n=240
c.SelectEvery(n)
GScript("""
for (i=1, n-1) {
stackvertical(c.SelectEvery(n, i))
}

""")

}

I have to add this to get it to work:

temp=last
""")
temp
}

yet this works fine:

#Requires gscript
v=colorbars(height=2).trim(0,240*20-1)
h=v.height
MergeLines(v,480)

function MergeLines(clip c, int h) {#as suggested by Gavino about 4 frames/sec, 449mb memory
n=h/c.height #n is number of frames to merge, e.g. 480 split to 2line has n=240
c.SelectEvery(n)
GScript("""
for (i=1, n-1) {
stackvertical(c.SelectEvery(n, i))
}
""")
}

Does it happen for you too?

[Gavino]

I'm getting "Not a clip" which makes no sense:

v=colorbars.trim(0,10)
h=v.height
splitlines(v,2)#split to lines of height 2
MergeLines(v,h)
...
function MergeLines(clip c, int h) {
n=h/c.height #n is number of frames to merge, e.g. 480 split to 2line has n=240
c.SelectEvery(n)
GScript("""
for (i=1, n-1) {
stackvertical(c.SelectEvery(n, i))
}
""")
}
It should just be MergeLines(h), as v is still the original clip.
Result is that n=1 inside the function, so the loop range is empty and so GScript (and hence the function) returns a 'void' result.

[jmac698]

oh, thank goodness! I'm glad your still online, I really wanted to do some actual video processing with this today.:thanks:

[jmac698]

I did it! I could swear I'm looking at a VHS SLP recording, amazing...

#VCR line jitter simulator ver. 0.1 by jmac698
#requires both gscript and grunt
colorbars(pixel_type="YUY2").trim(0,10)
pointresize(width,height*4)#get line resolution by working around min height 4 limitations
h=height
splitlines(4)#split to lines of height 2 - note the jitter function requires height>=4
jitter
#return last
MergeLines(h)#merge until height=h, note why do I need blankclip(height=480)?
blur(1).blur(1).blur(1)
pointresize(width,height/4)
temp=last
u=temp.utoy
v=temp.vtoy
u=u.bilinearresize(30,height).bilinearresize(320,height)#it's actually better than this, placement of transitions is finer
v=v.bilinearresize(30,height).bilinearresize(320,height)
ytouv(u,v,temp)

function SplitLines(clip c, int n) {
Assert(c.height%n == 0, "Clip height not a multiple of 'n'")
Assert(!(c.IsYV12() && n%2==1), "'n' must be even for YV12 clips")
nStrips = c.height/n
c = c.ChangeFPS(nStrips*Framerate(c)).AssumeFPS(c) # Repeat each frame 'nStrips' times
BlankClip(c, height=n) # template for ScriptClip result
GScriptClip("c.Crop(0, (current_frame%nStrips)*n, 0, n)", args="c, nStrips, n")
}

function MergeLines(clip c, int h) {
n=h/c.height #n is number of frames to merge, e.g. 480 split to 2line has n=240
c.SelectEvery(n)
GScript("""
for (i=1, n-1) {
stackvertical(c.SelectEvery(n, i))
}
""")
}

function shift(clip v, int sx, int sy) {
v#shift a video sx, sy pixels, sx<0 is left, sy<0 is down
pointresize(width*2,height*2)#to allow 1 pixel shifts in YV12
sx=sx*2
sy=sy*2
l=(sx<0)?-sx:0
r=(sx<0)?0:sx
t=(sy<0)?0:sy
b=(sy<0)?-sy:0
crop(l,t,-r,-b)
addborders(r,b,l,t)
bilinearresize(v.width,v.height)
}

function jitter(clip v) {#note: a good tape could typically be +-3 without a TBC, distribution is not realistic
ScriptClip(v,"""
shift(rand(6)+4,0)
""")
}

[Wilbert]

This conversion is not lossless and, according to the documentation, the one done in Overlay is of "slightly worse quality".
What do you mean by "slightly worse quality"? Less accurate/precision?

[Gavino]

What do you mean by "slightly worse quality"? Less accurate/precision?
I don't know - I was only quoting from the wiki page (http://avisynth.org/mediawiki/Overlay).
That bit has been there since the earliest version of the page
(and, according to the history, you were the author :)).

[jmac698]

That's hilarious :)

[Wilbert]

I didn't write that. @jmac698, half of the history is missing (namely every edit from the old wiki; that holds for all pages not just for Overlay). Sh0dan wrote the original version, but it's very possible that that comment is added at a later stage. I guess we need to look at it.

[IanB]

Overlay is showing it's legacy history. It started life 29/12/2003. Back then there was no YV24 only RGB, YUY2, a relatively new YV12 and what filters were prepared to support internally.

Internally Overlay is strictly a planar YUV 4:4:4 filter, so it had to support all it's own format transforms and memory management. It was intended as the YV12 super implementation of Layer. Layer is for YUY2 and RGB32 formats.

Every instance of overlay maintains upto 3, YUV 4:4:4 full size frame buffers. If you iterate lots of these into existence then you use lots of memory.

The RGB <-> YV24 code is C only and uses the same standard algorithm still used by Avisynth but without any chroma subsampling. So there is only the normal RGB to YUV 8 bit precision loss you would get anyway but nothing worse. If you are using only RGB data then you should be using Layer. Overlay's RGB conversions are intended to allow the overlay and mask clips to be RGB and converted without chroma subsampling. The source clip inherit the same conversions set because it uses the same code, but you really should not be using Overlay for all RGB data.

const int b = srcP[RGBx+0];
const int g = srcP[RGBx+1];
const int r = srcP[RGBx+2];

const int y = (cyb*b + cyg*g + cyr*r + 0x108000) >> 16; // 0x108000 = 16.5 * 65536
const int scaled_y = (y - 16) * int(255.0/219.0*2048+0.5);
const int b_y = (b << 11) - scaled_y;
const int r_y = (r << 11) - scaled_y;

dstY[x] = y;
dstU[x] = (b_y * ku + 0x20000000)>>22; // 0x20000000 = 128 << 22
dstV[x] = (r_y * kv + 0x20000000)>>22;and const int Y = (srcY[x] - 16) * int(255.0/219.0*65536+0.5);
const int U = srcU[x] - 128;
const int V = srcV[x] - 128;

dstP[xRGB+0] = ScaledPixelClip(Y + U * cbu);
dstP[xRGB+1] = ScaledPixelClip(Y - U * cgu - V * cgv);
dstP[xRGB+2] = ScaledPixelClip(Y + V * crv);
_

[jmac698]

I think I could write a patch for that, but not for a month at least...

[Gavino]

The RGB <-> YV24 code is C only and uses the same standard algorithm still used by Avisynth but without any chroma subsampling. So there is only the normal RGB to YUV 8 bit precision loss you would get anyway but nothing worse.
The strange thing is it does seem to be much worse, as this test demonstrates.
function OverlayN(clip c, int n) {
n==0 ? c : c.Overlay(BlankClip(c, height=2), y=64).OverlayN(n-1)
}

function ConvertN(clip c, int n) {
n==0 ? c : c.ConvertToYUY2().ConvertToRGB32().ConvertN(n-1)
}

function Test(clip c, int n) {
StackHorizontal(c.OverlayN(n).Subtitle("Overlay"), c.ConvertN(n).Subtitle("Convert"), c.Subtitle("Orig"))
}

grey = $b4b4b4 # as used in ColorBars
BlankClip(10,128,128, color=grey)
Animate(0, 9, "Test", 0, 27)
Notice how after several repetitions with Overlay a yellow tinge progressively appears, while with standard RGB-YUY2-RGB conversions the result looks the same as the original.

[jmac698]

I dunno, I'm finding this version shorter and easier:

MergeLines(h/4,h/4)
function MergeLines2(clip c, int n,int i) {
i<2?c.SelectEvery(n):stackvertical(MergeLines2(c,n,i-1),c.SelectEvery(n, i))
}

[TheSkiller]

@jmac698 Tried your jitter simulator, very nice! :)

I realized that the line-shuffling that I mentioned a few posts back could be archived if the resulting clip of SplitLines() gets it's frames mixed up before the lines/frames get put back together with MergeLines(). Maybe it could also be done in MergeLines itself, because I'm not sure if it is possible to simply mix frames in a clip. Although the mixing of the frames should be predictable so that it can be undone (which is basically the point), I'd appreciate any help in mixing the lines. :)
Any ideas?

[jmac698]

Yep, I can do that, I think the easiest is with select(n,45,32...) you can have a long list there. It's really simple as long as select can handle the length.
I'll try.

[IanB]

There is no rounding in this code :- dstU[x] = (b_y * ku + 0x20000000)>>22; // 0x20000000 = 128 << 22
dstV[x] = (r_y * kv + 0x20000000)>>22;It should be "+ 0x20200000)>>22; // 0x20200000 = 128.5 << 22"

Also the reciprocal divisions are a bit light on for bits.

The code probably should be using the generic RGB to YV24 code now anyway.

[jmac698]

skiller
here ya, don't have time to finish right now but it will work

#German scrambler ver. 0.5 by jmac698
#Swap lines in a video. There is a single shuffle order for the whole video, which is saved as a key.
#Requires GrunT
fn="scramblekey.txt"
s=scramblegenerator(480)
WriteFileStart(blankclip,fn, "s")
colorbars
#MPEG2Source("G:\project001a\sampleproblem\sample1.d2v", cpu=0).converttoyuy2
splitlines(1)
eval(s)
MergeLines(480)

function SplitLines(clip c, int n) {#duplicates then crops each copy in a different spot
Assert(c.height%n == 0, "Clip height not a multiple of 'n'")
Assert(!(c.IsYV12() && n%2==1), "'n' must be even for YV12 clips")
nStrips = c.height/n
c = c.ChangeFPS(nStrips*Framerate(c)).AssumeFPS(c) # Repeat each frame 'nStrips' times
BlankClip(c, height=n) # template for ScriptClip result
GScriptClip("c.Crop(0, (current_frame%nStrips)*n, 0, n)", args="c, nStrips, n")
}

function MergeLines(clip c, int n) {MergeLines2(c,n,n)}

function MergeLines2(clip c, int n,int i) {
i<2?c.SelectEvery(n):stackvertical(MergeLines2(c,n,i-1),c.SelectEvery(n, i))
}

function scramblegenerator(int n) {#Create a SelectEvery(x,x,x,x...) command
s=countstring(n)
s=shuffle(s,n-1,n-1)
"SelectEvery("+dec(n)+leftstr(s,strlen(s)-1)+")"
}

function countstring(int n) {countstring2(n,n)}

function countstring2(int n, int i) {#create a decimal countup of n in 3 digits like 000,
i<2?dec(n-i):dec(n-i)+countstring2(n,i-1)
}

function dec(int n) {#Create a 3 digit string of n
string(n,"%03.f")+","
}

function shuffle(string s, int n, int i) {#Shuffle a string in groups of 4 characters, call with i=count
i<2?swap(s,i,rand(n)):swap(shuffle(s,n,i-1),i,rand(n))#Swap with a random index less than current
}

function swap(string s, int i1, int i2) {#Swap subtrings i1,i2 in grid of 4 characters
t=i1
i1=i1>i2?i2:i1#i1 is changed now
i2=t>i2?t:i2
i1==i2?s:lefts(s,i1)+extract(s,i2)+mids(s,i1,i2)+extract(s,i1)+rights(s,i2)
}

function extract(string s, int i) {#Extract 4 char substring at index i
midstr(s,i*4+1,4)
}

function lefts(string s, int i) {#Extract everything to the left of i*4
leftstr(s,i*4)
}

function mids(string s, int i1, int i2) {#Extract between i1 and i2*4
midstr(s,i1*4+5,(i2-i1)*4-4)
}

function rights(string s, int i) {#Extract all to right of i*4
midstr(s,i*4+5,strlen(s)-4-i*4)
}



ps a true random shuffle is swapping current with one random index higher

UPDATE: better but still not working
UPDATE2: It's working

[TheSkiller]

Cool, but I'm getting a syntax error when trying to run the script:


function shuffle(string s, int i) {#Shuffle a string in groups of 4 characters
i<2:swap(i,rand(n)):shuffle(s,i-1)
}

The error message complains about that spot. Something seems to be wrong there.

[Gavino]

Replace ':' by '?'.

[clacker]

try

i<2?swap(i,rand(n)):shuffle(s,i-1)

Crap! Sorry Gavino, posted while you were posting.

[jmac698]

My problem is getting random numbers. They are the same at compile level, you need to generate them in runtime.

[Gavino]

Can you explain what you mean?
Each call to rand() generates a different number (and if called inside a function, generates a different number on each call to that function).

[jmac698]

It's ok I solved it. By now why can't I write s to a file?
The key isScript error: expected a , or )
((null), line 1, column 254)
See version 0.4 above.

What I mean is, the rand has to be in a scriptclip to get a different sequence per frame.

[Gavino]

WriteFileStart(blankclip,fn, """ "The key is" """,s)
should be
WriteFileStart(blankclip,fn, """ "The key is" ""","s")

[TheSkiller]

Looking great so far. Yes, the shuffle order really should change per frame.

[Gavino]

An observation: I think the key output to the file is the one used to 'encode' the video.
For 'decoding', don't you actually need the inverse of this?
To recreate the first line, you need to know where it 'went to', not where it 'came from'.
Or am I just confused? :confused:

[TheSkiller]

Just tried it on a small scale, here are my thoughts:

line - content
original:
1 - a
2 - b
3 - c
4 - d

(line -> gets content from line)
(1 -> 3) (2 -> 1) (3 -> 4) (4 -> 2) #this is basically the key

scrambled:
1 - c
2 - a
3 - d
4 - b

Let's do the same thing again
(1 -> 3) (2 -> 1) (3 -> 4) (4 -> 2)

descrambled:
1 - d
2 - c
3 - b
4 - a

Basically correct, but inverted, so yeah, I guess the key has to be inverted. But if the result is just upside-down, then a simple FlipVertical() should do, shouldn't it? :confused:

[Gavino]

Never draw general conclusions from a single example.
Other permutations do not have this property.
Eg (1->4, 2->3, 3->2, 2->1) when applied twice reproduces the original input, not an upside down version.
I'm pretty sure there are others that do neither of these things.

[jmac698]

Aha, I thought of that - it seems you could take the scramble and sort it. However, it might be easier to generate the answer key while your generating the scramble key. I'm just not finished yet. What if I had the order 1 5 3 2 4 the reverse is 1 4 3 5 2 let me think more.

[jmac698]

Wow, you just posted. Yeah, I had the original impression along other lines too, that it would be invertible as is, but no, only a sort will get it back. You could force it to have certain properties howevre. I was also thinking of making my own RNG so I could supply just one x bit key, that's even easier ;)

[TheSkiller]

jmac698, I'd really appreciate it if you would finish this script. :)

[jmac698]

Sure, been sick and not feeling much like scripting, more like sleeping :(

[-Vit-]

I know the OP was about splitting into lines, but if you just want a reversible encryption script then use just use AddGrainC random numbers as seeds for a simple random number generator on each pixel. It's not so useful though because encoding random noise is a worst-case scenario.

EDIT: At first I thought the encrypted vid would need lossless encoding, but a quick try and it does kinda work with x264 compression. With a decent CRF the decryption works OK - vid becomes noisy as the CRF increases. But in any case the compression rate is terrible: 1Gb for 2 or 3 minutes of SD!!! I imagine the same compression problem would affect the split into lines idea, just not so badly.

[Updated code...]

c = YourSource( "your.vid" ) # Whatever source filter

# Use same key to encrypt / decrypt, range is -2147483647 to 2147483647, use a key with many digits to deter brute force attack
en = c.Encrypt( 867144283 )
de = en.Decrypt( 867144283 ) # Try using wrong decryption key and see the output

stackhorizontal( en.Subtitle("Encrypted"), de.Subtitle("Decrypted") )

# Key range is -2147483647 to 2147483647
function Encrypt( clip Input, int Key )
{
random = BlankClip( Input, color=$808080 ).AddGrainC( 32767, 32767, seed=Key )
return mt_lutxy( Input, random, "x y + 256 %", U=3,V=3 )
}


function Decrypt( clip Input, int Key )
{
random = BlankClip( Input, color=$808080 ).AddGrainC( 32767, 32767, seed=Key )
return mt_lutxy( Input, random, "x 256 + y - 256 %", U=3,V=3 )
}


To crack the encryption with brute-force you could create a vid that tried a different key for each frame. It would take several weeks to watch and go through all the keys (less with multiple machines). To make that totally unfeasable use a 2-pass version. EDIT: This one is also more statistically secure:


# Each key range is -2147483647 to 2147483647
# Slightly more secure versions: cycling though all the keys is unfeasable with two large keys
function Encrypt2( clip Input, int Key1, int Key2 )
{
random1 = BlankClip( Input, color=$808080 ).AddGrainC( 32767, 32767, seed=Key1 )
random2 = BlankClip( Input, color=$808080 ).AddGrainC( 32767, 32767, seed=Key2 )

pass1 = mt_lutxy( Input, random1, "x y + 256 %", U=3,V=3 )
return mt_lutxy( pass1, random2, "x y + 256 %", U=3,V=3 )
}


function Decrypt2( clip Input, int Key1, int Key2 )
{
random1 = BlankClip( Input, color=$808080 ).AddGrainC( 32767, 32767, seed=Key1 )
random2 = BlankClip( Input, color=$808080 ).AddGrainC( 32767, 32767, seed=Key2 )

pass1 = mt_lutxy( Input, random2, "x 256 + y - 256 %", U=3,V=3 )
return mt_lutxy( pass1, random1, "x 256 + y - 256 %", U=3,V=3 )
}

[jmac698]

ha - I've done this before. I did something even more clever though, I made a decryption video and encrypted video and the addition of the two would be the clear video. So all you have to do is plug two wires together and connect to your TV. Each signal looks like noise.
I did something further, put two videos into one. Each video has to be greyscale. The videos are stored in the color channels. The decryption involves simple wiring again, this time with component video cables. For old movies, this is something even practical - two complete mono soundtracks and two complete video tracks in one DVD, with a video distribution amplifier and a switchbox you can switch between them.
And yes you can see a ghost image in the noise between the two of them, due to compression artefacts, but quite watchable.

I think cracking this technique is pretty simple though, any black of black screen basically reveals the pattern, you can try possibilibies and check of the ones with the best pixel correlation, etc.

I have a challenge for you, how to decrease correlation of difference of two videos? I mean x and y are clear videos, create a signal z=f(x)-f(y), to recover x=g(z-f(y)), g(x)=f'(x)
(I think.. )

[-Vit-]

I think cracking this technique is pretty simple though, any black of black screen basically reveals the pattern, you can try possibilibies and check of the ones with the best pixel correlation, etc.
You mean it's easy to crack your splitting scheme, or the one I posted above?

I don't think knowing where there was a black screen would help an attacker against the method I just posted. All you would find was the random pattern for that frame [not even that, you'd get the moduluses of the LCG]. You wouldn't know the pattern for the next frame without knowing the seed. There might be a mathematical analysis if you could get several frames of random data but that's pretty esoteric. Bigger problem is the limited random cache in AddGrain.

As to the splitting method, I would be worried about statistical analysis of the individual streams, even before we think about correlation. The [snip] embedding of random data in my version forces the statistical properties of the encrypted form to be almost indistinguishable from random. I don't think a subdivision could ever do that no matter how clever the split. For example, aren't dark areas always going to be dark with your system?

[jmac698]

You might find this interesting:
http://www.techmind.org/vdc/
videocrypt decoded

[-Vit-]

Thanks. A good illustration that encryption schemes that rely merely on cutting up the image are amenable to brute-force image analysis. Per-pixel randomization can't be attacked that way, as long as it's done properly there should be no correlation to find in the encrypted pixels. I recently rewrote AddGrainC, for a completely different reason - to make it use less memory - I never posted it because it didn't really save that much memory. But this experiment interested me - I might add the option to output cryptographically secure random data for this kind of purpose. I've already based it on the Mersenne Twister random number generator, which afaik can be trivially turned into a secure sequence. I'd have to drop the cache, which would make that mode slower though...

[TheSkiller]

Vit, the encryption script you posted works very well, just tried it, thanks for that.
This kind of encryption is safe and easily reversible.

However, I'm still more interested in the original line shuffling type encryption "Syster" which is by the way related to Videocrypt (Syster is basically the predecessor). I like the way the video is unwatchable, yet occasionally you can still detect one or the other original picture structure very distantly. I know it's not really safe due to brute force hacking, but it surely is safe enough for the average Joe, and, you gotta admit that, it's pretty cool.