A man claims to store multiple movies on 64kb

[scharfis_brain]

the formala, that greates the hash , and then is able to recreate the file from that hash will be at least as large as the file itself -> no compression gain.

[stephanV]

Originally posted by callmeace
Just like maths, the result is computed and there is only one answer to the sum (that is the important concept). Essentially the complex formula works like so 'if the input = 'this' then the result must = 'that'.
That's not just like maths. While 2^2 = 4, the square root of 4 can be 2 and -2, so reversibility would fail. A derivative of a formula has only one answer, but an integral can in principle have an infinite amount of answers. Math is reversible yes, but in above cases there's no way of telling where you are coming from, unless you start to exclude certain inputs, which would the destroy the compression.


Yes, the formula is complicated, but feasible. It does not work like common used methods which generate an compressed archive. It generates a character string derived from the contents of the file and unique to it, and by reversing can recreate the file from the string by determining what it must have been derived from.

As said before, the string created has to be just as unique as the input, and therefor should contain just as many bits. There is just no way you can represent all possible combinations of a certain amount of bits, with a lower amount of bits by using a conversion formula. With a codebook? Yes. By using a formula? No.

[callmeace]

Originally posted by stephanV
That's not just like maths. While 2^2 = 4, the square root of 4 can be 2 and -2, so reversibility would fail. A derivative of a formula has only one answer, but an integral can in principle have an infinite amount of answers. Math is reversible yes, but in above cases there's no way of telling where you are coming from, unless you start to exclude certain inputs, which would the destroy the compression.....

In Maths there is only one correct answer to a sum; It's true :) what you have presented is not differing answers to a sum where a formula has been defined.

I liken the character string when inputted to the formula being effectively maths, and the file is able to be recreated as 'the original file must be this'

[stegre]

Originally posted by callmeace Just like maths, the result is computed and there is only one answer to the sum (that is the important concept)... Yes, the formula is complicated, but feasible. It does not work like common used methods which generate an compressed archive.I think I understand what you're describing - the compressed file is essentially the complex "equation" itself, as opposed to a relatively simple algorithm that reads "data" to create the output. What you are describing, if I understand correctly, is actually the controversial "fractal image compression", which I see was briefly mentioned once in a post by Phoenyx (http://forum.doom9.org/showthread.php?s=&postid=545167&highlight=fractal#post545167) earlier in this thread. The guy's name he's trying to remember is Michael Barnsley, his company is or was Iterated Systems Inc, and he has patents, books, and working programs - some of this dating back to the early 90's. It's all rather controversial, though. More info here (http://fractals.iut.u-bordeaux1.fr/sci-faq/compression.html).

[stephanV]

Originally posted by callmeace
In Maths there is only one correct answer to a sum;
this is just not true, in maths there is very often not only one answer.

y = f(x), where f(x) = sqr(x) has 2 answers for y for each x (except when x = 0). In maths we normally ignore the negative value, but that is not to say it is not a valid answer of course.

[callmeace]

If you have defined the 'rules' (in simple terms i.e. like BODMAS) for a formula then there is only one correct answer to a mathematical sum

[callmeace]

Okay, I have considered another way to further try and explain how the 'compression' works. My apologies also for having failed to give the details coherently prior, I do not mean also to cause offence in any rebuffs to criticisms of this theory (so apologies). Thankyou for the link stegre


So, this is a very, very simplified way of how it would work (the basic idea)

Imagine you had some data which is a long number (maybe even a long sequence of binary code). Of course the number is unique in the sense that only it is that number (please I hope you grasp that). So you put that number into a formula and it breaks it down and produces a character string. Now think of that character string as being an equation which when executed via the formula would give you back the original number. Basically it's a sum. The rules of the formula would have been defined so that it is the case that there is only one correct answer. You can have incredibly long numbers represented as relatively short equations.

That is hardly revolutionary, but of course it's a much much more complex task than this in that the actual breakdown via the formula would certainly be very advanced mathematically (i.e. not just simple multiplication and use of powers)

That's very simplified, but I hope I did a reasonable job of explaining how the actual concept would work. Just imagine what a dedicated and very skilled mathematician and coder could achieve over 2 decades. Huge amounts of (probably binary) data can be broken down into just a string of characters which when put back through the formula are calculated back to reassemble the orginal data, This is far beyond 'standard' ways of compression that are currently popular. (The complexity of the formula does not mean that it has to be that big codewise)

[stegre]

This presented an intriguing possibility. If, in the forward direction, fractal mathematics is good for generating natural looking images, then, in the reverse direction, could it not serve to compress images? Going from a given image to an Iterated Function System that can generate the original (or at least closely resemble it), is known as the inverse problem. This problem remains unsolved.

Barnsley, however, armed with his Collage Theorem, thought he had it solved. He applied for and was granted a software patent and left academia to found Iterated Systems Incorporated (US patent 4,941,193. Alan Sloan is the co-grantee of the patent and co-founder of Iterated Systems.) Barnsley announced his success to the world in the January 1988 issue of BYTE magazine. This article did not address the inverse problem but it did exhibit several images purportedly compressed in excess of 10,000:1. Alas, it was not a breakthrough. The images were given suggestive names such as "Black Forest" and "Monterey Coast" and "Bolivian Girl" but they were all manually constructed. Barnsley's patent has come to be derisively referred to as the "graduate student algorithm.

"Graduate Student Algorithm"
o Acquire a graduate student.
o Give the student a picture.
o And a room with a graphics workstation.
o Lock the door.
o Wait until the student has reverse engineered the picture.
o Open the door.

Attempts to automate this process have met little success. As Barnsley admitted in 1988: "Complex color images require about 100 hours each to encode and 30 minutes to decode on the Masscomp [dual processor workstation]." That's 100 hours with a _person_ guiding the process.John Kominek, in an excerpt from comp.compression FAQ (http://www.faqs.org/faqs/compression-faq/part2/section-8.html)

[callmeace]

I believe that this concept I mention goes beyond that kind of thing stegre - it operates at a raw low level - i.e. it doesn't care and it doesn't matter to it what the data 'represents', it just observes data as being a series of numbers for example.
However, the article is very impressive in it's claims!

[DigitAl56K]

callmeace: I held of posting about the hash-like aspect of your post several days ago, I'm glad others also saw the similarity though.

Here's the problem.

What you are suggesting is that you take some huge amount of data and run it through some algorithm that produces a unique small piece of data that you can run through a reverse algorithm to reproduce the original huge amount of data.

What you have just described by the unique small piece of data can be thought of as a hash, and hashes are not uniquely reversible. We can have a function:

O = f(I) ; where i is the input, f is the hashing algorithm, and o is the output hash

O does not represent a unique result that can be reversed to generate I, it is simply a value that is always generated when f() is applied to I, and f() is usually designed in such a way that is is in fact difficult to generate O for an arbitraty I.

Hashing is not reversible because of hash collisions. It is entirely possible (and in fact likely) that there are are many I's that can generate one O.

Think about it like this: If O is a number from 0 to 999999 (i.e. a million possible combinations), but I can be a huge volume of data that when represented numerically could range 0 to 9999999999999999999999999999999, then you some how have to match 10000000000000000000000000000000 I's to only 1000000 O's. O can not be uniquely reversed.


You can, of course, have f() be some compression algorithm, let's take zip as an example, which can reduce the data in I to a unique O.

O = zip(I)

However, the data in O must be large enough to reverse O to I through the inverse zip:

I = izip(O)

The problem here is that there is no known lossless algorithm that can do this with anywhere near the compression required to achieve this 64Kb miracle.

But we're talking about video here. This doesn't nead to be lossless. However, no lossless algorithm achieves this kind of compression either.

You can talk about some "magical advanced mathematical formula", but in truth you still have to record enough parameters (co-efficients, variables, etc.) to create each frame.

E.g. a formula might look like this:

Pixel(X,Y)=f(X,Y,g(Z)) ; where x and y are the pixel co-ordinates, z is the single parameter we're passing to our magical formula, and g() is the magical formula/algorithm.

When I did my original calculation on this thread, I calculated that you could not even store a single 90 minute movie with one parameter/variable per frame (i.e. "Z") in 64kb, even if the value of the single variable in the formula was limited to the range 0 - 536,870,911. I.e. what this essentially says is that for this magical advanced mathematical formula to exist, there would have to be only 536,870,912 unique frames of video in the entire world (equivelent to ~4000 90 minute movies), and even then, you would not fit it into 64Kb without some further compression.

I'm not trying to suggest there aren't some great video compression algorithms just waiting to be discovered, but some things border so closely on the realm of impossibility that even debating them is almost pointless.

[callmeace]

DigitAl56K
I know this sounds 'oblique' of me, but I can immediately see that (I believe anyway) your belief of it being impossible is not correct. Once again, it is because I have not explained why and even how it works in enough detail. It is so hard to explain it - I will post again when I have thought how to put it down 'better' in a post.

Believe me, the approach to it is not being understood (my fault)

[stephanV]

man, callmeace, don't be so stubborn :)

what you want is really impossible, your formula is just not uniquely reversible unless you exclude certain inputs, which wouldnt make sense as you then can only compress certain kinds of predefined data.

unless you prove the theory right here, i think no one will believe you.

[callmeace]

Okay! let me take another stab at this, here are some items of information which I would like you to bear in mind :)

It is hard for me to communicate how it works to you as a concept because (a)I have noticed most of you have not got away from the common ideas of compression that are in use, I repeat "this way does not work like most of you seem to think" &(b)I am not that good at explaining things like this simply and the formula itself is unknown - I do not have it, I only know the idea and believe it works.

DigitAl56K, I must especially use your post as an example, to repeat - the 'compression' method I refer to does not work like that, it is not done in that way. Your information is valid I'm sure, but it is not an argument against this concept because it is not trying to compress like that.

Here in a differing explanation is how it all would work:

(1)Note that the string of characters (which will be part of the 'archive') is not generated in one stage directly from your input file - the formula it is processed through involves several stages. It is broken down, and you will end up with the character string.

(2)Now, here is the most important thing - the approach to how the 'compression' works - I repeat again, clear your mind of existing ideas because this is not the way that some of you think
As always, this is heavily simplified just for the purposes of myself attempting to convey the concept to yourselves - the maths involved would be complex and advanced and the result of much time and work (years and years worth), so bear that in mind when deciding on the credibility of all this


In the first stage of the input data entering the formula the data is going to be represented as a number. It could be a continuous number but it will more likely be broken up in to smaller sequences of numbers. Mapping is done at this stage only for the purpose of tracking what number sequence belongs to which block of data.
Now, most of you will know what a code-book is. Essentially it's to decipher inputted data that has been 'scrambled' or hidden or represented in a different way or whatever. This is used in the formula (of course it is the rules so the data is not 'meaningless' to the formula as some had thought)

To further clarify and emphasise the above paragraph, the input data has been represented and mapped to long sequences of numbers.

Now here is a simple thing to explain the concept of how it can go through a formula both ways. One way input data is broken down, and the reverse way input character string is rebuilt back up to the original bigger data.
maths says a big number can be represented in fewer characters as an equation like algebra and so on.

If the maths is complex and carefully considered and the mathematician is prepared to dedicate the time it can be done that by use of 'code-books', mapping and breaking big numbers down to sums Much data can go through stages and be finally represented as a character string. But here again is a point. In other methods, sure a hash could be meaningless for the purpose of recreating data, but here it is not! My character string is created by methods involving code-books and mapping, it is not a useless 'hash', it can be processed when fed back into the formula.

I will also point out that I am not denying anyones points or maths in their posts! I am not arguing against what they say, but they were not arguments against this way of compression because you had all seemed to not really understand how this works.

Now, this is maybe not an actual example, This is grossly simplified but it might help explain the concept.
Imagine you had some mapped binary data.
Now imagine that you were literally showing the code as a single big number like 11 trillion 1 hundred thousand billion 10 million and 10 or something :). Well imagine a much much much longer than that. That's what I mean as a concept of representing input data as a number - very long sequences of numbers. Such a long number can be said much shorter by using powers and the like.

Understand that a very advanced and dedicated mathematician will be processing the data through several stages. There will be rules as I've explained. The end data will not be meaningless to the formula when put back into it to recreate the file. I hope noone will deny this point now I've explained. Maths (reversing) can work both ways when there are prompts and rules defined, and neither does data have to be meaningless in those cases.

I almost expect some derision :) and don't blame anyone (my inadequate explanations ;) )but I know most of us probably visit several of the same websites. I have witnessed some very impressive advancements in the last few years and when I look around me I am amazed at some of the technical advancements. Therefore I am more willing to accept what dedication and good use of brains can do especially when people approach things in a brave new way (and commit themselves to a long slog). I believe in this concept strongly, I know it would work.

[fccHandler]

You're saying that we treat the file as one gigantic number, then derive a formula whose answer is that number?

But even for a small file, the number will be too impossibly huge for any human mathematician to work with.

[Manao]

Let's explain once more time. Tell me at which point you disagree with what I say. There's no need for a strong mathematical background to understand :

(1) compression is a process which transforms a file into another file, whose size is inferior to the first file
(2) a file is a finite serie of bits ( let's name it b(i), i = 0..N )
(3) a finite serie of bits can be associated to a unique couple of positive integers by the following formula : b(i), i = 0..N ---> ( N , P = sum(b(i) * 2^i , i = 0..N) )
(4) the formula can be reversed to obtain the finite serie of bits : (N, P) ---> b(i) = (P / 2^i) modulo 2 , i=0..N ( remark that P ranges from 0 to 2^(N+1) - 1 )
(5) a couple of positive integer can be associated to a unique positive number by the formula : (N, P) ---> m = 2^(N+1) - 2 + P ( the uniqueness is because of the previous remark )
(6) there again, the formula can be reversed : m ---> (N, P), where N is the biggest positive integer such as 2^(N+1) <= m + 2, and P is m + 2 - 2^(N+1).

(7) a file can be associated to a unique positive integer ( (2)->(3)->(5) ).
(8) that integer has the following property ( let's call F1 and F2 two files, and M1 and M2 the associated numbers ) : If (size(F1) < size(F2)), M1 < M2.
(9) if we consider all the files whose size is below M, it amounts to consider all positive integer ranging from 0 to 2^(M+2)-2 ( strictly )
(10) compression can then be seen as a function going from positive integers to positive integers.
(11) if it was possible to compress every file, it should be possible to find a function (f) that is defined from [|0..2^(M+2)-2[| = G to [|0..2^(M+2)-2[| \ { at least one integer } = im(f) = H
(12) for the compression to be reversible, the function would have to be surjective ( meaning : for each element of h in H, there is only one element g in G such as f(g) = h ). If it was not so, from the compressed file, you wouldn't be able to recontruct the initial file.
(13) Since G and H are finite spaces, a surjective function is bijective.
(14) f is bijective from G to H, which are finite, so cardinal(G) = cardinal(H), which contradicts (11). So (11) is wrong, so it's not possible.

I hope it's understandable ( it's the first time I write maths in english ), and that there are no mistake.

The false assumption you make in your reasonning is that you claim you can obtain a simple mathematical formula to represent the file. But you never say how you'll write that formula into a file. For example, a very simple formula to code a number N is the following : x = N, which takes as much bits as N to code.

You'll see that I never used any theorem specifics to compression in my explanations, nor do I limit myself to a compression scheme. It's just plain and simple maths.

[callmeace]

You have got the essence of the right path fcchandler, yes! :)

:stage 1 when the big data is inputted is to have the data represented numerically

Mapping of course would be done, in order to have that first huge number broken into sections of smaller numbers where wanted

In fact though, it might be advantageous to leave as many huge numbers as can be dealt with, from the point of view that there are more savings to be had in relation to the size differences to the characters in the equations that add up to it ;)

But really, I am very pleased that you have seen the concept fcchandler. the idea being that Maths adds up to a result and only a result (when using defined rules)in order to both compress and decompress is the essence and this approach is challenging yes, but unobtainable? no!

[fccHandler]

Originally posted by Manao
The false assumption you make in your reasonning is that you claim you can obtain a simple mathematical formula to represent the file. But you never say how you'll write that formula into a file.
I don't think it's a false assumption, though it is far-fetched. The compressed data could be a tokenized string. The tokens (characters) would represent numbers and mathematical functions. We'll still need an executable (decompressor) to interpret the tokens, act on them, and calculate the result of the equation(s). The result of its calculations will be the uncompressed file.

The main obstacle is how to break down an impossibly massive number into a series of simple equations, if that's even possible. And the idea of waiting years and years for such a compressor to work out an equation for my movie doesn't appeal to me. :p

[duartix]

@DigitAl56K:
Even though I didn't quite follow the "believer's" explanations, I can see you are having a bit of a resistance mainly because you're not opening yourself to diferent approaches. Compressions techniques might not be descriptive but constructive, right?

Allow me to shut the sound in the drawer. :p

As I pointed before, I could create a flash movie that would play for 5 days at 400 FPS with infinite resolution, and that well under 64K. Think South Park... :)

You can look at fractal compression as an more advanced constructive approach than Flash. You can have a 300000000 by 200000000 pixel image with half a dozen parameters.

Current compression techniques are meeting their limits as are computer physics, unless we turn our heads to new ways we can never balance an egg on a table... ;)

[callmeace]

I can understand that most of you do doubt, but the only reason that you do is because you are approaching the means of how the calculations would be carried out from the 'wrong angle'

More on that when I get a decent post together
@Manao I will likewise try and answer your post.

I am pleased that so far progress has been made towards perhaps addressing the 'challenge', which can only be started after realising how it could, and indeed that it could, work! :) It's all about realising a concept, though I doubt if anyone is going to sit down for maybe the next decade or so but perhaps it wouldn't take that long with 'open source'? ;)

[fccHandler]

OK, here's my working example of what (I think) callmeace may be talking about...

Let's say we have a file composed of 14 bytes:

0x06, 0x68, 0x02, 0x04, 0xEC, 0xBF, 0x42, 0x28, 0xA3, 0x55, 0x05, 0x0D, 0x36, 0x91

We treat it as a gigantic decimal number:

129,934,811,447,123,020,117,172,145,698,449

We run it through the compressor to discover that it just happens to be the result of the equation 7 to the power 38. We code this as a tokenized string:

0x07, 0xFF, 0x38

This is similar to BCD. Here, we let the byte values 0 to 99 represent decimal integers (or portions thereof), and byte values 100 to 255 represent math functions, operators, or other tokens yet to be defined. (In this case, we'll let 0xFF represent exponentiation.) The decompressor parses the string, performs the requested function(s), and it's output is the original number (file).

Thus 14 bytes have been losslessly reduced to 3. Granted this is a very simple (and forced) example, but my proposal is that one could extrapolate this idea to even larger numbers which may somehow be reduced to very simple equations.

[virus]

Finding that 1 out of 129,934,811,447,123,020,117,172,145,698,449 sequences can be represented by 3 bytes is not compression, at all. Please, let's not confuse "synthesis" with "compression", because they are different beasts.

"Compression" simply means "taking arbitrary data of any length and reduce its size". The amount of reduction depends on the "features" of the source, of course, and it can be zero.

So, you find that this particular number (let's call it N for the sake of simplicity) can be represented by just 3 bytes calculating a power (7^38). How do you plan to code N-1, then? Because you also need it. You also need to code at least the numbers from 0 to N if you want to have a true compression algorithm (better would be "0 to the smallest power of 2 >= N", and even better "0 to infinity").

Simple. You code it as (7^38)-1. That means that you're applying a different operation (power+subtraction), no? Then, to be able to reconstruct the data you also need to code what type of operation you need to perform. Saying that

7, 255, 38, 255, 1

can reproduce the number is plain wrong. You also need to code the "power+subtraction" thing (the "formula type"), and it takes more bits.

And how do you plan to code N - 1600, then?
Simple: (7^38) - (40^2)!

7, 255, 38, 255, 40, 255, 2, of course, plus the overhead to describe what to do with your data (power, power, then subtraction)

And what about N - 1601 then? ... (and so on) ;)

Eventually you'll end up with a formula for every integer in [0, N], but as you can see you'll need several types of formulas. Eventually much more complex than calculating a couple of powers. So you'll have the overhead for the "type" and all the operands. Do you really think that the average code length (computed on all the codewords, not just 1) will be so low?

(BTW 0x38 is 56 in decimal... that's why I've switched to plain decimal my numbers)

[fccHandler]

Originally posted by virus
Finding that 1 out of 129,934,811,447,123,020,117,172,145,698,449 sequences can be represented by 3 bytes is not compression, at all. Please, let's not confuse "synthesis" with "compression", because they are different beasts.
Fine, so let's forget compression and talk about synthesis.

So, you find that this particular number (let's call it N for the sake of simplicity) can be represented by just 3 bytes calculating a power (7^38). How do you plan to code N-1, then? Because you also need it. You also need to code at least the numbers from 0 to N if you want to have a true compression algorithm (better would be "0 to the smallest power of 2 >= N", and even better "0 to infinity").

Simple. You code it as (7^38)-1. That means that you're applying a different operation (power+subtraction), no? Then, to be able to reconstruct the data you also need to code what type of operation you need to perform. Saying that

7, 255, 38, 255, 1

can reproduce the number is plain wrong. You also need to code the "power+subtraction" thing (the "formula type"), and it takes more bits.
Errm, that code means 7^38^1. We need to define a new token to represent subtraction, let's choose 254. So then we can code it as

7, 255, 38, 254, 1

And yes it will take more bits to code it that way, but let's suppose that the compressor is powerful enough to find a simpler equation for 7^38-N? I don't know that one exists, but if it does then it will be the compressor's job to reduce it to the simplest form.

And how do you plane to code N - 1600, then?
Simple: (7^38) - (40^2)!

7, 255, 38, 255, 40, 255, 2, of course, plus the overhead to describe what to do with your data (power, power, then subtraction)
No, we code it as 7, 255, 38, 254, 40, 255, 2. And to save some bits we can set a fixed rule for the codec that exponentiation is always performed before subtraction (just like programming languages assume). If we must specify operational order, we can define some tokens to represent parentheses, say 253 and 252. (We still have 152 unused symbols.)

Eventually you'll end up with a formula for every integer in [0, N], but as you can see you'll need several types of formulas. Eventually much more complex than calculating a couple of powers. So you'll have the overhead for the "type" and all the operands.
How many operators do we really need?

+ - * / ^ ( )

Wouldn't that just about cover every "simple" equation? Also, I think only the "integer" type should be used. We don't want to get into the hassle of floating-point imprecision.

(BTW 0x38 is 56 in decimal... that's why I've switched to plain decimal my numbers)
oops ;)


P.S. Just to clarify my stance, I don't honestly believe that this idea is at all feasible in the real world, but I can sort of visualize the approach, and I think it has merit. Kind of like how I can imagine travelling to other stars, even though we may never be able to do it...

[fccHandler]

Originally posted by virus
And how do you plan to code N - 1600, then?
And what about N - 1601 then? ... (and so on) ;)
Thinking about this further, (7^38)-(40^2) would be better coded as follows:

7 255 38 254 16 0

(where the BCD "16, 0" = 16 * 100 + 0)

Then (7^38)-1601 would be:

7 255 38 254 16 1

[virus]

Originally posted by fccHandler
And yes it will take more bits to code it that way, but let's suppose that the compressor is powerful enough to find a simpler equation for 7^38-1?
You look a bit too much optimist to me ;)
Simpler than 7^38-1? Using integer numbers only? :eek:

And doing that for all the numbers between 0 and 7^38? nah, you don't need to be a mathematician to see that it's impossible.
Keep in mind, for example, that all the numbers in the form x^y, with x and y in [1, 255] are just a bit more than 65000. Compare this number with the N you wrote before and see how lucky you need to be to find a number which is close to a certain power of an integer number.
If you don't believe me, just draw a random number between 1000 and 9999 and try to do the calculations manually. The formulas will be quite a bit complex, and they'll often need many operators and large operands. And no matter how efficient your bitstream format is going to be, you are going to use a lot of bits...

Look at prime numbers for example: you cannot factorize them so powers and multiplications cannot bring you the result you want (you need to do "something more" to reach it). So you'll probably need to combine several operators here, and probably large operands, too. (and not just three small numbers like 7, 38, 1 ;))

Just to clarify my stance, I don't honestly believe that this idea is at all feasible in the real world, but I can sort of visualize the approach, and I think it has merit.
I can visualize the approach, too. But I don't think it will really compress anything, much in the same way you cannot reproduce efficiently whatever video clip you want using Flash ;)


EDIT: oh, I see you improved your bitstream format :D
what about N - 14897131 then? :devil:
(note that this is still just a small "perturbation" of 7^38, given how large 7^38 is)

EDIT2: stupid grammar error corrected

[fccHandler]

Originally posted by virus
I can visualize the approach, too. But I don't think it will really compress anything
Well, just like other compression ideas it takes advantage of weaknesses in the data. The "weakness" of my chosen number is the fact that it just happens to be equal to 7^38. But other numbers (including the perturbation you posted) may have hidden weaknesses we don't know about. In fact, I seem to remember a theory that all primes > 3 are of the form 6N + 1, so that may be one "weakness" that the compressor (oops, I mean synthesizer) could exploit.

Also, my bitstream format isn't the main obstacle here (I admit it can stand a lot of improvement)! As I see it, the problem is that even with today's technology it would take an impossibly powerful computer to break apart any giant random number and express it as the solution of a simplistic equation. Heck, 7^38 is only 14 bytes. How to synthesize 4GB of a DVD movie? :eek:

But who knows... Maybe the Star Wars DVD just happens to be the solution to the equation (742987 ^ 80175908972457823712) - 1. :D


edit; drat, another math error

[virus]

Originally posted by fccHandler
As I see it, the problem is that even with today's technology it would take an impossibly powerful computer to break apart any giant random number and express it as the solution of a simplistic equation. Heck, 7^38 is only 14 bytes. How to synthesize 4GB of a DVD movie? :eek:
yep, there must be some reason if I limited my analysis to the "pure efficiency" viewpoint ;)

How can we classify such an algorithm?

Really-NP-hard? :D

[fccHandler]

Thinking about it further, virus is right. You only have to experiment a little bit to realize that statistically, the chances of being able to represent any large arbitrary number as the solution of a simple equation, with the representation of the equation being smaller than the representation of the number, are hopelessly small.

@callmeace: Confound it, you almost had me convinced!

It was fun to think about, though. I'm going away now to work on my perpetual motion machine. :)

[Joe Fenton]

Well, I can record 256 movies in just 256 bytes. Just tell me which 256 movies, then I'll send you 180G drive with the player program on it (the ONLY thing on it), and a floppy disk with the 256 byte movie file. :D ;)

[callmeace]

Okay, very interesting reading your posts :) I had kept thinking of how to respond and put across what I wanted, so here goes...

Many of you seem stuck on a problem of how to allow for 'all possible combinations' and of random numbers and so forth. These are presented as disapproving the concept I present, but in my concept, These things are not an issue which will need to be dealt with.

Let me explain I hope :) I think this will answer Manaos post.

Data compression is a challenge. It's hard to compress data but it's not so hard to represent a long number by much less characters which can be put back into a formula, so we first represent the data which we want to compress as a number.

Do you recall in earlier posts here that I said about unique character string? And also that a number 'is only that number'?

Do you see that immediately we have a case in maths where (so long as you set the rules) equations will only give one result.

In stages, What the concept does is first present, via code-book and mapping, the data which you want to compress as a series of numbers, unique to that data for being derived from it. Note that the mapping and code-book use also allows essential 'reversing'. Then in following stages the number is broken down into equations which would use powers and algebra for example, until one is left with a character string.

When you want your original file back, the character string when fed back into the formula generates back up to your original data number, This is because it 'adds up' to the number mathematically, which at the end is reversely written (possible because of the code-book and mapping) back into your file.

Now, how could one get a number from some data? Hmm..... Well, here is one way just as a concept suggestion. On any computer.

Firstly, understand that functions can be performed on the hosting computer, so you want to get a number unique to any file? will you allow that the binary information of the file must exist with any PC? We could use that! It will be a unique number unique only to that file and derived from it.

So in stage one The formula maps this binary information of the input file as continuous code being a long number consisting only of the digits 1 and 0. Let me explain what I mean by that. We literally want to use this binary code as a long numbers without spaces, but there are naturally going to be limits hit on how big a number could be. imagine a string consisting of 0s and 1s in a text document being as an actual number (like 1001 is 'one thousand and one') but being much longer. So rather than having the binary data of the input file as a huge unmanageble number in the form of an incredibly long sequence of binary, this would be cut in sections which are mapped to blocks, as a number at the biggest size which can be handled.

So regarding the above, as the numbers only contain 1s and 0s, even better! How much easier to handle and compress :) We know that they all must be multiples of ten plus with or without an added one!, and the numbers beginning with 0s must have their S.F. (significant figures count) accounted for too.

So, we have lots of mapped blocks each consisting of a big number whose digits are either 1s or 0s.
In case I have lost you, Remember that these blocks are just cut sections of the original huge continuous number which is the binary code of the input file. We want to break this number down through a series of formulas and various Mathematical and other compression techniques like mapping, to a string of characters. The string of characters when put back into the formula will then generate that number (which will then be put back to your original data)

For the sake of argument lets say that a thousand thousands are a million, a thousand millions are a billion, a thousand billions are a trillion, and a thousand trillions are a zillion. This is much much smaller than it would really be but lets say each number (block) doesn't exceed the zillions.
I will insert commas here just for clarity, Imagine then that the first block has the number
Block01: 10,010,001,010,011,100
the second has
Block02: 110,101,001,110,101,001

and so on


in this stage it would maybe be that every block is given a multiple or power of ten number and flagged as having plus one or not and a SF number - this is just a suggestion

Remember that all this work is taking place running through the process, to be under 64KB its only the formula and the character string total we are concerned about.

In the following stages I'll leave to your imagination the advanced calculations from years of mathematical thinking which could be employed. But you would end up with a character string.

In all this quite briefly stated above I think I have shown the concept better than before. Can you see that there is no need to worry about infinite combinations of numbers and the like, there is no concerns about hashes not being reversible or anything like that. Those issues would not be encountered. The maths itself is simple but the mathematical techniques and knowledge would assuredly be complex and advanced to generate the character string.


Dificult and a painstaking task requiring dedication, yes. But it's not unrealistic, impossible or anything like that.


:edited add some clarity I hope

[stephanV]

i could understand fcc, with callmeace im lost...:confused:

Do you see that immediately we have a case in maths where (so long as you set the rules) equations will only give one result
rules create exceptions and you cant have that with compression. its not acceptable to not being able to compress a random set of data.

[Manao]

In what way did you answer my previous post ? Did you find a point you could contradict ? If not, then whatever compressing schemes you could propose, my reasoning would invalidate it. Now that I gave a proof, you either have to proove what you're claiming ( which you don't ) or proove me wrong ( which you didn't ). Any other attempts would be only trolling attempts.

And when will you understand that '139' written as a charecter string ( 3 bytes ) is three times longer that '139' written in hexa decimal ( 0x8b : only one bytes )

Don't you see that you'll have to store your formula ? And that you're greatly underestimating the size such a formula would take to be stored ?

And how can you say such things as "So regarding the above, as the numbers only contain 1s and 0s, even better! How much easier to handle and compress" ???

[callmeace]

@manao

I hoped when I explained my 'approach' to compression I was explaining why what you describe in your post

i.e. "...The false assumption you make in your reasonning is that you claim you can obtain a simple mathematical formula to represent the file. But you never say how you'll write that formula into a file. For example, a very simple formula to code a number N is the following : x = N, which takes as much bits as N to code...."

would not be a problem

It is because the original code is seen as a different number (binary is strung together and seen as a very long number consisting of 1s and 0s)

@stephanV :) the forumla which is only going to have numbers consisting of 1s and 0s should not encounter any problems - what exceptions to the rules could you suggest (for example)?

[virus]

@callmeace

man, you're really hopeless at explaining stuff! :D

Anyway, saying "the advanced calculations from years of mathematical thinking which could be employed" or "the mathematical techniques and knowledge would assuredly be complex and advanced" or maybe "I'm sure a man will come on his white horse and will save the world" :D looks like hot air to me.

No offence intended, but I need something more than hope ;)

So in stage one The formula maps this binary information of the input file as continuous code being a long number consisting only of the digits 1 and 0.
Well, the input sequence is already a binary string made up of 0s and 1s. Using an "one-to-one" function to remap to another binary string is not going to change anything.
Some inputs would be mapped in a string that compresses better, some others not. On average, you'll end up on par... so this step already looks useless ;)

Anyway: please give us a simple example, so we can understand step by step what you want to do with your input sequence. Here's a 6 bytes sequence:

159, 174, 181, 148, 136, 131

translated in binary, this is:

10011111 10101110 10110101 10010100 10001000 10000011

so uncompressed, the input string is 48 bits, as you can see.

Note that the data present correlation (= similarity) between the samples and that means you can compress them. The data looks quite like the luma taken along a row of a video frame, so we're dealing with something real here.

Please shows us your idea and produce an output binary string (shorter than 48 bits, possibly :)). You can do whatever you want, up to and including splitting the string into a couple of pieces to ease your calculations.
Later on I'll show you a very simple and fast way to compress it down, so we can compare the results.

[callmeace]

@Virus :) I can immediately see that my way is not understood (the 1s and 0s are mapped to continuous numbers no spaces), but for now:

Further clarification, I believe some of you will now 'get this'. Here are some of the advantages of the way.

Firstly, surely none of you are going to deny that a weakness of normal methods of compression is that they only perform best with data that they are optimised for. And as some of you mentioned, there is a weakness with some organisations of data - unexpected or unallowed for circumstances etc that will 'foil' the plan.

Do you recall in a previous post when I mentioned that this way of compression doesn't care what data is fed into it? Which is why I say any data will be compressed well with my way. All data is transferred to long numbers of 1s and 0s. Full Stop for now, think about that.

Manao, I can only reiterrate that I'm not arguing over your maths or denying its correctness or whatever, but I will point out that your maths is simply not applicable to my way of compression. Those 'limits' do not apply to it's processes. I urge you (and others) to please get away from traditional ideas and try and understand this new way.

There will be no problems with some data not fitting into the formula accurately or some set of data not being compressible or the problems or limitations that are currently faced. Note - I have mentioned that there will probably be a limit on file input size, vaguely suggested as 4TB, just for the sake of the character string & mapping data fitting into a few KB.

Also, I don't know why you can't see that as the numbers are all multiples of ten and with our without plus one, that this immediately makes it easier for the formula to break down much more? I still think this suggests that people aren't understanding how it works.

We are all here to learn and want to have better technology right? :) I will strive to try and be clearer at explaining, but please be patient! Some of you keep getting hung up on problems that are not going to occur with the concept I mention.

[Manao]

I will point out that your maths is simply not applicable to my way of compressionAlright, so you didn't even bother to read what I wrote. If you did, you'd have remarked that my proof was valid for any compression scheme which takes a file as input and gives a file as output.please get away from traditional ideas and try and understand this new wayPlease, it's your only argument is this discussion, and it's not even a valid one, so could you stop repeating it over and over ?

[callmeace]

Originally posted by Manao
...Alright, so you didn't even bother to read what I wrote. If you did, you'd have remarked that my proof was valid for any compression scheme which takes a file as input and gives a file as output...

Manao, I detect some irritability in your post. I understand that and I don't mind because I can see you haven't understood my explanations.
I can't say any other way than nicely to you though, that your statement above is just not correct "my proof was valid for any compression scheme which takes a file as input and gives a file as output".
In my concept the code is not being compressed 'as code', it is literally being translated to something which is not code but a number. However that number is still unique to the code and meaningful to it. Do you see? The binary digits are strung together (mapped) to make long numbers (no longer being binary code) and these numbers are broken down via maths.

You see, your maths is fine - I don't deny it! But we are approaching compression in a different way where old rules and limitations don't have to apply.

We are all here to learn and make things better so I hope there is no hostility over a misunderstanding :)

[virus]

Originally posted by callmeace
The binary digits are strung together (mapped) to make long numbers (no longer being binary code) and these numbers are broken down via maths.
:rolleyes:

And how the hell are you supposed to work with these "long numbers" on a digital finite state machine like a Pentium-class processor, with binary registers with a finite length, if you don't represent them as a binary sequence?

And how the hell are you supposed to work with your numbers ("break them down via maths") if you can't even manipulate them? Representing exactly a real number requires up to infinite bits!

Seriously, either give a practical example, or shut up. I'm tired of reading these sentences "my way is better, my way has no limits, my way is not traditional, my ideas are new and old approaches don't matter". This is BS at the most irritating level. You can go on forever with that but you won't find any listeners anymore.

If you cared to give an example instead of repeating "my way is better, you don't understand me" over and over and over I would already have torn it in pieces, showing you the practical problems which arise when you finally stop dreaming and start working with something real.

[Manao]

In my concept the code is not being compressed 'as code', it is literally being translated to something which is not code but a numberThank you for making my point. Go read my post, especially (7).we are approaching compression in a different way where old rules and limitations don't have to apply.Maths still apply : you're the one using an uber-mathematical compression scheme, and you don't like it when the same maths prove you wrong ?I detect some irritability in your post.Indeed, there is some. Because I more and more think you're deliberatly trolling here, and that by doing so, you're misleading other people.you haven't understood my explanationsBut I have. In the other hand, you, obviously, didn't either understand or read mine nor any other explanations given by others.

[stephanV]

Originally posted by callmeace
I will insert commas here just for clarity, Imagine then that the first block has the number
Block01: 10,010,001,010,011,100
the second has
Block02: 110,101,001,110,101,001

and so on


in this stage it would maybe be that every block is given a multiple or power of ten number and flagged as having plus one or not and a SF number - this is just a suggestion



callmeace, this just doesnt make sense

lets try your way of compressing on the first block (at least in the way i understand you, if not you should correct me.

first we would need to define how many bits we need for all our characters:

we have all numbers ---> 0,1,2,3,4,5,6,7,8,9
a sign for 10^ ---> *
a sign to add 1 ---> +

so we have 12 signs and thus we need 4 bits for each character in our string.

the first thing i see as a power of ten is 100 so that would become , *2 (10^2), continuing like this we get:

*2*3*1*3+1*2 (if i didnt make a mistake somewhere)

so we have 12 characters and thus need 48 bits to store 17... thats not compression.

of course you will say i got it all wrong and i dont understand you, but now you give a better example then.

stay away from vague terms like "advanced maths", that doesnt mean anything. we cant use maths that doesnt excist (yet) and neither can we make any true statements about such maths.

[r6d2]

I really don't see why Manao, Virus and others just don't see the brilliance of callmeace's ideas. I just fed his posts through the latest version of Turing's Natural Language interpreter (look it up at the Magic Forum) and it created an output which is an algorythm that works!

In fact, I just compressed LOTR (the three movies) and stored them on my 128-MB pendrive. In fact I'm playing them back right now on my 7" HDTV. Quality is awesome!

[Joe Fenton]

r6d2: love the sarcasm man. :D

callmeace: you need to take a couple basic math classes, dude. You don't know the first thing about math. You're talking sh*t. It's not that people don't understand "your way", it's that YOU don't understand math. You're spewing nonsense. Come back after 4 to 8 years of math and computer science at a reputable university.

[callmeace]

:) @ all

I won't return any hostility towards some of you. I won't respond to a couple of efforts to degrade this thread into one of those ridiculous 'ego-wars' either ;)

Observe the title - "A man claims to store multiple movies on 64kb"

I saw that, I have a way and decided to drop in here and discuss how it can be done. To defend myself ;) , I did apologise a few times in advance knowing that I'm not so great at putting across and explaining things, yet some people decided to be rude and a bit nasty to me, although I have kept it friendly.

Well, most of you think you have your maths which prove that this kind of compression is impossible. Okay then! You have it in your minds that it can't be overcome, and are blocked to approaching the task in a different way where your Maths needn't apply. I know my way will work, doubt all you want but I've re-read all the posts in here several times. My posts explain overall pretty well I think (when all read), but I'm not happy to receive rudeness and ridicule, so I thought "well forget it then" as they say. Probably one day someone will reveal that they've been working on such code and then you'll see :) But seriously, I find it odd the resistance to what is actually quite easy to grasp, oh well.

[virus]

Originally posted by callmeace
You have it in your minds that it can't be overcome, and are blocked to approaching the task in a different way where your Maths needn't apply.
have you considered moving to a different universe? :)

The one we currently live in is so boring, so blocking, so maths-oriented...

[Nic]

@callmeace: Well done in keeping your cool and keeping the conversation civil. Your quite right that having different opinions shouldn't lead to "rudeness and ridicule".

@Joe Fenton: I can see what your trying to say, but your comments were perhaps a little harsh.

I must agree with the majority and think that callmeace's notion of compression is not possible (at least not in the way he thinks, due to too many reasons to list here). But I doubt he can be convinced otherwise, so lets keep it as friendly as possible. :)

Cheers,
-Nic

[Manao]

Okay then! You have it in your minds that it can't be overcome, and are blocked to approaching the task in a different way where your Maths needn't applyYou never said why they couldn't apply. We're still waiting.But seriously, I find it odd the resistance to what is actually quite easy to graspBut you still don't question your reasoning and keep saying we're wrong. And that's even odder.

[stephanV]

Originally posted by callmeace
Observe the title - "A man claims to store multiple movies on 64kb"

I saw that, I have a way and decided to drop in here and discuss how it can be done.

Then discuss it, and stay away from vagueness. Until so far you're ideas have been either wrong or not understandable. You're best argument till so far has been that with a lot of effort somewhere in the future it could be done. Sorry, that's just not good enough.

[virus]

A few quotes from callmeace:
Originally posted by callmeace
In the following stages I'll leave to your imagination the advanced calculations from years of mathematical thinking which could be employed. The maths itself is simple but the mathematical techniques and knowledge would assuredly be complex and advanced to generate the character string. You have it in your minds that it can't be overcome, and are blocked to approaching the task in a different way where your Maths needn't apply.
So, maths is advanced, maths is powerful, maths is needed to build this "algorithm", yet it doesn't need to apply. Looks a bit strange to me... and now an invaluable quote from Manao:
Originally posted by Manao
Maths still apply: you're the one using an uber-mathematical compression scheme, and you don't like it when the same maths prove you wrong?
Well said. Damn well said... :)

...do we really need to further "discuss" this topic? ;)


Oh and BTW, I've invented the perfect compression algorithm. Here it goes, step by step:

1) found a company.
2) output several press releases claiming that your compression scheme is the best and that it will work on any data, up to a 1000:1 compression ratio.
3) collect investors' money in a Swiss bank account.
4) state that you're going to demo it privately for Philips and that they're going to buy an exclusive license.
5) collect Philips' rival companies money to have it demoed a few days before Philips.
6) the day before the demo, claim the money collected at points 3 and 5 from your bank.
7) change your name into something less suspicious.
8) fly to Maldives with a couple of top-models.
9) done!

...finally an algorithm that works without that boring math stuff... anyone available for beta testing it? :D

virus

[Manao]

I'm ready to test 8), it seems to be the crucial point of the algorithm :D

[Joe Fenton]

Originally posted by Nic
@Joe Fenton: I can see what your trying to say, but your comments were perhaps a little harsh.

Yes, perhaps. One thing always ticked me off - I'd be sitting in a partial differential equations class and there's ALWAYS one student who just doesn't get it. The prof writes 2 + 2 on the board, then in the next step writes 4, and sure as the sun shines, he's got his hand up - How'd you get that?

AAAAAAAAHHHHHHHHHHH!!!!!!! What are people with no concept of math doing in these classes?

Anywho, this just struck me the same way. People may dream of shortcuts, but sometimes you just HAVE to do three pages of Eigen-function expansions.

I'll try to be more civil in the future.

[fccHandler]

Originally posted by virus
Here's a 6 bytes sequence:

159, 174, 181, 148, 136, 131

translated in binary, this is:

10011111 10101110 10110101 10010100 10001000 10000011

so uncompressed, the input string is 48 bits, as you can see.

Note that the data present correlation (= similarity) between the samples and that means you can compress them. The data looks quite like the luma taken along a row of a video frame, so we're dealing with something real here.

Please shows us your idea and produce an output binary string (shorter than 48 bits, possibly :)). You can do whatever you want, up to and including splitting the string into a couple of pieces to ease your calculations.
Egad, I'm still wrestling with these numbers, trying to derive a simple equation. That shows you what a pathetic geek I am. ;)

Bytes 159 174 181 148 136 131 strung together, translates to decimal 175,572,719,536,259. That number is evenly divisible by 359, which gives 489,060,500,101. But is that a prime? (I haven't been able to divide it any further.) I'm losing hope that this can be reduced to any kind of equation which I can represent in fewer than six bytes.

@callmeace: If you have some advanced maths which can solve this problem, please post it!

@virus, you're an evil dude. :sly: