A man claims to store multiple movies on 64kb

[ripper64]

Hi,

just watched some TV here in the Netherlands, and I'm absolutely surprised.

In 1999, someone named "Jan Sloot" claimed to have stored a number of ten complete movies on 64 kilobyte. In this documantary they said that Philips and some other companys invested millions in this "Sloot Coding System". Some days after he showed his invention he died all at once. There is even a book written about this mystery.

Myself, I don't believe in this "Sloot Coding System".

If you want more information, I do have links, but they're all in Dutch.

[stef24]

hah you beat me to it. i was about to post this:

i just saw a most intresting program on Dutch TV about the "sloot Digital Coding System". Electronics expert Jan Sloot spend 20 years developing a coding system that compresses video to mind boggling extent.

SDCD is a new alphabet for storing digital information which no longer uses the binary system of 1 and 0, but a much more efficient method. Just like with text, movies are composed of a finite number of colors and sounds. All those finite elements were encoded in 5 different algorithms of 74MB resulting in a total of 370MB - this is the core of the invention. The only thing needed to complete the system would be a fitting key which takes up just 1KB. Like this, dozens of keys can be stored on a simple chip.

So after making a payment, somebody could download the keys to a dozen movies via cellphone to then watch the movies at home on a player containing the basic algorithm. Should this ever reach the commercial market all CD, DVD, tapes and diskettes would cease to exist, aswell as fiber optics. The coper wire and the cell phone would be worth more than gold. Everything would change

Jan Sloot presented his Digital Coding System to a bunch of investitors who eventually took him to the Philips headquarters, where he showed his invention to Philip's CEO Roel Pieper. The demonstration on the top floor of the Philips headquartes involved playing 16 movies at the same time from a box containing a 64KB chip, attached to his laptop - no hard drive.

Roel Pieper took Jan Sloot to the headquarters of Computer Associates in the States where he was told "you are going to be the richest man on earth"

But two days before the source code was to be handed over he was mysteriously found dead in his garden. The source code has never been found by the security agencies assigned to try and recover it, but the box was apparently previously opened by Philips without Jan Sloot's permission ... stolen?

The story has been written in a book ("De Broncode" transl The Source code) by a journalist who was present during much of the demonstrations and followed everything closely.

several online links document the story and the technology in detail but unfortunatly there is no english information!

the final part of the TV program is on Sept 12

i hope the dutch readers of this forum can start a discussion about this. here are some links. alternatively, google for "sloot digital coding system". very much looking forward to discussions about this

article in the press: http://www.nrc.nl/W2/Nieuws/2001/01/23/Vp/07.html
website of the tv program: http://www.netwerk.tv/index.jsp?p=items&r=netwerk&a=131206
detailed story from Quote Media: http://www.gids.nl/techno/jan-sloot.html
article by a non believer: http://www.exp-math.uni-essen.de/~immink/interviews/ed2001.htm
another article: http://www.uitgeverijpodium.nl/index.html

and technical discussions can be found on usenet in nl.comp.programmeren

[ripper64]

Yes, just the links I meant.

Stef24, I must say your post is much better than mine ;)

But, if this invention of Jan Sloot is real, then it's certainly stolen, however I can't believe Philips commiting "a murder with robbery".

But, throwing the binary system of 0's and 1's away, how is this possible, since all processors and what else uses 0's and 1's. A whole new machine is required, while he showed his invention on a regular laptop.

[stef24]

hey i just translated parts of text i found in those links. the program didnt really go into much detail about the technology

speaking of the technology, most of it goes way beyond what's in my head. i guess if he used a system other than the binary one, it must have been coming out of this box. thats what he reluctantly gave to Philips and he got very pissed after he found out it had been opened.

if you think of the implications this technology could have im sure alot of CEO's would put some money into a kitty to "bust a cap in his ass". would not be the first time ey

[Wilbert]

Yeah, I also saw it. It's a strange story. Apperently it happened several years ago (July 1999).

Anyone knows about any investigations, police reports? Is he murdered (I guess there's no evidence for it), signs of burglary (I assume all the stuff was in his house)?

[stef24]

one of the above links says they suspect him having a safe somewhere that stashed the source code. they didnt find anything at home and obviously it wasnt all in his memory

very strange almost disturbing story indeed

[DigitAl56K]

65536 bytes to play with
Divided by 10 movies = 6553 bytes per movie

It takes 3.625 bytes to address 387973120 (370MB) bytes of "magic video data"

There are 129600 frames for a 90 minute movie @ 24fps

Which means if you only needed to address one single piece of video data per frame (which is completely rediculous), for a complete movie you would need:
129600 * 3.625 bytes = 469800 bytes

If you then wanted this to fit into the aforementioned 6553 bytes per movie you would need some further kind of arithmetic compression of (469800/6553) = 71 times - and what you would be compressing would essentially resemble a bunch of random numbers (indicies into the magic video data), which we all know doesn't compress well losslessly even with the best known algorithms.

This story is a complete non-starter. The only disturbing element is that anyone would actually believe it is possible, let alone make a TV show on it...

[Wilbert]

The only disturbing element is that anyone would actually believe it is possible, let alone make a TV show on it...
Not only that. It was demonstrated to several people (also IT compagnies). Investors invested millions in it (Pieper from philips for example).

Second part starts how, so I have to watch TV ...

[theReal]

Sounds like this works similarily to MIDI??? Addressing existing samples??

However even if it does work you'd loose all the things like color finesse or certain types of grain in a movie - they'd all basically look the same...
I don't think film makers would approve...

[ripper64]

This compression of Jan Sloot doesn't only conern to video or audio, to all sort of data, is what I make up of the story.

But, if you haven't seen the TV programme, you wouldn't believe the whole story anyhow.

I'm really amazed of all the investments. Philips invested millions, so did ABN Amro (Dutch Bank), even Computer Associates invested in it. Unbelieveable, you wouldn't just invest millions if someone says that he invented a compression factor million, so I'm curious what it makes so reliable.Originally posted by Wilbert
Anyone knows about any investigations, police reports? Is he murdered (I guess there's no evidence for it), signs of burglary (I assume all the stuff was in his house)? According to the TV Program he had a truckload of paper in his room and this is all disappeared after his death. He claimed that he had his invention stored in a safe, not found yet. There wasn't an autopsy performed, so his immideate heart-attack isn't cleared up.

**Thinking to myself: Why can't I just go talking Dutch here if this discussion is bound by almost only Dutch people. :D

[Wilbert]

Incredible story. It was on tv again, because someone published it in a book:

Eric Smit, adjunct-director at the business magazine Quote. (ISBN 90 5759 156 1). The book is called 'the sourcecode'.

It turns out that after he gave a demonstration to Roel Pieper (who was at that time in the board of directors of Philips (vice-president), and he would be the next president of Philips when Cor Boonstra would leave), Roel Pieper flew him several times to USA to all the big IT-guys and investors. But, not in the name of Philips, but for himself (at least that's what the book claims the evidence points to).

Moreover Pieper established a compagny (I think it's duch) called 'the Fifth Force Inc.', which still exists today, and already started on a business plan.

That time there were also some dutch investors that were willing to invest (50 million dollar by ABN-AMRO bank as a startfund, and others).

Two days before he would give his system to Pieper and Co, he died because of a heart attague (it is known he had heart problems, but strange no section was done). Strange thing is that his room is cleaned, and all the paperwork is gone (no idea whether the were traces of burglary). Btw, this sourcecode was not in a house, but somewhere in a safe (which nobody can find). In 2001, he doesn't work at Philips anymore, but I can't find the date when he left.

some other links (dutch):
http://www.parool.nl/nieuws/2004/SEP/06/eco2.html
http://www.gids.nl/techno/jan-sloot.html

[gircobain]

Has anyone heard of the product (http://www.theproduct.de)?

To anyone who doubts that such amazing compression technique could ever be developed, well that's amazing too...

[DigitAl56K]

That's completely different.

It's one thing to have some code that can generate a movie, it's another thing entirely to be able to encode an arbitrary source.

The Produkt isn't compression at all. Ok, so it does probably contain many compression routines and is probably coded extremely efficiently, but all it is really doing is generating graphics/sound based on some code somebody has written to do just that. The amazing compression factor that was derived was based on the fact that "This movie is w minutes long and has x,y,z funky things going on, so if you worked out how much space it would take to store that instead of our 64kb of code that generates it, that's pretty awesome compression."

In the same vein, I could say that my favorite WinAMP VIS plugin has uber-better-than-anything-on-earth compression because it's been generating video for years yet the download was only 3MB!

I think if this "Jan Sloot" did ever really exist, then this whole "conspiracy" theory is completely wrong. My guess would be that if this story is true, then Jan Sloot is one of the best con artists in history - walking away with many millions of dollars after convincing some of the biggest businessmen in the industry that he had stumbled upon the impossible. And hey! With that kind of cash in his pocket, I bet he could have faked his own death and disappeared real easy...

[gircobain]

Well, I didn't say it was the same thing nor was I even referring to video compression specifically
I meant that it is amazing what could be done with so few bytes - let's reckon, better than many games which take 1, 2 or more full cds
Imagination and creativity have no limits - there are more things in heaven and earth than are dreamt of in our philosophy, already said Shakespeare :)

[DigitAl56K]

Yes, it is very impressive - I've seen some pretty neat stuff done with 256 bytes also. Or this one (http://www.scene.org/file_dl.php?url=ftp://ftp.scene.org/pub/parties/2004/0a000h/256b/blobdistort201.zip&id=205275), which is only 201 bytes! (probably shorter than this post).

[Phoenyx]

Originally posted by gircobain
Well, I didn't say it was the same thing nor was I even referring to video compression specifically
I meant that it is amazing what could be done with so few bytes - let's reckon, better than many games which take 1, 2 or more full cds
Imagination and creativity have no limits - there are more things in heaven and earth than are dreamt of in our philosophy, already said Shakespeare :)

Shakespeare should have said "There are more things in Heaven and Earth than *can* be dreamt of in your philosophy, Horatio." (extra word is mine and I accept the consequences ;) )

[dani82]

al capone's vault, the holy grail, the cure for AIDS (maybe), and now "sloot's code" all said to exist, but never proven or otherwise

[Mug Funky]

lol! you forgot about the Higg's boson behind the grassy knoll...

it's a nice story, and would make a good movie itself (though they should trim back the fantastic elements a bit if they want a market outside of Hollywood).

...but i don't think there's much in it apart from "wouldn't that be cool" factors.

perhaps it's an attempt by 3 1/2" diskette manufacturers to get back into the market :)

[Doom9]

I've heard this before.. I guess it's the urban legend of the video encoding biz. My memory is extremely hazy already, but what I recall from college is that given a certain amount of information, there's a theoretical limit to how much this can be compressed losslessly. Existing coding systems are obviously less efficient, but by that theorem, it's simply impossible to compress a movie to 64k (assuming you expect presentable quality).

[unmei]

you forget the 64Kb are only the "key", the data is in the 370Mb.

How about a chaotic function with 370MB parameter data that takes let's say a few descriptors as initial contidion, maybe 16K position, 16K velocity, 16K acceleration, 16K a' (whats that called, curbation?).
The "video frame" at time t would then correspond to the position calculated for that time.
Anyway, construction for the parameter data as well as finding the start values to use to get a certain movie, i think, can be said to be impossible :)

[Mug Funky]

doom9 - i think the theory is that there is a limit to the amount of predictible data in a set of data, and the rest is entropy. as entropy can't be predicted, it must be stored as is. the more entropy, the less coding efficiency.

of course, given a Grand Theory of Everything, we'll find a lot of stuff that looked like entropy and chaos, but was in fact perfectly predictible, but the only problem here is setting the initial conditions. just think of how you'd use a theory of everything to determine the orientation of silver halides in film grain... you'd need a data set approaching the complexity of the entire universe to derive so much from a set of laws.

still, it's all food for thought. i wouldn't think something like this was possible, and if that guy was murdered, i'm willing to bet it was a pissed off venture capitalist who talked to a computer scientist and realized he'd been duped bad.

but i can't see how big companies could have invested in this - it's high risk, and didn't pay off. shareholders simply wouldn't allow that kind of irresponsible use of their money - R&D is not fashionable in investment circles even when it has solid science on it's side. why else would sony and philips have to get together to make CD and SACD, rather than just one of the two companies doing it off their own bat?

[ripper64]

Originally posted by DigitAl56K
The only disturbing element is that anyone would actually believe it is possibleThat is one disturbing thing.
That the man died all at once is disturbing.
That millions were invested in it is also disturbing.

[LordRPI]

There were a few posts back like this in 2001 at the DivX.com forum. It basically boiled down to the fact that they were not actually movies, but DirectX sequences that were generated from a very small amount of data. You can find these movies, generally starting at 64Kb here (http://www.theprodukkt.com/demoscene.html).

Ooops, been posted in this thread before.

[virus]

Originally posted by Mug Funky
doom9 - i think the theory is that there is a limit to the amount of predictible data in a set of data, and the rest is entropy. as entropy can't be predicted, it must be stored as is. the more entropy, the less coding efficiency.
ehm, *cough*, no... :)
There's a statistical measure called "entropy rate" of a source ("source" = a set of generic data = a sequence of numbers, to put it simply), which is the lower bound for the average code length you can spend to represent that source with a compression algorithm. You cannot do lossless compression at a bitrate lower than the entropy rate, exactly in the same way you cannot make your car travel faster than light ;)

Entropy is a measure of the amount of information delivered by a source. The more the source outcome is unpredictable (= "random"), the more information it will deliver, the higher the entropy rate and the less you can compress it (and vice versa for highly predictable sources, which are insanely compressible).

The unit of measure of the entropy is "bits/symbol" (that is!... in the world of image/video compression, generally 1 symbol represents 1 pixel, so you can read it as "bits/pixel" if you want - well, I'm oversimplyfing things a bit too much here... but you get the idea ;))

[Neo Neko]

Has anyone heard of analog computing? An analog computer should be capable of storring a rather large number of discrete values without needing to transcribe them to some less efficient digital representation. For that matter what about computing with numerical bases other than 2? Either one would require much more complex hardware than what is in a traditional system. But would be capable of representing more with much less. Binary encoding is not the height of efficiency especially when your data is not generally binary in nature. And heck not much is truly binary in nature. :p Perhaps this box was an analog computer capable of storring and computing with a set of 256 or more discrete values.

And in the future what will happen with quantum computing? Where each quantum bit is capable of describing 5 unique states simultaneously? Or where actual hardware network latency is zero and the transport incurrs all overhead. With quantum networks it would be possible to send data from one end of the universe to the other simultaneously! Interesting stuff. Interesting stuff that is on the horizon and will change the way we think about computing drastically.

[avih]

Originally posted by Neo Neko
Has anyone heard of analog computing? An analog computer should be capable of storring a rather large number of discrete values without needing to transcribe them to some less efficient digital representation. For that matter what about computing with numerical bases other than 2? Either one would require much more complex hardware than what is in a traditional system. But would be capable of representing more with much less. Binary encoding is not the height of efficiency especially when your data is not generally binary in nature. And heck not much is truly binary in nature. :p Perhaps this box was an analog computer capable of storring and computing with a set of 256 or more discrete values.
.
.
.


analog calculations might have something, but analog value can be represented pretty good with few bits ("few" depends on the resolution you actually need, and the problem domain. typically though, 64 bits would be enough for most range-limited analog values. without getting into account errors that sum-up due to quantization of the values, but also not taking into account the naturall inaccuracy of analog values)

regarding elementary memory elements that are not binary (i.e. each basic element represents 4/5/8/16/whatever descrete values), not much to gain, as all it would save is the space that the momory consumes (whether it be silicon or ano other matter). so u get 1/8 the size for the same memory. "big deal" (at least in our context).

the issue was rather if at all such algorithm could exist (well, it depends on the input imo, for some inputs it might, for others, not. as someone mentioned, it depends on the inherent entropy of the input). using more states/cell will only make it compute better (=faster or on smaller base die).


appart from kkrieger, nothing comes close ;) (well, there was also this raytracer in 256 bytes ;) )


well, if we could only write down the parameters of the universe right after the big bang, we could have done it though. we would only need then a large and fast enough memory and a fast enough computer to calculate the state of the universe at any given time. then we could have calculated (by prediction) the demonstration that took place, its content (= the films), and hey, we could have even predicted the actual 64K that he used to compress the data :)

[Doobie]

Originally posted by DigitAl56K
There are 129600 frames for a 90 minute movie @ 24fps

Which means if you only needed to address one single piece of video data per frame (which is completely rediculous), for a complete movie you would need:
129600 * 3.625 bytes = 469800 bytes

You wouldn't need a single bit encoded to address each frame. The decoder knows that the next frame starts where the last one ends.

[DigitAl56K]

Originally posted by Doobie
You wouldn't need a single bit encoded to address each frame. The decoder knows that the next frame starts where the last one ends.

That's not what I said :) I said for each frame, you would have to dig into your magic 370mb store of video data at least once (or you have nothing at all to decode), and in order to address 370mb of data you need 3.650 bytes. In reality, you would probably have to dip into it several thousand times, so multiply the unliklihood of this story by a facter of 1000x at least :)

With regards to number systems other than two, it doesn't matter what your number base is. When you describe a number of bits you are describing the representation in base 2. For comparitive evaluation (ok he said 64KB not 64Kb), the value would also have to be described in terms of conventional wisdom, i.e. base 2, for it to be meaningful.

[Doobie]

Originally posted by Neo Neko
Has anyone heard of analog computing? An analog computer should be capable of storring a rather large number of discrete values without needing to transcribe them to some less efficient digital representation.

That would be video tape. Using technology several decades old, six hours of movies can be put on one ultra-cheap tape at near broadcast quality. Using modern technology, a high quality analogue VHS-size tape might be able to hold a thousand hours at near DVD quality. But, analogue cannot be used to store computer data because of imprecision.

There is no such thing as digital in the natural world. Transisters that are used to build computer chips are analogue devices. They're digital only like your water faucet is digital. With no water, or only a little water (a small or even a big leak), you have a zero. With some high level of water flow, you have a one (you can't even know for sure what that high level will be. Water pressure varies, even if you open the faucet up all the way). There is no need for a precise water flow to have a precise one or zero. This is why computers are base-2 and why they can't practically be anything else.

With analogue bits, two bits could represent each pixel vs. 24- or 32-bits as it is with binary computers. And, these analogue bits could be packed much more densely and without the overhead of error-correcting onto megnetic media.

And in the future what will happen with quantum computing?

Quantum computing is cool. :cool:

[LordRPI]

Originally posted by Doobie
That would be video tape. Using technology several decades old, six hours of movies can be put on one ultra-cheap tape at near broadcast quality. Using modern technology, a high quality analogue VHS-size tape might be able to hold a thousand hours at near DVD quality. But, analogue cannot be used to store computer data because of imprecision.



If I remember correctly, the general rule of thumb is that a $2 CD player at Walmart is equivalent to a $2K record player.


There is no such thing as digital in the natural world. Transisters that are used to build computer chips are analogue devices. They're digital only like your water faucet is digital. With no water, or only a little water (a small or even a big leak), you have a zero. With some high level of water flow, you have a one (you can't even know for sure what that high level will be. Water pressure varies, even if you open the faucet up all the way). There is no need for a precise water flow to have a precise one or zero. This is why computers are base-2 and why they can't practically be anything else.

With analogue bits, two bits could represent each pixel vs. 24- or 32-bits as it is with binary computers. And, these analogue bits could be packed much more densely and without the overhead of error-correcting onto megnetic media.



Everything can be experessed in binary. If one bit isn't enough, you string many of them together. Lets say I want to dedicate one bit to say if the water on my faucet is on, and 7 bits represent the water pressure... and another 10 bits represent the chemical impurity of the water coming out of the faucet.I don't recall exactly who proposed this, but this idea is approaching 100 years of age, predating computers. It's funny though, the math behind algorithm analysis has been around much longer than the computer itself.

I've never been familiar with the term analog bits, but old designs have suggested going beyond binary numbers to decimal and octal. These designs, however proved extremely inefficient. But getting back on track, the analog greatness focuses on that everything be ideal: A storage medium that can store infinite values with signals of infinite lowness and infinite highness: that the playback head of a record player has zero mass, and your speaker produces no air resistance while moving even though it relies on making air move for your own feeble analog ears to hear it. We live in the real world though and have to accept that to some extent, we will be limited by physics, at least in the near future. As some of you might have realized, the non digitalness of computer microprocessors have caused Intel and IBM to fall behind Moore's law on grounds of excessive wire delay caused by slowing speeds due to the resistivity of modern interconnects.

[Neo Neko]

Originally posted by avih
analog calculations might have something, but analog value can be represented pretty good with few bits ("few" depends on the resolution you actually need, and the problem domain. typically though, 64 bits would be enough for most range-limited analog values. without getting into account errors that sum-up due to quantization of the values, but also not taking into account the naturall inaccuracy of analog values)

You make it sound as if analog is somehow less accurate than digital. It isn't. The only thing Digital has over Analog is it's error resilliance. Other than that Digital quantization tends to be innacurate and innefficient in comparison. For a limited example of how analog can improve transmission/storrage etc look at modems. Part of the use of analog is out of necessity because of the medium. But rather than just sticking with trying to make the analog signal mimic the digital signal as close as it can, techniques like QAM have become common. QAM takes advantage of analogs nature. Using several discrete voltages to represent up to 256 different values. My digital cable box and cable modem both use 256 QAM. That's lossless compression of about 4:1. 4 digital values represented by 1 analog value. Which in and of itself is nothing to sneeze at. If this were further extended from the realm of transmission and storrage to psychovisual/acoustic compression algorythms and general computation who is to say we would not see a similar efficiency increase?

Originally posted by avih
regarding elementary memory elements that are not binary (i.e. each basic element represents 4/5/8/16/whatever descrete values), not much to gain, as all it would save is the space that the momory consumes (whether it be silicon or ano other matter). so u get 1/8 the size for the same memory. "big deal" (at least in our context).

Actually that seems like a rather big deal to me. The best lossless codecs etc generally don't do better than halving the bitrate required to represent the original stream. And at the cost of sometimes demanding computations. Here we are talking about a reduction in size that is 4 times greater while still being lossless. And at the same time being far less computationally taxing etc. That would be a huge boon! If you were then able to craft a lossless codec that were able to losslessly describe the original with half the bitrate using our new coding system that already reduces the size by to 1/8. Why you would have a lossless compression to 1/16th the original size! That is massive! That is about the equivalent of what MP2 and MP3 provide in a lossy way. Only losslessly. In my context that is downright monumental. :p

Originally posted by avih
the issue was rather if at all such algorithm could exist (well, it depends on the input imo, for some inputs it might, for others, not. as someone mentioned, it depends on the inherent entropy of the input). using more states/cell will only make it compute better (=faster or on smaller base die).

Actually computing with more base states to me looks to have the same advantage no matter what the entropy. At least in the base sense. Then there is the possibility that it would further benefit everything else. Including entropy based encoding etc. ;)

[int 21h]

Originally posted by LordRPI
...As some of you might have realized, the non digitalness of computer microprocessors have caused Intel and IBM to fall behind Moore's law on grounds of excessive wire delay caused by slowing speeds due to the resistivity of modern interconnects...

You posted the same thing on the DivX forums a few months ago, and I am inclined to disagree.

The reason Intel and IBM are falling behind on Moore's law is due to heat dissipation problems. Limits due to the resitivity of modern interconnects will not happen for another decade or so...

[Phoenyx]

Could I point out one thing? Think of a program that can create an infinitely complex picture using a single equation. Fractal compressors are even now being worked on. There was a mathemetician (sp?) (and I don't remember his name) that said that an image of photographic quality could be encoded using about 10k of equations and 500k of base data. As I remember (could be wrong) that was supposed to be a 1200dpi equivalent 16 million color picture.

[RadicalEd]

Originally posted by avih
well, if we could only write down the parameters of the universe right after the big bang, we could have done it though. we would only need then a large and fast enough memory and a fast enough computer to calculate the state of the universe at any given time. then we could have calculated (by prediction) the demonstration that took place, its content (= the films), and hey, we could have even predicted the actual 64K that he used to compress the data :)

Only if the universe is deterministic, of course, which is currently up in the air according to quantum physics :D

[Neo Neko]

Aaaaah yes. The old cat in the box trick. Yor task is to determine whether the cat is alive or dead. A quantum theorist would postulate that there is a 50/50 chance at any one time that the cat is dead. An average person would just go over and look inside the box. At which point the quantum theoris would go off on a rant about how they ruined the result by observing it. :p

[RadicalEd]

Real determinism would make the universe so much simpler IMO. Unfortunately, you get people like Heisenburg ruining it with their uncertainty principles. All in all, it comes down to whether or not quantum systems are just too complex for us to currently predict or if there truly are random factors involved (or the more boring possibility that some things are just physically impossible for us to ever predict -- enter Heisenburg). A good TOE should solve this problem, and of course we're still waiting on that one...

[Mug Funky]

i'm kinda glad there's no determinism to speak of.

means we get some kind of free will...

[Didée]

Originally posted by Neo Neko
Aaaaah yes. The old cat in the box trick. Yor task is to determine whether the cat is alive or dead. A quantum theorist would postulate that there is a 50/50 chance at any one time that the cat is dead.
No, that's not what the quantum theorist would postulate.
There is not a 50/50 chance that the cat is dead. According to quantum mechanics, the cat is 50% dead, and 50% alive.

Another one:
Assume a tree is falling in a forest, but miles and miles around there is no living being present that could hear something.
Does the tree make some noise when it falls, or does it fall in perfect silence?

And Mug ... your fate is determined for every single moment. But nonetheless, you do have a free will. In fact this is not a contradiction in terms.

[stephanV]

Originally posted by Didée
No, that's not what the quantum theorist would postulate.
There is not a 50/50 chance that the cat is dead. According to quantum mechanics, the cat is 50% dead, and 50% alive.

That's not completely correct either, there is no such thing as 50% dead or 50% alive. The cat is just both dead and alive untill its state is made an observation, after which it turns out to be either one. :)

[Didée]

Ah, an expert :cool:

Yep, correct - seems I was caught in a rush of 50's while typing ...

So, what about the tree?

[Cyberman]

How is this "dead and alive at the same time" supposed to fit into the real world?

Obviously, the cat can be either dead or alive - it can´t be both at the same time, these terms contradict each other.

Once I look into the box and see the cat´s alive, I KNOW it´s been alive all the time - because it´s next to impossible it´s been resurrected.

--

As for the tree - why shouldn´t it make a noise? It doesn´t make noises for us, so it won´t care if someone´s listening or not.

[virus]

Originally posted by Cyberman
How is this "dead and alive at the same time" supposed to fit into the real world?
probably it's more funny if you apply it to bits, instead of cats (can't use a cat to run XviD, anyway :D). Example: a couple of quantum bits can be at the same moment into a combination of four different "states", indicated with <00>, <01>, <10> and <11>, that is, all the combinations at the same time. If you perform a logic operation on those quantum bits (e.g. a NOT) you'll get all the four results at once. You only need to extract the result you want (I still haven't understood how you're supposed to extract only the result you want from a quantum register with all the states merged, but hey, I'm not a physicist :))

This huge, intrinsic level of parallelism should be the main point behind quantum computers. So huge that with them you'll be able to break whatever cryptographic system is currently used, no matter how long the key you choose. I've heard that some of these experimental quantum machines are already working (with registers just 4 or 5 bits wide and very simple algebraic operations). There's still a long, long way to come before the MPAA can use them to prevent our backups ;)

[Didée]

Originally posted by Cyberman

As for the tree - why shouldn´t it make a noise? It doesn´t make noises for us, so it won´t care if someone´s listening or not.
Are you sure of that. How _can_ you be sure of that.

One just cannot port those quantum behaviours into the "real world". That would be the same as saying "an electron can appear either as a wave or as a particle. Therefore, my pencil possibly can appear as a wave as well." - It makes no sense to look at it this way.

To shorten a long story: guys, don't raise sciences to a religeous state. Natural sciences are good and all, but they will never lead to the truth behind. The're just a tool, and nothing more. And each tool has its specific scope of usage, but is pretty worthless outside that scope ("a screwdriver comes handy for screwing screws, but one should not try to saw any wood with it").
The problem is that their basic set is not complete. If the basic set is not complete, one cannot extrapolate those other parts of reality that are dependendant on the missing elements of the basic set - especially if it is falsly assumed that the incomplete basic set was complete. And the mathematical way of abstraction won't help anything, either. It is griping much too short. Mathematical abstraction only enables exrapolations within the span of its given premisses. One can endlessly find "new" relations by doing math - but only those that are describeable by math at all. Describing "Love" through math has little prospect of being successful. Very little.
Also, Physics in last instance is based on man's expectancy. But again, man's expectancy has little to do with the true nature of things. In the expectation of a "smallest particle" being present, the hunt will go on forever, and whatever the physicians will find, will be dividable once more (here, the "string theorem" was an astonishingly good approach - however, it has become pretty silent in that area). Within physics, time is nothing more than a vector. But a vector always can be reversed, and in fact physics has little problems with reversing time. However, in practice it doesn't work all that well. Or gravity. It's one of the most basical things around us, each baby has to learn it is there, and how to deal with it. But physic is rather clueless. Any theory of particles transmitting a "tearing" force can't work out for several reasons. The new theory of omnipresent particles transmitting a pushing force, and gravity being the result of materia shielding these particles, doesn't work out on large magnitude. And Albert's explanation of gravity not being a property of materia but of bowed space, gives some nice visualizeable models - but it doesn't offer any explanation where the force emerges from. And actually, what is "force" after all? There are so much models, so much formulas to describe everything. But there are very little explanations. Take two magnets, bring them so close to each other that you feel the force, and then look at it and try to describe whats *really* going on there. We have no explanations. Last instance of the double gap experiment - miracle. The experiment of amplifying an electromagnetic impulse by transmitting it through inert gas - miracle. Virtual particles - miracle. Latest modification of the doubled photone experiment - miracle.

No, I would not put much hope in sciences to explain the world. They may explain that world that arises out of their rules. But they will not explain the world that IS. We need completely new sciences for that - and brains that are not bounded to the expectations of what seems to lie obviously before the eyes.

The nature of things is not of that kind we use to think it would be. Not at all. It is completely different.

In other words:

The tree falls quietly. Not noiseless, but quietly.

[Neo Neko]

Originally posted by Didée
No, that's not what the quantum theorist would postulate.
There is not a 50/50 chance that the cat is dead. According to quantum mechanics, the cat is 50% dead, and 50% alive.

You can't be 50% dead and 50% alive. Just like you can't be 50% here and 50% there at the same time. ;) A particle can only be in one place at one time. But untill we observe or measure the position the particle has the potential to be in more than one place. Just as the cat has the potential to be either alive or dead but not both. This is a basic principle for quantum mechanics and explains how light can be composed of particles that wave. :D

Originally posted by Didée
Another one:
Assume a tree is falling in a forest, but miles and miles around there is no living being present that could hear something.
Does the tree make some noise when it falls, or does it fall in perfect silence?

No one can say they have personal knowledge of such situations. But we know for a fact the laws of physics do not caese to exist when no one is observing. And since the phenomina of a tree making a sound when falling is easily explainable using physics. We can indeed be sure that a falling tree does indeed make a sound regardless if anyone is there to observe it or not. :p

[Soulhunter]

Assume a tree is falling in a forest, but miles and miles around there is no living being present that could hear something.
Does the tree make some noise when it falls, or does it fall in perfect silence?

Same opinion as Neo Neko here...

If you would know all the variables, you could even calculate the exact dB of the tree impact !!!


Bye

[stephanV]

Didee is right, although he explained it wrong. Quantum mechanics suggests that by an equal chance of possibilities (life and death), both are true until the observation is made. Of course, the only way too confirm this is to *not* make the observation/measurement, which is also a nice paradox in itself. :)

About the tree:

Every assertion about something we do not observe, is just pure speculation. We only *assume* it makes sound, because trees falling make sound when we are around. But of course, pure assumption is not enough to be sure that the tree actually makes sound. At least, according to Bohr. :) (there are quantum scientists who disagree with this view BTW)


But - of course this is all just theory anyway. :rolleyes:

[theReal]

Assume a tree is falling in a forest, but miles and miles around there is no living being present that could hear something.
Does the tree make some noise when it falls, or does it fall in perfect silence?I think this philosophical question was asked in a time where there wasn't physical certainty about moving objects causing sound waves. The question is still asked today in philosophy seminars at university but I think it's a bad example. Although there might be the slightest possibility that one tree in one case won't make any sounds, but, as said before, we know that the laws of physics don't cease to exist when no human is around (or do we? ... I still think it's a nonsense question, there are better philosophical questions than this...)

[eb]

Tree falls in a perfect silence when it falls in a pure vacum.

eb

[b0b0b0b]

This reminds me of the informal ongoing contest at comp.compression (I think) where the moderator offers some nominal prize if you can compress to any extent, some amount of random data.

So this one guy offers a solution where he splits the random data into a number of files and encodes some of the data into the file names and tries to claim the prize.

Hi-larious.

[Soulhunter]

Originally posted by eb

Tree falls in a perfect silence when it falls in a pure vacuum.

But the tree is in a forest, and it produces oxygen, no...


Bye

[unmei]

even in the strange case of a (dead) forest in vacuum, the tree falling probably produces sound in the trees it touches during that process. The sound will just not propagate to onjects not connected by some medium.

Another question, why do you even load that forum page to read ..how come you assume you can read it by entering the url and pressing return before you are actually "observing" it. Computer users assume a lot of physics to just work all the time by pure experince. And i for myself don't even know how all the processes are supposed to work, i just assume they do.

[LordRPI]

Originally posted by unmei
And i for myself don't even know how all the processes are supposed to work, i just assume they do.

They don't work. It's a conspiracy. They found a way to read all the minds of people and put what they think they should see on the computer monitor. Click here for more details. (http://sprott.physics.wisc.edu/pickover/esp.html)

[Soulhunter]

Originally posted by LordRPI

They don't work. It's a conspiracy. They found a way to read all the minds of people and put what they think they should see on the computer monitor. Click here for more details. (http://sprott.physics.wisc.edu/pickover/esp.html)

Uhm... :eek:

EDIT: Hehe, after some sleep I got it... ;)


Bye

[stephanV]

Originally posted by theReal
I think this philosophical question was asked in a time where there wasn't physical certainty about moving objects causing sound waves.

You are forgetting that we hardly understand this sort of physics at all. From my study i can see we already have trouble enough calculating the heat flow through a perfectly spherical object whilst air is flowing around it. All sorts of arbitrary constants come into play then (e.g. Reynolds). We certainly don't have the knowledge to predict what kind of sound an immensely complex object like a tree would produce when falling down.

Remember that from experience you could also conclude that the earth is flat.

I'm not saying that trees dont make sound if they fall when we aren't around; there's just no way to be sure. Science is at its best highly inaccurate.

[Soulhunter]

Crash-tests...

Feeding all info into a computer and you will get a perfect pre-construction of the incident !!!

Real crash-tests are only made afterwards to be 100% sure that the pc simulation was right...

Sould be possible to do the same with the tree as well, no ???


Bye

[stephanV]

Originally posted by Soulhunter
Crash-tests...
Feeding all info into a computer and you will get a perfect pre-construction of the incident !!!

You perhaps get an approximation of the incident. But can you be sure you are gathering all relevant info? No computer simulation has ever perfectly predicted the outcome of such complex events. Try to make a perfect re-creation of any currently existing tree in a computer: you will fail. Modeling can only go so far, all models have a limited range in which they can be used.

A computer is nothing more than a overpowered calculator. But what if there is relevant data to take into account that we have forgotten to calculate? Or even worse, what if there is relevant data that cannot be calculated with?

Another thing is: most models are not build by understanding the physical nature of things (those models that are, are usually highly inaccurate), but from experience. Of course, most of what we see as our understanding of the physical nature of things is just experience too.

If you create a model from experience you are only confirming what you already thought. If your model goes against your test results, you say the model is wrong, but if your model confirms them you say it is right. But almost no one ever considers that the test results themselves might be wrong.

[Soulhunter]

There are even more complex simulations (http://www.es.jamstec.go.jp/esc/eng/ESC/mission.html) than a "ordinary" tree-fall !!!

But sure, you would still need dozens of parameters to do the tree-fall simulation...


Bye

[stephanV]

:)

computers have difficulty prediciting the weather... what they are trying to do is hopeless ;)

[unmei]

Wow that test is pretty cool. Smart guy. But the comments on the explanation page are even better.. i wonder whether he picked only the dumbest or produced them himself :) ..and the second or so is plain out lying by claiming it were wrong once out of two times (as it is bound to work everytime until the user cheats).

[Zap_Bee]

i cannot believe that a hundred or so people tried this test and not even one found out the trick. i don't post it because i don't want to spoil this for other people, but i know there are enough guys who couldn't expect to show off their greatness. Strange.

Zap

[stephanV]

Originally posted by unmei
(as it is bound to work everytime until the user cheats).

How can you cheat this test? It's impossible... :)

[Didée]

Yeah, what a cheek. If they would sell that stone old card trick just as the lame trick it is, one could laugh about it. But selling it as an ESP experiment, clothed in a science-like appearance ... that's already impertinent. I could get angry about such things. Less because of the bumbledom of the site's creators, but more because of the possible damage done to the not-so-bright people that stumble about it and believe that crap. That's about comparable of telling all sorts of lies to a small child, just because it will believe them. Which is no good way of getting acknowledgment, and for sure its not funny.
There are really poor minded people out there.

[Didée]

>> [Soulhunter]
>> But sure, you would still need dozens of parameters to do the tree-fall simulation

You forgot typing the word "centillion" - some dozen centillions of parameters.

Any assumptions that the flow of things could be predicted, at least theoretically, if only the start conditions were known close enough, are naive. In one word: Mandelbrot. Or more verbose: Not only that many physical systems show natural chaotic behaviour in general and alone therefore can't be predicted. You can't even know the start conditions close enough, since this is simply impossible: Firstly, you would need to know the exact state of your object down to all of its smallest possible elements. But remember Heisenberg - you cannot evaluate the exact status of small particles. So you'd already fail here. Secondly, in order to make an exact prediction, you would need to know the exact state of EVERYTHING in the universe, since you cannot assume that interaction of things doesn't exist beyond a certain distance.
Of course one can do rough approximations with a much simpler set of data. But then you're only doing a rough approximation, not an exact prediction. But for the ones claiming that a physically based prediction would be theoretically possible, the above applies. (Apart from the fact that all discussion about prediction and/or predetermination is superfluid, when looking at it from the point of a wider spanning framework - but that's another story.)

No. That thinking of "gimme a big enough computer, and gimme enough data, then I'll compute you everything that will happen in future" belongs to childs in sandpits - read: scientific childs in scientific sandpits.


>> [stephanV]
>> If you create a model from experience you are only confirming what you already thought.
>> If your model goes against your test results, you say the model is wrong, but if your model
>> confirms them you say it is right.
>> But almost no one ever considers that the test results themselves might be wrong.

Ah, we start getting closer. Although the wording of "wrong test results" is not too lucky. If you made a test, and the test finished with a result, then there *is* a result, being the necessary product of the test setup, and can't be lied away. However, what might be wrong (and ever is) is the test setup. To create a correct test setup, everything belonging to the test system has to be considered. This includes man itself, and that's the point that always gets dismissed. (I'm not aiming at humans making errors during the experiment. I'm aiming at humans being part of the system.)
It's been so long now since the equivalence of energy and matter has been revealed (to stay on and view from the scientific angle), I wonder why it is so hard to draw the conclusion. If nothing else, at least this one simple conclusion: There is no matter. Not really. Not how we usually understand it. Matter is a special appearance of energy - thus there lastly is no point in differentiating them. There is energy, only energy, and nothing else but energy. There is no such (discrete) thing as matter. That's the point that really must be understood. Understood not only by cognitive thinking, but by heart, mind and soul. Only if one has really internalized this, a way leading out of the dead-end-road of our current sciences appears, and the possibility of getting a faint idea about the spanning interrelations arises. Which, in fact, make the traditional sciences appear a little ... obsolete.

Hell, the scientists can't even give a senseful explanation what "energy" is, after all. All our technics are up and running, but in last instance, no one can explain why it is so. Even our "hard sciences" are grounded on way too much things that "are just like they are, don't ask" - which demands for nothing more than "believing", making the sciences sort of a religion themselves. This isn't realized by most, but it is like that indeed.

[duartix]

If there is still anyone interested in the topic of the original post, as Phoenyx pointed out:
Could I point out one thing? Think of a program that can create an infinitely complex picture using a single equation. Fractal compressors are even now being worked on. There was a mathemetician (sp?) (and I don't remember his name) that said that an image of photographic quality could be encoded using about 10k of equations and 500k of base data. As I remember (could be wrong) that was supposed to be a 1200dpi equivalent 16 million color picture.

Can you extrapolate this to 64K of equations (not keys, there's a big diference!) and 370MB of base data? I believe this has some grounds even though the encoding/decoding process could take ages...

But if you allow me to follow the key path, I could do it myself if I wasn't a honest person. For 10 movies, I could have used 37MB of data per movie and just 10 bytes of keys. Give me 37MB of Flash and tell me how long do you want the movie to be...

Without knowing exactly what kind of content Jan Sloot presented, there is no way to guess whcih technical approach was taken.

We don't even know if the box, which was opened later without is consent, was feeding his laptop with just 64K of data or 20GB of video that only his portable could understand, and Philips decided to hide the fact that they were taken...

I guess we'll never know. :(

[Phoenyx]

Originally posted by duartix
If there is still anyone interested in the topic of the original post, as Phoenyx pointed out:


Can you extrapolate this to 64K of equations (not keys, there's a big diference!) and 370MB of base data? I believe this has some grounds even though the encoding/decoding process could take ages...

But if you allow me to follow the key path, I could do it myself if I wasn't a honest person. For 10 movies, I could have used 37MB of data per movie and just 10 bytes of keys. Give me 37MB of Flash and tell me how long do you want the movie to be...

Without knowing exactly what kind of content Jan Sloot presented, there is no way to guess whcih technical approach was taken.

We don't even know if the box, which was opened later without is consent, was feeding his laptop with just 64K of data or 20GB of video that only his portable could understand, and Philips decided to hide the fact that they were taken...

I guess we'll never know. :(
^^^^^^^^^^^^^^^^^^^^^^^^^

And that is the key to this thread. :D

(At least until someone invents a 3D wide area past time viewer, anyway:p )

[unmei]

How can you cheat this test? It's impossible...
By remembering a card that is not in the presented set ..that is actually how i realised the trick as this card then showed up in the "set with your card removed" :rolleyes:

[Soulhunter]

Originally posted by Didée

>> [Soulhunter]
>> But sure, you would still need dozens of parameters to do the tree-fall simulation

You forgot typing the word "centillion" - some dozen centillions of parameters.

Any assumptions....


Common, this was only about -> "Does the tree-fall produce a sound..." !!!

Maybe with a "simple" simulation you can't tell if it produces a sound of 74.964565 dB...

But Im sure you could calculate that it produces a sound somewhere between 70-80 dB !!!


Yes, its a rough approximation as you said...

But as more parameters you know the more accurate the simulation is !!!

For a 100% accurate simulation you would need to know... uhm, everything... :rolleyes:

But even with some dozen parameters you would know that the tree-fall produces a sound !!!


EDIT:

Some "tree-fall-question" links... 1 (http://www.physlink.com/Community/Forums/viewmessages.cfm?Forum=7&Topic=1326), 2 (http://www.eadon.com/phil/fallingtreetb.php), 3 (http://www.spectacle.org/396/scifi/tree.html), 4 (http://www.iidb.org/vbb/archive/index.php/t-84854) !!!

Interesting...

Visualizing (measuring) the sound from far away without actually hearing it would solve the problem !!!

But to visualize the sound, there must be some sort of microphone that "hears" it... :rolleyes:

Wait, the question was "someone" and not "something" !!!


Bye

[SeeMoreDigital]

There's nothing like a good conspiracy theory!

The bloke probably never even existed and was created by some nerd somewhere to see how far the story would run ;)


Cheers

[Klendathu]

the length of this discussion undermines the reputiation of this site.

please stay tuned for our upcoming topic:
my hamster was an agent for the federal governmnent and spied on me.

[Mug Funky]

uh? it was just starting to get interesting... i love arguments about determinism and stuff. it's a bridge between philosophy and science.

[scharfis_brain]

multiple movies into 64k?

you are not even able to store multiple soryboards/scripts (Drehbuecher) in plain ASCII format into 64kb, even if they are heavily compressed (RAR, ZIP or whatever)!

how should fit video & audio in 64k then?

[DK64_MASTER]

I love reading the comments about people who think the card trick is real ESP. It's a well known illusion, and I bet all those testemonials were fake.

[Soulhunter]

Originally posted by Klendathu

please stay tuned for our upcoming topic:
my hamster was an agent for the federal government and spied on me.

I have a better one...

All TV's are working backwards -> The CRT is giant camera-lens !!!

This way the broadcasting stations are able to see what you do while watching TV... :D


Bye

[Mug Funky]

Soulhunter: that's already been done

http://www.online-literature.com/orwell/1984/

Winston Smith! excercise harder! (says the TV)

[Soulhunter]

Originally posted by Mug Funky
Soulhunter: that's already been done

http://www.online-literature.com/orwell/1984/

Winston Smith! excercise harder! (says the TV)


Ok, whats with this one...

All politicians are actors !!!

A dictator controls the whole world... :eek:


Bye

[SeeMoreDigital]

Originally posted by Soulhunter
A dictator controls the whole world... :eek: Don't you know... a modern democracy is a dictatorship ;)


Cheers

[LordRPI]

Originally posted by scharfis_brain
multiple movies into 64k?

you are not even able to store multiple soryboards/scripts (Drehbuecher) in plain ASCII format into 64kb, even if they are heavily compressed (RAR, ZIP or whatever)!

how should fit video & audio in 64k then?

Ok, new idea here. You don't store anything in your movie file. That file just unlocks part of your decoder which already has your movie information in it :)

[Joe Fenton]

I think some people here realize the "trick" of fitting 10 movies into 64K. You work the problem backwards.

Instead of a 64K video player in the computer and ten 37M movies on the media, you just turn it around. Ten 37M movies in a database in the computer and the 64K player program on the media. 64K - 1 byte representing the movie you want to play for the player... I could do that easy. Take a single codec and strip all non-essentials out of it. No input other than selection of 1 of x movies, simple output to display device. It's hardcoded for everything, including looking for the movie data in the 370M database in the PC. Depending on the complexity of the codec I choose and the size and length of the movie, I can determine how good the movies will look in 37M. Easy, easy, easy. Philips probably whacked the guy when some sniggering engineer told the PHBs how they were taken by a con-artist.

[callmeace]

:cool:

Think of an MD5 sum - it's essentially a 'hash' number that is generated by the contents of the file.

Think logically about that.

If you could have a system where a unique number is generated by the contents of a file (for arguments sake we could put a limit of 4 TB on a file size), then any file could be represented uniquely in maybe less than 4KB - size of a text file to hold the hash number and maybe small amount of other data like original filename and maybe some prompts too (sort of like cue file that you get with BIN files)
Now all you would need is to put that together with a well optimised code generator of some kind that would reassemble the original file from the unique hash. Code Generator could maybe be less than 60KB

The concept is there :cool: I believe it could be perfectly feasible as I've explained above.

[Mug Funky]

hashes don't work like that. otherwise we'd be seeing ZIP and RAR using these techniques, and nobody would ever need more than a 56k connection.

[Cyberman]

Originally posted by scharfis_brain
multiple movies into 64k?

you are not even able to store multiple soryboards/scripts (Drehbuecher) in plain ASCII format into 64kb, even if they are heavily compressed (RAR, ZIP or whatever)!

how should fit video & audio in 64k then?

The text isn´t stored at maximum efficiency(sp?). The compression had to check for whole words, storing THEM instead of the characters.

For example:

1 = codec
2 = computer
3 = video
4 = encoding
5 = using
6 = my

"I am 5 a 1 for 4 of 6 3 on 6 2."

Quite short, ain´t it? Reversed, it´ll become:

"I am using a codec for encoding of my video on my computer."

The downside is that everything has to be known in advance.

[theReal]

The text isn´t stored at maximum efficiency(sp?). The compression had to check for whole words, storing THEM instead of the characters.

For example:

1 = codec
2 = computer
3 = video
4 = encoding
5 = using
6 = my

"I am 5 a 1 for 4 of 6 3 on 6 2."
That's about how Zip and Rar work (only they'd probably first use a number for words like and "am" or "my" because they are the most frequent in a text and so profit most from abbreviation). Every repetitve pattern can be stored much shorter than its actual length.

Now check what scharfis_brain said:
(not) even if they are heavily compressed (RAR, ZIP or whatever)!He wasn't talking about plain ASCII...

[DigitAl56K]

Originally posted by Cyberman
The text isn´t stored at maximum efficiency(sp?). The compression had to check for whole words, storing THEM instead of the characters.

For example:

1 = codec
2 = computer
3 = video
4 = encoding
5 = using
6 = my

"I am 5 a 1 for 4 of 6 3 on 6 2."

Quite short, ain´t it? Reversed, it´ll become:

"I am using a codec for encoding of my video on my computer."

The downside is that everything has to be known in advance.

Good explanation :) If you go back and look at the math I did originally, remembering it was equivelent to looking up just one word per frame, using the minimum possible address size. I.e. in your example, you used 7 indexed lookups to compress part of a sentence "5146362" - modify that to one indexed lookup to represent an entire frame, and you'll get what I got, and still be nowhere near 64k, even for a single movie!

[callmeace]

Originally posted by Mug Funky
hashes don't work like that. otherwise we'd be seeing ZIP and RAR using these techniques, and nobody would ever need more than a 56k connection.

Well, in fact they could work like that if someone took the time (20 years ;) )

It is possible:- given that you put a maximum size on an input file, to have a string of characters together with a sum 'formula' which can recreate any file. It works by reversing, like so:

You have your input file. You have your formula which generates a unique string of characters derived by the contents of the file. Assume this string of characters can be held in 4kb. The formula (which is under 60KB) can recreate the file because the character string is unique and was generated by the contents of the file - there is only one true answer for the 'sum' to add up to; just like any mathematical sum if you like.

Now interestingly, like with any compressor or decompressor it works more efficiently if there are hints given as to what the data will be in advance. So your <64KB of data in addition to the character string and the formula probably would contain a few hints about the file that needs to be recreated, which would surely speed this process up massively.

It's amazing what can be done by the dedicated :)

[Manao]

callmeace : you claim it could be possible to store any file whose size is under M Bytes on N bytes, with N < M. That's impossible. You have 2^M possible different files as input, and only 2^N possible key strings. It's bound to have key strings which reference at least two different input file.

[Mug Funky]

mm. listen to manao...

hashes are useless without the file they are hashed from. all they can do is check whether they were derived from a given set of data. they cannot reproduce that data, any more than you can recover a dead hard disk using only the parity bits.

we can't recover CD-R data if all we have is the ECC codes in the sector and no actual data, but we can often recover the sector if there's enough intact data to make the error-correction work.

[Phanton_13]

It's posible to do this tipe of good compresion but unlike the actual compresors than uses redundancy in files, they need not only to obtains a data string, they need to optain a complex matematical expresion an then store both. But this is not a trivial problem, this is very computational expensive, complex and had some limitation but it can deliver greater performance than actual tecniques, for a 10 megabyte files you need days in the nowdays most power computer on the heart to obtain a good compresion ratio, but you can decompres it it seconds.


Sorry for may bad english.

[Manao]

No, it's simply not possible. Consider compression as a function which associates to a number x another number y=f(x).

If x can range from 0 to M, and y from 0 to N, with N < M, you are bound to have x1 != x2 with f(x1) = f(x2), which means f is non invertible, which means you can't use f for compression.

[virus]

@Manao

maybe it's worth reminding that your "proof" only stands for f(x) defined in a discrete domain and with discrete values for every x, taken from a finite set of values (example: the numbers you can represent with N bits).

(obviously it won't work for continuous functions... take y=0.1x as an example ;))

cheers :)
virus

[Didée]

Originally posted by magicclue

quote:
----------------------------------------------------------------------
Originally posted by Didée
One just cannot port those quantum behaviours into the "real world". That would be the same as saying "an electron can appear either as a wave or as a particle. Therefore, my pencil possibly can appear as a wave as well." - It makes no sense to look at it this way.
----------------------------------------------------------------------

Remember school? Remember physics?
Remember LIGHT?
Light is being transmitted as wave as well as particle.

Now what?


Yes? What "what"?

Draw a conclusion, state a hypothesis, ask a question. But don't sound out things wildly.

(I'm not going to develop a new sight of the world in this thread, never fear. But I'm sure enough of mine.)


edit: added quote, since magicclue deleted his post.

[Atamido]

Originally posted by Manao
callmeace : you claim it could be possible to store any file whose size is under M Bytes on N bytes, with N < M. That's impossible. This has been brought up before, but there is a standing reward of several thousand dollars to anyone that can break this. Basically, you ask for a piece of random data that is X size. Then you must make a program and a "compressed" file that can be used to reconstruct the original file and whose total size together is less than the size of the original file.

Of course people want this to be wrong, but there is no reason to believe that it is. If you can prove it, go collect your money.

[Manao]

Pamel : it's not the same. callmeace was speaking of on program that could compress anything to a lesser size.

Anyway, in your case, it's still not possible. You're bound to find a file that won't be compressed under it's own size. If it was not so, you'd have found an invertible function defined on a finite space, which goes into another finite space whose size ( cardinal ? ) is stricly inferior to the first one. And that doesn't exist.

[callmeace]

:)
Manao & Mug Funky

I was not talking about traditional means of compression - as in 'pointers' and repetitive sequences of data etc, nor when I used the term 'Hash' was I meaning it in as an MD5 type hash (or at least not to be used for a purpose like that) which is only useful to check existing data that you already have.

No! It is a lot more advanced than that, it would need a lot of work and dedication to create such a formula as I suggest.

Let me try and explain it again with a bit more detail :)

The formula would have to be very complex.
The formula would probably have to work at the binary code level (as in analysing the data).
There would have to be given limits on the size of the input file so that logically there would be a limit on the variations of data one could ever have - and hence it is actually possible to make the formula we wnat in the first place ;)
The 'hash' or code which will be generated will so be unique for every variation of code there could be.
The 'hash' will be a unique sequence of letters and numbers generated from the data of the file, the formula will recreate the original file as there can only be one correct answer to the 'result' - that's how the formula works as in 'input data' - unique code generated - reverse using code to recreate what must be the original file as the only answer to the 'sum' (the code) using the formla.

[virus]

Come on guys, 50+ years of development in the realm of information theory are still not enough to prove that you cannot compress arbitrary sequences of random data? There's no algorithm that can do that, and this has been proven a long time ago.

Anyway, I want to take the chance of launching a contest myself. I'm ready to award 1.000.000.000 euro to the one who will build a spacecraft that can travel faster than light ;)

Who knows, maybe with a very complex formula... :)

[Soulhunter]

Originally posted by virus

I'm ready to award 1.000.000.000 euro to the one who will build a spacecraft that can travel faster than light ;)

Is bending the space allowed ???

Or is this cheating... :D


Bye

[callmeace]

Originally posted by virus
Come on guys, 50+ years of development in the realm of information theory are still not enough to prove that you cannot compress arbitrary sequences of random data? There's no algorithm that can do that, and this has been proven a long time ago......

But this is a different approach to 'compression' :)

a unique string is generated from the data contents (to be used to gether with a formula)

The formula can recreate the original data because mathematically with the formula there 'is only one result that it must add up to'

Complex formula, yes :) but skill & dedication would pay off. But it seems the one who did something like this got killed

[fccHandler]

I agree with Manao and virus. It's just not possible to losslessly compress everything that exists. For example, already compressed data isn't able to be further compressed. (And if it is, it only means the original compressor wasn't 100% efficient.) I believe there must exist data that cannot be compressed by ANY method, thus all of the methods we have are compromises in a way. Note how they all seem to be optimized only for specific types of data streams.

ZIP is actually a suite of compression methods, but I know I've seen MPEG videos which can't be zipped to a smaller size. I've also seen Huffyuv produce larger files than uncompressed video (try capturing the static from an unused TV channel). ;)

[r6d2]

Interesting story. Reminded me of this other one:
Once the great mathematician David Hilbert was invited to give a talk on any subject he liked during the early days of air travel. His subject: The Proof of Fermat's Last Theorem.

Needless to say, his talk was eagerly anticipated. The day arrived, the talk was given, and it was brilliant -- but it had nothing at all to do with Fermat's Last Theorem.

After the talk, someone asked Hilbert why he had picked a title that had nothing to do with the talk. His answer: "Oh, that title was just in case the plane crashed." :)

[stegre]

The 'hash' or code which will be generated will so be unique for every variation of code there could be.
If you have a unique "hash" for every possible variation of code, well, that's not really a "hash", but more to the point, there would be no associated compression. It would take as many bits to represent your hash as it would to represent your original item. If there are exactly a million variations of your item, and you uniquely assign each to one of a million codes, what have you gained? You've just "numbered" them.

With the standard meaning of "hash", you have a "many to one" relationship of items to codes. Once you have fewer codes than items, then sure, that's compression, and using the hash "backwards" (to go from a code to its item) is the decompression. But since the codes can map to more than one item, the scheme is either lossy or requires you to put limits on the data.

Say my hash for the 256 possible 8-bit numbers was simply "use the 7 most significant bits". Nice 12.5% compression - but when I decompress, each maps to 2 numbers. If "always choose the even number" - that's lossy compression (result is always within "1" but may not be exact). If my needs were such that odd numbers would never come up (socks in my drawer [in a perfect world!]) then the compression could be considered lossless - but only because of the limitations placed on the original data.

[callmeace]

aargh! :) Okay, I realise I still did not really explain the 'concept' or how it works very well, and my use of the term 'hash' was poor - sorry.

Here is a further attempt to explain how it works (would work)

(1)This way of compression would take very long, and a very dedicated person for sure! It does not use your 'normal' common ways of compression! It does not work by finding repetition in the data or anything like that! This is a different concept of approach to compression, probably because it would take so long to develop the formula. This is the idea I would like you to realise

(2)It is not a way of compression where the method can be compared as being like zip, ppmd, or anything similar. The hash I mention is not like some of you thought I meant (my misuse of the term I guess)

(3)Okay, here is how it can be done. Please bear with the concept.

You have a formula which calculates a result you might say, from an input.(a)If input is some data like a big file, then the result will be a character string of say letters and numbers (what I called 'hash' before :D) (b)or reversing, if the input is character string then the result will be original data file.

Just like maths, the result is computed and there is only one answer to the sum (that is the important concept). Essentially the complex formula works like so 'if the input = 'this' then the result must = 'that'.

Yes, the formula is complicated, but feasible. It does not work like common used methods which generate an compressed archive. It generates a character string derived from the contents of the file and unique to it, and by reversing can recreate the file from the string by determining what it must have been derived from.

(4)The maths could be worked out and indeed I believe has - like I said, dedication and commitment of time and energy, but in the long term a far superior way to what we have currently!

Please I hope you get the concept of how it works. I guess a starting point is consider how an MD5 is generated but realise it is beyond that.

:edited MD5 (had typed MDG )

[scharfis_brain]

the formala, that greates the hash , and then is able to recreate the file from that hash will be at least as large as the file itself -> no compression gain.

[stephanV]

Originally posted by callmeace
Just like maths, the result is computed and there is only one answer to the sum (that is the important concept). Essentially the complex formula works like so 'if the input = 'this' then the result must = 'that'.
That's not just like maths. While 2^2 = 4, the square root of 4 can be 2 and -2, so reversibility would fail. A derivative of a formula has only one answer, but an integral can in principle have an infinite amount of answers. Math is reversible yes, but in above cases there's no way of telling where you are coming from, unless you start to exclude certain inputs, which would the destroy the compression.


Yes, the formula is complicated, but feasible. It does not work like common used methods which generate an compressed archive. It generates a character string derived from the contents of the file and unique to it, and by reversing can recreate the file from the string by determining what it must have been derived from.

As said before, the string created has to be just as unique as the input, and therefor should contain just as many bits. There is just no way you can represent all possible combinations of a certain amount of bits, with a lower amount of bits by using a conversion formula. With a codebook? Yes. By using a formula? No.

[callmeace]

Originally posted by stephanV
That's not just like maths. While 2^2 = 4, the square root of 4 can be 2 and -2, so reversibility would fail. A derivative of a formula has only one answer, but an integral can in principle have an infinite amount of answers. Math is reversible yes, but in above cases there's no way of telling where you are coming from, unless you start to exclude certain inputs, which would the destroy the compression.....

In Maths there is only one correct answer to a sum; It's true :) what you have presented is not differing answers to a sum where a formula has been defined.

I liken the character string when inputted to the formula being effectively maths, and the file is able to be recreated as 'the original file must be this'

[stegre]

Originally posted by callmeace Just like maths, the result is computed and there is only one answer to the sum (that is the important concept)... Yes, the formula is complicated, but feasible. It does not work like common used methods which generate an compressed archive.I think I understand what you're describing - the compressed file is essentially the complex "equation" itself, as opposed to a relatively simple algorithm that reads "data" to create the output. What you are describing, if I understand correctly, is actually the controversial "fractal image compression", which I see was briefly mentioned once in a post by Phoenyx (http://forum.doom9.org/showthread.php?s=&postid=545167&highlight=fractal#post545167) earlier in this thread. The guy's name he's trying to remember is Michael Barnsley, his company is or was Iterated Systems Inc, and he has patents, books, and working programs - some of this dating back to the early 90's. It's all rather controversial, though. More info here (http://fractals.iut.u-bordeaux1.fr/sci-faq/compression.html).

[stephanV]

Originally posted by callmeace
In Maths there is only one correct answer to a sum;
this is just not true, in maths there is very often not only one answer.

y = f(x), where f(x) = sqr(x) has 2 answers for y for each x (except when x = 0). In maths we normally ignore the negative value, but that is not to say it is not a valid answer of course.

[callmeace]

If you have defined the 'rules' (in simple terms i.e. like BODMAS) for a formula then there is only one correct answer to a mathematical sum

[callmeace]

Okay, I have considered another way to further try and explain how the 'compression' works. My apologies also for having failed to give the details coherently prior, I do not mean also to cause offence in any rebuffs to criticisms of this theory (so apologies). Thankyou for the link stegre


So, this is a very, very simplified way of how it would work (the basic idea)

Imagine you had some data which is a long number (maybe even a long sequence of binary code). Of course the number is unique in the sense that only it is that number (please I hope you grasp that). So you put that number into a formula and it breaks it down and produces a character string. Now think of that character string as being an equation which when executed via the formula would give you back the original number. Basically it's a sum. The rules of the formula would have been defined so that it is the case that there is only one correct answer. You can have incredibly long numbers represented as relatively short equations.

That is hardly revolutionary, but of course it's a much much more complex task than this in that the actual breakdown via the formula would certainly be very advanced mathematically (i.e. not just simple multiplication and use of powers)

That's very simplified, but I hope I did a reasonable job of explaining how the actual concept would work. Just imagine what a dedicated and very skilled mathematician and coder could achieve over 2 decades. Huge amounts of (probably binary) data can be broken down into just a string of characters which when put back through the formula are calculated back to reassemble the orginal data, This is far beyond 'standard' ways of compression that are currently popular. (The complexity of the formula does not mean that it has to be that big codewise)

[stegre]

This presented an intriguing possibility. If, in the forward direction, fractal mathematics is good for generating natural looking images, then, in the reverse direction, could it not serve to compress images? Going from a given image to an Iterated Function System that can generate the original (or at least closely resemble it), is known as the inverse problem. This problem remains unsolved.

Barnsley, however, armed with his Collage Theorem, thought he had it solved. He applied for and was granted a software patent and left academia to found Iterated Systems Incorporated (US patent 4,941,193. Alan Sloan is the co-grantee of the patent and co-founder of Iterated Systems.) Barnsley announced his success to the world in the January 1988 issue of BYTE magazine. This article did not address the inverse problem but it did exhibit several images purportedly compressed in excess of 10,000:1. Alas, it was not a breakthrough. The images were given suggestive names such as "Black Forest" and "Monterey Coast" and "Bolivian Girl" but they were all manually constructed. Barnsley's patent has come to be derisively referred to as the "graduate student algorithm.

"Graduate Student Algorithm"
o Acquire a graduate student.
o Give the student a picture.
o And a room with a graphics workstation.
o Lock the door.
o Wait until the student has reverse engineered the picture.
o Open the door.

Attempts to automate this process have met little success. As Barnsley admitted in 1988: "Complex color images require about 100 hours each to encode and 30 minutes to decode on the Masscomp [dual processor workstation]." That's 100 hours with a _person_ guiding the process.John Kominek, in an excerpt from comp.compression FAQ (http://www.faqs.org/faqs/compression-faq/part2/section-8.html)

[callmeace]

I believe that this concept I mention goes beyond that kind of thing stegre - it operates at a raw low level - i.e. it doesn't care and it doesn't matter to it what the data 'represents', it just observes data as being a series of numbers for example.
However, the article is very impressive in it's claims!

[DigitAl56K]

callmeace: I held of posting about the hash-like aspect of your post several days ago, I'm glad others also saw the similarity though.

Here's the problem.

What you are suggesting is that you take some huge amount of data and run it through some algorithm that produces a unique small piece of data that you can run through a reverse algorithm to reproduce the original huge amount of data.

What you have just described by the unique small piece of data can be thought of as a hash, and hashes are not uniquely reversible. We can have a function:

O = f(I) ; where i is the input, f is the hashing algorithm, and o is the output hash

O does not represent a unique result that can be reversed to generate I, it is simply a value that is always generated when f() is applied to I, and f() is usually designed in such a way that is is in fact difficult to generate O for an arbitraty I.

Hashing is not reversible because of hash collisions. It is entirely possible (and in fact likely) that there are are many I's that can generate one O.

Think about it like this: If O is a number from 0 to 999999 (i.e. a million possible combinations), but I can be a huge volume of data that when represented numerically could range 0 to 9999999999999999999999999999999, then you some how have to match 10000000000000000000000000000000 I's to only 1000000 O's. O can not be uniquely reversed.


You can, of course, have f() be some compression algorithm, let's take zip as an example, which can reduce the data in I to a unique O.

O = zip(I)

However, the data in O must be large enough to reverse O to I through the inverse zip:

I = izip(O)

The problem here is that there is no known lossless algorithm that can do this with anywhere near the compression required to achieve this 64Kb miracle.

But we're talking about video here. This doesn't nead to be lossless. However, no lossless algorithm achieves this kind of compression either.

You can talk about some "magical advanced mathematical formula", but in truth you still have to record enough parameters (co-efficients, variables, etc.) to create each frame.

E.g. a formula might look like this:

Pixel(X,Y)=f(X,Y,g(Z)) ; where x and y are the pixel co-ordinates, z is the single parameter we're passing to our magical formula, and g() is the magical formula/algorithm.

When I did my original calculation on this thread, I calculated that you could not even store a single 90 minute movie with one parameter/variable per frame (i.e. "Z") in 64kb, even if the value of the single variable in the formula was limited to the range 0 - 536,870,911. I.e. what this essentially says is that for this magical advanced mathematical formula to exist, there would have to be only 536,870,912 unique frames of video in the entire world (equivelent to ~4000 90 minute movies), and even then, you would not fit it into 64Kb without some further compression.

I'm not trying to suggest there aren't some great video compression algorithms just waiting to be discovered, but some things border so closely on the realm of impossibility that even debating them is almost pointless.

[callmeace]

DigitAl56K
I know this sounds 'oblique' of me, but I can immediately see that (I believe anyway) your belief of it being impossible is not correct. Once again, it is because I have not explained why and even how it works in enough detail. It is so hard to explain it - I will post again when I have thought how to put it down 'better' in a post.

Believe me, the approach to it is not being understood (my fault)

[stephanV]

man, callmeace, don't be so stubborn :)

what you want is really impossible, your formula is just not uniquely reversible unless you exclude certain inputs, which wouldnt make sense as you then can only compress certain kinds of predefined data.

unless you prove the theory right here, i think no one will believe you.

[callmeace]

Okay! let me take another stab at this, here are some items of information which I would like you to bear in mind :)

It is hard for me to communicate how it works to you as a concept because (a)I have noticed most of you have not got away from the common ideas of compression that are in use, I repeat "this way does not work like most of you seem to think" &(b)I am not that good at explaining things like this simply and the formula itself is unknown - I do not have it, I only know the idea and believe it works.

DigitAl56K, I must especially use your post as an example, to repeat - the 'compression' method I refer to does not work like that, it is not done in that way. Your information is valid I'm sure, but it is not an argument against this concept because it is not trying to compress like that.

Here in a differing explanation is how it all would work:

(1)Note that the string of characters (which will be part of the 'archive') is not generated in one stage directly from your input file - the formula it is processed through involves several stages. It is broken down, and you will end up with the character string.

(2)Now, here is the most important thing - the approach to how the 'compression' works - I repeat again, clear your mind of existing ideas because this is not the way that some of you think
As always, this is heavily simplified just for the purposes of myself attempting to convey the concept to yourselves - the maths involved would be complex and advanced and the result of much time and work (years and years worth), so bear that in mind when deciding on the credibility of all this


In the first stage of the input data entering the formula the data is going to be represented as a number. It could be a continuous number but it will more likely be broken up in to smaller sequences of numbers. Mapping is done at this stage only for the purpose of tracking what number sequence belongs to which block of data.
Now, most of you will know what a code-book is. Essentially it's to decipher inputted data that has been 'scrambled' or hidden or represented in a different way or whatever. This is used in the formula (of course it is the rules so the data is not 'meaningless' to the formula as some had thought)

To further clarify and emphasise the above paragraph, the input data has been represented and mapped to long sequences of numbers.

Now here is a simple thing to explain the concept of how it can go through a formula both ways. One way input data is broken down, and the reverse way input character string is rebuilt back up to the original bigger data.
maths says a big number can be represented in fewer characters as an equation like algebra and so on.

If the maths is complex and carefully considered and the mathematician is prepared to dedicate the time it can be done that by use of 'code-books', mapping and breaking big numbers down to sums Much data can go through stages and be finally represented as a character string. But here again is a point. In other methods, sure a hash could be meaningless for the purpose of recreating data, but here it is not! My character string is created by methods involving code-books and mapping, it is not a useless 'hash', it can be processed when fed back into the formula.

I will also point out that I am not denying anyones points or maths in their posts! I am not arguing against what they say, but they were not arguments against this way of compression because you had all seemed to not really understand how this works.

Now, this is maybe not an actual example, This is grossly simplified but it might help explain the concept.
Imagine you had some mapped binary data.
Now imagine that you were literally showing the code as a single big number like 11 trillion 1 hundred thousand billion 10 million and 10 or something :). Well imagine a much much much longer than that. That's what I mean as a concept of representing input data as a number - very long sequences of numbers. Such a long number can be said much shorter by using powers and the like.

Understand that a very advanced and dedicated mathematician will be processing the data through several stages. There will be rules as I've explained. The end data will not be meaningless to the formula when put back into it to recreate the file. I hope noone will deny this point now I've explained. Maths (reversing) can work both ways when there are prompts and rules defined, and neither does data have to be meaningless in those cases.

I almost expect some derision :) and don't blame anyone (my inadequate explanations ;) )but I know most of us probably visit several of the same websites. I have witnessed some very impressive advancements in the last few years and when I look around me I am amazed at some of the technical advancements. Therefore I am more willing to accept what dedication and good use of brains can do especially when people approach things in a brave new way (and commit themselves to a long slog). I believe in this concept strongly, I know it would work.

[fccHandler]

You're saying that we treat the file as one gigantic number, then derive a formula whose answer is that number?

But even for a small file, the number will be too impossibly huge for any human mathematician to work with.

[Manao]

Let's explain once more time. Tell me at which point you disagree with what I say. There's no need for a strong mathematical background to understand :

(1) compression is a process which transforms a file into another file, whose size is inferior to the first file
(2) a file is a finite serie of bits ( let's name it b(i), i = 0..N )
(3) a finite serie of bits can be associated to a unique couple of positive integers by the following formula : b(i), i = 0..N ---> ( N , P = sum(b(i) * 2^i , i = 0..N) )
(4) the formula can be reversed to obtain the finite serie of bits : (N, P) ---> b(i) = (P / 2^i) modulo 2 , i=0..N ( remark that P ranges from 0 to 2^(N+1) - 1 )
(5) a couple of positive integer can be associated to a unique positive number by the formula : (N, P) ---> m = 2^(N+1) - 2 + P ( the uniqueness is because of the previous remark )
(6) there again, the formula can be reversed : m ---> (N, P), where N is the biggest positive integer such as 2^(N+1) <= m + 2, and P is m + 2 - 2^(N+1).

(7) a file can be associated to a unique positive integer ( (2)->(3)->(5) ).
(8) that integer has the following property ( let's call F1 and F2 two files, and M1 and M2 the associated numbers ) : If (size(F1) < size(F2)), M1 < M2.
(9) if we consider all the files whose size is below M, it amounts to consider all positive integer ranging from 0 to 2^(M+2)-2 ( strictly )
(10) compression can then be seen as a function going from positive integers to positive integers.
(11) if it was possible to compress every file, it should be possible to find a function (f) that is defined from [|0..2^(M+2)-2[| = G to [|0..2^(M+2)-2[| \ { at least one integer } = im(f) = H
(12) for the compression to be reversible, the function would have to be surjective ( meaning : for each element of h in H, there is only one element g in G such as f(g) = h ). If it was not so, from the compressed file, you wouldn't be able to recontruct the initial file.
(13) Since G and H are finite spaces, a surjective function is bijective.
(14) f is bijective from G to H, which are finite, so cardinal(G) = cardinal(H), which contradicts (11). So (11) is wrong, so it's not possible.

I hope it's understandable ( it's the first time I write maths in english ), and that there are no mistake.

The false assumption you make in your reasonning is that you claim you can obtain a simple mathematical formula to represent the file. But you never say how you'll write that formula into a file. For example, a very simple formula to code a number N is the following : x = N, which takes as much bits as N to code.

You'll see that I never used any theorem specifics to compression in my explanations, nor do I limit myself to a compression scheme. It's just plain and simple maths.

[callmeace]

You have got the essence of the right path fcchandler, yes! :)

:stage 1 when the big data is inputted is to have the data represented numerically

Mapping of course would be done, in order to have that first huge number broken into sections of smaller numbers where wanted

In fact though, it might be advantageous to leave as many huge numbers as can be dealt with, from the point of view that there are more savings to be had in relation to the size differences to the characters in the equations that add up to it ;)

But really, I am very pleased that you have seen the concept fcchandler. the idea being that Maths adds up to a result and only a result (when using defined rules)in order to both compress and decompress is the essence and this approach is challenging yes, but unobtainable? no!

[fccHandler]

Originally posted by Manao
The false assumption you make in your reasonning is that you claim you can obtain a simple mathematical formula to represent the file. But you never say how you'll write that formula into a file.
I don't think it's a false assumption, though it is far-fetched. The compressed data could be a tokenized string. The tokens (characters) would represent numbers and mathematical functions. We'll still need an executable (decompressor) to interpret the tokens, act on them, and calculate the result of the equation(s). The result of its calculations will be the uncompressed file.

The main obstacle is how to break down an impossibly massive number into a series of simple equations, if that's even possible. And the idea of waiting years and years for such a compressor to work out an equation for my movie doesn't appeal to me. :p

[duartix]

@DigitAl56K:
Even though I didn't quite follow the "believer's" explanations, I can see you are having a bit of a resistance mainly because you're not opening yourself to diferent approaches. Compressions techniques might not be descriptive but constructive, right?

Allow me to shut the sound in the drawer. :p

As I pointed before, I could create a flash movie that would play for 5 days at 400 FPS with infinite resolution, and that well under 64K. Think South Park... :)

You can look at fractal compression as an more advanced constructive approach than Flash. You can have a 300000000 by 200000000 pixel image with half a dozen parameters.

Current compression techniques are meeting their limits as are computer physics, unless we turn our heads to new ways we can never balance an egg on a table... ;)

[callmeace]

I can understand that most of you do doubt, but the only reason that you do is because you are approaching the means of how the calculations would be carried out from the 'wrong angle'

More on that when I get a decent post together
@Manao I will likewise try and answer your post.

I am pleased that so far progress has been made towards perhaps addressing the 'challenge', which can only be started after realising how it could, and indeed that it could, work! :) It's all about realising a concept, though I doubt if anyone is going to sit down for maybe the next decade or so but perhaps it wouldn't take that long with 'open source'? ;)

[fccHandler]

OK, here's my working example of what (I think) callmeace may be talking about...

Let's say we have a file composed of 14 bytes:

0x06, 0x68, 0x02, 0x04, 0xEC, 0xBF, 0x42, 0x28, 0xA3, 0x55, 0x05, 0x0D, 0x36, 0x91

We treat it as a gigantic decimal number:

129,934,811,447,123,020,117,172,145,698,449

We run it through the compressor to discover that it just happens to be the result of the equation 7 to the power 38. We code this as a tokenized string:

0x07, 0xFF, 0x38

This is similar to BCD. Here, we let the byte values 0 to 99 represent decimal integers (or portions thereof), and byte values 100 to 255 represent math functions, operators, or other tokens yet to be defined. (In this case, we'll let 0xFF represent exponentiation.) The decompressor parses the string, performs the requested function(s), and it's output is the original number (file).

Thus 14 bytes have been losslessly reduced to 3. Granted this is a very simple (and forced) example, but my proposal is that one could extrapolate this idea to even larger numbers which may somehow be reduced to very simple equations.

[virus]

Finding that 1 out of 129,934,811,447,123,020,117,172,145,698,449 sequences can be represented by 3 bytes is not compression, at all. Please, let's not confuse "synthesis" with "compression", because they are different beasts.

"Compression" simply means "taking arbitrary data of any length and reduce its size". The amount of reduction depends on the "features" of the source, of course, and it can be zero.

So, you find that this particular number (let's call it N for the sake of simplicity) can be represented by just 3 bytes calculating a power (7^38). How do you plan to code N-1, then? Because you also need it. You also need to code at least the numbers from 0 to N if you want to have a true compression algorithm (better would be "0 to the smallest power of 2 >= N", and even better "0 to infinity").

Simple. You code it as (7^38)-1. That means that you're applying a different operation (power+subtraction), no? Then, to be able to reconstruct the data you also need to code what type of operation you need to perform. Saying that

7, 255, 38, 255, 1

can reproduce the number is plain wrong. You also need to code the "power+subtraction" thing (the "formula type"), and it takes more bits.

And how do you plan to code N - 1600, then?
Simple: (7^38) - (40^2)!

7, 255, 38, 255, 40, 255, 2, of course, plus the overhead to describe what to do with your data (power, power, then subtraction)

And what about N - 1601 then? ... (and so on) ;)

Eventually you'll end up with a formula for every integer in [0, N], but as you can see you'll need several types of formulas. Eventually much more complex than calculating a couple of powers. So you'll have the overhead for the "type" and all the operands. Do you really think that the average code length (computed on all the codewords, not just 1) will be so low?

(BTW 0x38 is 56 in decimal... that's why I've switched to plain decimal my numbers)

[fccHandler]

Originally posted by virus
Finding that 1 out of 129,934,811,447,123,020,117,172,145,698,449 sequences can be represented by 3 bytes is not compression, at all. Please, let's not confuse "synthesis" with "compression", because they are different beasts.
Fine, so let's forget compression and talk about synthesis.

So, you find that this particular number (let's call it N for the sake of simplicity) can be represented by just 3 bytes calculating a power (7^38). How do you plan to code N-1, then? Because you also need it. You also need to code at least the numbers from 0 to N if you want to have a true compression algorithm (better would be "0 to the smallest power of 2 >= N", and even better "0 to infinity").

Simple. You code it as (7^38)-1. That means that you're applying a different operation (power+subtraction), no? Then, to be able to reconstruct the data you also need to code what type of operation you need to perform. Saying that

7, 255, 38, 255, 1

can reproduce the number is plain wrong. You also need to code the "power+subtraction" thing (the "formula type"), and it takes more bits.
Errm, that code means 7^38^1. We need to define a new token to represent subtraction, let's choose 254. So then we can code it as

7, 255, 38, 254, 1

And yes it will take more bits to code it that way, but let's suppose that the compressor is powerful enough to find a simpler equation for 7^38-N? I don't know that one exists, but if it does then it will be the compressor's job to reduce it to the simplest form.

And how do you plane to code N - 1600, then?
Simple: (7^38) - (40^2)!

7, 255, 38, 255, 40, 255, 2, of course, plus the overhead to describe what to do with your data (power, power, then subtraction)
No, we code it as 7, 255, 38, 254, 40, 255, 2. And to save some bits we can set a fixed rule for the codec that exponentiation is always performed before subtraction (just like programming languages assume). If we must specify operational order, we can define some tokens to represent parentheses, say 253 and 252. (We still have 152 unused symbols.)

Eventually you'll end up with a formula for every integer in [0, N], but as you can see you'll need several types of formulas. Eventually much more complex than calculating a couple of powers. So you'll have the overhead for the "type" and all the operands.
How many operators do we really need?

+ - * / ^ ( )

Wouldn't that just about cover every "simple" equation? Also, I think only the "integer" type should be used. We don't want to get into the hassle of floating-point imprecision.

(BTW 0x38 is 56 in decimal... that's why I've switched to plain decimal my numbers)
oops ;)


P.S. Just to clarify my stance, I don't honestly believe that this idea is at all feasible in the real world, but I can sort of visualize the approach, and I think it has merit. Kind of like how I can imagine travelling to other stars, even though we may never be able to do it...

[fccHandler]

Originally posted by virus
And how do you plan to code N - 1600, then?
And what about N - 1601 then? ... (and so on) ;)
Thinking about this further, (7^38)-(40^2) would be better coded as follows:

7 255 38 254 16 0

(where the BCD "16, 0" = 16 * 100 + 0)

Then (7^38)-1601 would be:

7 255 38 254 16 1

[virus]

Originally posted by fccHandler
And yes it will take more bits to code it that way, but let's suppose that the compressor is powerful enough to find a simpler equation for 7^38-1?
You look a bit too much optimist to me ;)
Simpler than 7^38-1? Using integer numbers only? :eek:

And doing that for all the numbers between 0 and 7^38? nah, you don't need to be a mathematician to see that it's impossible.
Keep in mind, for example, that all the numbers in the form x^y, with x and y in [1, 255] are just a bit more than 65000. Compare this number with the N you wrote before and see how lucky you need to be to find a number which is close to a certain power of an integer number.
If you don't believe me, just draw a random number between 1000 and 9999 and try to do the calculations manually. The formulas will be quite a bit complex, and they'll often need many operators and large operands. And no matter how efficient your bitstream format is going to be, you are going to use a lot of bits...

Look at prime numbers for example: you cannot factorize them so powers and multiplications cannot bring you the result you want (you need to do "something more" to reach it). So you'll probably need to combine several operators here, and probably large operands, too. (and not just three small numbers like 7, 38, 1 ;))

Just to clarify my stance, I don't honestly believe that this idea is at all feasible in the real world, but I can sort of visualize the approach, and I think it has merit.
I can visualize the approach, too. But I don't think it will really compress anything, much in the same way you cannot reproduce efficiently whatever video clip you want using Flash ;)


EDIT: oh, I see you improved your bitstream format :D
what about N - 14897131 then? :devil:
(note that this is still just a small "perturbation" of 7^38, given how large 7^38 is)

EDIT2: stupid grammar error corrected

[fccHandler]

Originally posted by virus
I can visualize the approach, too. But I don't think it will really compress anything
Well, just like other compression ideas it takes advantage of weaknesses in the data. The "weakness" of my chosen number is the fact that it just happens to be equal to 7^38. But other numbers (including the perturbation you posted) may have hidden weaknesses we don't know about. In fact, I seem to remember a theory that all primes > 3 are of the form 6N + 1, so that may be one "weakness" that the compressor (oops, I mean synthesizer) could exploit.

Also, my bitstream format isn't the main obstacle here (I admit it can stand a lot of improvement)! As I see it, the problem is that even with today's technology it would take an impossibly powerful computer to break apart any giant random number and express it as the solution of a simplistic equation. Heck, 7^38 is only 14 bytes. How to synthesize 4GB of a DVD movie? :eek:

But who knows... Maybe the Star Wars DVD just happens to be the solution to the equation (742987 ^ 80175908972457823712) - 1. :D


edit; drat, another math error

[virus]

Originally posted by fccHandler
As I see it, the problem is that even with today's technology it would take an impossibly powerful computer to break apart any giant random number and express it as the solution of a simplistic equation. Heck, 7^38 is only 14 bytes. How to synthesize 4GB of a DVD movie? :eek:
yep, there must be some reason if I limited my analysis to the "pure efficiency" viewpoint ;)

How can we classify such an algorithm?

Really-NP-hard? :D

[fccHandler]

Thinking about it further, virus is right. You only have to experiment a little bit to realize that statistically, the chances of being able to represent any large arbitrary number as the solution of a simple equation, with the representation of the equation being smaller than the representation of the number, are hopelessly small.

@callmeace: Confound it, you almost had me convinced!

It was fun to think about, though. I'm going away now to work on my perpetual motion machine. :)

[Joe Fenton]

Well, I can record 256 movies in just 256 bytes. Just tell me which 256 movies, then I'll send you 180G drive with the player program on it (the ONLY thing on it), and a floppy disk with the 256 byte movie file. :D ;)

[callmeace]

Okay, very interesting reading your posts :) I had kept thinking of how to respond and put across what I wanted, so here goes...

Many of you seem stuck on a problem of how to allow for 'all possible combinations' and of random numbers and so forth. These are presented as disapproving the concept I present, but in my concept, These things are not an issue which will need to be dealt with.

Let me explain I hope :) I think this will answer Manaos post.

Data compression is a challenge. It's hard to compress data but it's not so hard to represent a long number by much less characters which can be put back into a formula, so we first represent the data which we want to compress as a number.

Do you recall in earlier posts here that I said about unique character string? And also that a number 'is only that number'?

Do you see that immediately we have a case in maths where (so long as you set the rules) equations will only give one result.

In stages, What the concept does is first present, via code-book and mapping, the data which you want to compress as a series of numbers, unique to that data for being derived from it. Note that the mapping and code-book use also allows essential 'reversing'. Then in following stages the number is broken down into equations which would use powers and algebra for example, until one is left with a character string.

When you want your original file back, the character string when fed back into the formula generates back up to your original data number, This is because it 'adds up' to the number mathematically, which at the end is reversely written (possible because of the code-book and mapping) back into your file.

Now, how could one get a number from some data? Hmm..... Well, here is one way just as a concept suggestion. On any computer.

Firstly, understand that functions can be performed on the hosting computer, so you want to get a number unique to any file? will you allow that the binary information of the file must exist with any PC? We could use that! It will be a unique number unique only to that file and derived from it.

So in stage one The formula maps this binary information of the input file as continuous code being a long number consisting only of the digits 1 and 0. Let me explain what I mean by that. We literally want to use this binary code as a long numbers without spaces, but there are naturally going to be limits hit on how big a number could be. imagine a string consisting of 0s and 1s in a text document being as an actual number (like 1001 is 'one thousand and one') but being much longer. So rather than having the binary data of the input file as a huge unmanageble number in the form of an incredibly long sequence of binary, this would be cut in sections which are mapped to blocks, as a number at the biggest size which can be handled.

So regarding the above, as the numbers only contain 1s and 0s, even better! How much easier to handle and compress :) We know that they all must be multiples of ten plus with or without an added one!, and the numbers beginning with 0s must have their S.F. (significant figures count) accounted for too.

So, we have lots of mapped blocks each consisting of a big number whose digits are either 1s or 0s.
In case I have lost you, Remember that these blocks are just cut sections of the original huge continuous number which is the binary code of the input file. We want to break this number down through a series of formulas and various Mathematical and other compression techniques like mapping, to a string of characters. The string of characters when put back into the formula will then generate that number (which will then be put back to your original data)

For the sake of argument lets say that a thousand thousands are a million, a thousand millions are a billion, a thousand billions are a trillion, and a thousand trillions are a zillion. This is much much smaller than it would really be but lets say each number (block) doesn't exceed the zillions.
I will insert commas here just for clarity, Imagine then that the first block has the number
Block01: 10,010,001,010,011,100
the second has
Block02: 110,101,001,110,101,001

and so on


in this stage it would maybe be that every block is given a multiple or power of ten number and flagged as having plus one or not and a SF number - this is just a suggestion

Remember that all this work is taking place running through the process, to be under 64KB its only the formula and the character string total we are concerned about.

In the following stages I'll leave to your imagination the advanced calculations from years of mathematical thinking which could be employed. But you would end up with a character string.

In all this quite briefly stated above I think I have shown the concept better than before. Can you see that there is no need to worry about infinite combinations of numbers and the like, there is no concerns about hashes not being reversible or anything like that. Those issues would not be encountered. The maths itself is simple but the mathematical techniques and knowledge would assuredly be complex and advanced to generate the character string.


Dificult and a painstaking task requiring dedication, yes. But it's not unrealistic, impossible or anything like that.


:edited add some clarity I hope

[stephanV]

i could understand fcc, with callmeace im lost...:confused:

Do you see that immediately we have a case in maths where (so long as you set the rules) equations will only give one result
rules create exceptions and you cant have that with compression. its not acceptable to not being able to compress a random set of data.

[Manao]

In what way did you answer my previous post ? Did you find a point you could contradict ? If not, then whatever compressing schemes you could propose, my reasoning would invalidate it. Now that I gave a proof, you either have to proove what you're claiming ( which you don't ) or proove me wrong ( which you didn't ). Any other attempts would be only trolling attempts.

And when will you understand that '139' written as a charecter string ( 3 bytes ) is three times longer that '139' written in hexa decimal ( 0x8b : only one bytes )

Don't you see that you'll have to store your formula ? And that you're greatly underestimating the size such a formula would take to be stored ?

And how can you say such things as "So regarding the above, as the numbers only contain 1s and 0s, even better! How much easier to handle and compress" ???

[callmeace]

@manao

I hoped when I explained my 'approach' to compression I was explaining why what you describe in your post

i.e. "...The false assumption you make in your reasonning is that you claim you can obtain a simple mathematical formula to represent the file. But you never say how you'll write that formula into a file. For example, a very simple formula to code a number N is the following : x = N, which takes as much bits as N to code...."

would not be a problem

It is because the original code is seen as a different number (binary is strung together and seen as a very long number consisting of 1s and 0s)

@stephanV :) the forumla which is only going to have numbers consisting of 1s and 0s should not encounter any problems - what exceptions to the rules could you suggest (for example)?

[virus]

@callmeace

man, you're really hopeless at explaining stuff! :D

Anyway, saying "the advanced calculations from years of mathematical thinking which could be employed" or "the mathematical techniques and knowledge would assuredly be complex and advanced" or maybe "I'm sure a man will come on his white horse and will save the world" :D looks like hot air to me.

No offence intended, but I need something more than hope ;)

So in stage one The formula maps this binary information of the input file as continuous code being a long number consisting only of the digits 1 and 0.
Well, the input sequence is already a binary string made up of 0s and 1s. Using an "one-to-one" function to remap to another binary string is not going to change anything.
Some inputs would be mapped in a string that compresses better, some others not. On average, you'll end up on par... so this step already looks useless ;)

Anyway: please give us a simple example, so we can understand step by step what you want to do with your input sequence. Here's a 6 bytes sequence:

159, 174, 181, 148, 136, 131

translated in binary, this is:

10011111 10101110 10110101 10010100 10001000 10000011

so uncompressed, the input string is 48 bits, as you can see.

Note that the data present correlation (= similarity) between the samples and that means you can compress them. The data looks quite like the luma taken along a row of a video frame, so we're dealing with something real here.

Please shows us your idea and produce an output binary string (shorter than 48 bits, possibly :)). You can do whatever you want, up to and including splitting the string into a couple of pieces to ease your calculations.
Later on I'll show you a very simple and fast way to compress it down, so we can compare the results.

[callmeace]

@Virus :) I can immediately see that my way is not understood (the 1s and 0s are mapped to continuous numbers no spaces), but for now:

Further clarification, I believe some of you will now 'get this'. Here are some of the advantages of the way.

Firstly, surely none of you are going to deny that a weakness of normal methods of compression is that they only perform best with data that they are optimised for. And as some of you mentioned, there is a weakness with some organisations of data - unexpected or unallowed for circumstances etc that will 'foil' the plan.

Do you recall in a previous post when I mentioned that this way of compression doesn't care what data is fed into it? Which is why I say any data will be compressed well with my way. All data is transferred to long numbers of 1s and 0s. Full Stop for now, think about that.

Manao, I can only reiterrate that I'm not arguing over your maths or denying its correctness or whatever, but I will point out that your maths is simply not applicable to my way of compression. Those 'limits' do not apply to it's processes. I urge you (and others) to please get away from traditional ideas and try and understand this new way.

There will be no problems with some data not fitting into the formula accurately or some set of data not being compressible or the problems or limitations that are currently faced. Note - I have mentioned that there will probably be a limit on file input size, vaguely suggested as 4TB, just for the sake of the character string & mapping data fitting into a few KB.

Also, I don't know why you can't see that as the numbers are all multiples of ten and with our without plus one, that this immediately makes it easier for the formula to break down much more? I still think this suggests that people aren't understanding how it works.

We are all here to learn and want to have better technology right? :) I will strive to try and be clearer at explaining, but please be patient! Some of you keep getting hung up on problems that are not going to occur with the concept I mention.

[Manao]

I will point out that your maths is simply not applicable to my way of compressionAlright, so you didn't even bother to read what I wrote. If you did, you'd have remarked that my proof was valid for any compression scheme which takes a file as input and gives a file as output.please get away from traditional ideas and try and understand this new wayPlease, it's your only argument is this discussion, and it's not even a valid one, so could you stop repeating it over and over ?

[callmeace]

Originally posted by Manao
...Alright, so you didn't even bother to read what I wrote. If you did, you'd have remarked that my proof was valid for any compression scheme which takes a file as input and gives a file as output...

Manao, I detect some irritability in your post. I understand that and I don't mind because I can see you haven't understood my explanations.
I can't say any other way than nicely to you though, that your statement above is just not correct "my proof was valid for any compression scheme which takes a file as input and gives a file as output".
In my concept the code is not being compressed 'as code', it is literally being translated to something which is not code but a number. However that number is still unique to the code and meaningful to it. Do you see? The binary digits are strung together (mapped) to make long numbers (no longer being binary code) and these numbers are broken down via maths.

You see, your maths is fine - I don't deny it! But we are approaching compression in a different way where old rules and limitations don't have to apply.

We are all here to learn and make things better so I hope there is no hostility over a misunderstanding :)

[virus]

Originally posted by callmeace
The binary digits are strung together (mapped) to make long numbers (no longer being binary code) and these numbers are broken down via maths.
:rolleyes:

And how the hell are you supposed to work with these "long numbers" on a digital finite state machine like a Pentium-class processor, with binary registers with a finite length, if you don't represent them as a binary sequence?

And how the hell are you supposed to work with your numbers ("break them down via maths") if you can't even manipulate them? Representing exactly a real number requires up to infinite bits!

Seriously, either give a practical example, or shut up. I'm tired of reading these sentences "my way is better, my way has no limits, my way is not traditional, my ideas are new and old approaches don't matter". This is BS at the most irritating level. You can go on forever with that but you won't find any listeners anymore.

If you cared to give an example instead of repeating "my way is better, you don't understand me" over and over and over I would already have torn it in pieces, showing you the practical problems which arise when you finally stop dreaming and start working with something real.

[Manao]

In my concept the code is not being compressed 'as code', it is literally being translated to something which is not code but a numberThank you for making my point. Go read my post, especially (7).we are approaching compression in a different way where old rules and limitations don't have to apply.Maths still apply : you're the one using an uber-mathematical compression scheme, and you don't like it when the same maths prove you wrong ?I detect some irritability in your post.Indeed, there is some. Because I more and more think you're deliberatly trolling here, and that by doing so, you're misleading other people.you haven't understood my explanationsBut I have. In the other hand, you, obviously, didn't either understand or read mine nor any other explanations given by others.

[stephanV]

Originally posted by callmeace
I will insert commas here just for clarity, Imagine then that the first block has the number
Block01: 10,010,001,010,011,100
the second has
Block02: 110,101,001,110,101,001

and so on


in this stage it would maybe be that every block is given a multiple or power of ten number and flagged as having plus one or not and a SF number - this is just a suggestion



callmeace, this just doesnt make sense

lets try your way of compressing on the first block (at least in the way i understand you, if not you should correct me.

first we would need to define how many bits we need for all our characters:

we have all numbers ---> 0,1,2,3,4,5,6,7,8,9
a sign for 10^ ---> *
a sign to add 1 ---> +

so we have 12 signs and thus we need 4 bits for each character in our string.

the first thing i see as a power of ten is 100 so that would become , *2 (10^2), continuing like this we get:

*2*3*1*3+1*2 (if i didnt make a mistake somewhere)

so we have 12 characters and thus need 48 bits to store 17... thats not compression.

of course you will say i got it all wrong and i dont understand you, but now you give a better example then.

stay away from vague terms like "advanced maths", that doesnt mean anything. we cant use maths that doesnt excist (yet) and neither can we make any true statements about such maths.

[r6d2]

I really don't see why Manao, Virus and others just don't see the brilliance of callmeace's ideas. I just fed his posts through the latest version of Turing's Natural Language interpreter (look it up at the Magic Forum) and it created an output which is an algorythm that works!

In fact, I just compressed LOTR (the three movies) and stored them on my 128-MB pendrive. In fact I'm playing them back right now on my 7" HDTV. Quality is awesome!

[Joe Fenton]

r6d2: love the sarcasm man. :D

callmeace: you need to take a couple basic math classes, dude. You don't know the first thing about math. You're talking sh*t. It's not that people don't understand "your way", it's that YOU don't understand math. You're spewing nonsense. Come back after 4 to 8 years of math and computer science at a reputable university.

[callmeace]

:) @ all

I won't return any hostility towards some of you. I won't respond to a couple of efforts to degrade this thread into one of those ridiculous 'ego-wars' either ;)

Observe the title - "A man claims to store multiple movies on 64kb"

I saw that, I have a way and decided to drop in here and discuss how it can be done. To defend myself ;) , I did apologise a few times in advance knowing that I'm not so great at putting across and explaining things, yet some people decided to be rude and a bit nasty to me, although I have kept it friendly.

Well, most of you think you have your maths which prove that this kind of compression is impossible. Okay then! You have it in your minds that it can't be overcome, and are blocked to approaching the task in a different way where your Maths needn't apply. I know my way will work, doubt all you want but I've re-read all the posts in here several times. My posts explain overall pretty well I think (when all read), but I'm not happy to receive rudeness and ridicule, so I thought "well forget it then" as they say. Probably one day someone will reveal that they've been working on such code and then you'll see :) But seriously, I find it odd the resistance to what is actually quite easy to grasp, oh well.

[virus]

Originally posted by callmeace
You have it in your minds that it can't be overcome, and are blocked to approaching the task in a different way where your Maths needn't apply.
have you considered moving to a different universe? :)

The one we currently live in is so boring, so blocking, so maths-oriented...

[Nic]

@callmeace: Well done in keeping your cool and keeping the conversation civil. Your quite right that having different opinions shouldn't lead to "rudeness and ridicule".

@Joe Fenton: I can see what your trying to say, but your comments were perhaps a little harsh.

I must agree with the majority and think that callmeace's notion of compression is not possible (at least not in the way he thinks, due to too many reasons to list here). But I doubt he can be convinced otherwise, so lets keep it as friendly as possible. :)

Cheers,
-Nic

[Manao]

Okay then! You have it in your minds that it can't be overcome, and are blocked to approaching the task in a different way where your Maths needn't applyYou never said why they couldn't apply. We're still waiting.But seriously, I find it odd the resistance to what is actually quite easy to graspBut you still don't question your reasoning and keep saying we're wrong. And that's even odder.

[stephanV]

Originally posted by callmeace
Observe the title - "A man claims to store multiple movies on 64kb"

I saw that, I have a way and decided to drop in here and discuss how it can be done.

Then discuss it, and stay away from vagueness. Until so far you're ideas have been either wrong or not understandable. You're best argument till so far has been that with a lot of effort somewhere in the future it could be done. Sorry, that's just not good enough.

[virus]

A few quotes from callmeace:
Originally posted by callmeace
In the following stages I'll leave to your imagination the advanced calculations from years of mathematical thinking which could be employed. The maths itself is simple but the mathematical techniques and knowledge would assuredly be complex and advanced to generate the character string. You have it in your minds that it can't be overcome, and are blocked to approaching the task in a different way where your Maths needn't apply.
So, maths is advanced, maths is powerful, maths is needed to build this "algorithm", yet it doesn't need to apply. Looks a bit strange to me... and now an invaluable quote from Manao:
Originally posted by Manao
Maths still apply: you're the one using an uber-mathematical compression scheme, and you don't like it when the same maths prove you wrong?
Well said. Damn well said... :)

...do we really need to further "discuss" this topic? ;)


Oh and BTW, I've invented the perfect compression algorithm. Here it goes, step by step:

1) found a company.
2) output several press releases claiming that your compression scheme is the best and that it will work on any data, up to a 1000:1 compression ratio.
3) collect investors' money in a Swiss bank account.
4) state that you're going to demo it privately for Philips and that they're going to buy an exclusive license.
5) collect Philips' rival companies money to have it demoed a few days before Philips.
6) the day before the demo, claim the money collected at points 3 and 5 from your bank.
7) change your name into something less suspicious.
8) fly to Maldives with a couple of top-models.
9) done!

...finally an algorithm that works without that boring math stuff... anyone available for beta testing it? :D

virus

[Manao]

I'm ready to test 8), it seems to be the crucial point of the algorithm :D

[Joe Fenton]

Originally posted by Nic
@Joe Fenton: I can see what your trying to say, but your comments were perhaps a little harsh.

Yes, perhaps. One thing always ticked me off - I'd be sitting in a partial differential equations class and there's ALWAYS one student who just doesn't get it. The prof writes 2 + 2 on the board, then in the next step writes 4, and sure as the sun shines, he's got his hand up - How'd you get that?

AAAAAAAAHHHHHHHHHHH!!!!!!! What are people with no concept of math doing in these classes?

Anywho, this just struck me the same way. People may dream of shortcuts, but sometimes you just HAVE to do three pages of Eigen-function expansions.

I'll try to be more civil in the future.

[fccHandler]

Originally posted by virus
Here's a 6 bytes sequence:

159, 174, 181, 148, 136, 131

translated in binary, this is:

10011111 10101110 10110101 10010100 10001000 10000011

so uncompressed, the input string is 48 bits, as you can see.

Note that the data present correlation (= similarity) between the samples and that means you can compress them. The data looks quite like the luma taken along a row of a video frame, so we're dealing with something real here.

Please shows us your idea and produce an output binary string (shorter than 48 bits, possibly :)). You can do whatever you want, up to and including splitting the string into a couple of pieces to ease your calculations.
Egad, I'm still wrestling with these numbers, trying to derive a simple equation. That shows you what a pathetic geek I am. ;)

Bytes 159 174 181 148 136 131 strung together, translates to decimal 175,572,719,536,259. That number is evenly divisible by 359, which gives 489,060,500,101. But is that a prime? (I haven't been able to divide it any further.) I'm losing hope that this can be reduced to any kind of equation which I can represent in fewer than six bytes.

@callmeace: If you have some advanced maths which can solve this problem, please post it!

@virus, you're an evil dude. :sly:

[fccHandler]

Originally posted by Joe Fenton
The prof writes 2 + 2 on the board, then in the next step writes 4
:confused: 2 + 2 = 4?

Nonsense. 2 is a prime number, and 4 is a perfect square. Besides, quantum theory would never allow that.

[virus]

Originally posted by fccHandler
Egad, I'm still wrestling with these numbers, trying to derive a simple equation. That shows you what a pathetic geek I am. ;)
...and also what a pathetic algorithm you're using ;)

That number is evenly divisible by 359, which gives 489,060,500,101. But is that a prime?
Hints are not allowed...! :D
But anyway, here's mine: if you multiply a number that takes M bits with one that takes N bits, your result requires up to M+N bits... now try to find prime factors of a number that takes M+N=48 bits... ;)

@virus, you're an evil dude. :sly:
Not only I'm evil, but I can also shrink that sequence down to 39 bits using a trivial algorithm that uses just a few clock cycles per byte :devil:


:)

virus

[fccHandler]

Originally posted by virus
I can also shrink that sequence down to 39 bits using a trivial algorithm that uses just a few clock cycles per byte :devil:
I don't doubt it. But you won't get a movie to fit in 64K using that algorithm either. ;)

[MrTVideo]

Thats the first I heard of that, most interesting. Just to throw a another slant. There are other types of computers, Analog for instance and lets not forget what we have on our shoulders. I believe we store in pattens and even I have more cells than any computer made, shame I cannot recall digitally. So when I went over this thread I get the feeling that this box of his was indeed an alternative memory device "Programed to store and play movies". Which we do in dreams. Don't make sense but movies just the same and in colour. I said I would like to throw a slant

[callmeace]

Originally posted by virus
A few quotes from callmeace:

So, maths is advanced, maths is powerful, maths is needed to build this "algorithm", yet it doesn't need to apply. Looks a bit strange to me...

;) The maths that needn't apply is that which has been used by many of you to 'prove' the impossibility of my compression claims. Selectively quoting me so as to misrepresent the context in which those quotes were originally posted is rather naughty virus but no hard feelings ;)

I have been working on a post which will help most of you understand.

To give stephanV some answers in the meanwhile:
with my way you wouldn't have to store all the information at each stage you only tell it what to do with the result. Example: lets say you have the input. We know it is going to be seen as blocks of numbers which must be a multiple of ten with or without +1. Lets say we break the blocks down to what adds up to each number. You don't have to store this number becaused it's only used while the formula is running. Each step generates the content to be processed by the next step (either way through the formula). In other words, the point none of you seem to understand - you don't have to store the contents of the input file in my way, I'm not using normal methods there is not any 'magic' compression. I have said this before.
When you have your end character string the original file can be built back up in stages through the formula - each result being processed in the next step.

I await more ridicule, but either way I will get an explanatory post finished which tries to give an overall concept view of how it might work.

[duartix]

Hey! I can't compress "top models" to less than 10 bytes. :(
Oh s**t! I forgot "top models" are already fully zipped! :o
Wouldn't mind unziping one, though! :cool: :D

[MrTVideo]

callmeace your theory makes sense in as far as the 64k(if I get your drift) holds the formular and the assembly code to dissasemble and then reassemble for input and output. But is'ent that just a "codec" anyway, where the actual information is kept in in another location.?

[gldblade]

I think your basic idea is that given a set of data, you can represent it with a smaller set of data that can be used to reconstruct the original data. Then you hope to perform multiple passes, each time needing less and less data to represent previous passes, in the end reaching an extremely small file size.

Based upon this synopsis, you will require an algorithm that satisfies two requirements: it can indefinitely reduce filesize, it is completely reversible.

I believe that you stated it was up to math to find such an algorithm. However, I will state that it is ignorance that suggests that such an algorithm exists in the first place. My rudimentary understanding of information theory suggests that such a process cannot possibly exist, and if you believe otherwise, please prove it by locating such an algorithm. Then proceed to contact the many information theory scientists across the globe to tell them that their decades of research is wrong.

*EDIT*
Just to clarify, you can come up with an algorithm that can indefinitely reduce filesize (approximately at least, you can't get down below 1 bit). You can come up with an algorithm that is reversible. But you can't come up with an algorithm that can do both.

The reason is that reducing filesize intrinsically loses information. You can have lossless compression up to a certain (mathematically proven) limit using various tricks, afterwards lossy compression is required to further decrease filesize. Since information is lost in the process, you cannot fully reconstruct previous data. With each pass, more and more information would be lost, meaning less and less of the original data.

For example, let's take an algorithm where you merely divide by 10 and round.

123456
12346
1235
124
12
1

Information is clearly lost with each iteration such that one would have less and less hope of reconstructing the original data the more iterations you run.

You may claim that there is an algorithm out there somewhere that will somehow meet both requirements of indefinite filesize reduction and full reversibility. Once again, I would state that this is due to a lack of understanding on the limits of mathematics.

An alternative approach to having a single algorithm is to have multiple formulas that will each individually reduce the size of the data, which is probably one of the ideas that you were trying to get across.

For example,

120, formula: divide by 5
24, formula: divide by 4
6, formula: divide by 3
2, formula: divide by 2
1

In which case, if you knew every single formula in the chain, you could theoretically reduce the orignal data to a very small filesize. However, even this approach has some major issues. First, will these formulas be predetermined, or will they be generated on the fly?

If the formulas are predetermined, then the process can only represent a *very* limited subset of data. For example, the chain above cannot represent a starting value of 121. Therefore, to be practical, the formulas would need to be generated on the fly.

However, how then do you tell the decoder what to do with the end data? You would need to store the formula chain somehow, probably alongside the processed data that you will use to reconstruct the original data. But then have you really reduced the filesize? Doesn't the storage of the formula chain count towards the filesize? If you take this into account, I think you will realize that this imposes a significant limit on your compression efficiency, and may even exact a severe penalty in some cases. Using the example above, storage of the number 120 requires only 8 bits. Storage of the number 1, along with the formula chain, would require many more than that.

I must applaud anyone who read through all of this. I can't believe I actually wrote this much considering its 4:25 in the morning. I must apologize for lack of coherency anywhere :P

[MrTVideo]

No if its possible at all then it would be necessary to start backwards 64000 bytes each byte how many bits ( lets give ourselves a chance say 32) so we have 64000 x 32 bits of information to work with
so how could that array of bits be used to store one movie say a 4.7 gig mpeg2 already heavily compressed. Here is where, if it can be believed that it was done at all. obviously not by compression algorithms but what?. Consider a movie of 4.7 gig 4,700,000,000 explained in 64000x32 = 2,048,000
which would be a compression ratio of 4,700,000,000/2,048,000
229,492:1 I think again that if it was done at all you would have to ask how much information could be stored. Even if some form of look up table was used that would have to be stored external.

I think turning water into wine would be easier. wait a bit I brew my own beer. I use water add flavour add sugar and after a bit drink it.
Nothing is impossible

[gldblade]

@MrTVideo

Sorry, I was talking to callmeace :P I don't think what he is proposing is possible

[Soulhunter]

Dunno what you all complain about..:confused:

Ive tested this compression method, and it works excellent !!!

Just watch this screenshot (http://img93.exs.cx/my.php?loc=img93&image=341.jpg) from a Matrix Reloaded -> 3x floppy-disk encode... :D


Bye

[stephanV]

Originally posted by callmeace
In other words, the point none of you seem to understand - you don't have to store the contents of the input file in my way, I'm not using normal methods there is not any 'magic' compression. I have said this before. When you have your end character string the original file can be built back up in stages through the formula - each result being processed in the next step.


Yes, but in my example you havent reached any compression at all. You have actually inflated the file by a factor of 3... so what would the next step do?

[MrTVideo]

Ah! this is where we have to forget about digital. in a previous post I said that we think in pattens and mentioned other types of computers hoping for a response. Ok well I will propose another massive storage device. You have to start moving into electronics and analog.
Suppose that 64K is made not with gates as in digital computers, holding a two state code, but capacitors that can be individualy programed to a set voltage(Patten) assume each bit(capacitor) can hold 10 volts(Back to digital notation) and we use a mathmatical notation to base 10 then the limitation of the code is the number of places of decimal the voltage can be written and read to. In theory if the actual size of the storage device is not critical then I cannot see a problem other than speed of info transfer.
There are other options but the theory's starts to get bit thick.

Digital storage is finite not that you cannot create more storage area just exactly how much you can put in a given space.

1. Would involve being able to iradiate and un-iradiate (if thats the term)a passive substance and measure the level. I do not know much about nuclear theory but believe it possible.

2. Using a photo electric cell and store a light value converted to a voltage. IE: An optical computer.

Impossible has never been a term I can be comfortable with, especially, as in my 62 years, having seen the impossible occur many times.

[Manao]

MrTVideo : "Suppose that 64K is made not with gates" ---> then, you're not speaking of 64Kbits anymore :)

[fccHandler]

"Gates"? You mean Bill? :D

[MrTVideo]

Of course not, we are now in the realm of analog computing and remember a figure of 64k was originaly stated. A Rat has more memory cells than a cray computer maybe he used a rat. (Only Joking) I am just throwing alternative theory's around. I read a fiction story which had a truth about it. A group of scientists were told another govenment had solved the anti_gravity problem. Because the story was supported by that countries leader they believed it was true The outcome a device crude but working was made. Now that was fiction. When I was a boy I submited a essay to my science teacher on what was the most significant event I would witness. I wrote an account of how a man would make the journey, land on and return from the moon. That was in 1958 I gave a date of 20 years later. I was penalised to write 500 times a mans body could not stand the escape velocity of what? I think 22,000 miles/hr. Just goes to show what is thought impossible today is reality tomorrow. All of you can attest to that, we now have a global world not a flat one. Sorry about going on so much. I will not even attempt to tell you where I think Einstien went wrong only to quote that fantastic sentance "I think therefore I am"

[Doobie]

Originally posted by MrTVideo I was penalised to write 500 times a mans body could not stand the escape velocity of what? I think 22,000 miles/hr. Just goes to show what is thought impossible today is reality tomorrow.

Escape velocity has nothing to do with sending men to the moon. We use rockets, not cannons.

[MrTVideo]

I agree but remember back then they only had v1's and Verna Von Braun
to go on, It was the speed that scientists of the day could'ent get a handle on and remember it was much later that the "G" suits were perfected to stop a persons blood from being pushed to one side of their body. Canon or rocket a person still has to reach that speed (before the fuel/Kinetic energy runs out) in order to escape the earths gravity ergo "escape velocity". There was a lot of work done on rocket sleds and yes some were like cannons. but this is getting off track.

[r6d2]

Originally posted by Doobie
Escape velocity has nothing to do with sending men to the moon. We use rockets, not cannons. We use rockets because we cannot use cannons. If we did, men on board would end up like an omelette. :)

AFAIK, we use rockets to achieve escape velocity (http://en.wikipedia.org/wiki/Escape_velocity) gradually. In fact, we take advantage of the fact that escape velocity decreases significantly with distance to the Earth center of gravity.

If at any point during flight the rocket stops propelling, and speed is lower than escape velocity at that point, it falls down. You know what they say: the higher they fly, the harder they fall. ;)

Maybe that's one possible way of compressing a movie into 64-KB. :D:D:D

[MrTVideo]

Whats that got to do with compression :D I am still waiting to hear from some out there who have a real wide personal paradyne. I scored a response but not in the right direction

[r6d2]

Originally posted by MrTVideo
Whats that got to do with compression :D Well, the falling rocket would end up very compressed. Just like the astronauts on the cannon. :D:D:D

[MrTVideo]

WoW I looked at that article on ESCAPE VELOCITY at 9000 km you still need a 7km/sec kick in the pants. You can see why I had to write those lines (7km/sec. 420km/hr) I read an article on just how a rocket can get in space. All to do with how fast they can dispose of their fuel (A super garbage disposal device). This really stuffs the original thread. I apologuise but could not resist the banter, Good will to all.
Terry ;)


Spelling correction I hate it when I do that

[Monkeychops]

If I remember my Astrophysics lectures correctly an escape velocity of 11Km/s (not sure if this is correct but dont want to work it out) is necessary if and only if the rocket etc is not propelled in any way after launch.

Effectively rockets can get into space because they expend the amount of energy necessary to increase their potential energy such that they escape Earth's gravity field (not totally but good enough.)

By a similar token, if you had a long enough ladder you could climb it to the Moon by expending the energy to raise your potential energy.

[Foofaraw]

Wow the arrogance in this thread :) "I can't imagine any way to do this, so it will never be possible" - Tsk tsk.

Well it was always thus, most people saying something was impossible and then suddenly someone has a brilliant idea and something new becomes possible.

I'd like to think there are great compression ideas yet to be discovered, and unless 3D storage becomes so cheap that nobody cares about size I'm sure we'll find them eventually (unless we nuke the planet)

And hey, the people who make Stuffit just made a new compression algorithm that will losslessly compress JPEG files by an average of 30% (as opposed to zip's 0%-1%)

What do you know :D
http://compression.ca/act/act-jpeg.html

[Joe Fenton]

Originally posted by Foofaraw
Wow the arrogance in this thread :) "I can't imagine any way to do this, so it will never be possible" - Tsk tsk.

And hey, the people who make Stuffit just made a new compression algorithm that will losslessly compress JPEG files by an average of 30% (as opposed to zip's 0%-1%)

It's not arrogance, it's intelligence. He didn't claim his movie compression saved 30%, he claimed it saved 10,000%. If you can't see the difference, you would certainly mistake intelligence for arrogance. :rolleyes: