I really should sleep instead of thinking about stuff like this

So I ran into something a bit new for me, a comment about entropy as a measure of information density, or, a bit more commonly phrased, as variability and unpredictability in random numbers and whatnot.

How do unpredictability and information density relate to eachother? This is how I see it. If something is predictable, you can describe it in a shorter length than itself... Er, that's not well phrased. An example, then. Take a string of 0's about five million characters long. That's stupidly predictable, but I described it exactly with only a bit over 50 characters. If the string of characters were more unpredictable, like, say, the first few million digits of an arbitrary irrational number, it would be much harder to condense it down.

Let's switch examples a second. Suppose I have a long piece of text that has the word "kaleidoscope" in it quite a bit. If I want to save space, I could just stick a short description at the beginning of it saying "KDS = kaleidoscope" and the replace all other instances of kaleidoscope with KDS. As long as the amount of space I save (The number of times I say kaleidoscope multiplied by the difference in space between kaleidoscope and KDS) is greater than the amount of space the explanation requires, I profit. Or at least increase information density. And if another phrase happens to pop up a lot with KDS in it, you can define that by some other term, and so on and so forth, building it up until you reach the limit.

Of course, there are problems. If I drop a letter in "TLA", it would be a lot harder to comprehend what I meant than if I dropped a letter in the phrase "three letter acronym". The redundancy of natural language helps us understand it despite occasiona mistakes. (I actually left that l out unintentionally, but it helps serve the point, right?) That, and it should also be considered that it still takes a while to grok all that information. Condensing information is different than simplifying it. If I have "foo means quuuuuz", you still need to understand quuuuuz to understand what I'm talking about each time foo comes up. And the "key" (the cipher analogy is actually pretty relevant) has a lot more important information than the rest of the book, or at least, losing it will make a lot of the rest of the book completely incomprehensible. And, at the extreme ends, one begins to run up against the fact that there's only so many letter combinations at the short levels one can use as abbreviations of words and neologisms expressing phrases, especially letter combinations you haven't already attributed some other meaning to. That bit's important, as the whole point is condensing information without losing meaning, and if you create ambiguity you lose meaning.

Anyway. I don't know quite what the point of most of these ramblings where. Probably, as the subject line implies, that I should get some sleep. But I wanted to get these thoughts down as I was having them, or a reasonably short time thereafter. And share, in case you're interested. A few key things here I'll have to think about later, including meaning attribution, and ambiguity. Hmmm. But now sleep.