A serious paper on bits as Joules per Kelvin

My ramblings about thermodynamics aren't so off-base, it turns out!

Remember this one? From 2009? Where I explained how Joules per Kelvin (energy per unit temperature) is a valid measure of information (or entropy, effectively "missing information"), which is normally measured in bits?

Well, now there's a paper that formalizes that idea and related ones. As the title ("Temperature as Joules per Bit") indicates, it looks at a rearranged version of the same insight (mine was "bits as Joules per temperature"). But, it also goes a lot deeper and derives thermodynamics starting from entropy to understand temperature, rather than the other way around, as is conventionally done.

Related thought: I remember back in that 2009 thermodyanmics/info theory frenzy, one of my goals was rederive the Carnot limit based on information-theoretic considerations -- that is, show it as a simple implication of the amount of knowledge you have about a system in a case where only know the temperature difference. (Naturally, I assumed someone had already done this and tried to find it but it was very hard to google for.)

Background: The Carnot limit tells you the maximum amount of mechanical work ("useful energy") you can extract from heat -- like, through an engine -- and, as it turns out, it's a function of the ratio of absoute temperatures you're working between. You don't face this limit when extracting work from a flywheel (spinning disc with grooves). Inspired by an counterintuitive insight in an Eliezer Yudkowsky LessWrong post, and my thoughts about it, I figured you could draw a more direct line from "knowledge of a temperature difference" to "how much energy is extractable".

Now I'll give it a go with ChatGPT, and post my findings!

NAND to Tetris: Nothing short of awesome

Been a while since I posted, huh?

Well, here's an interesting development in my life: I finally started the NAND to Tetris course (part 2) on Coursera, which is based on the instructors' site.

I can say, whole-heartedly, that the course is awesome, and the first thing in a long time that has gotten me into a flow state.

The idea behind the course is this: they take you through (virtually) building a computer from a very basic level, showing all the abstraction layers that produce a computer's behavior.

It starts with logic gates -- specifically, the NAND gate (hence the name), since it's universal. Your first project is to use a hardware simulator to build several other logical gates from NAND. (NAND is just an AND gate with the outputs inverted, so that true and true yield false, everything else yields true.) The second project is to to build the arithmetic logic unit (ALU) in a CPU out of the components, so you basically have a configurable circuit: based on six control bits, you perform one of several possible functions on two 16-bit inputs.

The next project incorporates "flip flops", which allow you to repeatedly execute that circuit while also writing to and reading from some some persisted memory. Your inputs to the circuit then function as a kind of machine code.

The project after that then has you implement functions in that machine code, writing in an assembly language the authors created for the course that has a precise mapping to the machine code inputs in the CPU from the previous lesson. I can honestly say it was a really fun project to implement multiplication directly in assembly language!

Later on you write a compiler from a high level language into assembly (which then converts simply into machine code), in a way that's broken into two steps: a compiler from the high level language into a virtual machine that works on a stack with memory blocks, and a compiler from the virtual machine commands to assembly. I just recently finished that latter part (which comes first).

Those layers then build up to making a game that runs on your CPU, then an OS, and some other stuff I haven't delved into.

In the course of the projects, you use a hardware simulator, an assembler (for converting the assembly language into machine code), a CPU simulator, and a virtual machine simulator.

In order to catch up with the class an have enough leeway for a weekend trip, I went through much of the course over the weekend, and enjoyed every minute of it! I especially like how the break the projects into manageable pieces. For example, in the VM-to-assembly part, it first has you implement simple push/pop/add operations on a stack and run a test to verify that you can compile those commands. Further test suites give you a manageable set of operations to add.

I've written an actual compiler now! (albeit limited use...)

(Consider how fun this is, and how naturally it comes to me, maybe I picked the wrong field.)

One interesting challenge is that the CPU only has two registers call A and D, and only the A register is used when accessing memory, to know which word of memory to look at. It took me a while to figure out how you could say "look at the value in the memory location refered to in the memory location zero." Before I saw how you could do it, I implemented one project by having two separate code branches for the two possible values at the memory location!

Would love to link my code for the projects, but they discourage that to leave the challenge for others to solve.

Some of my geeky tech jokes -- with explanations!

I know the line: explaining a joke is like dissecting a frog; you understand it better, but it dies. Still, not everyone will get these, and I figure I might as well have a place where you at least get a chance. So here are some of my own creations, explained.

---

Girl, you make me feel like a fraudulent prover in a stochastic interactive zero-knowledge proof protocol ... because I really wish I had access to your random private bits!

Explanation: In a stochastic zero-knowledge proof protocol, there is a prover and a verifier, where the former wants to convince the latter of something. But for proof to work, the verifier must give the prover unpredictable challenges. Think of it like a quiz in school -- it's not much of a quiz if you know the exact questions that will be on it.

The information to predict the challenges is known as the verifier's private random bits Those with a legit proof don't need this, but a fraudulent prover does. Thus, a fraudulent prover in a stochastic interactive zero-knolwedge proof protocol wants access to the verifier's "random private bits".

---

A historian, a geologist, and a cryptographer are searching for buried treasure. The historian brings expertise on practices used by treasure hiders, the geologist brings expertise on ideal digging places, and the cryptographer brings expertise on hidden messages.

Shortly after they start working together, the cryptographer announces, "I've found it!!"

The others are delighted: 'Where is it?'

The cryptographer says, "It's underground."

'Okay, but where underground?'

"It's somewhere underground!"

'But where specifically?'

"I don't know, but I know it's underground!"

'Slow down there. If all you know is that it's underground, then in what sense did you "find" anything? We're scarcely better off than when we started!'

"Give me a break! I just gave you an efficiently-computable distinguishing attack that separates the location of the treasure from the output of a random oracle. What more could you want?"

Explanation: In cryptography, an encryption scheme is considered broken if an attacker can find some pattern to the encrypted message -- i.e. they can identify telltale signs that it wasn't generated by a perfect random number generator, a "random oracle". Such a flaw would be called a "distinguishing attack". So in the cryptography world, they don't care if the attack actually allows you to decrypt the message; they stop as soon as they find non-randomness to the encrypted data. Applied to a treasure hunt, this means they would give up as soon as they conclude that the treasure location is non-random, which the cryptographer here things s/he's done simply by concluding that it's "underground".

---

So, 16-year-old Johnnie walked into an Amazon Web Services-run bar...

"Welcome," said the bartender. "What are you drinking?"

Johnnie replied, 'What've you got?'

"Well, we have a selection of wines and the beers you see right here on tap. But if you prefer, we also have club soda and some juices."

Johnnie thought, Wait a second. Why is he telling me about the wines and beers? Does he even realize ... ?

'Okay, I'll take the Guinness.'

"Bottle or draft?"

'Draft.'

"Alright, and how will you be paying?"

Johnnie only had large bills from his summer job and gave the bartender a C-note.

"Sorry, but I gotta check to make sure this is real." The bartender took out a pen and marked it, then counted out the change. Johnnie reached for the beer.

"Hold on a second! Make sure to use a coaster!" The bartender slipped one under the glass. "Okay, now enjoy!"

Johnnie lifted up the glass to drink. Before he was able to sip, the bartender swatted it out of his hand.

"WHAT ARE YOU THINKING!?! Don't you know 16-year-olds can't drink!"

Explanation: On the AWS site, they will gladly let you click on the "Launch server" button and go through numerous screens and last-minute checks to configure it, and only at the very last stage does it say, "oops, turns out you don't have permission to do that" -- so it's like a bartender that takes you through a entire transaction, even verifying irrelevant things (like whether the money is real), while knowing the whole time he can't sell to you.

---

How is a Mongo replica set like an Iowa voter?

In primary elections, they only vote for candidates they think are electable!

Explanation: Databases can have "replica sets" where there are multiple servers that try to have the same data; secondary servers depend on an agreed-upon "primary" to be the "real" source of data. Often times, the primary server goes down, so they have to decide on a new primary, known as a "primary election". But there are some restrictions on who they will vote for -- if they e.g. have reason to believe that a server can't be seen by other members, and in those cases it will regard that server as unelectable. So you can get funny messages about "server42 won't vote for server45 in primary election because it doesn't think it's electable".

Funny Slashdot exchanges, before they're lost to time

In the time that I was a regular reader of Slashdot, I saw a few exchanges that stayed in my mind. I later went back to find them, but was never able to. So that they're not lost to time, I figured I'd post all the ones I remember. What follows is from memory, and prettied up a bit. (Not trying to plagiarize, if you can find the original post for any of these, let me know.)

Enjoy.

---

[Story: Armadillo Aerospace has a failed rocket launch.]

A: Well, I think we can close the books on Carmack's little project.
B: Come on, now. Private space travel is still in its infancy. There are growing pains. Not everything works the first time. But what's important, is that we're learning from these events. Armadillo is learning. They'll adapt. And the next voyage will be better and safer!
C: You mean, even safer than a big orange fireball?

---

A: [long rant] So that's the problem with this ban on incandescent light bulbs.
B: Whoa whoa whoa, slow down. There is no "ban" on incandescent light bulbs. It's just that the government passed new efficiency standards, and incandescents don't meet them.
C: Oh, that's clever! I should try that some time: "See, I'm not breaking up with you! I'm just raising my standards to the point where you no longer qualify."

---

[Story: a pedophile was caught because he took pictures of his acts and tried to blur out the victims' faces, but police analysts were able to unblur them.]

A: Hah! What an amateur! Everyone knows you have to do a true Gaussian blur to destroy the information content of the picture!
B: Yeah, or entropize it by blacking out the whole face.
C: Right. Or, you know, you could just ... not molest children.

(IIRC, C was heavily voted down and criticized for assuming guilt.)

---

[Story: police used "big data" analytics techniques and discovered that most robberies occur on paydays near check-cashing places, which allowed them to ramp up arrests.]

A: I don't know, this seems kind of big-brothery...
B: Not at all! This is the kind of police work we should applaud! Working only off publicly available, non-private data, they found real, actionable correlations. It wasn't just some bigoted cop working off his gut: "Oh, this must be where the thugs go ..." No, they based it on real data. What's more, it let them avoid the trap of guessing the wrong paydays, which can actually vary! Some people get paid weekly, some biweekly, some of the 1st and 15th. For example, I get paid on the 7th and 21st.
C: So, uh ... where do you cash your checks, by chance?

Broken Windows, Part I: The Pain of Hard Choices

This will be the first in a series where I spell out an underappreciated concept in economics and how it leads many economists astray in proposing solutions to economic problems. I figured I better get a start on it before the New Year.

Recently, I've gained some insight into the economic debates between the various camps that claim to have a solution to our current problems. In addition to tying up some loose ends regarding a century-old debate, this insight gave me a good explanation of why standard dismissals of the so-called recalculation story (in explaining recessions like the current one) are making a subtle error.

First, a high-speed recap: Way back in the 1800s, Bastiat described what is known as the "Broken Window Fallacy" to refute the prevailing economic wisdom of the age. Many believed that a vandal who broke a window could be doing the economy a favor, reasoning that the owner would have to hire a glazier to fix the window, who would have new money he could use to buy new shoes, which would give the shoemaker the chance to buy something he wanted, and so on. (Note the early shades of the "multiplier effect" argument.)

Bastiat replied, basically, that no, this doesn't quite work, because you have to account for the "unseen" loss to the window owner, who would have engaged in the exact same economic stimulation as the glazier, had the window not broken, because he would have been able to buy something he wanted -- and we'd get to keep the window, to boot!

This mention of the Broken Windows Fallacy is often brought up in response to proposed Keynesian solutions (involving government stimulus spending), where their opponents say that it makes the same error, neglecting the unseen economic activity that would go on in the absence of the government's spending.

Keynesians, in turn, reply that the Broken Window Fallacy only applies at "full employment", where there is no "crowding out" (i.e. forgone projects due to the government's use of resources for different ones). In a depressed economy, they argue, the alternative to a metaphorical broken window (along with its fixing) is not "the window owner buys something else", but rather, "the window owner hoards that money", providing no economic benefit. Therefore, breaking a window in such a case would not have an economic opportunity cost, and so could indeed be good for the economy -- though Keynesians of course admit there are much better ways to increase employment than breaking a window.

The back-and-forth goes on, of course, with each side claiming that the other's position implies or relies on an absurdity. Keynesians accuse the free-market/"Austrian" types of thinking the economy is always optimally using resources, while Austrians accuse the Keynesians of calling a hurricane "God's gift to depressions".

But here, I think, I've noticed something that tremendously clarifies the debate, and gives us insight into why economic activity does or doesn't happen, and why certain events are or aren't good. So, here goes.

*******

Let's go back to the original Bastiat thought experiment about the broken window. Ask yourself this: Why are we assuming the window will be fixed at all?

Don't misunderstand me: it's a reasonable assumption. But we have to be careful that this assumption isn't fundamentally ignoring relevant economic factors, thereby baking in a desired conclusion from the very beginning. And here, I think we have good reason to believe that's exactly what's going on.

So let's start simple: under what circumstances would it be not be reasonable to assume that the window will be fixed, (i.e. that the owner will choose to pay someone to fix it), even during a depression? That's easy: if the neighborhood (along with that building) is run-down to begin with, already littered with broken windows. A lone broken window merits a quick repair, but if it's yet-another-broken-window, why bother? (Note here the substantive similarity to the homonymous "broken window" effect!)

So here we see the crucial, unappreciated factor: the obviousness of certain production decisions. What these thought experiments -- carefully constructed to make a different point -- actually prove is the importance of being able to confidently decide what is the best use of resources. And we can step back and see the same dynamic in very different contexts.

For example, say an unemployed guy, Joe, is trying all different kinds of things to find a job, and nothing is working. Then while driving one day, makes a wrong turn and steers his car off a bridge into the river below. Not good. But there is one teensy-weensy good part: it's a lot easier to prioritize! Previously, Joe didn't know what he should do to make optimal use of his time. Now, he knows exactly what he needs to work on: avoiding death from falling into a river!

And we can step back even more and generalize further: what we are seeing is but a special case of the law of diminishing returns. Abstractly, each additional unit of satisfaction requires a greater input of factors: land, labor, capital ... and thought (sometimes called "entrepreneurial ability"). Generally, the further up you pick the fruit, the harder it is to pick the next branch up, in terms of any factor of production, including and especially thought. Conversely, if you suddenly face a sharp drop in satisfaction by being deprived of more fundamental necessities, it becomes easier to decide what to do: replace those necessities!

***

That should give you a taste of what I think is missing from discussions of the economic impact of natural disasters and inability to reach full employment. In the next entry, I'll go further to illustrate how deeply this oversight impacts the ability to perform good economic analysis.

Setting the signaling model of education straight(er?)

Note: free business suggestion below.

You might have heard about the so-called "Signaling model of eduction", promoted by Bryan Caplan at GMU (among others!), and it's something I find plausible.

First, some background: The problem is to explain why people who get a college education are more able to get jobs, and better paying ones. The traditional explanation is that colleges provide you with knowledge skills that allow you to be more productive. (This has always seemed suspicious to those of us who have remarked, throughout our education, that "I'm never gonna use this stuff" ... and been mostly right.)

The signaling model, in contrast, says that completion of college simply reveals your possession of good traits for hiring that you already had before, but could not convincingly claim to have until you completed college, since a college degree indicates some combination of intelligence, willingness to do boring stuff that doesn't make sense, and capacity to be indoctrinated into and conform with a group (I'm simplifying a bit). These things are hard to test in a job interview, or, in the case of intelligence, usually illegal to test for.

A few years ago, I pointed out (HT: Bob Murphy [1]) that one usefully testable implication of the signaling model is that you should be able to earn big profits by running a business that provides high school graduates with the same "signals of good qualities" that a college provides, but at significantly lower (monetary) cost to them, simply by "cutting out the fat" -- all the stuff that doesn't help to signal the student's ability. You would just set up some school that filters students by IQ, and then puts them through hell, gives them difficult assignments, poor living conditions, etc. No way an unemployable person could survive through that kind of regimen, right?

So there's your idea: you make students just as employable, but they don't have to take on nearly as much debt.

Interesting caveat: in one discussion of my idea, someone mentioned that this business model is already in widespread use: specifically, the military! Let's go through the checklist:

- Cheaper than college? Check. (Heck, in terms of money, they pay you!)
- Enforces indoctrination and unquestioning following of direction? Check! [2]
- Selects for people who are willing to give a lot to a big organization? Check.
- Employers regard service therein as equivalent to college experience? Check (usually).
- Gives experience doing boring tasks because you were told to? Check.
- Generally puts you through hell? Check.

Wait, this can't be right, can it? This comparison fails in that the military doesn't filter people based on an IQ test! Hah!

Not so fast -- they've got that one covered: in the US, it's called the ASVAB, which determines whether you can get in, and then which branch, role, or officer status you're eligible for. (My mom used to pass on her dad's remark that, "the army'll take anyone who can crawl there, but not the Coast Guard! An exaggeration, of course, though the branches do have different score cutoffs.) The ASVAB is, in content, an IQ test.

Now, if you can provide a better value than the military (say, to people who don't want to possibly be put in harm's way), here's your business idea!

[1] Yes, a hat tip for pointing me to my own post,.
[2] Note: this isn't always a bad thing. As Eliezer Yudkowsky put it in that article:

Let's say we have two groups of soldiers. In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy. In group 2, everyone at all levels knows all about tactics and strategy.

Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?

In this case I have to question how much group 2 really understands about military theory, because it is an elementary proposition that an uncoordinated mob gets slaughtered.

Setting Arrow's Impossibility Theorem Straight

Okay, by now, you might have noticed the econ blogosphere cooing over how awesome and insightful and useful the Arrow Impossibility Theorem is: Here, here, here, here, and here (in random order).

Um, to put it mildly ... no.

First, a summary of the theorem: let's say you want to convert individual preference rankings over outcomes into a social preference ranking that faithfully reflects these individual preferences as best as possible (i.e., create a voting system). You place a few "obvious" constraints on it that it voting system should meet, and it turns out -- you can't! Boo hoo, democracy sucks. (Well, in many senses, it does ... just not for this reason.)

This issue was discussed almost exactly one year ago on LessWrong. Long story short, the result has much less practical application than you might think. The requirements it asks of an aggregation system are far too strict. For one thing, the "determinism" requirement rules out the use of randomized tie-breakers. Keep in mind, there's always the possibility of some hopeless tangle involving a preference ordering like:

Person 1: A > B > C
Person 2: B > C > A
Person 3: C > A > B

Such preferences are completely intransitive, so no method of aggregation has any hope of being faithful. Normal people react to this by saying, "Okay, in the occasional pathological case, just use some tie-breaker that's not slanted in favor of any option -- in the end, it all averages out, so no problem". But Arrow's Theorem throws up its abstract arms and says, "Gosh, how hopeless. You can never satisfactorily aggregate preferences. Look how insightful I am!"

Needless to say, "We are not impressed."

It gets better though. "Black Belt Bayesian" makes the point that the "independence of irrelevant alternatives" (IIA) requirement is undesirable in the first place. (IIA means basically, if you remove some option, it should not change the aggregated ordering of the remaining options.) Why is it undesirable? Because so-called "irrelevant alternatives" aren't. Rather, they give evidence about the relative _strengths_ of preferences and therefore SHOULD affect the aggregated preference ordering!

Why was the econ blogosphere talking about Arrow's Theorem in the first place? Because someone had asked about underappreciated ideas in economics. Well, I think it's clear by now that this one doesn't suffer from a lack of deserved appreciation.

But what's even worse is that Amartya Sen's celebrated Liberal Paradox is viewed as a corrolary to the Arrow Theorem, and is just as ridiculous. It basically says you can't *both* respect people's rights *and* achieve Pareto optimality. Now, how do you imagine that works out? Well, you cheat by equating rights with obligations -- that is, you eliminate the possibility of people waiving a right when it's infringement would make everyone -- everyone -- weakly better off.

But who cares about that case? Not me. The very reason that rights allow for Pareto-optimality is because people can trade them as necessary when they find welfare-improving opportunities! If you equate "property rights in a specific apple" with "the obligation never to trade the apple away" ... well, you kinda throw a kink in all that.

As I said a year ago, if a transaction really is Pareto-efficient, then rights won't get in the way, because the relevant parties will waive the relevant rights! (Epic tongue-twister, too.)

Reassuringly, the folks on the opposite end of the ideological spectrum from me come to the same conclusion.

So are we set straight now?

When you can't go back to sleep: thermodynamics

Did you miss my posting? Well, it's been one of those days when you wake up early and can't go back to sleep. My mind's running wild this morning, and I figured I'd make some good use out of it. (It is no longer early as of completing this post because of interruptions from a playful kitty.)

I'm going to continue the lesson about thermodynamics that last left off about a year ago, by discussing some more interesting implications of the idea that "energy per unit temperature" is a measure of degrees of freedom. You see, in the time since then, I read John S. Avery's book Information Theory and Evolution, which, as you might have inferred, discusses life from the perspective of that ever-so-useful field of information theory. He also applies it to cultural (often called "memetic") evolution.

The first interesting insight that this book alerted me to is about molar entropy. Some background: in your chemistry class, you might have learned about the Gibbs free energy of a reaction, ΔG, which is calculated from ΔH - TΔS, where H is the molar enthalpy (internal + flow energy per mole), T is absolute temperature, and S is the molar entropy. For a chemical reaction, you look up the molar enthalpies of the products and subtract off the enthalpies of the reactants. Then you do the same for molar entropies, multiplying by the absolute temperature at which the reaction takes place, and add them. A negative sign for ΔG means the reaction happens spontaneously (well, as long as there is an available pathway).

With that out of the way, what are the units for S? Most tables give them as J/K*mol (Joules per Kelvin per mole, or energy per unit temperature per quantity of molecules). But, as the last post in this series showed, energy per unit temperature measures degrees of freedom, which can also be expressed in bits. So, as Avery neatly derives on pages 81-82, you can also express molar entropy in bits per molecule. (The conversion factor is 1 J/K*mol = 0.1735 bits/molecule.) I find this a much more intuitive way to think about it, because it connects the concept of molar entropy to the underlying dynamic: how many bits of information (on average) do you need to specify a molecule's current state, beyond that which you know from the temperature?

Also, rather than having to empirically derive this value directly (either from reaction data or by integrating its specific heat capacity per unit temperature from 0 K to its current temperature), it can be inferred from the known properties of the molecule: its shape, size, and bond strength. The stronger ("stiffer") its bonds are, the lower the entropy of the molecule, because large deviations from its equilibrium configuration are less probable. (Diamond, with its very strong covalent bonds, has the incredibly low molar entropy of 0.24 bits per carbon atom at STP, meaning you need less than one bit of information to specify every four atoms.)

[ADDENDUM: Avery also adds that if you divide the Gibbs equation through by T, you can describe a reaction in terms of the "information lost", i.e., the greater number of degrees of freedom you have permitted by letting the reaction take place.]

By recognizing this interconnection between molecule properties and complexity (needing more information to fully specify = more complex), one sees more unity ("consilience") to the science as a whole: entropy and bond properties aren't just off in their own domains, but have a lawful relationship. Unfortunately, however, I haven't worked out how to derive entropy from stiffness of a degree of freedom, and I haven't found a text that does it either.

Next in the series: A discussion of Eric J. Chaisson's Cosmic Evolution: The Rise of Complexity in Nature, which proposes specific energy flux (energy flow through a system per unit mass) as a measure of complexity that is applicable to everything from stars to planets to life to vehicles to computer chips to culture.

Mixing economics, thermodynamics, and heterogeneity

... or METH, as some call it. And if you want some more drug innuendo, read on.

On Brad DeLong's blog, a commentator named "MJ" deftly applies insights from thermodynamics to the issue of heterogeneity of goods in economics:

What would statistical mechanics be without a quantitative model of heterogeneous vs. homogeneous distributions? Such a statistical mechanics would miss a few subtle but crucial concepts. Such as entropy.

Note how the negentropic development of increasingly heterogeneous capital allocations over the past decade was accomplished through entropy production: bundling good with bad, compromising tranches, etc... Goldman Sachs made a killing, basically, off of knowingly producing entropy. The entropy production, of course (2nd law), far exceeded the negentropy production of their wealth aggregation- as reflected in the order of magnitude between financial industry's gains and the over all loss.

We're now learning Fannie and Freddie also engaged in entropy production, obscuring the distinction between scores over and under 660.

I can vouch for that as showing a good understanding of entropy, and it gives a good perspective for viewing economics:

1) An efficient economy produces as little net entropy as possible: the entropy it generates (destruction of heterogeneity) should be offset by the entropy it destroys in organizing inputs for their uniquely optimal roles.

2) A sign of inefficiency is when economic actors destroy distinctions (like in MJ's example of very different tranches and borrowers being made indistinguishable) without making a corresponding useful distinction or organization.

Definitely some issues worth fleshing out. I know I've seen papers that try to view economics from a thermodynamic perspective, but they invariably have me rolling my eyes.

Silas Barta, information theorist by night

UPDATE 12/17/09: Steven Landsburg, after responding several times in the comments section here, posts a defense of his position on his blog, although without mentioning me or Bob Murphy. Hey, I can understand: if I were in his position, I'd hide the existence of me and Bob too!

***

Bob Murphy invokes my expertise on information theory to criticize (yet) another bizarre argument from Steven Landsburg, that the natural numbers are more complex than human life. Here's the mistaken part of Landsburg's reasoning:

...the most complex thing I’m aware of is the system of natural numbers (0,1,2,3, and all the rest of them) together with the laws of arithmetic ...

If you doubt the complexity of the natural numbers, take note that you can use just a small part of them to encode the entire human genome. That makes the natural numbers more complex than human life. Unless, of course, human beings contain an uncodable essence, like an immortal soul

Naturally, I don't necessarily agree with the broader theological points Bob makes in his reply, and such issues will remain even scarcer on this blog than on his. However, I will expand on point I made in discussion with Bob.

The error in Landsburg's line of reasoning is: the fact that you can use instances of X to build Y does not mean X is more complex than Y. Just the opposite, in fact: in order to describe Y, you must describe X as a substep. Like in the analogy I gave, you can use bricks and mortar to build a house, but that means it's the house that's more complex. To fully specify the house you must describe not only the bricks and mortar, but the form they take as a house -- how they're supposed to be put together.

As for arithmetic and natural numbers, it's their lack of complexity that makes them so useful. By appealing to it, you can make sense of a diverse array of phenomena. The more complex arithmetic were, the less helpful it would be in making sense of things.

Just to be clear, this doesn't mean it's easy to learn math (different people have different problems in different topics and levels), or that you can't do anything complex with math. The point is that no amount of complexity produced in using arithmetic could ever imply arithmetic's complexity, for the same reason that no matter how complex a house you make with one kind of brick, you can't make the brick more complex.

But of course, Landsburg's errors don't end there. He wants to go so far as to say that by merely encoding the genome in base 4, you've described human life. That's certainly the impression people get from discussions of DNA in the popular media and movies like Jurassic Park. Hey, all you need is a string of letters made up of A,G,C,T, and you've described someone completely!

To put it mildly: that's not how it works. First of all, you need to say what the letters actually mean. And then, even if you know that much, all you have are empty labels -- suggestively named LISP tokens. So you know that C is cytosine? Okay, but what's that? Now you need to describe where the carbons and nitrogens and oxygens go to make up cytosine. But wait -- what's this "nitrogen" thing, anyway? And so on.

Don't worry -- the process terminates: once you've described the generative model that puts all of these concepts together in a way that yields a description of human life as its output.

Needless to say, you're using more than a few integers by that point!

What "interference" with radio signals really means, and its implications for property rights

A common confusion often arises: people talk of the "interference" with radio tower transmissions, without understanding what physical process the term refers to. This misunderstanding makes it hard to see the logic in my analogy between intellectual property and rights to radio frequencies.

In a recent debate on intellectual property, I finally decided to set the record straight, and what follows in this post borrows heavily from what I said in the debate.

First, how does radio communication actually work? I'll admit that I don't know the answer all the way down to the nuts-and-bolts level. But I can explain it from the perspective of information theory.

Radio communication works, to the extent that it works, because a listener can perform a measurement, and thereby learn something about the source, i.e. the message transmitted. (This "something" they learn is called the "mutual information" between the two points, and is equivalent to so-called "Bayesian evidence".)

And when it comes to your radio, what is that actual measurement? Setting a dial on it that changes a circuit's properties so that it resonates when the surrounding area is filled with electromagnetic (EM) waves around a certain frequency. And when it resonates, an electrical signal in the radio follows a certain pattern that's correlated to the signal the radio tower is sending. Your radio then converts the circuit's electrical signal into sound that is meaningful to you.

All of this functioning relies on an assumption: that by performing the measurement, you do in fact learn something about the source. That assumption is violated when more than one tower transmits with enough intensity at the frequency you perform a measurement on. In this case, no measurement result tells you anything about either source: the transmitted waves overlap each other, coming across a gibberish on your radio. (In the lingo, there's no "mutual information" between you and either source.)

So whenever you talk about "interference" with radio communication, what you really mean is "violation of an assumption some parties were using to communicate which, when violated, makes them unable to communicate."

To understand the significance of using the term "interference" in this way, let's look at a more practical, intuitive example with the same dynamic, but unrelated to the EM spectrum.

An Illustrative Example

Let's say that I live in a small village where I have a few friends. I want an easy way to communicate to them that I expect a rainstorm today. So, I work out an "encoding scheme" with them in advance: if they hear me hit my gong before 8 am, I predict rain. If they don't hear me hit my gong, I don't predict rain. So, instead of having to tell them all individually, I can just hit the gong. They'll hear it, and they'll get a message from me. By "measuring" the sound they hear before 8 am, they learn the "signal" I'm sending.

So far, so good.

But there's a little snag: my friends will hear a gong sound as long as anyone hits a gong not just me! So, our communication scheme only works as long as we can rely on no one else hitting a gong before 8 am. If we can't rely on that, I can't send them the message, at least not as reliably. Because when they hear a gong, sure, it could be me, but it could also be anyone else with a gong. Hearing the gong sound is no longer a reliable sign that I think it will rain.

So there you see it: our communication system can be defeated by "interference" from other people, either because they're trying to set up their own similar system, or because they just like being mean. But this "interference" simply means: violating an assumption that we, rightly or wrongly, thought we could rely on.

And how does this relate to radio communication? Simple: the existence of the gong sound before 8 am is just like a radio signal within a given frequency range: it can provide information to others, but only if others don't try to use the same means to communicate.

Conclusion

So do you think people should be able to "homestead" such "communication assumptions" like that? Should I be able to assert rights as "the only one who can hit a gong in this area before 8 am"? (Or, to be less greedy, the right to hit a gong in this area in a certain five-minute window, with a certain rhythm.) Your answer to that question tells you a lot about how you should look at other issues.

For example, how about asserting rights as "the only one who can broadcast radio waves in this area within a particular frequency band"? How about asserting rights as "the only one who can distribute books containing Harry Potter stories"?

Hey! That last one kinda sounds like intellectual property rights...

Another interesting thermodynamics result

Here's another interesting insight on thermodynamics and information theory to add to my previous: I realized why "joules per kelvin" is a measure of entropy. Not exciting? Wait, you'll see.

In the previous post on this topic, I mentioned all the parallels between entropy in information theory and entropy in thermodynamics. Also, some properties can be calculated by their information-theoretic definition or their thermodynamic definition, such as the thermodynamic availability, which can be calculated as the Kullback-Leibler divergence, a measure from information theory. But what's interesting is that this value can be expressed in terms of bits, or in terms of Joules per Kelvin, which has units of energy over temperature, with a simple constant multiplier for conversion.

Huh?

You see, there's the hard part: why on earth would bits -- which measure how much memory your computer has -- possibly refer to the same property as "Joules per Kelvin", the way that inches and meters refer to the same property?

And that's where we get to the interesting part. First of all, what is temperature? It's not how much internal energy something has, but rather, it's internal energy per degree of freedom. In this context, a "degree of freedom" is a distinct way that something can be modified at the molecular level. A single-atom molecule may be viewed as having three degrees of freedom, since it can translate in three dimensions. Once the molecule has shape, however, it can rotate in addition to translating. So, two different substances at the same temperature can have different internal energy, because one of them may be stuffing that energy into more degrees of freedom.

So where does that get us with Joules per Kelvin and energy per unit temperature? Well, watch what happens when you expand out temperature in the entropy expression:

energy
------------------------
energy/degree-of-freedom

= energy * degree-of-freedom/energy

= degree-of-freedom (!)

So there you have it! Once you expand it out, energy per unit temperature is simply a roundabout way of saying "degrees of freedom".

Now you may ask, "Nice, but that still doesn't explain what that has to do with bits." But then, what is a bit but a binary degree of freedom? When you have memory of n bits, then there are n values that you can independently set to one of two possible values, making it likewise a measure of degrees of freedom. (Note that this capability allows you to store 2^n possible states.) And informational entropy, in turn -- also expressed in bits -- is the logarithm of the number of possible states a system can be in, making it proportional to the degrees of freedom as well.

The two lessons to take away are that:

1) The number of degrees of freedom a system has depends on the arbitrary choice of what you count as a degree of freedom, just like the number of "units of length" something is.

2) Whichever consistent method you use of counting degrees of freedom, the number of degrees of freedom is proportional to the logarithm of the number of possible states.

Mystery solved! (No, I don't know if this discussion is given in any textbook treatment of the issue.)

Oh, and: Happy Saint Patty's Day!

My plan to destroy the universe won't work

And I'll bet you're relieved!

Maybe a little background is in order.

A question of interest to philosophers and theoretical physicists is whether or not the universe is just a simulation running on some computer, one level up. (See e.g. Nick Bostrom's Simulation Argument.) Of course, many ridicule this idea as being non-falsifiable and thus non-scientific.

Not so fast! I said. Of course it's falsifiable. Here's how: if the universe is a simulation, then its programmers probably try to economize on computational resources (computing cycles, memory, disc space, time, etc). And to do that, they will make the program reveal to "us" (the conscious entities) the minimum required to make everything appear "believable". That in turn, means that as long as we "wouldn't know the difference" if some physical process developed in a way contradicting the rest of our observations, the simulator won't bother to churn through the calculations needed to make the process match up with known universal laws. In other words: "If we're not looking, why bother making sure something's there?"

And that tells us how to test the Simulation Hypothesis: have everyone set up as much measurement equipment as they can, and therefore observe as much as they can. This will force the simulator do many more calculations than it would otherwise have to, since now it has to keep consistent with that many more observations. The programmers then have to devote an ever-increasing amount of resources to keep it running, which will eventually force them to "cut corners" in implementing the laws of physics, revealing violation of Standard Model physics, or ... um, make them pull the plug on our existence.

Hence, my "plan to destroy the universe".

Now, the good news: the plan wouldn't work, based on what we already know about how the universe would react to such a "hypermeasurement" scenario! And the reason is shocking: because we can't actually increase our total knowledge.

"What in the hay-ll? I did me some book-larnin' not but three yurs ago!"

Sorry, that was Cletus, our resident country bumpkin.

Well, I'll need to some more background now to justify that claim. First, I want to point you to a post on OvercomingBias.com that introduced me to a lot about what I'll discuss here: Engines of Cognition.

Now, consider the 2nd law of thermodynamics. There are many ways to express it, but a simpler way is: "The amount of disorder ('entropy') in the universe must always increase." Sure, you can increase the order any one specific place -- say, when you form crystals -- but it will always be counterbalanced by an increase in disorder somewhere else. The most common application of this law is in heat engines (such as the one in your car): when you burn fuel to turn your engine and thus your tires, you are extracting a kind of order: the useful mechanical "work" (as it is called in physics) of a spinning engine. However, to do so, you burn fuel and transfer heat to the environment, which, when tabulated, generates entropy/disorder exceeding that which you destroyed in extracting mechanical work from the system to drive.

Now, here's the kicker: there are deep parallels between the concept of entropy in thermodynamics, and the concept called "entropy" in information theory. In the latter, it refers (roughly) to the uncertainty one has about the content of a message before reading it. Any knowledge that some kinds of messages are more likely than others therefore reduces that "entropy". Similarly, entropy is at a maximum when all messages are equally likely.

And the truly mind-blowing part is that the connection between the two kinds of entropy is so deep that entropy in the information-theoretic sense affects entropy in the thermodynamic sense. (This is going somewhere, just be patient.) In short, if you are able to reduce your uncertainty (information-theoretic entropy) about the "message" contained in the molecules of a system, that knowledge can actually be exploited to reduce the thermodynamic entropy of the system and thereby extract useful work! (For reference, and early exploration of this idea is called the Maxwell's Demon thought experiment, and a hypothetical engine that extracts work this way is the Szilard engine.)

But this hypothetical capability of decreasing the entropy of a system does not actually contradict the 2nd Law, which, you'll remember, says that total entropy must increase. Rather, for reasons I won't go into, this acquisition of knowledge itself is limited by the 2nd Law. Just as the extraction of "organized" mechanical work from fuel requires the generation somewhere else, of at least as much counterbalancing disorganization, so too does the collection of information that could permit extraction of the same work without the fuel require a counterbalancing loss of information somewhere else, i.e. increased uncertainty.

This principle reveals a fundamental limit that your brain (in a deep sense, a "cognitive engine") faces: in order to learn something true about your environment (whether via the senses or inferences), you must sacrifice knowledge somewhere else. Fortunately, nothing requires you to care much about that lost knowledge, which takes the form of "lost certainty about aggregate statistical properties of thermodynamic variables".

Now, back to the main point: from the perspective of hypothetical beings running the universe's simulator, my idea to gather more measurements has no impact. Any time we make a measurement, we are gathering knowledge, which must therefore correspond to lost knowledge somewhere else. So, far from threatening the computer's ability to simulate our universe, all our measurements will (amazingly) decrease the computational resources the simulator requires.

Which neatly returns the Simulation Hypothesis to non-falsifiability, and assures us that even if people acted on my idea, we're still safe and sound. Alternatively, it reveals the universe's programmers to be really, really clever :-)