Rocks to Rules: Climbing a Rickety-But-Interoperable Semantic Ladder

Summary

(Epistemic status: A promising but weird research direction in agent foundations and philosophy of language. A half-built conceptual tower whose mortar and paint are both still wet: please excuse our mess while we innovate. Some of it, I'm confident in as pretty much right; some of it is merely suggestive; probably much of it is wrong or incomplete. Certainly it lacks the math it'd need to be truly correct, which gnaws at me. The prior art here is strange and thin: Gärdenfors, Tarski, Gentner, and the later Mohists make for odd bedfellows, and I can't tell whether that's a warning sign or a moat I can peculiarly easily clear. I gave the simplest parts of this as a talk extremely recently in Berkeley at a luminously-named venue for a conference with a Grecian name. For @johnswentworth, but also @abramdemski, @Morphism, @Gretta Duleba, JRM, @Dalcy, MM, and EA; I hope they enjoyed the oranges. With thanks to @WhatsTrueKittycat as well. As an attention conservation note... this is maybe 12k words, and that's the abbreviated version. Take it slow, take breaks, have some water on hand, and feel free to pause anywhere that looks like a natural break. I'm sure this won't vanish before you get back. Probably.) TL;DR: Where I start: two reasoners with different world-models but the same predictions trying to talk (Wentworth's interoperable semantics), and who can thus at least agree on nouns as clusters in thing-space. What's new: a ladder from there up through verbs to rules (constraints on whole sets of trajectories) and pacts (rule-sets several agents bind themselves to), with operators on each rung. Additionally: a conjecture with bounties on it - no rung's machinery runs without the rungs below. What's testable: wherever two rungs meet, the world's languages (or relevant cognitive technology, like contract law) should show a smeared word class straddling the gap. What's fun: Marvel arguing that the X-Men aren't human, Borges's Tlön, and worlds made of waves, where verbs come before nouns. Why should you care: the values we'd want to hand an AI are solely expressible as rules and pacts, and nobody has the theorem that says when your "murder" is a function of mine. (That's in "Why care"; the conjecture is in "The summit".) The board from the talk! Intro Have you ever seen a pop culture corporation argue itself into a laughably self-defeating position, twisting ontology and linguistic philosophy around, undermining the very framework they painstakingly set up in their stories, just to make a few bucks? You're about to. Picture this: it's 2003, in the United States Court of International Trade, and the court has just now been asked to rule on whether the X-Men are human. According to US law, "dolls" are figures representing humans, and "toys" are anything else; under the tariff schedule of the day, it's the former that gets taxed at a higher rate. If you're Marvel, what do you do? Obviously, you send lawyers to argue the X-Men aren't humans and never were! Never mind that you've cut the legs out from under the entire frame your superhero stories live and breathe: you've made a quick buck, dodged import taxes, and ignored narrative in favor of legislative victory, and that's the real win. When unstoppable real money met an immovable class of noun, the noun class moved. Now swap the tariff schedule out for a reward function and read that paragraph again. Specification gaming is the exact same maneuver run by a different litigant in a different court to similar ends: hostile category re-drawing, adversarial downward-binding, with categories relitigated from above by whatever's got stakes and leverage. (Hold on to that idea; we'll ground it better in the next post.) One more piece of scene-setting, because we love our historical witnesses to our hot fresh epistemological constructions, here: some 2,300 years ago, the School of Names was one of the foremost philosophical traditions in China. We know this because any of their works survived at all. The Disputers, as they were sometimes known, spent entire careers on the paradox of whether a white horse is a horse; they were mocked for it by every other respectable school of the Warring States period from the Confucians to the Daoists, and then Qin Shihuang (probably) burned their scrolls. Here's the thing: I think that they were on to something - mockery, fire, white horses, and all. Blog posts are a little harder to burn than scrolls, and in the modern day I can appoint a great many more Disputers. (Gentle reader: your job is to break what follows and hand me the repair. Dispute like - and with - the best of them.) As for what follows? What binds; in what order; how it is shared; what deforms it; and given that this is a research direction and not a victory lap, what theorems are missing. (What shame, what agony, for JSW to point out that it is now me who lacks sufficient math in their work.) What do I want this frame for? To take a leaf out of my own book (see "32. W