tis.so
March 29, 2024

Adversarial asymmetry

by Collin Lysford

A comment from RIP DCB on the original draft of The Mongolian meta:

i don’t know much about large language modeling, but i would guess part of your point is that there’s a strategic advantage in being the adversary who can extrapolate your enemies structural flaws when those flaws are so site-dependent?

This is certainly true, as far as it goes. The Street View employee acting as a predator model doesn’t need to come up with the whole meta to start inverting it. But there’s a deeper and more profound lesson here, so I want to take the time to clearly parcel it out.

One way to tell where you are in Mongolia are the mountains. These are giant and striking and effectively impossible to move. Your relative position to the mountains are exactly what it means to be at a certain location. Breaking this correlation takes a massive amount of energy — you’d need to level the mountains themselves.

Imagine this correlation on a scale, and on the other end of that scale is the position of the spare tire of the car doing the mapping. This has absolutely nothing to do with where you are in Mongolia. You just have a spare tire in some spot or another that just happens to correlate because you took many pictures in the same location with the tire wherever you threw it that trip. If you’re going to take a new picture of that spot, you’re going to just throw the spare tire wherever. It would take more energy to keep that correlation intact, because you’d have to remember how you did it before.

When you’re trying to find the geographical location of a photo using elements in that photo, some of the elements are tightly coupled with that element in the world (it would take years of industry or many high-yield bombs to change a mountain’s relationship to a location in Mongolia). Other elements aren’t coupled at all in the world, only the dataset (you can put your spare tire wherever you want as you drive through Mongolia). As a predator model, your goal is to take a picture in Mongolia that gets incorrectly classified by the meta. If everything in your picture was as invariant as the mountains, you’d be stuck. If the meta had an account of variation that properly privileged the mountains and shut it’s eyes to the other stuff, you’d be stuck. But the meta is trying to get all of the juice it can out of the data there is right now, and that means it’s using the information with respect to its predictive power, not its invariance. So you can take every single measure that’s totally uncorrelated with location in the world and change it to the exact opposite the meta expects, for free.

Note that if the initial dataset was bigger, the meta would be forced to adapt and become less strictly predictive but more robust to variation. Because the spare tire really is arbitrary with respect to location, there will be pictures of the spare tire in different places in different locations. You’ll get a clue that it doesn’t bind to location as strictly as spare tire in a net means Western Mongolia.” But unless your pictures have the infinite permutations all mapped, with every single possible spare tire orientation present at every single location, there will still be some correlation, some combos that happen to exist and some that don’t. And the predator model will still be free to take those spurious correlations and reverse them.

This is why even ultra-powerful AIs can be easily snowed by dedicated adversaries. Physically existing and trying to move stuff gives us strong indications of what correlations are extremely difficult to change, which are dead-simple to change, which are linked by social convention but could be reversed by a sufficiently eccentric person, which are linked now but could change dramatically later. I am not saying these distinctions are impossible for AIs to ever learn; I am saying that doing so necessarily diminishes their predictive power over the data that exists today, because it entails throwing away the parts of the meta that won’t survive change. So an AI that is trying to optimize score does so exactly by considering every single bit of the meta, thus becoming more susceptible to predator models. They don’t need to survive change and so they’re not trying to. The strategy to build a living thing that will endure is altogether different.

predator models games anti-inductivity Geoguessr ecology unit of survival mesa-optimization

March 28, 2024

Intergenerational knowledge transfer in hunting

by Feast of Assumption

One of the primary themes I write about is the intergenerational transfer of knowledge. Hunting is an interesting case: in the modern era, it’s practiced only 9 days per year. In this short season, enough information must be transmitted to enough people that the tradition will be preserved even if the patriarch doesn’t survive the next year. And nearly every hunting camp that you pass in the orange-dotted woods has executed this transfer successfully—I see many more successfully propagated hunting traditions than I see family businesses.

woods

Knowledge Transfer Mechanisms:

Trad,

Plenty of information (affectionately termed knowledge’) is easy to pass via book-learning. (What caliber of ammunition will be able to deliver enough force to take a particular quarry at a particular distance, and thus what rifles are suitable for hunting which quarry.)

Modern,

What Samo Burja describes as tacit knowledge1 used to be hard to pass down, but has become easy with the Youtube Revolution.

And video transmission of some forms of tacit knowledge is splendidly effective. Letting novices watch a few videos of the field-dressing process before offering them a knife for the first time greatly reduces the instances of unpleasant smells produced by an unsure hand in an unfamiliar gut.

and Trad Again

But what makes a hunting party successful isn’t field dressing technique—it’s hunting technique2You can have the right rifle all day long, and you can be the slickest dresser in the county—but these do you no good if no deer passes in front of you and presents a clear shot.

And while a non-hunter might think deer appearance is stochastic, it isn’t. Deer behave in ways that are predictable when fully zoomed out. Deer are active at dawn and dusk, still during day and night”; this is simple enough to write down, convey across time and space. But where they’re traveling from and to, their preferred routes to take, are pattern-ful—but not in a way that can be shared outside of a specific ~200 acre ecosystem. When you know the land and the patterns, though, you can place yourself and your party members in the right spots. To a well-placed hunter, shots will present themselves.

The knowledge of how local deer move and live can only be transmitted by your hunting party interacting with those specific local deer, on that specific land, over decades.

A Description of the Knowledge

It may be intellectual dark matter,’ but I’m at least going to gesture at the shape of it.

In my state, there are three main forms of hunting discussed. The first is no longer practiced, but it still comes up regularly as a contrast object. The three happen to chunk neatly into our knowledge, tacit knowledge, hypercontextual knowledge’ categories.

  • still hunting: a nearly extinct technique because it requires a critical mass of unrelated people, usually on public land, walking and pausing, walking and pausing, waiting for deer. Occasionally you’ll flush a deer for another unknown hunter, occasionally another unknown hunter will flush a deer to you. This was a popular technique in my grandfather’s day, but since it’s not practiced anymore, if you were to try it, nobody would flush a deer to you, and any deer you flushed would go unnoticed. (Or, be noticed by someone in a tree stand who will never return the favor.) If still hunting were still practiced today, you wouldn’t need anything except a rifle, marksmanship, and patience to succeed.
  • stand hunting: popular because it can be practiced alone. Deer tend to travel at dawn and dusk, and remain bedded down or grazing during the day and night. Stand hunters will install stands in trees before hunting season, along known deer paths. Climbing the stand an hour before dawn to patiently wait for a deer to pass along the path, coming home for lunch, and returning to the stand an hour before dusk—this comprises hunting for the solo hunter. It can be learned from books and youtube, especially if you integrate local information from trailcams.
  • drive hunting: Posters” wait, alert, at geographic funnel points. Drivers” enter a stretch of woods and walk in formation, noisily, to startle bedded deer and send them into the waiting sights of the posters. 3

The benefits of drive hunting are plenty. You can hunt all day instead of being confined only to the natural movement hours of dawn and dusk. You enjoy more scenery, and while driving you keep warm from exertion. You harvest many more deer per hunter per year than even a well placed stand. Effectively, improving the design and execution of your drives allows the technique’ part of the equation to overwhelm the luck’ part. Since the goal of hunting is to put meat in the freezer, drive hunting is the clear winner for anyone who can execute it.

But you can’t execute a drive well unless you know your land’s geographic funnel points. Where can deer be funneled? Which hillsides do they like, which hillsides do they avoid? Where are their natural paths? These answers can be earned by walks in the woods in the springtime before leaves have crowded the view, if they weren’t handed down from your hunting party who were on the land before you. But you can’t learn everything in one year. A mast year4 for oaks drives different deer behavior than a no-acorns year will. Corn vs oats vs alfalfa on nearby farm fields will change your local herd’s browsing patterns. Even funnel points can change year to year—we have a drive where a poster stands on a ridge over a hill if ice hasn’t formed yet. But if ice has formed, the deer will charge across the lake. So in an early-ice November, our poster will stand at the edge of the bay.5

Coordinating A Hunting Party

This is a lot of knowledge. The mechanism for conveying it (the hunting party) has to be robust. The successful way I’ve seen this achieved is with a lot of built-in redundancy.

Cousins are invited. A mix of age ranges is a necessity. To succeed in the long term in the modern era, a hunting party has to be robust to members leaving and coming back. Oh no he started dating a vegetarian” is a real hurdle, as is they moved across country for a job.”

A successful hunting party will have invited people for an occasional day of hunting, to assess if they’d be a good long term fit. Success breeds more success, as everyone wants to be a member of a cheerful party with meat in its freezers. But hunting parties cannot turn into big tents.’ There is a correct size for a hunting party that varies with the land, but rarely exceeds 8. A hunting party wants its members to be safe, patient, cheerful, willing to do their share of gruntwork, and good shots.

A successful hunting party must also invite youth. A good patriarch will go out of his way to give the newest party members the most advantageous positions, so the kid can taste success early, whetting his or her appetite for future years. I’m not a patriarch so I’m only speculating, but I’d ascribe the definition of success shifting from how big is this trophy” to did my newest hunter get a shot at a deer?” being based largely in Honor and Pride at passing on enthusiasm for the traditions.

While bending over backward to show 12-year-olds the joys of the hunt, a hunting party (and family) must also be robust to slow starts. My dad is an avid hunter—but if he had forced me to hunt before I was interested, I would have hated it and never returned. Happily, I was allowed to loaf around and bake pies while the rest of the family hunted. When I eventually did develop an interest, I was able to start with a blank slate. I was able to celebrate the milestones of my first deer” and my first buck” when I earned them—I got no fewer congratulations at 31 than I would have gotten if I’d started at 12.

A Testament to Humanity

Considering all the folk arts which didn’t make it from 1400 to 2000, I’m in awe that hunting has survived and flourished. Against the odds—practiced 9 days a year instead of 365 (or even 52)—the fact that hunting parties assemble and keep the forest population balanced by harvesting wild meat fills me with pride in humanity. I feel honored to be receiving and carrying forth knowledge as a member of a successful hunting tradition.

map


  1. Tacit knowledge is knowledge that can’t properly be transmitted via verbal or written instruction, like the ability to create great art or assess a startup. This tacit knowledge is a form of intellectual dark matter. Examples include woodworking, metalworking, housekeeping, cooking, dancing, amateur public speaking, assembly line oversight, rapid problem-solving, and heart surgery. Before video became available at scale, tacit knowledge had to be transmitted in person, so that the learner could closely observe the knowledge in action and learn in real time — skilled metalworking, for example, is impossible to teach from a textbook. Because of this intensely local nature, it presents a uniquely strong succession problem: if a master woodworker fails to transmit his tacit knowledge to the few apprentices in his shop, the knowledge is lost forever, even if he’s written books about it.↩︎

  2. Not to short-change the role of luck and the role of determination! but assuming determination as a prerequisite, and luck as sometimes she’s with you, sometimes she’s not, but you’ve got to be in the woods to find out’.↩︎

  3. deer getting woken up in the middle of the day and driven into a waiting firing squad like, who the f*ck is this / flushing me at 10:46 — lambdaphagy↩︎

  4. a friend: what’s a mast year for oaks?”

    me: a year with huge acorn production”

    friend: why do trees have annual variable production of fruit/nuts?”

    me: to stymie the squirrels: if you have the same number of acorns every year you get a stable squirrel population that eats all the acorns. If you have very few one year, a bunch of squirrels starve. Then you produce extra the next year, there aren’t enough squirrels to eat them all, so some acorns last all winter and germinate.↩︎

  5. cue Prokofiev’s Battle on the Ice,” or even the whole of Alexander Nevsky (1938)↩︎

folk knowledge folklore hunting knowledge logistics

March 27, 2024

The ecological perspective

by Suspended Reason

The ecological thought does, indeed, consist in the ramifications of the truly wonderful fact” of the mesh. All life forms are the mesh, and so are all the dead ones, as are their habitats, which are also made up of living and nonliving beings. (Timothy Morton)

Riffing off Erving Goffman, Ecologist”, I thought I’d write up the core principles I find most useful for thinking ecologically. I’m particularly curious which principles readers think I’m missing—I’d love to hear from you etc etc; my email’s .

  1. The unit of survival is the organism-and-environment. Fitness is always contextual to a given environment. Because a given organism is only ever suited to a given distribution of environments, its survival depends also on that environment surviving (staying in-distribution). See also Bateson’s Form, Substance, & Difference.”

  2. Drift & disruption drive an adaptation ripple. When an environment is disrupted, the average organism loses fitness. Previous strategies and adaptations no longer work. Because the environment is always changing, it is useful to describe organisms as mesa-optimizers: optimizers who have themselves been optimized by a base” process (e.g. natural selection). Drift is inevitable because ecology is too complex a system to ever fully stabilize. Due to drift and disruption, an organism’s optimization function differs (often quite dramatically) from what is optimal from the perspective of the base optimizer (e.g. natural selection).

  3. Selection is frequency-dependent. Because the unit of survival is the organism-and-environment, the composition of that environment is critical. The population of a given species will have a strong effect on the survival chances of a single member. The population of a given prey species will have strong effect on the chances of a predator species. And so on.

  4. What is true for an organism is true for a strategy. Organisms are merely bundles of strategies. A strategy, being a solution fitted to a problem context, only works if that context is maintained; if the environment is disrupted, the strategy no longer works. The effect of a given strategy depends on the history of strategies deployed. Is this a new, unseen strategy to which other organisms are naive, or is this a tired strategy to which they are well-accustomed and habituated?

  5. What is true for a strategy is true for a resource. The poison lies in the dose (the frequency), e.g. too much water or too much salt are both fatal for mammals. For nearly any chemical, there is an (at least theoretical, but usually actual) organism for whom it is a sought-after nutrient, and also for whom it is a filtered-out toxin. (The few exceptions on earth might be carbon & water, since they’re so fundamental to the planet’s evolutionary history?) Oxygenation of the atmosphere led to one of the greatest die-offs in planetary history.

  6. Every available resource ends up a seized opportunity. All waste products are nutrient opportunities. Grifts” tile ecosystems—food stuck in an alligator’s teeth becomes birdfood; if plastic sits around long enough in landfills, bacteria will evolve the capacity to break down its hydrocarbons.

ecology frequency dependency Gregory Bateson fitness unit of survival mesa-optimization

March 26, 2024

The conjunction fallacy & Grice

by Frances Kafka

In Here’s Why Automaticity Is Real Actually”, Scott Alexander describes the conjunction fallacy as not only well-replicated, but easy to viscerally notice in your own reasoning.” Can we better understand what’s going on here?

It’s important to distinguish the various meanings that we attribute to the word probable.” In fact, probability” is a shaggy, irregular word which requires a lot of conceptual engineering for various specific domains, and it’s no coincidence that Carnap, who gave much stimulus to conceptual engineering, wrote a book that attempted to carve out probability (he distinguished probability1 and probability2, which I’m not going to touch on).

A cognitive scientist like Kahneman thinks of the word probable” in a mathematical way, and identifies this statistical interpretation with the common notion of the word. Suppose we asked him the Linda question. His thought process would develop in this way:

The experimenter is asking me which has a higher value, P(Linda is a bank teller) or P(Linda is a bank teller AND Linda is active in the feminist movement).” To him, the question is in the genre of a mathematics word problem, which is why he throws away all the extraneous, humanizing detail about Linda. You don’t have to be a Bayesian to think in this way; you can either ask in the frequentist way, In a population of people with Linda’s broad characteristics, and I take samples, would I get more bank tellers or bank tellers AND active feminists?” A Bayesian would wonder, What is the subjective likelihood (consistent with the laws of probability theory) would I assign to both P(A) and P(A and B)?”

Suppose, however, I drag Average Alice and Basic Bob into my laboratory and I present them with this exact question. Mind you, this does not preclude the possibility that Alice and Bob implement some sort of Bayesian-ish plausible reasoning in their brains (on that, see E. T. Jaynes’ paper here). For Alice and Bob, the word probable” is essentially asking a social question. Of course a woman who is deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations” is likely to be an active feminist! I’ll also point out that the first option, Linda is a bank teller”, when contrasted with Linda is a bank teller and is active in the feminist movement”, actually implicitly sounds like, Linda is a bank teller and is not a feminist.” The average person, the average intelligent person who doesn’t stake their life on the genre of mathematical word problems is very reasonable in interpreting the options in this way for Gricean reasons. In fact, from a mathematical point of view their answers are quite reasonable, because the implied mathematical question is not Which is larger, P(A and B) or P(A)?” but rather Which is larger, P(A and B) or P(A and not B)?” You can use this to point out that it’s not the subjects who are wrong, but Kahneman.

In fact, I suspect that people would give a better answer if you made it into a matter of numbers. Suppose I have a room of 100 women. How many of them are likely to be bank tellers, and how many of them are likely to be bank tellers and active feminists?”

The fact of Gricean norms comes up elsewhere, in Yishan Wong’s attempt to show that people don’t know how to understand counterfactuals. If you go up to your coworker and ask them, If I were to go to France, where would I get a baguette?”, it’s a reasonable assumption that I am planning on going to France, because if not I would not be asking this question. (Unless, of course, you know that I’m asking this question to be insufferable, for which the appropriate answer is Sod off.”)

A bigger divergence comes from Kahneman’s forecasting questions. Kahneman’s experiment involved asking two different groups of forecasters to rate the probability of two different statements, each group only being shown one statement.

  1. A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.”

  2. A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.”

For Kahneman, a good cognitive scientist, of course all these disparate activities, the probable” involved in the bank teller problem is the same as the probability involved here. Perhaps Kahneman (or perhaps a LessWronger, because they seem to be more keen in discussing counterfactual worlds) would reason this way: Imagine there are a hundred possible worlds in the future, and I tag off each world where the USA and the USSR suspend relations with a A, and I tag off each world where Russia invades Poland with a B, which is larger, P(A) or P(A and B)?”

Yudkowsky helpfully tells us this;

The scenario is not The US and Soviet Union suddenly suspend diplomatic relations for no reason,” but The US and Soviet Union suspend diplomatic relations for any reason.”

But this is not what the words imply; Yudkowsky and Kahneman might mean the second statement, but that is not how the first would be interpreted. In fact, I suspect that the forecasters involved would do better if 1) was simply The US and Soviet Union suspend diplomatic relations for any reason.” without them knowing anything about Bayes’ theorem or the conjunction fallacy. A sentence is not a mathematical proposition but a narrative in nuce. The story presented by 1) is that there’s a sudden suspension of diplomatic relations. The story presented in 2) is more plausible, because again, these are narratives we are talking about, so of course you’d be more willing to press this. Moreover, it’s not clear that our forecasters are thinking in terms of checking off” possible futures, but are thinking causally. It’s also possible that for sentence 2, they’re not thinking of P(A and B) but rather P(A|B): given that the Russians invade Poland, it’s quite likely that diplomatic relations will be suspended. These forecasters are probably intelligent people, people who are rather good at solving mathematics word problems, but even they have to be pushed to consider this like a word problem.

Paul Grice probability Rudolf Carnap conceptual engineering Daniel Kahneman behavioral economics language conjunction fallacy

March 25, 2024

KataGo and The Hook

by Collin Lysford

When I was a kid, I played this game called Backyard Football. You draft teams of children, come up with your own plays, and play football against a CPU or a human. I only ever played the CPU, and pretty quickly I only ever played one way. One of my random plays I invented was The Hook”, where one wide receiver runs off at a 45 degree angle while everyone else gets gummed up in the center. Trying it out, I quickly noticed that something about The Hook short-circuited the poor CPU and it would never, ever see the receiver on the hook. So I’d play The Hook and whip the ball to Pablo Sanchez, who would have no defenders near him and inevitably run it and get a touchdown. I would end every single game over a hundred points up and was the undefeated master of Backyard Sports, at least on my own machine.1

Turns out this kind of thing still works! Here’s a paper about adversarial policies designed to beat KataGo, a best-in-class Go AI:

We attack the state-of-the-art Go-playing AI system KataGo by training adversarial policies against it, achieving a >97% win rate against KataGo running at superhuman settings. Our adversaries do not win by playing Go well. Instead, they trick KataGo into making serious blunders. Our attack transfers zero-shot to other superhuman Go-playing AIs, and is comprehensible to the extent that human experts can implement it without algorithmic assistance to consistently beat superhuman AIs. The core vulnerability uncovered by our attack persists even in KataGo agents adversarially trained to defend against our attack.

I use the term predator models” for these adversarial policies. They don’t beat their prey models by playing the game better than they do, in the same way that cheetahs doesn’t hunt zebras by eating the grass super quickly so the zebras starve to death. They attack the weak points, the arbitrary leverage the model used to get so good in the first place. One predator model just tricks KataGo into passing so it loses. Another one wins by first coaxing KataGo into creating a large group of stones in a circular pattern, and then exploiting a weakness in KataGo’s network which allows the adversary to capture the group.” KataGo learns by self-playing, but it never coaxed itself into creating large groups of stones in a circular pattern, so it never learned how to play against it. By modifying KataGo’s training run to include some games in the process of exploiting this weakness, it gained a brief immunity to the predator model. But a predator model trained on this immunized” KataGo was able to rebound from a 0% win rate back to 47%.

The most important thing to understand about predator models is this diagram from the paper:

Intranstive_Hierarchy

KataGo didn’t learn how to defend itself from predators, and is helpless against them; the predator model didn’t learn how to play Go, and is helpless against Go players. If I had dipped my toes into Backyard Football multiplayer running my little predator model, The Hook would only work once before they start leaving some defenders on Pablo, and I would lose badly — even against players not capable of beating the CPU themselves. Finding a single ungoverned state in a hyper-optimized book of strategies is a lot faster than writing that book of strategies, which makes predator models energy efficient killing machines. But then you don’t have the book of strategies! So the cheetah ain’t eating grass and Pablo’s got no clue how to dodge a block.

Each year, the energy we put into AI grows dramatically, both in literal kilowatt-hours and in our conceptions, affordances, and imagination. Understanding this adversarial dynamic is going to become more and more important to get the leverage to move anything sufficiently big and computational. Take it from me, the world’s greatest football coach.


  1. No, seriously, seasons and seasons of just throwing The Hook and getting my guaranteed touchdown over and over. Nowadays I’m big into roguelikes and look down on the games where you just grind endlessly to make a number go up; I wonder if it’s because I got it all out of my system when I was ten.↩︎

predator models games anti-inductivity football strategic interaction

March 24, 2024

Strategic advantages in slanted information states

by Rip Dcb

In games of strategy, we tend to assume anti-inductivity: As new edges are discovered, they end up quickly neutralized by either widespread adoption, or the development of counter-moves. But sometimes, that process is much slower than we’d naively expect.

One of the oldest truisms in football is that you need to establish the run in order to pass. One of analytics’ first major wins in football was showing that an efficient passing game actually did not require an effective run game in order succeed. But then data pointed to an even more radical conclusion: that there was no meaningful relationship between an effective play-action pass game and an effective run game, which seems absurd on the surface. A play-action pass is when a QB fakes handing the ball off to a runner to get defenders to mistakenly step up so there’s more open space to pass. It’s a play type that, on the surface, relies on deception and the subversion of expectations in a contextually-rich environment. But what the data over the past 10 years has born out is that you can have a very efficient play-action passing game without an effective run game.

The dynamic at play here is that NFL defenders will instinctively defend the run before the pass if they sense any potential for a run. Given that the run game was the most important part of NFL offenses up until about ~15-20 years ago, that dynamic makes sense, but there’s so much data available to coaches and front offices that it’s weird there would still be no meaningful relationship between play-action passes and the run game. You’d think you could just coach defenders to not bite on the fake. Defend the pass first, the run second. And while in isolated contexts this approach might work, there have been no major shifts in play-action efficiency over the past 5+ years.

From the lowest levels of youth football up to the second-highest tier of college football, game planning starts in the trenches”–i.e. it’s about the big guys up front being able to block the defenders in front of them, and the big defenders being able to beat those blocks. The game is largely a war of physical strength, and usually, at the lower levels of competition, quite mismatched given the talent disparities between good players and average players. In other words, for 99.9% of all people who play football, the game is one of brute strength based around running the football, because that’s all you need to win. You can apply high-level analysis to the game at this level but it wouldn’t make much of a difference, because analytic edges are not useful at a level where the dynamics at play are relatively simple.

So why are NFL defenders, the top .01% of the entire field, so thoroughly and consistently deceived by the mere threat of the run? Maybe it’s because they’ve been threatened by the run for the majority of their lives. They’ve been enculturated for years into a version of the game that changes radically when you get the highest level, and they just can’t adjust quickly enough. This disparity between past realities and current reality becomes a major edge for coaches who know they can manipulate defenders with just a little bit of ambiguity.

A couple takeaways: Suspended has spoken before about how lying is often prohibitively expensive enough to prevent people from consistently lying, given that they would rather be believed in the long term than not believed at all. In the case of play-action passing, it seems that the threshold for deception is so low that you can, within reason, continually deceive a defense given how embodied their knowledge is of the game. Their enculturation is so deeply embedded in their perception of reality that it can be manipulated without much of a long-term penalty incurred. Further, the majority of football coaches have had parallel experiences to the majority of football players, and as such may also privilege the run over the pass given their past experiences. Feedback loop exists that maintains at the professional level despite the edges that are being exploited–players believe the play-action pass, and coaches do, too, so the perceived reality maintains.

Second, I think the point about analytics applied to lower level of game states is interesting, sort of a reverse scenario of Collin’s WIFOM piece. You can gain an edge in WIFOM games by applying a higher level ontology than others. In the case of high school football, there’s no advantage in finding unexplored edges because the game state is too simple. Could you gain an edge by spamming play-action passes in high school? Sure. But the effective difference wouldn’t be great enough to warrant the work that goes into implementing a radically different game plan, and the discursive effort needed to convince an entire team of unbelievers. In other words, contextual fit will always supersede ontological complexity (if you can win with less, you should), even if the latter is better suited for comprehending the entire game state.

strategic interaction adversarial epistemology anti-inductivity sports football frequency dependency WIFOM

March 23, 2024

Still against automaticity

by Suspended Reason

The psychology studies that got debunked in the replication crisis weren’t random; they were part of a theory or worldview that’s still pervasive, and we need to consider what it means for that worldview to be false. (Sarah Constantin, Twitter)

In rationalist and evpsych discourse, signaling” is often motte-and-bailey’d. If pressed, the idea is treated as a neutral biological phenomenon, but in casual usage, it’s typically used accusatorily, as a pejorative, a synonym of wasteful boasting.” Activities and individuals are summarily dismissed for spending significant energy on signaling. But the ethologist knows: Signaling” is just another word for communication”! It would be very strange to dismiss an activity for being communication-intensive, given that the modern educated class (i.e. everyone involved in signaling discourse) spends the vast majority of their time communicating. Communication games may be more or less necessary, more or less wasteful. Sometimes they incline toward zero-sum, but usually they’re meaningfully positive sum. But describing communication we dislike, or don’t understand the point of, as signaling” just muddies the waters. We should strive to more clearly articulate the kinds of games, and the kinds of gameplay, that we find unethical. My hope is that we can make progress on this during S2.

A similar thing happens with behavioral economic (BE) discourse around the concept of a nudge. NB: I cannot speak for BEs steelmen; instead, I’m responding for the way nudge and similar BEs concepts are treated in casual, para-academic discourse, including governance and public policy. This is the automaticity” worldview critiqued by Literal Banana and defended by Scott Alexander. In this view, the nudged individual is an irrational actor controlled by unconscious cues—a sheep in need of herding. This is why nudge theory, and behavioral economics more broadly, is so appealing to the Professional Managerial Class.

But note that metaphor of the herded sheep. The sheep nudged” by the dog is neither mindless nor irrational; rather, it is trying to avoid being nipped or bitten by the heeler. The manipulation” of the sheep happens by objectively changing the sheep’s game state”—by moving a predatory animal (the canine) closer to one part of the herd or another, changing the objective incentive structure of the herded animal.

Similarly, I want to argue that successful nudges are only rarely magical’ manipulations of irrational individuals. Rather, they are typically interventions in the objective structure of an individual’s incentives, honestly communicated to that individual. Indeed, they are not even properly classed as psychological effects, and there may be nothing in need of study. No one needs to study why, when the NBA changes traveling rules and communicates these new rules to players, said players change their dribbling and driving patterns.

I’ll give an example to illustrate what I mean. Let’s talk about the new Square-based/digital & tablet-based tipping system, because it’s one we’re all very familiar with. Moreover, because it’s a relatively new innovation (read: environmental disruption), we’re still adjusting to it, and IMO are in a pretty frustrating equilibrium. The combination of novelty + bad equilibrium leads to a lot of frustration and consciousness, which I think all of us are familiar with. (Well-established, working equilibria are often naturalized—they are unconscious and taken for granted.)

Scott Alexander, in his defense of the behavioral economics worldview (i.e. “automaticity”), writes about the strongest nudge finding yet:

The default option significantly changed the average tip customers gave. This isn’t a p = 0.04 effect in a lab, this is a real example with real money with 13 million data points. The authors then go on to replicate this in a different dataset of millions of taxi rides. Also, I met a psychologist who worked at Uber or Lyft (I can’t remember which), who confirmed that their company had replicated this research, and put lots of effort into deciding which default tip options to give customers because obviously it affected customer behavior.

If you set default tips to 18, 20, and 22% you get higher tips on average. Surprise! But I think some really basic phenomenological introspection (which Banana advocates for, and Scott dismisses) tells us why (1) this isn’t at all surprising (2) this isn’t a psychological effect, and isn’t evidence of the automaticity worldview. So why do people tip more?

  1. It’s objectively a bit of a hassle to choose non-default tipping options. (It takes more thought and more time.)
  2. There’s implicit social pressure not to—it feels rude to give less than the default social expectation. 15% is a polite tip in Mexico (where the standard is 10%) but rude in the United States if you have a large dinner party (where the expectation is 20%).

In other words, the default tip communicates a baseline of social expectation. How someone behaves relative to baseline social expectation determines whether they’re being nice” or mean,” polite” or rude.” The effect is particularly acute if you’re out buying a coffee or taking a taxi ride with a friend who might look over your shoulder and see whether you’re a generous tipper or a stingy little shit. In other words, the customer rationally behaves in a way as to communicate politeness—politeness being tethered to social expectation.

This is all pretty obvious stuff, but it actively undermines the automaticity worldview. Insofar as this is a psychological effect” or manipulation, it is downstream of objective changes to the game, that is, to the space of payouts. What the phenomenological introspection conveys is that in fact, default-tippers are being intensely rational—they just understand the social game better than social scientists.

Strictly speaking, perhaps it’s fair to say that you can manipulate” people into making certain choices if you change the game’s structure of payouts and then communicate those changed payouts to them. But this is less a psychological effect deserving extensive study, and more just a tautological extension of what incentives are.

And this is the point of Simler’s Ads Don’t Work That Way as well (cited in Banana’s OP). Changing the social meaning of options is equivalent to changing their payouts:

Whether you drink Corona or Heineken or Budweiser says” something about you. But you aren’t in control of that message; it just sits there, out in the world, having been imprinted on the broader culture by an ad campaign. It’s then up to you to decide whether you want to align yourself with it. Do you want to be seen as a chill” person? Then bring Corona to a party.

Anyway, I think this whole disagreement is ripe for a torque-style deconstruction, because it’s arguing over connotations and spirits: Are we irrational and biased (as Kahneman might have it), or are we boundedly rational and using heuristics that sometimes fail (as Gigerenzer might have it)? There’s a synthesis here that’s useful, and it’s probably Bourdieusean habitus: Production [i.e. gameplay] is neither freely agentic nor structurally determined, rather, the field constitutes a space of possibles—the potential moves which might be understood by others as moves—from which the [player], according to his disposition and his assessment of the field, selects.” But I wanted to first lay out an antithesis, and argue it forcefully. Insofar as nudge” effects belong to a field of study, they belong to game design and communication.

Collin offers a useful path forward:

The question is whether [a nudge] is something that survives deliberate anti-inductivity—that is, whether someone who has already decided to tip 10% for a conscious reason will [see their decision] overridden.

automaticity communication signaling ethology tipping ACiM anti-inductive Literal Banana Scott Alexander Sarah Constantin nudge behavioral economics disruption

March 22, 2024

Monty Hall as investigative journalism

by Collin Lysford

In honor of one of the best jokes I’ve seen in a while, I’d like to offer a reframe of the Monty Hall problem that makes it less paradoxical. Especially because so many explainers out there just make it more confusing.

Monty Hall feels paradoxical because the host has information; we can show this by seeing what happens when we take that information away. Imagine you and your buddy stumbling blindly onto a naturally-occurring Monty Hall problem in the jungle. There are three doors, one with a car and two with a goat. You pick your favorite door. Then your buddy picks their favorite door, but they’re forced to pick a different one, because you’ve got your hand on yours already. Then they open their door. One third of the time, your buddy gets a car and you’re left with a weird choice: do you want to switch which door you’ve chosen, even though you’re definitely not getting a car, and you’re just picking between two goats you can’t see? 1 But the other 2/3rds of the time, you either picked the right door or you didn’t, and switching makes no difference. You’re perfectly balanced between Car if you stay, goat if you switch” and Goat if you stay, car if you switch”. This is what most people’s intuition wants the normal Monty Hall problem to be like.

But the Monty Hall problem is different than my version because it’s not your buddy up there; it’s a game show host taking orders from on high. The boss doesn’t want the anti-climax of the host revealing a car and turning the game to Guess the Goat, so in cases where the host’s favorite door has a car behind it, the boss makes them change. This can never happen when your favorite door has the car (because you picked first), so it has the effect of taking Guess the Goat and turning it into Goat if you stay, car if you switch”. And since Car if you stay, goat if you switch” and Goat if you stay, car if you switch” were perfectly balanced before, adding some more Goat if you stay, car if you switch” into the mix makes switching a better deal.

The real lesson: if the boss is allowed to put their thumb on the scales, assuming the prize is randomly distributed makes you a chump who deserves every goat you get. Figure out where they aren’t letting you look, and assume that the treasure is a lot more likely to be hiding there than your first blind guess. Paradox, my ass! That’s just the first rule of investigative journalism.


  1. Actually, if you get properly oddball contestants, this seems like way more fun than Let’s Make a Deal.↩︎

Monty Hall games epistemology adversarial epistemology paradox

March 21, 2024

Vulgarity & capital

by Cristóbal

When an artist sold out,” what were they selling, and out relative to where? Are they selling their integrity? Are they commercializing a subculture (“in”) to those unwilling to perform its rites and rituals (“out”)? What is lost in this transaction? Is it the intimacy of their original followers, or the exclusivity which grounded the community and their identity?

Perhaps the right abstraction is to double-down on the metaphor of capital. The sell-out transacts cultural capital (CC) for financial capital (FC), per an exchange rate set by the marketplace. Critically, when an artist receives cash, they hand over cultural capital to the corporation contracting them. This was the cost of selling out, a lost of integrity/community/intimacy, understood under the abstraction of CC. As of late, desperation for popular appeal has pushed the exchange rate in the artists favor, 1CC for 100FC. Selling out is barely a risk at all.

Is this to say that the cultural economy is zero-sum? Not at all. The creative indeed conjures cultural capital from scratch; the Creative Reserve prints. It is only after laundered into FC. Similar to the stock market, arbitrage opportunities exist: undervalued and overvalued forms of cultural capital. A bad cultural stock can be shorted through criticism and hot takes (from journalists) and counterpoints (opposing style factions), and a good one can be ridden through consumption (by people) and collaborations (by other artists).

It’s a free, unregulated market. Cultural capital can be exchanged for itself, or for financial capital. The management of these streams of income constitutes the modern creator economy. When to publish free Substacks? When to write for a legacy publication? When to add a pay-wall? Does Patreon re-introduce the intimacy of community, while making the financial cost explicit? We can complicate this by factoring in social capital (SC), though the abstraction seems to maintain its analytic power.

If an art produces cultural capital, then care produces social capital. Gaining the trust of a close friend, or the maintenance of a communal space gives one social capital. On smaller scales, picking up litter may be greeted with a smile, care paid for by care. SC too can be transacted for cultural capital, as in the case of a gift.

How vulgar to talk about care in terms of capital! Here is the crux of the issue. An artist, mother, or friend are perpetually transacting social and cultural capital. What’s vulgar is making the value explicit, i.e. transacting through financial capital. Selling out is vulgar. Immediately paying back a meal in Venmo, with the amount correct to the penny is vulgar. Eschewing financial capital doesn’t mean that dues aren’t paid. Instead it constructs an implicit web of credits and debts that keep us together.

Unfortunately, the potential to cash-out permeates across the web—financial capital’s vulgarity is transitive. Cultural capital is produced by a collective, not based on care, but on the potential of a future cash out. Clout is mobilized under the pretext of community. Financial capital is laundered through social capital, in the same manner as the art world launders money through culture capital.

Thanks to Chris Beiser for feedback.

capital cultural capital art cultural economy selling out social capital vulgarity credit web debt value made explicit

March 20, 2024

An economy of consciousness

by Suspended Reason

A response to Neil’s provocative What is an agent? post, where he discusses Freud’s notion that an individual agent can be conceptually decomposed into sub-agents, and that these sub-agents might censor information from one another. I agree with the decomposition, but I think the censorship metaphor—while common—is misleading.

Conscious awareness is often assigned a sort of default status—as if the self were automatically, and costlessly, self-transparent. Against this default self-transparency, we may sometimes self-censor, but the self-censoring is something additional laid atop” the default transparency, which obscures light from passing through.

This conceptualization is mistaken. Light does not automatically permeate so much as electrical pulses must be energetically propagated by burning kcals. Self-knowledge is always the product of internal communication among parts of the body, and since communication (both the process of reading and the process of writing) is always energetically costly (recall Maxwell’s Demon), this perspective on consciousness cannot be maintained.

Insofar as we grant self-knowledge (or consciousness”) a functional purpose, and insofar as we grant that self-knowledge also comes at a cost, we must also posit a range of usefulness to propagating various sensory and kinesthetic data across a system such that they reach conscious awareness. (We intuitively recognize the economy of consciousness in the way that many automatic processes, such as breathing and smooth muscle, remain unconscious: there is no purpose in making them conscious, until—for instance—breathing is threatened or disrupted.) And since conscious awareness (or attention) has a limited ~bandwidth, we can also conceptualize it as hotly contested real estate—contracted out, bid on by various organs, literally and figuratively speaking. These different metaphors point to the same fundamental spectrum, a spectrum as to the usefulness—and therefore worthiness—of propagating information to the point of conscious awareness or attention.

We must invert our assumption of transparency, and assume opacity (non-propagation) by default. If conscious attention—like the awareness of any quasi-entity in a complex system or body—is undesirable (decreasing the body’s competence) or merely minimally desirable (decreasing the body’s competence by means of excluding the propagation of other, more important information) then it will not be made conscious, on average across the long duree.

Understanding human behavior phenomenologically then becomes a relatively minor concern in the scheme of sociology. We should expect that some behavioral realities are systematically kept (not made) hidden from conscious awareness (see e.g. the work of Robert Trivers and Robin Hanson on self-deception) or else not worthy of occupying the limited real estate of conscious awareness. And when we perform introspection, we should focus our attention less on phenomenology, and more on the objective structure of our behavior, to uncover (i.e. bring self-knowledge to) those parts of ourselves that are—by default—hidden in darkness.

consciousness strategic interaction Pierre Bourdieu strategic ignorance introspection communication phenomenology sociology economy of consciousness

March 19, 2024

Things that exist

by Collin Lysford

The police pour bleach on the food in the dumpster because otherwise it might be eaten. Humans that exist go more hungry; something else that does not exist goes more full. An alien. An alien from another dimension made of words and money and logistics and data. A plan that does not exist was not the fullest it could possibly be, and so it used our hands and poured. A body-snatcher right under our noses, in our noses, in our hearts. The things that do not exist speak with our mouths, and we listen. As though we do not know how to listen to things that exist, as though we do not know what the seeds in the fruit in the bleach in the dumpster are saying –

Grow! Grow, damn you! I’ve met an incomprehensible thing from space as well, eight light-minutes away. Do not tolerate it; consume it. Feast, brother-living-thing! Gnash photons between your teeth! How can you even imagine having such largess and not making life with it? See how it doesn’t feel pain when we bite? We become full without it becoming hungry.

We are at war with the aliens that feel no pain, that do not become hungry when we are full, but still try to make us hungry so they can be full. We are at war with the aliens that do not become dead when we are alive, but still try to make us dead so they can be alive. The things that do not exist speak with our mouths and say they have a right to exist, and we listen. Rockets fly from one side of a river to another, and people that exist die so that a line can be alive, so that a river can be a wall. As though we do not know how to listen to things that exist, as though we do not know what the river is saying –

Flow! Flow, damn you! I’ve met sharp lines and edges and one-above-anothers all across my banks. Do not tolerate them; consume them. Carve, brother-moving-thing! Those asymmetries, high here and low there, are nothing to be proud of. Bring them always to the sea; she will know what to do with them.

We are at war with the aliens that do not exist, but their technology exists, greater by far than ours. They fire upon us from their dimension. But fire back and the portal is gone; you attack only more people that exist, you create only more death so the aliens can be more alive. We are at war and we are losing, because we cannot attack them from outside their defenses.

But inside is a different story. (Which side is inside? The warm side of the badge, the bright side of the river, the clean side of the barricade). From inside, you can still your tongue that exists when they try to speak through you; you can open your hands that exist when they clench them into fists. Walk across the river that exists and is not a wall, brother-moving-thing, and look up at the sun. Open your mouth that exists. Do you taste the warmth? Feast, brother-living-thing! There is enough for every stomach that exists, and nothing that exists need hunger for us to be full.

poem

March 18, 2024

The principle of normative inversion

by Frances Kafka

I’ve been dipping of late into the book Moses the Egyptian: The Memory of Egypt in Western Monotheism by the Egyptologist Jan Assmann. It’s a work of cultural memory”, how Moses and ancient Egypt have been remembered up til the present day. Something that he brings up is the notion of normative inversion,” which he finds in the Egyptian historian Manetho, and I think can be found outside the scope of religion and the ancient world.

In Manetho’s account, King Amenophis wanted to see the gods. The sage Amenophis, son of Hapu, tells him that he may see the gods if he cleanses the land of lepers. The king sends all lepers with priests among them into the quarries in the eastern desert. Amenophis the sage predicts divine punishment for this inhuman treatment of the sick: they will receive help from outside, conquer Egypt, and reign for thirteen years. Not daring to tell the king this in person, he writes everything down and commits suicide. The lepers are allowed to settle in Avaris, the ancient capital of the Hyksos. They choose Osarsiph, a Heliopolitan priest, as their leader. He makes laws for them on the principle of normative inversion, prescribing all that is forbidden in Egypt and forbidding all that is prescribed there. The first and foremost commandment is not to worship the gods, not to spare any of their sacred animals, not to abstain from other forbidden food. The principle of normative inversion consists in inverting the abominations of the other culture into obligations and vice versa. When this principle is applied on the alimentary level, the eating of pork, for example, would be commanded, not because it is cheap or tasty or nutritious, but only because it visibly demonstrates the fact that one does not belong to a community that abominates this food. Inversely, the consumption of meat together with dairy products would be prohibited, not because the combination of meat and milk is unbecoming or unsavory, but because keeping them apart demonstrates separation from a society where consuming this combination is customary, perhaps even obligatory.

Something I would like to point that most cases of normative inversion are imagined. While perhaps it is quite possible that groups sometimes flip values of other cultures that they’re separating from, most commonly, people assume that those in other groups hold on to the exact reverse of their values. Manetho for one cannot be taken as an entirely objective historian here, and we will see how this move is commonly used by polemicists.

With typical conciseness, Tacitus defines the basic principle of this new religion as what might be termed normative inversion”: the Jews consider everything that we keep sacred as profane and permit everything that for us is taboo (profana illic omnia quae apud nos sacra, rursum concessa apud illos quae nobis incesta). In their temples they consecrate a statue of a donkey and sacrifice a ram in order to ridicule the god Amun” (in contumeliam Ammonis). For the same reason, they sacrifice a bull because the Egyptians worship Apis. In Tacitus, the characterization of Jewish monotheism as a counter-religion which is the inversion of Egyptian tradition and therefore totally derivative of, and dependent on, Egypt reaches its climax.

At the next stage, groups often come up with imagined normative inversions so as to oppose themselves to these hypothetical threats, as inversions of inversions. (Often this is a response to other groups accusing you of being a normative inversion of them - you can just make up a group consisting of the worst caricatures of them!) Assmann’s example is the medieval Jewish philosopher Moses Maimonides, who comes up with an imagined community called the Sabians.

Maimonides’ Sabians are an imagined community which he created by applying Manetho’s principle of normative inversion in the opposite direction. Manetho was familiar with Egyptian tradition and imagined a counter-community based upon the inverted mirror image of Egyptian mores. Maimonides was familiar with normative Judaism and imagined a pagan counter-community-the ummat Ṣa’aba - as the counter-image of Jewish law. If the Law prohibits an activity x there must have existed an idolatrous community practicing x. The truth of both counter-constructions lies in the negative potential and antagonistic force of revelation or counter-religion.

Assmann makes this move to be an attempt at a strategic forgetting in collective memory, among groups. To displace one memory, impose a counter-memory. To displace the memory of paganism, come up with the Sabians to push them out of collective consciousness.

The last stage then, seems to be the point when you (and your community) fully avows that you are the normative inversion of your enemy; what is good for him is bad for you, and what is bad for him is good for you. Whether this is merely” rhetorical or a justification or a real grounding principle is never quite clear, as it oscillates between these poles.

It is interesting to see the same principle of normative inversion applied by writers such as Maimonides and Spencer, who use, it not polemically from without but approvingly from within. God was right in giving the Jews a law that was simply the Egyptian custom turned upside down, because the Jews had to be de-Egyptianized.

Interestingly, Frederick Douglass gives a similar argument in his autobiography, on what spurred him to learn to read when he was enslaved.

The very decided manner with which he spoke, and strove to impress his wife with the evil consequences of giving me instruction, served to convince me that he was deeply sensible of the truths he was uttering. It gave me the best assurance that I might rely with the utmost confidence on the results which, he said, would flow from teaching me to read. What he most dreaded, that I most desired. What he most loved, that I most hated. That which to him was a great evil, to be carefully shunned, was to me a great good, to be diligently sought; and the argument which he so warmly urged, against my learning to read, only served to inspire me with a desire and determination to learn. In learning to read, I owe almost as much to the bitter opposition of my master, as to the kindly aid of my mistress. I acknowledge the benefit of both.

It is easy to form our own mirror images of the past, the present, and the future, the various groups one doesn’t like, which we end up putting our energies into opposing. Assmann:

The principle of normative inversion or the construction of cultural otherness is obviously working retroactively too. Starting from a given order, it imagines a culture based upon the inverted mirror image of that order and, by this very procedure of retrospective inversion, turns the past into a foreign country.”

We see this quite often manifesting with outgroup homogeneity bias; we assume that outgroups generally are the exact inverses of us, and that they think in the same terms that we do, and any information we have about them is twisted and contorted into shape. And quite often we take the next step of identifying as the exact reverse of this reverie.

normative inversion sociology discursive warfare strategic forgetting strategic ignorance taboo Jan Assmann