Notes on Compositional Agency

Compositional agency is about agents that come to exist as a result of composition/combination/aggregation of "smaller" agents. Here are my growing, scattered notes/thoughts on this topic.

Terminology

When I want to distinguish between "levels",¹ I may say "sub-agent", "super-agent", "sub-sub-agent", etc.

Is agency universally/necessarily compositional?

Some people are going to claim that "all intelligence is collective intelligence", e.g. Falanday et al. (2022) or Michael Levin. Every intelligent system must be a collective of subsystems that are appropriately coordinated and those subsystems need to be intelligent in the right way.

My understanding of Michael Levin is that he means that somewhat metaphorically: it is not strictly/literally the case that every agent is composed of sub-agents that are themselves agents in whatever fixed sense of the word. Rather, there is no strict line between agency and non-agency or intelligence and non-intelligence or mindness and non-mindness or life and non-life.

Therefore, if you were to grade a system's intelligence/agency as a scalar, we would say that a system with score $x$ is composed of some number of sub-systems with a score $x-\epsilon$ with $\epsilon$ relatively small.

One suggestion of where to draw a bright-ish line was given to me by Richard Ngo (personal communication): for a part to be considered a subagent, it needs to have situational awareness, i.e. be appropriately/strategically responsive to its situatedness in the broader world to advance its purposes (that's at least my understanding of Richard's view).

Agency of human collectives

Conjecture: Large human collectives (say >1000 people) have too low communication bandwidth to have any complex value structures. Therefore, to the extent that they appropriately cohere as agents, their chief value/objective/purpose is the simplest and the most primordial one, namely, to keep themselves existing, keep being what they are, and perhaps enhance their being of what they are, likely including power-seeking (to the extent that this is useful/viable/realistic).

Rough argument: If your ability to establish value-related common knowledge is limited, you compensate for what you can't establish as common knowledge via Schelling point coordination. The most reliable Schelling point is the lowest denominator, i.e. keep the system as it is, ensure it keeps existing roughly in its current form.

This is not necessarily the case of small-ish teams or companies which can therefore maintain sufficiently rich value structures. Kongō Gumi has existed for almost 1500 years which Geoffrey West attributes to it being a rather small family business for most of its history, therefore being able to resist some scale-of-company-related tendencies.

Minor point of evidence: It seems to be the case that collective agents relatively quickly pivot towards whatever attractor is going to keep them existing. Ostensibly AI Safety labs easily turn to AI Capabilities labs (OpenAI, Anthropic). Communist countries, founded on principles that are rather opposed to nationalism, quickly pivot to nationalism because it is adaptive if you want to build a strong state apparatus (Soviet Russia, PRC, North Korea).