Ground of Agency/Mindness

Question: What are agents/minds?¹

Standard/orthodox type signature

In standard decision theory, the "type signature" of an (idealized) agent/mind is a triple consisting of probabilistic beliefs, a utility function over the world's states or trajectories over world states (or a preference ordering satisfying such-and-such criteria), and a policy (we can also call it a decision function) formalizing what it means to maximize utility under the agent's subjective beliefs. Various decision theories, such as causal decision theory (CDT), evidential decision theory (EDT), and updateless decision theories (UDT 1.0, 1.1, 1.01) are orthodox examples of such a policy, where the value being maximized is a linear expectation of the utility function, as weighed by the agent's probabilistic belief.

This type signature is taken as an assumption in parts of AI discorse which are trying to think of advanced AI systems (in particular, advanced AI agents) as approximating idealized agents (often, approximating them so closely that it is impossible for humans to distinguish them from actually idealized agents in a way that has any actionable implications).

There are various ways to "loosen up" or generalize these assumptions. (The following list might be expanded in the future.)

• Probabilistic beliefs can be generalized to or replaced by:
  • Imprecise probabilistic beliefs, e.g. Infra-Bayesian homogeneous ultra-contributions (HUCs) and several independently invented equivalent frameworks.
  • Dempster-Shafer belief functions.
  • Joseph Halpern lists a few more formalisms for representing beliefs and updating those representations in Reasoning about Uncertainty.
• Utility functions can be generalized to or replaced by:
  • Imprecise utilities (infra-utilities).
  • Non-VNM preferences, e.g., incomplete preferences.
• Standard decision theories can be generalized to or replaced by:
  • Less orthodox policies dispense with the need to decide based (solely) on expected value, e.g. Lara Buchak's risk-weighted expected utility.

What reasons do we have to think that the standard triune type signature of an agent (or something close to it) is the right abstraction for thinking about agents?

It seems like the core intuition is that an agent is an entity that robustly wants something in the sense of effectively trying to get a lot of something. A nice mathematical formalization of this is: the agent is choosing actions that maximize some function's expected value. The choice of the decision theory comes in to clarify the trickiness of modeling the dependence between actions and states that depend on them.²

An appealing justification of this mathematical operationalization is that it can be derived from several more modest operationalizations (e.g., frameworks of von Neumann-Morgenstern, Savage, Jeffrey-Bolker) with some (more or less important) differences depending on the exact choice of the operationalization.

Questions:

• What are some alternative intuitions behind what agents/minds are "fundamentally"?
  • Where do we get if, instead of starting with this core intuition, we start with a different intuition?
    • It doesn't have to be an intuition for what minds are. It could be an intuition for what distinguishes minds from non-minds.
• Is "mindness"/"agency" the right basic concept or is there a more basic concept/category within which minds/agents form a subclass ("natural" or "conventional")?³
  • For example, adaptive systems.

Alternative intuitions

• Kantian rationality (?): An agent is supposed to behave so as to follow some logic / reasons.
• Progency:⁴ Homeostasis, self-maintenance, adapting to changing situations, while preserving one's own shape/constitution during transitions.
• Mind-first: Start with mind, rather than agency. A mind is something that is capable of thinking, understanding, learning, &c. (All of these are in want of explication, of course.)
• Minds are distinguished by internal sharing of elements: "to what extent does a mind share / connect / interface / translate elements with other elements when that would be useful?".
  • The requirement is not that the mind share elements whenever it is useful. This is not possible given logical non-omniscience, boundedness, etc. The requirement is only that it does it to a large extent.