Core of Mindness

Beliefs (usually expressed as probability distributions) and values (usually expressed as utility functions (equivalent to some consistent preference ordering) or similar) are first-class-attributes of an agent/mind. The implicit belief seems to be something like: An agent is an entity that maximizes the expected value of its utility function where that expectation is taken under its probabilistic beliefs about what actions lead to what outcomes and so on; "mind" is roughly what we call the agent's computation in service of this goal. Another free parameter is the agent's decision theory, i.e. its way of modeling the dependence between its decisions (regarding its policy or individual actions or whatever) and utility-relevant outcomes.

These three components, probability, utility, and decision theory, came to represent the core of agency/mindness, i.e. it is a working assumption that they serve as a solid backbone for building a theory of intelligent agents.

What reasons do we have to think this?

It seems like the core intuition is that an agent is an entity that robustly wants something in the sense of effectively trying to get a lot of something and a nice mathematical formalization of this is choosing actions that maximize some function's expected value. The choice of the decision theory comes in to clarify the trickiness of modeling the dependence between actions and states that depend on them. (It's interesting that the very concept of utility was birthed in order to handle the St. Petersburg paradox. See Moscati, 2023)

Alternative core intuitions:

• Kantian rationality (?): An agent is supposed to behave so as to follow some logic / reasons.
• Progency:¹ Homeostasis, self-maintenance, adapting to changing situations, while preserving one's own shape/constitution during transitions.
• Mind-first: Start with mind, rather than agency. A mind is something that is capable of thinking, understanding, learning, &c. (All of these are in want of explication, of course.)