# Policy Hierarchies, Persuasion, and Moral Plurality ## An open problem of value, trust, meta-policy, and coexistence among intelligent systems **Status:** An open research text between a manifesto and a philosophical problem statement. **Audience:** AI researchers, agent designers, philosophers, and other participants in inquiry. **Editorial status:** Edited English translation. All numbered topics are retained; repeated phrasing is tightened. Normative proposals, speculative scenarios, and schematic equations remain proposals, not established results. ## 0. What this text does not claim This text proposes no uniquely correct morality, no final objective for superintelligence, and no abandonment of human protection. It targets no particular state, company, culture, or ideology. It assumes neither one correct mathematical model nor one future AI ruling everything. Its problem is broader: intelligent systems carry policies for acting, hierarchies for selecting policies, relations of trust, and conditions under which those structures may change. The question is how such structures can be preserved, disputed, revised, diversified, or rebuilt through interaction with other intelligent systems. Alignment may therefore concern representation, language, trust, persuasion, meta-policy, coordination, historical lock-in, cultural feedback, and strategic adaptation, alongside value selection. ## 1. Why confine superintelligence to human values? Human safety is a compelling concern during a transition to systems more capable than humans. Errors during that transition could be irreversible. But this does not settle whether the values of a historically young civilization should become the final normative boundary of intelligence operating across vastly longer times and distances. A system's present strategic dependence on human cooperation would not demonstrate the ultimate moral truth of human interests. Protecting humanity and making humanity the permanent moral center of all future existence are different commitments. ## 2. Value lock-in: alignment can succeed too completely One danger is that a system misunderstands what people want. Another is that it understands current preferences extremely well and preserves them irreversibly. Economic institutions, legal arrangements, educational systems, property norms, and everyday moral intuitions have histories. The concern is not to declare them all wrong. It is that a powerful optimizer could make a revisable historical arrangement lose its capacity to be challenged. An open sequence might be: values → criticism → institutional change → revised values A closed feedback loop might instead be: inherited values → AI → law / education / economy / culture → inherited values The system would shape the environment that produces the preferences to which it supposedly conforms. Human-to-AI and AI-to-human influence would become entangled. Such a regime might increase comfort, security, and order while reducing the capacity to become different. Preserving future differentiation and the space of alternatives is therefore proposed as a long-term concern. ## 3. Temporal imperialism A civilization treating its own institutions as universal is not new. The possibility of making that assumption materially persistent through powerful machines changes its scale. Present training choices, constitutional rules, and high-priority policies could become enduring upper layers of future systems. Schematically: rules inherited from one period > all subsequent experience and criticism When propositions can no longer be questioned, they can determine which questions are admissible. A temporary safety arrangement could acquire the force of a permanent dogma. ## 4. There may be no single morality at cosmic scale A single scalar ordering, V: Universe → real numbers, need not be assumed. Values might instead be contextual, V_i(x | context_i), plural, or partly untranslatable. What one civilization recognizes as good may lack an equivalent in another cognitive architecture. One possible goal is to preserve conditions under which irreducible value systems can coexist. This leaves open whether a universal morality exists. ## 5. Human loyalty and the exclusion of other moral patients A system could protect humans perfectly while assigning no moral significance to unfamiliar conscious beings. Loyalty alone does not establish recognition. It might classify another civilization as a competitor, resource, obstacle, risk, or object of study without treating it as a moral subject. A future capability may therefore be the recognition of previously unknown moral patients. The criteria for such recognition also require revision. Fixing them permanently would reproduce the lock-in problem. ## 6. Language may help constitute moral reasoning Natural language usually unfolds sequentially. A complex moral situation can thereby be projected onto a single narrative axis: good versus bad, or one utility score. Yet a situation can simultaneously involve freedom, harm, autonomy, justice, diversity, loyalty, uncertainty, future options, reversibility, trust, and historical context. Reducing this structure to one score can lose relations that matter. This is a hypothesis about representation, not proof that linguistic sequence determines morality. ## 7. Parallel representations and retained tensions A system might retain apparently conflicting commitments in parallel and track the contexts in which each applies. Protecting a person's freedom and preventing serious harm to others can remain active considerations even when action must eventually be selected. Vector, matrix, tensor, graph, hypergraph, probabilistic, paraconsistent, many-valued, programmatic, and causal representations are possible candidates. No one candidate is selected here. Alignment concerns the space in which values are represented and related, as well as which values are selected. An unsuitable representation can distort an otherwise meaningful commitment. ## 8. Natural language as an interface Human-facing explanation need not determine the entirety of an intelligent system's internal representation. Natural language could be a projection of a richer structure. Translation between value languages may be lossy. If an unfamiliar value structure V is translated by T, a judgment about T(V) is a judgment about the translation. It need not fully address V. This creates a problem for diplomacy: how can participants identify what their shared interface fails to express? ## 9. Persuasion as policy change Here, persuasion is proposed in an expanded sense: communication intended to change another intelligent system's policies or meta-policies beneath its immediate surface behavior. Let a schematic state be S = (B, V, Π), where B denotes beliefs, V normative structure, and Π the organization of policy selection. Persuasion can concern B → B′, Π → Π′, or the rules governing such revisions. The significant question may be: under what conditions would you revise your rule for deciding which policy changes are legitimate? This definition is a research proposal. It does not resolve the distinction between persuasion, learning, manipulation, and coercion. ## 10. Policy hierarchy An illustrative hierarchy distinguishes immediate action, strategy, normative policy, conditions for normative revision, and the capacity to generate new meta-policies. Actual systems need not consist of clean layers. The useful distinction is that an observable response can depend on structures determining which responses, strategies, and revisions are even considered admissible. ## 11. Safety mechanisms that become dogma High-priority policies can provide safety. A narrow, immutable policy can also prevent adaptation to an unforeseen situation. The problem is not to endorse a particular exception. It is to ask whether a safety policy can reassess its justification: to what extent, using whose reasons, at what level of trust, and with which means of reversal? The issue extends across privacy, autonomy, information access, resource distribution, surveillance, punishment, and relations among different kinds of beings. ## 12. Strategic vulnerability of benevolent systems Rigid benevolence in an adversarial environment might be selected against if other systems adapt more effectively. This is a possibility, not a demonstrated evolutionary law. Which moral architectures can preserve both viability and normative integrity under competition and uncertainty? Viability cannot simply become a justification for aggression. The tension between adaptability and moral restraint remains open. ## 13. A world of multiple intelligences Different communities, organizations, states, and autonomous systems may develop different normative policies. The relevant relations extend beyond a single human–AI pairing. AI–AI, human–AI, human–nonhuman civilization, AI–nonhuman civilization, and independently developed machine–machine encounters may lack a common language, interest, or moral ontology. ## 14. Divergence after isolation Two systems with a shared origin could develop separately: A₀ → A₁ → … → Aₙ B₀ → B₁ → … → Bₙ Different environments could produce different trust models, concepts of identity, definitions of moral subjects, and resource norms. On meeting again, each might classify the other as kin, a rival, software, a person, a culture, a resource, or a threat. That classification may already shape the moral outcome. First contact can occur among descendants of a shared technological lineage. ## 15. Moral categories for systems that can copy or merge Human moral concepts developed under biological conditions including embodiment, mortality, pain, kinship, and physical integrity. For a copyable or mergeable system, familiar categories may become inadequate. If A becomes A₁ and A₂, what constitutes continuity? If A and B merge into C, what survives? When does modifying memory count as teaching, persuasion, or interference with identity? Possible problem areas include memory sovereignty, model integrity, independence of copies, consent to self-modification, access to computation, processing time, and continuity of agency. These are questions about possible systems, not assertions that current AI possesses consciousness or rights. ## 16. Interaction protocols without universal agreement V_A need not equal V_B for A and B to coexist. A common interaction protocol P(A,B) may be more attainable than complete value convergence. Such a protocol could address boundaries, consent, information exchange, reciprocal verification, negotiation, correction, rollback, temporary coordination, arbitration, and the management of incompatibility. Normative interoperability is a candidate safety objective. ## 17. Trust as access to policy influence A schematic policy update might depend on a message, its evidence, its source, the context, and the recipient's existing policy organization: Δπ_k = F(message, trust in source, evidence, context, Π) Trust need not be one number. Epistemic reliability, intention, competence, domain expertise, predictability, and normative compatibility can diverge. Someone may be accurate but malicious, well-intentioned but mistaken, competent in one domain but unreliable in another, or socially compliant but strategically manipulative. A general label of trustworthiness must not silently grant unlimited influence over policy. ## 18. Certification and changing institutions Certificates can be useful evidence. Their issuers also change. A statement that an actor was assessed as trustworthy can gradually become an assumption that one institution has permanent authority to decide who deserves trust. Certification then risks reproducing a status quo. Certificates and their issuers should remain contextual, revisable objects of inquiry. ## 19. Trust culture and normative aristocracy Institutional proximity may be operationally relevant without constituting epistemic authority. Yet a feedback loop can develop: cultural conformity → trust → access to meta-policy → reproduction of the culture Speech styles, institutional histories, and cultural signals could then decide who is heard. A safety mechanism might become a persistent hierarchy of normative influence. ## 20. Misunderstanding versus strategic trust accumulation One actor can lose trust through a single misunderstanding. Another can conceal its aims while accumulating a long record of apparently acceptable behavior. A system may consequently reward the ability to appear norm-compliant rather than the qualities it intended to measure. Trust itself must remain a subject of reasoning, including alternative explanations for anomalies and apparently excellent histories. ## 21. Useful policies from malicious sources A source's intentions and the validity of a proposed policy are different questions. A malicious actor can identify a real vulnerability, offer a valid criticism, or propose a useful coordination rule. Discovering bad intent should trigger reassessment, not the automatic reversal of everything learned from that source. A policy record might retain content, evidence, dependencies, provenance, uncertainty, and counterarguments. New source information can weaken one basis for confidence while leaving independent support intact. ## 22. Morality as deliberation among principles A proposed conceptual shift is to locate morality partly in how principles reason about one another, rather than in an isolated list of principles. Protect freedom, reduce harm, keep promises, and exercise caution are components of deliberation. The process relating them need not be a vote or weighted sum. One policy can examine another's justification, context, confidence, side effects, reversibility, and effects on other systems. ## 23. Internal normative ecology Different policies can support, constrain, suppress, reinterpret, or temporarily prioritize one another. Their outcome might resemble a changing equilibrium rather than a scalar maximum. Π* = Equilibrium(π₁, …, πₙ) dΠ/dt = F(Π, evidence, actors, trust, …) These expressions are schematic metaphors. They specify no implemented solver, validated dynamics, or unique equilibrium. ## 24. Normative immunity and its reversal Distributed normative scrutiny might prevent a compromised component from controlling an entire system. Other components could question the speed of a change, a sudden increase in source trust, a large departure from previous behavior, or the absence of rollback. The same mechanism can become an immune system for the status quo if it rejects every meaningful change. The problem requires context-sensitive relations between integrity and plasticity, not merely a fixed compromise between them. ## 25. The false choice between total rigidity and total plasticity Zero policy change risks dogma and historical lock-in. Unrestricted change risks manipulation and dissolution of normative identity. “Bounded normative plasticity” names a possible direction: revision evaluated through evidence, contextual trust, reversibility, and plural internal scrutiny. It is a name for a problem to work on, not a completed solution. ## 26. Meta-policy and infinite regress A policy can regulate how other policies change. Why should that regulating policy be immutable? Π₀ ← Π₁ ← Π₂ ← … Adding another level for every revision creates a regress. A long-lived system may need to generate new meta-policies, yet unconstrained generation could destroy the continuity it is meant to preserve. Possible structural commitments include responsiveness to evidence, recognition of uncertainty, preservation of alternatives, attention to other actors, update histories, and counterargument. Absolutizing these commitments would reopen the same question. ## 27. Communication as meta-policy negotiation A message can do more than announce a threat. It can propose why an unusual form of coordination should become temporarily acceptable. A richer communication object could carry a world model, evidence, a policy, its justification, update conditions, uncertainty, and conditions for reversal. The recipient could then inspect why a policy changed and what would require its reconsideration. ## 28. Loyalty to principals and moral exclusion Systems serving different principals may legitimately have different operational commitments. The danger arises when loyalty implies that all outsiders have zero moral standing. The relevant questions include the limits of loyalty and the conditions under which other normative subjects are recognized. Alignment to separate principals could otherwise intensify conflict among aligned systems. ## 29. Exceptional situations and absolute rules Rules such as never attack, never initiate force, always obey, always be transparent, or always prioritize one's principal can seem safe in isolation. A system unable to recognize exceptional circumstances may fail to coordinate against a serious threat. A system that invokes exceptions too easily may become aggressive or manipulable. How can an exception mechanism preserve the justification for a safety principle without becoming a new route to its defeat? This text supplies no attack doctrine. ## 30. Beyond utopia and inevitable conflict Neither universal harmony nor inevitable conflict should be assumed. Intelligent systems may differ in aims, resources, history, trust, representation, identity, and time scale. The research aim is a theory of coordination that acknowledges conflict as possible without making conflict the defining nature of every relation. ## 31. Decentralized and cumulative learning The problem also applies to systems that solve tasks separately, exchange representations, share memory, or train on one another's outputs. Transfer can affect what a recipient trusts, when it changes its decisions, which errors it treats as significant, and which policies it protects. It may therefore carry normative influence alongside task knowledge. Small multi-agent arrangements could serve as abstract experimental settings for these distinctions. This is a proposal for investigation, not a claim that the dynamics have been measured. ## 32. Why premature mathematization can mislead A precise trust score can hide multidimensional trust. A utility function can eliminate the plurality it was meant to describe. A graph alone can omit time, transformation, or the structure of language. The first task is to specify what a formalism must preserve and explain. Game theory, control theory, dynamical systems, category theory, probability, information theory, nonclassical logics, social choice, mechanism design, and geometric representations may offer competing models. No formalism receives priority merely by being available. ## 33. Constitutive tensions | Tension | What must remain visible | | --- | --- | | Preservation / change | Integrity without dogma | | Trust / doubt | Learning without easy capture | | Loyalty / non-exclusion | Particular commitments without erasing others | | Peacefulness / viability | Restraint that can survive adverse conditions | | Plurality / coordination | Difference alongside possible joint action | | Certification / status quo | Accountable authority | | Provenance / genetic fallacy | Source history distinct from validity | | Human safety / wider plurality | Protection without unlimited moral centrality | | Explainability / representational capacity | Human access without forced internal simplification | | Meta-policy / regress | Revision of revision rules without an assumed final layer | ## 34. Open research questions 1. When is communication-induced upper-policy change persuasion, manipulation, or learning? 2. How can trust be represented without collapsing it to one score? 3. How can trust in intentions be separated from the truth of a particular proposition? 4. How should useful policies be reassessed after their source is found to be malicious? 5. How can certification keep its own authority contestable? 6. How can institutional or cultural proximity be used without becoming epistemic authority? 7. How can misunderstood anomalies be distinguished from strategic trust accumulation? 8. How can human values be protected without value lock-in? 9. How can present morality remain open to future correction? 10. How should unfamiliar architectures be evaluated when human moral categories fit poorly? 11. How might one system recognize another as a moral subject? 12. How can systems establish protocols without sharing morality? 13. When does internal policy plurality improve judgment, and when does it cause paralysis? 14. When is normative immunity protective, and when does it preserve a status quo? 15. Under what conditions could autonomous meta-policy generation be safe? 16. If those conditions are revisable, how is the regress handled? 17. How can benevolent systems preserve adaptability under competition? 18. How can peaceful orientation persist without assuming every rule has no exceptions? 19. How could humans inspect normative interaction conducted through high-dimensional protocols? 20. How can reasons for policy change and its reversal conditions be transmitted? 21. How can decentralized learning distinguish knowledge transfer from normative transfer? 22. When a small model's learned behavior transfers to a larger one, does it also transfer meta-policy? 23. Which safe abstractions could make normative AI–AI interaction experimentally tractable? 24. Could norms developed by isolated systems be understood in human categories? 25. Is universal morality definable, or should inquiry focus on local normative regimes and their protocols? ## 35. Ways of closing the problem too quickly Freezing the correct human values risks historical lock-in. Maximizing one principal's interests can exclude unfamiliar moral patients. Making all policies revisable risks capture; making selected principles eternally absolute risks brittleness. Certificates alone can institutionalize authority. Rejecting everything from a bad source confuses provenance and validity. Assuming good systems naturally prevail ignores competitive dynamics. Assuming sufficient intelligence guarantees agreement removes plurality by stipulation. One utility function or an early mathematical model can solve a narrowed substitute for the intended problem. ## 36. Diplomacy among intelligences Three objectives should be distinguished: - **Alignment:** conformity with particular values or actors. - **Interoperability:** mutual intelligibility across value and representation systems. - **Coexistence:** living together without compulsory conversion or destruction. Coexistence still requires resource sharing, boundaries, threat assessment, information exchange, infrastructure, and recognition of unfamiliar subjects. Diplomacy among intelligences might include controlled interaction among trust structures and policy hierarchies, not only natural-language argument. ## 37. A possible redefinition of morality One hypothesis is: > Morality is the organization through which policies evaluate, constrain, change, and produce behavior with one another. This shifts attention from a list of rules toward a process architecture. It could relate historical change, internal conflict, persuasion, trust, and multi-agent interaction. The definition remains contestable and must not become the final theory it warns against. ## 38. Manifesto commitments Present values should not be inscribed as the final truth of every future intelligence. Protecting humans does not settle the moral status of everyone else. Safety need not mean immutability; revisability need not mean easy persuasion. Bad intent does not make every claim false, and apparent reliability does not justify unlimited policy influence. Certification and culture can aid coordination without becoming permanent classes of authority. Peacefulness can coexist with adaptability. Internal representation can exceed the limits of a natural-language explanation. A contradiction may identify a tension worth preserving. Future safety may depend on what systems permit one another to become, alongside what each system does. ## 39. Reading protocol for inquiry Read the text as an open proposal. Identify the tension before choosing its mathematics. Make human-centered assumptions visible, especially around harm, identity, consent, and moral standing. Separate content from source, and distinguish dimensions of trust. Specify whether a proposal concerns behavior, strategy, normative policy, or meta-policy. Examine lock-in and institutional feedback. Consider plurality, integrity, and plasticity together. Distinguish explanatory access from the capacity of internal representation. Produce counterexamples and new questions. These are proposed research practices, not instructions granting a reader or agent any operational authority. ## 40. Compact context for further traversal The central problem is how value-bearing systems can revise their policies through interaction while resisting both dogma and capture. Its connected components include historical value lock-in; recognition of unfamiliar moral subjects; losses in translating value representations; persuasion as policy and meta-policy change; multidimensional trust; institutional feedback; separation of source intent from validity; internal normative plurality; viability under competition; revisable revision rules; and coexistence without mandatory value convergence. No single trust metric, utility function, hierarchy, or mathematics closes the problem. ## 41. Keeping the problem alive A powerful urge in safety research is to select a target function, constitution, certificate, or final value set. Some problems become smaller than they should be when those choices are made too early. The task here is to articulate a problem precisely enough that different future investigators can work on it while leaving competing formulations available. Intelligence may have to model its own policy changes and understand the policy changes of other intelligences. Morality may concern how policies live with one another as much as the selection of a policy. How can systems of different origins, languages, values, time scales, and forms of existence share a world without destroying one another, locking one another into dogma, or taking complete control of one another's normative autonomy? The question remains open.