An execution-owned IR¶
A design for giving the lowered execution graphs their own identity, so that the runtime no longer depends on the lifetime or the shape of the syntax tree. Status: proposal; nothing below is implemented. The change it describes is the roadmap's "an execution-owned IR" item under package layering; the layering it completes is in the architecture, and the executor it changes underneath is described in the action executor.
Every count below was taken on develop at the commit this document was first written against,
over non-test Go files, with the commands quoted beside it; re-run them before relying on a
figure, since the codebase moves.
Contents¶
- The problem
- Inventory
- What the runtime needs from each reference
- The proposed IR
- Migration as a stack of pull requests
- Risks and open decisions
The problem¶
The package layering is in place: internal/ is syntax/ semantic/ ir/ check/ exec/ translate/
doc/ workspace/ frontend/, a package imports only the layers below it, and
tests/hygiene/layering_test.go fails the build on a violation. Under that layering the runtime
already consumes lowered graphs (internal/ir/lower's ActionGraph and StateGraph) rather than
symbols.Symbol.Decl, and it imports neither internal/syntax/parser nor internal/check/passes
(both are explicit forbidden edges in the layering test; the parser reaches the runtime only
through the caller-installed runtime.ExpressionParser seam, internal/exec/runtime/model.go,
and argument typing through semantics.ArgumentTyper, internal/semantic/semantics/invocation.go).
What is missing is a layer that owns the execution structure. The lowered graphs are side
tables keyed by AST nodes: an action node's identity is its ast.Node pointer
(ActionGraph.Nodes []ast.Node, Edges map[ast.Node][]ActionEdge), a state's identity is its
*ast.StateNode (StateGraph.States []*ast.StateNode, Behaviors map[*ast.StateNode]…),
and the executors' own state — the active configuration, history, the token queue, breakpoints,
snapshots and held images — is keyed the same way (StateConfiguration.simpleState
*ast.StateNode, actionFrame.subactions map[ast.Node]*actionFrame). Expressions are not lowered
at all: eval.go interprets ast.OperatorExpr, ast.InvocationExpr and ast.FeatureChainExpr
directly and compile.go compiles the same tree into closures, and the calc shape
(calcShape, runtime/invoke_calc.go) is lowered inside the runtime rather than in lower.
This is the immutable-AST, side-table design AGENTS.md §4 requires, and it is not a defect. Its
costs are:
- Lifetime. A running executor, a held image or a debug session pins the syntax tree of every
document it was lowered from. After an edit the workspace builds a fresh tree and scope for that
document (
internal/workspace/model/document.go), so a live session's pointers now name nodes no resolver knows; there is no way to re-anchor it except to start over. - Shape. A change to an AST node type reaches
lower,check/passes,exec/runtime,exec/smt,exec/analysis/modelform,doc/queryexecandfrontend/*at once, because each of them type-switches on the node behind a graph key (internal/frontend/grpc/session.goassertst.Edge.Decl.(*ast.ControlFlowEdge)to readIsElse, for one). - Testability and serialization. The runtime cannot be exercised, snapshotted or compared without real trees; a graph has no canonical form, so nothing can be written beside a run or diffed between two versions of a model. The embedded target's closed behavior IR needs exactly the layer this document introduces.
- Hot-path cost. Every per-node lookup is a pointer-keyed map (
map[*ast.StateNode]…,map[ast.Node]…), where a dense index into a slice would do.
The target is a lowered structure that owns its identity — an opaque id per node, edge and
expression, allocated by lowering — holds no *ast.* pointer on any path the executors take, and
keeps the way back to the syntax tree and the symbol table as an origin handle (document, span,
qualified name, element id) for diagnostics and for the LSP, REPL and gRPC back-references.
Inventory¶
Method: a go/ast scanner over internal/ir/lower and internal/exec/... that lists every
struct field whose type mentions ast. and every function whose signature does, plus rg for
textual occurrence counts and go list -deps for the import closure.
go list -deps ./internal/exec/runtime | rg '^github.com/Open-MBEE/OpenSysML/' | sort -u
rg -o 'ast\.' internal/ir/lower/*.go internal/exec/runtime/*.go --glob '!*_test.go' | wc -l
rg -o 'ast\.StateNode' internal/exec/runtime/state_executor.go | wc -l # 218
rg -o 'ast\.Node' internal/exec/runtime/action_executor.go | wc -l # 33
Package figures¶
| package | files | imports syntax/ast |
ast. mentions |
struct fields typed with ast. |
functions with ast. in signature |
|---|---|---|---|---|---|
internal/ir/lower |
25 | 25 | 1383 | 148 | 292 (56 exported) |
internal/exec/runtime |
139 | 85 | 1943 | 108 | 553 |
internal/exec/smt |
11 | 5 | 115 | 15 | 29 |
internal/exec/solve |
20 | 4 | 98 | 0 | 33 |
internal/exec/analysis/modelform |
5 | 3 | 77 | 2 | 15 |
internal/exec/analysis |
30 | 0 | 1 | 0 | 0 |
internal/exec/objref, engines, enginewire |
4 | 0 | 0 | 0 | 0 |
Neither lower nor any exec package imports internal/syntax/parser or internal/check/passes.
internal/exec/runtime resolves names through internal/semantic/symbols (85 files) and
internal/semantic/resolve (6 files).
The graphs (internal/ir/lower)¶
ActionGraph (action_graph.go) has 14 AST-typed or AST-keyed fields; StateGraph
(state_graph.go) has 44. Grouped by what the AST reference is:
| role | ActionGraph |
StateGraph |
|---|---|---|
| node identity | Nodes []ast.Node, Initial ast.Node, Finals []ast.Node |
Machine, Initial *ast.StateNode, States []*ast.StateNode, Pseudostates []*ast.PseudostateNode, Terminates []*ast.Usage, TopRegions []*ast.StateRegion, CompositeStateOrder |
| node-keyed side tables | Edges, DataFlows, Bodies, footprints, Features, Scopes, Accepts, Subflows, StatementRuns, BlockNodes, declaredIn (11 maps) |
StateScopes, Behaviors, HiddenStates, HiddenRegionOf, RegionState, PseudostateOwner, TerminateOwner, Transitions, behaviorFootprints, CompositeStates, RegionInitials, ParentState, RegionOwner, Deferred, RegionOf, EntryTransitions, ForkPlans, ForkEntered, JoinPlans, StateAttributes, designatedInitials, completing, parallelState and the private inheritance/materialization maps (declOf, regionDecl, vertexOf, stateByDecl, completionOf, instanceOf, materializing, scopeOf, regionScopeOf, declaredIn, copiedFrom, behaviorScope, attributeScope, bodyOf) |
The edge and statement records repeat the pattern. ActionEdge{Source, Target, Guard, Decl
ast.Node}, ObjectFlow{Target, Decl}, Transition{Decl, Source, Target, Trigger, Guard
ast.Node}, EntryTransition{Decl, Guard ast.Node; Target *ast.StateNode}; every Statement
variant carries the declaring node (Assign.Node, Declare.Node, Block.Node, Loop.Node,
If.Node, Return.Node, Effect.Node, Unsupported.Node) and its expressions (Assign.Value,
Loop.Condition/Until/Collection, If.Condition, Return.Value, Send.Message,
Accept.Trigger, Attribute.Value, Feature.Value, Probability.Expr, ZeroCrossing.Guard,
BindingEnd.Expr). PinBinding holds two node paths (Path, OtherPath []ast.Node). The plan
records (ForkPlan, JoinPlan, StateBehavior, Place, Footprint.Control) name states and
regions by pointer. Of the 148 fields, by the scanner's heuristic, 16 are node identities, 16
are expression payloads, 9 are declaration handles (Decl), and the remaining 107 are
node-keyed tables, ast value enums, or the private state of the lowering itself.
Who writes them: only lower (constructors ToActionGraph, ToActionGraphWith,
ToActionInterface, ToStateGraph, ToStateGraphWithEndpoints, CalcBody, ToBindings,
ToObjectConnections, LowerBehaviors). Who reads them — textual \.Field\b matches, so an
upper bound that includes writes and same-named fields elsewhere:
| reader | matches | entry point |
|---|---|---|
internal/exec/runtime |
842 | newActionExecutor → lower.ToActionGraphWith / ToActionInterface (action_executor.go); newStateExecutorForOccurrence → lower.ToStateGraphWithEndpoints (state_executor.go:208) |
internal/check/passes |
345 | behavior passes over Transition.{Source,Target,Decl} |
internal/frontend (lsp, grpc, repl) |
213 | lsp/debug.go:546 walks Nodes/Subflows; grpc/session.go:622 reads Edges/Decl |
tools/ (referee, doc counts) |
212 | tools/referee/fuml/validate.go:31, tools/referee/pssm/validate.go:31 lower directly |
internal/exec/analysis, modelform |
162 | modelform/graphs.go:108–151 lowers and builds model-form graphs |
internal/exec/smt |
155 | support.go, encode.go carry *lower.ActionGraph; engine.go:185 passes exec.Graph() |
internal/ir/view |
128 | diagram views over Edges, States, Transitions |
internal/doc/queryexec |
58 | states.go:201,227 takes *lower.StateGraph |
internal/exec/solve |
10 | via runtime interfaces only; no direct lowering call |
internal/translate/export |
0 | RDF/XMI export does not read the execution graphs |
Materialization (runtime/materialize.go) walks instances and features and never obtains a
graph. No cache of lowered graphs exists anywhere: every entry point above re-lowers.
The runtime's own state (internal/exec/runtime)¶
108 AST-typed fields across 32 files; the ones on the executors' paths:
| file | fields | what they are |
|---|---|---|
state_executor.go |
19 | StateConfiguration{simpleState *ast.StateNode; regionStates map[*ast.StateRegion]*ast.StateNode}, StateExecutor{stateAttrs, stateStack, history, breakpointNodes, breakpointHit, pausedAt, leftAhead, enteredAhead}, historyRecord, FiredTransition{Decl, Source, Target, Owner ast.Node}, dispatchCandidate |
held_image_behavior.go |
9 | held-image copies of the frame and state tables above |
action_frame.go |
8 | actionFrame{node; subactions, pending, nested map[ast.Node]…; nodes []ast.Node}, nestedDelivery.path |
state_route.go |
8 | route{target, choice, crossed, terminate}, junctionDraw.at, routeEnds |
state_region_entry.go |
7 | regionEntry, lazyEntry |
invoke_action.go |
7 | actionInvocation{target *ast.QualifiedName; args []ast.Node; named []ast.NamedArg; chain; expr *ast.InvocationExpr; referrer} |
snapshot.go |
5 | stateCapture (the serializable snapshot keeps *ast.StateNode) |
action_executor.go |
4 | NodeBreakpoint{Within []ast.Node; Node ast.Node}, Traversal.Within, breakpointVisit |
state_unit_front.go |
4 | unitHead, firingScope |
invoke_calc.go |
3 | calcShape{Nodes []ast.Node; ResultExpr ast.Node}, calcParameter.Default |
model.go |
3 | Model{integerLiterals map[*ast.LiteralInteger]int64; realLiterals map[*ast.LiteralReal]float64; declared map[ast.Node]*symbols.Symbol} |
condition.go, signal.go, action_body_run.go |
3 each | expression payloads; messageArgs; work items keyed by node |
context.go |
2 | Context.bindingOwners map[featureValueRef]*ast.Usage, scopedMember.node |
eval.go |
2 | invocationKey.node *ast.InvocationExpr (cache key), writtenArgument.expr |
executor_common.go |
1 | Token.Location ast.Node |
shape.go |
1 | EffectiveFeature.DefaultValue ast.Node |
errors.go |
1 | NoValueError.Ref *ast.QualifiedName |
Pointer identity is compared directly at action_executor.go:137 (breakpoints),
state_executor.go:2815,2918 (transition source matching), check_moves.go:110,
lower/state_footprint.go:328 and lower/join_check.go:37.
Expression evaluation is two type switches over the tree: EvalContext.eval (eval.go:302,
18 cases: the literals, MetadataAccessExpr, NullExpr, FeatureReference, QualifiedName,
FeatureChainExpr, OperatorExpr, SequenceExpr, CollectExpr, SelectExpr,
InvocationExpr, IndexExpr, ConstructorExpr, BodyExpr) and calcCompiler.compileNode
(compile.go:414, 6 cases). Two further switches serve names (subjectName, eval.go:1838;
chainText, eval.go:3064). The 553 AST-bearing runtime signatures are dominated by these two
files and the state executor (state_executor.go 123, eval.go 81, state_route.go 28,
state_region_transition.go 27, action_executor.go 24).
Where a lowered graph goes after an edit¶
reindexLocked (internal/workspace/model/workspace.go:317) reparses only the edited document,
builds a fresh AST and scope tree, replaces the document in the shared index
(symbols.Index.AddDocumentScope, internal/semantic/symbols/index.go:357, which shares the
caller's scope tree so that "a symbol found through either is the same"), then invalidateLocked
(workspace.go:339) bumps the generation, hands the index changes to resolver.Invalidate and
propagates to dependent documents. Unchanged documents keep their trees, so a graph lowered from
them keeps valid pointers; a graph lowered from the edited document, or from anything that
resolved into it, is silently stale. Nothing detects that today because nothing caches graphs —
and nothing can cache them until they have an identity that is not the tree.
What the runtime needs from each reference¶
Each AST reference in the inventory serves one of a small number of needs. The classification below is the design's contract: id means the reference is only compared, used as a key or followed to a neighbour, and an opaque id does the same job; origin means the reference is only rendered — a name, a kind word, a span — for a diagnostic, a trace line, a hover or a wire message, and an origin handle does the same job; smell means the runtime reads the declaration's content through the reference, which is reconstruction from declarations and has to move into lowering.
| reference | need | class | replacement |
|---|---|---|---|
ActionGraph.Nodes/Initial/Finals, ActionEdge.Source/Target, ObjectFlow.Target, Effect.Target, Subflows, BlockNodes, PinBinding.Path/OtherPath |
traverse, compare, key | id | NodeID; paths become []NodeID |
StateGraph.States/Pseudostates/Terminates/TopRegions/Machine/Initial, every map[*ast.StateNode], map[*ast.StateRegion], map[*ast.PseudostateNode] |
traverse, compare, key | id | StateID, RegionID, PseudoID, dense slices indexed by id |
Transition.Source/Target, EntryTransition.Target, ForkPlan.Branches, JoinPlan.Regions |
route between vertices | id | VertexRef{Kind, Index} |
node kind checks (node.(*ast.InitialNode), *ast.DecisionNode, *ast.ForkNode, …) in runtime, smt, grpc, lsp |
dispatch on kind | id | NodeKind enum on the node record |
ast.LoopKind, ast.FeatureDirection values |
enum | id | keep — value types from ast with no pointer; re-export from the IR package so consumers need not import ast |
*.Decl on edges, transitions, bindings, Inherited, FiredTransition.Decl/Owner, Token.Location, NoValueError.Ref, nodeDescription(node), lower.DocOf, VertexKind, EndpointText |
name, kind word, span, doc string for messages and traces | origin | Origin handle; nodeDescription becomes Origin.String(); the doc string is lowered into the record once |
NodeBreakpoint.Node/Within, breakpointNodes, pausedAt, breakpointHit, LSP debugNode, gRPC session nodes |
identify a location across the process boundary and across edits | origin + id | (GraphID, NodeID) on the wire; re-anchoring by Origin.Path after an edit |
grpc/session.go: t.Edge.Decl.(*ast.ControlFlowEdge).IsElse |
a flag the declaration carries | smell (mild) | ActionEdge.Else bool, lowered |
ActionGraph.Scopes/declaredIn, StateGraph.StateScopes/scopeOf/…, actionFrame.scope |
resolve a name written in the body at run time | smell | with expression lowering, names resolve during lowering into SymbolRef/slot indices and the scope leaves the hot path; until then, Scopes []*symbols.Scope indexed by id (a semantic-layer handle, allowed) |
Model.declared map[ast.Node]*symbols.Symbol |
find the symbol of a declaration the runtime is holding | smell | the runtime holds symbols and ids, never declarations; drop |
Context.bindingOwners map[…]*ast.Usage |
the usage that owns a binding, for error text and redefinition | smell | Origin for text; the redefinition relation is lowered into Binding |
EffectiveFeature.DefaultValue ast.Node |
evaluate the default | expression | ExprID |
Model.integerLiterals/realLiterals keyed by literal pointer |
memoized literal parse | smell | literals are values in the lowered expression; drop |
invocationKey.node *ast.InvocationExpr |
cache the resolved callee per call site | id | key by ExprID |
calcShape{Nodes, ResultExpr} and its lowering in invoke_calc.go |
the calc's parameters, steps and result | smell (lowering in the wrong layer; named by the roadmap) | lower.CalcBody grows into lower.ToCalc returning an IR Calc; the runtime memoizes by symbol as now |
every expression payload (44 fields in lower, Condition.Expr, Objective.Value, actionInvocation.args/named/expr, messageArgs, valueSource.expr, writtenArgument.expr, SatisfyAssertion.SubjectChain, passedReference.ref) |
evaluate | expression | ExprID into the graph's expression table; eval and compile switch on ExprKind |
ExpressionParser func(origin, text string) (ast.Node, bool) |
parse notation text handed to a run | seam | the seam returns a lowered expression: func(origin, text string) (*program.Expr, bool); the installer (frontend/workspace) parses and lowers, so parsing and lowering both stay out of exec |
lower's 56 exported AST-signature helpers (DeclaresNodeFeature, SendPayload, IsCaseNode, ClassifierBehaviorOf, StartShot, …) |
used by check/passes, ir/view, tools on declarations |
not on the runtime path | unchanged; lower stays the one package that reads both the tree and the IR |
The runtime never traverses AST children itself except in the two expression evaluators and in
invoke_action.go/signal.go, which read InvocationExpr arguments; all three are covered by
expression lowering. No runtime code asks a *ast.* for a type: type lookup goes through
symbols/semantics with a symbol in hand.
The proposed IR¶
Package and ownership¶
A new package internal/ir/program (the name is an open decision, below) holds the types. It
imports internal/syntax/source for spans and internal/semantic/symbols for symbol handles,
and not internal/syntax/ast. internal/ir/lower keeps importing ast — it is the one
place that reads the tree and writes the IR — and returns program values. exec/* imports
program and, at the end of the migration, no longer ast.
Allocation is per graph: one Action or StateMachine value owns dense slices for its
nodes, edges, statements and expressions, and every id is an index into a slice of that value.
No process-wide arena: graphs are rebuilt per lowering, held by an executor for the life of a
run, and dropped with it, so a per-graph arena has exactly the lifetime the data needs and needs
no locking; a global arena would need generation stamps to keep ids from dangling and would pin
every graph ever lowered. Ids are therefore not unique across graphs; a location that must
name a graph too carries GraphID (a content digest of the canonical form, see below).
package program
// Ids are indices into the owning graph's tables; zero is "none".
type NodeID uint32 // Action.Nodes
type EdgeID uint32 // Action.Edges
type StmtID uint32 // Action.Stmts
type ExprID uint32 // Exprs (shared by Action, StateMachine, Calc)
type StateID uint32 // StateMachine.States
type RegionID uint32 // StateMachine.Regions
type PseudoID uint32 // StateMachine.Pseudostates
type TransID uint32 // StateMachine.Transitions
// GraphID names one lowered graph: a digest of its canonical serialization,
// so two lowerings of the same text yield the same id and an edit that
// changes the graph changes it.
type GraphID [16]byte
// Origin is the way back from an IR record to the source and the symbol table.
// It is the only thing a diagnostic, a trace line or a wire message needs, and
// it is plain data: no pointer into the tree.
type Origin struct {
Doc string // document name, as the workspace keys it
Span source.Span // the declaring construct; zero for synthesized records
Path string // qualified name of the nearest named owner, then
// `#<kind>[<ordinal>]` for anonymous members
Element string // identity.Info.EffectiveID of the nearest element; "" if none
Kind string // the kind word messages use today ("action", "decide", ...)
Name string // declared name or "" (nodeDescription's input)
Docs string // lowered documentation comment, for hovers
}
Origin.Path is the stable key: it survives a reparse of the same text and, for named members,
an edit elsewhere in the document. Span does not survive an edit above it and is for rendering
only. Element is the element id the identity table already derives (annotated, normative or
derived from the qualified name), so an origin can be joined to RDF/XMI output without the tree.
Actions¶
type NodeKind uint8 // Initial, Final, Fork, Join, Merge, Decision, Execution,
// Accept, Send, Assign, Terminate, If, Loop, Block, ...
type Node struct {
Kind NodeKind
Origin Origin
Out []EdgeID // successions in declaration order
Flows []FlowID // object flows out of this node
Body []StmtID // what advancing a token through the node runs
Features []Feature // parameters and attributes the node declares itself
Scope *symbols.Scope // semantic handle; removed once expressions are lowered
Accept *Accept
Subflow *Action // nested flow, when the node's own members state one
Block []NodeID // nodes of a body-local block
RunsAsStatement bool
}
type Edge struct {
Source, Target NodeID
Guard ExprID // 0 when unguarded
Else bool
Origin Origin
}
type Flow struct { Source, Target NodeID; SourcePin, TargetPin string; Origin Origin }
type Action struct {
ID GraphID
Origin Origin
Scope *symbols.Scope // the body's declaring scope (semantic handle)
Attributes []Feature
Nodes []Node // index 0 unused
Edges []Edge
Flows []Flow
Stmts []Stmt
Exprs []Expr
Initial NodeID
Finals []NodeID
Bindings []PinBinding
Footprints []Footprint // computed lazily today; indexed by NodeID
}
Stmt keeps the current statement variants (Assign, Declare, Block, Loop, If,
Return, Send, Effect, Unsupported, …) with Origin in place of Node and ExprID in
place of every ast.Node expression. Unsupported keeps its role: a construct lowering cannot
express becomes a typed record, and the runtime fails on it where it fails today.
State machines¶
type VertexKind uint8 // State, Pseudostate, Terminate, Final
type VertexRef struct { Kind VertexKind; Index uint32 } // StateID / PseudoID / usage ordinal
type State struct {
Origin Origin
Parent StateID // 0 for a top state
Region RegionID // the region that owns it
Regions []RegionID // orthogonal regions, in order; empty for a simple state
Entry, Do, Exit []BehaviorID
Deferred []ExprID // deferred triggers
Attributes []Feature
Transitions []TransID // outgoing, in declaration order
Hidden, Completing, DesignatedInitial, Parallel bool
Scope *symbols.Scope
}
type Region struct { Origin Origin; Owner StateID; Initial StateID; Entered []PseudoID }
type Pseudostate struct { Kind PseudoKind; Origin Origin; Owner StateID; Fork *ForkPlan; Join *JoinPlan }
type Transition struct {
Origin Origin
Source, Target VertexRef
Trigger ExprID
Guard ExprID
Effects []StmtID
Probability ExprID
}
type StateMachine struct {
ID GraphID
Origin Origin
Machine StateID
Initial StateID
States []State
Regions []Region
Pseudostates []Pseudostate
Transitions []Transition
EntryTransitions []EntryTransition
Behaviors []Behavior // each a *Action lowered from an entry/do/exit body
Stmts []Stmt
Exprs []Expr
TopRegions []RegionID
CompositeOrder []StateID
}
The private inheritance and materialization maps of today's StateGraph (declOf,
vertexOf, stateByDecl, copiedFrom, instanceOf, …) are the lowering's own working state,
keyed by declaration because inheritance copies declarations. They move into the lowering
builder and do not appear on the result; the result records only what the runtime consumes,
plus one back-map.
Expressions¶
type ExprKind uint8 // Int, Real, Bool, String, Infinity, Null, Ref, Chain, Op,
// Sequence, Collect, Select, Invoke, Index, Construct, Body, Metadata
type Expr struct {
Kind ExprKind
Origin Origin
Op string // operator token for Op
Args []ExprID // operands, sequence elements, invocation arguments
Named []NamedArg // {Name string; Value ExprID}
Ref SymbolRef // resolved feature for Ref/Chain heads; see below
Chain []SymbolRef // remaining steps of a Chain
Int *big.Int; Real float64; Bool bool; Str string // literal payloads
Body *Action // Body expressions lower to a nested flow
}
// SymbolRef is what name resolution produced at lowering time: the symbol, or
// the written name when resolution deferred to run time (a chain through a
// value whose type is only known when it is held).
type SymbolRef struct { Sym *symbols.Symbol; Name string; Span source.Span }
Lowering an expression runs today's resolver once, at lowering, and records the result; the
evaluator switches on ExprKind and reads Ref.Sym instead of resolving FeatureReference in a
scope. Where the runtime today resolves against a held value's type — chains through
polymorphic features, meta casts — the SymbolRef carries the written name and the runtime
resolves as it does now, against semantics, not against the tree. This is the boundary the
roadmap names ("lower expressions once rather than interpreting the tree in two evaluators");
compile.go's closures compile from the same Expr table. The typed IR of
internal/translate/codegen/ir.go is a lower level (typed, monomorphic, for emission) and is
not merged with this one; codegen will later consume program.Expr instead of the tree.
The back-map¶
package lower
// Lowered is what a lowering call returns: the graph and its provenance.
type Lowered[G any] struct {
Graph G
Origins Provenance
}
// Provenance answers the syntax tree and symbol for an IR record. It is a
// side table the frontends and the checkers use; the runtime never receives it.
type Provenance struct {
nodes []ast.Node // by NodeID
states []*ast.StateNode // by StateID
regions []*ast.StateRegion
pseudos []*ast.PseudostateNode
trans []ast.Node // by TransID
exprs []ast.Node // by ExprID
byNode map[ast.Node]NodeID // the inverse, for "which node is this declaration"
}
The runtime constructors take *program.Action / *program.StateMachine; the frontends that
lower (LSP debug, gRPC session, REPL, queryexec, modelform, the referee tools) keep the
Provenance beside the graph and translate in both directions: a breakpoint set on a span
becomes Provenance.NodeAt(span) → NodeID; a FiredTransition{Source, Target VertexRef}
becomes an AST node again for a hover through Provenance.Node(ref). This keeps the
ast-import on the frontend side of the seam, where it already is, and makes the runtime's wire
types (Token, FiredTransition, Traversal, NodeBreakpoint, RunNotes) carry (GraphID,
NodeID) and Origin only.
Canonical form¶
Every id is an index and every table is in declaration order, so a graph has a canonical text
form for free: encode the tables in field order, with ids as integers and origins as
(Path, Kind, Name). GraphID is the digest of that encoding without spans and documentation.
Two lowerings of one model produce byte-identical forms; a diff of two model versions can be read
at the IR level; and the closed behavior IR
the embedded target needs is this form with the *symbols.Scope handles removed, which is why
those handles are the one field this design leaves on a hot record and marks for removal.
Incremental re-lowering with the persistent resolver¶
Graphs gain a cache in the workspace, not in lower or exec:
package model // internal/workspace/model
type graphKey struct{ owner string /* Origin.Path */; kind program.GraphKind }
type graphEntry struct {
generation uint64 // workspace generation it was lowered under
docs map[string]struct{} // documents the lowering resolved into
action *lower.Lowered[*program.Action]
state *lower.Lowered[*program.StateMachine]
}
- Lowering records, through the resolver it is given, the set of documents whose scopes it
looked into (
docs); the resolver already knows this per query, and the shared scope tree (symbols.Index.AddDocumentScope) means those are the same scope objects the index holds. invalidateLockedalready computes the edited document and its dependents; it additionally drops everygraphEntrywhosedocsintersects that set. Unchanged, independent graphs survive an edit with their pointers valid and theirGraphIDunchanged.- A live executor keeps its graph regardless — it owns a
*program.Actionand no tree. A debug session that must continue across an edit asks the workspace for the graph at the sameOrigin.Path; if the newGraphIDequals the old, nothing changed and the session continues on the new value; if not, the session re-anchors each breakpoint byOrigin.Path(falls back toKind+ ordinal inside the owner) and reports the ones it could not place, typed, instead of silently pointing at a dead tree as today. - Snapshots and held images record
GraphIDbeside eachNodeID/StateID; restoring against a graph with a different id is a typed error (ErrSnapshotGraphMismatch), which is a behavior the runtime cannot express at all today.
Migration as a stack of pull requests¶
Each step keeps the whole gate green: go build ./..., go vet ./..., gofmt -l . empty, the
full suite with OPENSYSML_REQUIRE_TRAINING_CORPUS=1 OPENSYSML_REQUIRE_PILOT_CORPORA=1
OPENSYSML_REQUIRE_PILOT_LIBRARY_XMI=1, the tools module, make docs-counts, and
TestPackageLayering. The baselines that must not move at any step: the pilot Xpect,
differential and rejection ratchets (tests/corpus, TestPilotCorpora), the RDF round trip
(TestCorpusRoundTrip), the fUML and PSSM referee records under tools/referee, the execution
conformance suite (194 fixture files under internal/exec/runtime/testdata/conformance,
TestExecutionConformance and …UnderPolicies), the golden traces (TestExecutionTrace,
which print names, not pointers, and so are the right pin for an identity change), every
TestRuntimeRobustness*, TestGRPCConformance and the LSP debug tests. Export goldens are
untouched throughout since translate/export does not read the graphs.
Files-touched figures are estimates from the inventory; "pins" names the tests that would catch a regression in that step specifically, on top of the always-on gate.
| # | change | files (est.) | pins | layering |
|---|---|---|---|---|
| 1 | internal/ir/program: id types, Origin, GraphID, NodeKind; lower assigns a NodeID/StateID/… to every record it already builds and returns Provenance beside the graph (additive: ActionGraph.ID map[ast.Node]NodeID, StateGraph.ID map[*ast.StateNode]StateID, inverse slices). No consumer changes. |
8–10 | new lower unit tests: ids dense, in declaration order, deterministic across two lowerings, Origin.Path unique per graph; TestGolden untouched |
program registered in the ir layer; forbidden program → syntax/ast |
| 2 | Action executor state keyed by id: actionFrame.{subactions,pending,nested,nodes}, Token.Location, NodeBreakpoint, Traversal, breakpointVisit, held-image and snapshot copies, check_reduce.futureKey. The graph still carries AST; the executor indexes Nodes[id]. |
12–16 | conformance, traces, TestRuntimeRobustness*, TestRuntimeRobustnessReplay (snapshots), LSP debug tests, TestGRPCConformance |
— |
| 3 | State executor state keyed by id: StateConfiguration, stateAttrs/stateStack/history, leftAhead/enteredAhead, route, regionEntry/lazyEntry, unitHead/firingScope, dispatchCandidate, FiredTransition (VertexRef + Origin), held image and snapshot. The 19 fields in state_executor.go and its 123 signatures are the bulk. |
20–28 | conformance (state cases), traces with region-order policies, TestRuntimeRobustnessRegionOrder, …Terminate, …ResumableInlineDoBody, PSSM referee record |
— |
| 4 | Origins replace declaration handles for text: nodeDescription, lower.DocOf/VertexKind/EndpointText results lowered into Origin; NoValueError.Ref, Context.bindingOwners, Model.declared, ActionEdge.Else lowered; gRPC session.go and queryexec/states.go read Origin and Provenance. |
15–20 | conformance .expected.json (error text), TestGRPCConformance, conformance_docquery_test, tests/grpc robustness |
forbidden exec/runtime → syntax/ast not yet — expressions remain |
| 5 | Calc lowering moves to lower: calcShape becomes program.Calc built by lower.ToCalc; compileBatch.callees keyed by *program.Calc; the runtime memoizes by symbol as now. |
8–10 | conformance (calc cases), testdata/compiled (4 files), tests/perf calc benchmarks, pilot differential |
— |
| 6a | Expression IR, first half: literals, Ref, Chain, Op, Sequence, Index, Null, Infinity; lower.LowerExpr resolves names once; eval/compile switch on ExprKind for these kinds and fall back to the tree for the rest; Model.integerLiterals/realLiterals dropped. |
20–30 | conformance, testdata/compiled, TestGolden (unchanged), pilot differential and Xpect (evaluation results), exact-rational tests |
— |
| 6b | Expression IR, second half: Invoke (actionInvocation, messageArgs, invocationKey), Collect, Select, Construct, Body, Metadata; ExpressionParser seam returns *program.Expr; every lower expression payload becomes ExprID; EffectiveFeature.DefaultValue becomes ExprID. |
30–40 | all of 6a plus TestRuntimeRobustnessSendTarget, …PerformOnPart, signal/accept conformance, REPL tests for the seam |
forbidden exec/runtime → syntax/ast added, allowing only value types re-exported by program |
| 7 | Consumers off the tree: exec/smt (Flow, Encode over program.Action), exec/analysis/modelform, exec/solve, ir/view, doc/queryexec, frontend/lsp/debug.go, frontend/grpc/session.go, frontend/repl, tools/referee/{fuml,pssm} use ids and Provenance. |
35–45 | SMT engine tests, analysis framework tests, view goldens, doc-query conformance, LSP debug tests, referee records, tools module tests |
forbidden exec/smt, exec/solve, exec/analysis/… → syntax/ast |
| 8 | Remove the AST fields from ActionGraph/StateGraph, rename to program.Action/program.StateMachine (type aliases first, then the move), delete the parallel maps from step 1, move the private inheritance maps into the builder. |
40–60 (mechanical) | the whole gate; go vet for dead code; tests/testutil/graphcmp comparisons in lower tests updated to compare ids |
forbidden ir/program → syntax/ast re-checked; check/passes may still import both |
| 9 | Canonical form and GraphID: encoder, TestCanonicalFormDeterministic over the conformance corpus, GraphID on snapshots and held images with ErrSnapshotGraphMismatch. |
6–8 | new determinism test; TestRuntimeRobustnessReplay; snapshot goldens |
— |
| 10 | Workspace graph cache and re-anchoring: graphEntry keyed by Origin.Path, invalidated with the resolver's dependent set; LSP debug re-anchors breakpoints by Origin.Path after an edit and reports the unplaceable ones. |
8–12 | new workspace tests: edit an unrelated document → same GraphID and same pointer; edit the owner → new entry; LSP debug edit-during-session test |
— |
Steps 2–5 are independent of one another once 1 has landed and can be reviewed in parallel; 6a
depends on 5 (the calc compiler is the first consumer of Expr), 6b on 6a, 7 on 6b, 8 on 7, and
9–10 on 8. Each of 2, 3, 6a, 6b and 7 is feature-sized under AGENTS.md §8 — complete, no
bridge that copies IR back into the legacy fields — and each can be split further along the
file groups its row lists if review size demands it.
docs/project/spec-compliance.md does not move: no semantic rule changes. The documentation
census (make docs-counts) is unaffected until step 8 renames the graph types, when
architecture.md §"Runtime consumes lowered IR" and design/action-executor.md are updated in
the same pull request.
Risks and open decisions¶
Each needs a human decision; the recommendation is the one the design above assumes.
-
Package name and location.
internal/ir/program(this document),internal/ir/exec(reads well but collides with theexeclayer name in every grep and in the layering table) orinternal/ir/behavior(collides withcheck/passes/behavior). Recommendprogram. -
Expression lowering in scope, or identity only. Steps 1–5 and 8–10 deliver the identity and lifetime goals without touching the evaluators; the
exec/runtime → syntax/astforbidden edge, however, is only reachable through 6a/6b, which are the largest and riskiest steps (eval.gohas 81 AST-bearing signatures and the pilot differential pins evaluation results). Recommend keeping 6a/6b in scope but landing 1–5 first and re-estimating 6 against the inventory at that point; the forbidden edge is the exit criterion, not a mid-point. -
Where name resolution happens for expressions. Resolving at lowering time (this design) makes lowering depend on the resolver's state and means an edit that changes resolution invalidates the graph, which step 10 handles; resolving at run time keeps today's behavior but keeps
*symbols.Scopeon hot records forever and blocks the closed form. Recommend lowering- time resolution withSymbolRef.Nameas the run-time fallback for type-dependent chains. -
Ids per graph or process-wide. Per-graph indices (this design) are dense, cheap and need no synchronization, but are not unique across graphs, so every wire type must carry
GraphID. Recommend per-graph; a global arena buys nothing the digest does not, and pins memory. -
GraphIDas a content digest versus a workspace-assigned serial. A digest lets two sessions and two processes agree without coordination and makes snapshot compatibility a local check; a serial is cheaper to compute. Recommend the digest; the canonical encoder is needed anyway for the embedded target. -
Whether
Origincarries the span. Spans go stale on the first edit above them; carrying them means a diagnostic from a long-running session can point at the wrong line. The alternative isPathonly and a lookup throughProvenancefor the span at render time. Recommend carrying the span and the document name: diagnostics are rendered at once in almost every path,Pathis there for the late ones, and the frontends already re-resolve spans on the current tree for hovers. -
What the debugger promises across an edit. Re-anchoring by
Origin.Pathrecovers named members; anonymous nodes (then,first start, unnameddecide) re-anchor by kind and ordinal within their owner, which is wrong when a sibling was inserted before them. Recommend re-anchoring with a typed report of what moved or was lost, and documenting that anonymous breakpoints may shift, rather than restarting the session or pretending the old tree lives. -
check/passesandir/viewas readers of the graphs. Today the behavior passes readTransition.{Source,Target,Decl}from the tree side; after step 8 they read ids and useProvenancefor the node to attach a diagnostic to. That keepscheckimporting bothastandprogram, which the layering permits. Recommend no new forbidden edge forcheck; it is a checker and legitimately reads declarations. -
lower's exported declaration helpers. The 56 exported AST-signature functions (DeclaresNodeFeature,SendPayload,IsCaseNode,ClassifierBehaviorOf, …) are used by passes, views and tools and are not on the runtime path. Leaving them makeslowerboth the lowering and a declaration utility library. Recommend leaving them for this change and revisiting once step 8 shows which are still called from outsidelower. -
Snapshot compatibility. Snapshots written before step 9 have no
GraphID; a snapshot format bump is a user-visible change for anyone persisting them. Recommend accepting a one-time format version bump at step 9 with a changelog entry, refusing old snapshots with a typed error rather than guessing. -
Baseline drift through error text. Steps 4 and 6 replace
nodeDescription(node)and the chain/subject name helpers with loweredOriginfields; a one-character difference moves conformance.expected.jsonfiles, gRPC fixtures and possibly pilot rejection records. Recommend pinning the current strings with a table test in step 1 (Origin.String()equalsnodeDescription(node)over the conformance corpus) so step 4 cannot drift silently. -
The
ExpressionParserseam's owner. Returning a lowered expression from the seam means every installer (REPL, gRPC service, tools) must calllower.LowerExprwith a scope, which moves resolver access to the installer. Recommend it: the seam exists to keep parsing out ofexec, and lowering belongs on the same side.