Three time scales#

An autonomous drone makes decisions at three very different rates.

The control loop runs at hundreds of hertz. It stabilises attitude, tracks velocity setpoints, and reacts to disturbances. It is deterministic, well understood, and must never wait on anything slow.

The navigation layer runs at a few hertz. It fuses perception into a local map, plans collision-free paths to a waypoint, and issues setpoints to the control loop. It is real-time but tolerates some latency.

The mission layer runs on the scale of seconds to minutes. It decides what the drone should be trying to do: which area to survey next, whether an observed object warrants a closer look, when to abort and return. It is where the ambiguity lives.

A language model is slow, expensive, and non-deterministic. It belongs at exactly one of these levels, the mission layer, and the entire architecture follows from keeping it there.

What the LLM owns#

At the mission layer the LLM does three jobs.

Interpreting intent. A mission arrives as natural language: “survey the east field, prioritise anything that looks like standing water, come back before the battery drops below a safe margin.” The model turns this into a structured mission plan: an ordered set of goals, each with a success criterion and a priority.

Replanning on new information. Perception produces events the mission did not anticipate. An area is obstructed, an object of interest appears, weather changes. The model receives a compact description of the event and the current plan, and emits a revised plan. This is where language models earn their place: they handle open-ended situations that no finite state machine anticipated, and they do it with a plan that a human can read and check.

Explaining decisions. Every replanning step produces a rationale alongside the plan. In a research setting this is how we debug the system. In any deployed setting it is how an operator would decide whether to trust it.

What the LLM never touches#

The model never emits a velocity, an attitude, or a motor command. It never decides whether a path is collision-free. It never overrides a safety limit. These are the domain of the navigation and control layers, which are conventional, tested, and deterministic.

The interface between the mission layer and the layers below is a small, typed vocabulary of goals: fly to a waypoint, orbit a point at a radius, hold position, return to launch. The LLM composes missions out of this vocabulary and nothing else. If it emits something outside the vocabulary, the plan is rejected and the model is asked again. This is the same discipline as validating at the edges in an agent graph: constrain the output to a schema, check it before it acts, and fail locally.

Perception in, language out#

The mission layer needs to know what the drone sees, but feeding raw sensor data into a language model is both wasteful and unreliable. Instead, the perception stack produces symbolic summaries: detected objects with classes, positions, and confidences; a coarse occupancy description of the surroundings; battery and link state. The LLM reasons over this summary.

The consequence is that the perception stack, not the language model, determines the ceiling on what the system can respond to. If perception cannot represent standing water, no mission phrasing will make the drone find it. This is a feature, not a limitation. It keeps the question “what can this system perceive?” answerable, which a vision-language model reasoning over raw pixels would not.

Latency and the fallback plan#

A replanning call may take seconds. The drone cannot hover in indecision for that long, and it certainly cannot do so during an emergency. Two mechanisms address this.

First, the current plan always remains valid until a new one replaces it. The navigation layer keeps executing the last accepted goal while the model thinks. Second, a small set of safety behaviours is hard-wired below the mission layer and triggers on sensor state alone: low battery, lost link, geofence violation. These preempt the mission layer entirely. The language model can propose returning to launch; it cannot prevent it.

Simulation first#

All of this is developed in simulation, using ROS with Gazebo for the vehicle and environment and Open-RMF for scenarios involving more than one vehicle. Simulation is not just cheaper than flight testing; it is where the mission layer can be exercised against thousands of scenario variations to find the situations where the model produces an inappropriate plan. Those failures are then turned into validation rules at the interface, so that the same class of bad plan is rejected before it reaches the navigation layer in the future.

Open problems#

The interesting unsolved questions are about trust calibration. When should the system ask the operator rather than replan on its own? How should the model’s confidence in its interpretation of an ambiguous mission be surfaced? And, as the mission vocabulary grows to cover more capable behaviours, how do we keep the guarantee that every composition of vocabulary items is safe? Those are the questions the next phase of this work is aimed at.

The overall lesson so far is unglamorous. The language model is the least important component for making the drone fly and the most important component for making it useful. Keeping those two facts separate, architecturally, is what makes the system work.

Where the chain breaks#

A chain encodes three assumptions at once. It assumes the order of steps is known before execution starts. It assumes each step has exactly one successor. And it assumes control flows forward only, so that nothing upstream ever needs to be revisited.

Real tasks violate all three within minutes. A research agent discovers that the question was ambiguous and needs the planner to disambiguate it. A tool call fails and the right response is to try a different tool, not to pass the failure downstream. Two sub-questions are independent and could be answered in parallel, but the chain forces them to run in sequence. A reviewer rejects a draft, and the honest fix is to loop back to the writer with feedback rather than to bolt a second reviewer onto the end.

Framework authors know this, which is why chains grow escape hatches: conditional edges, retry wrappers, “supervisor” agents that route dynamically, and callbacks that mutate the pipeline while it runs. Each patch works locally. Collectively they turn the chain into an implicit graph whose structure lives in scattered control-flow code, where it cannot be inspected, visualised, tested, or reasoned about as a whole.

Make the graph explicit#

The alternative is to admit that the structure is a graph and to make that graph the first-class object the framework executes. In GraphBit, agents, tools, and memory stores are all nodes. Edges carry data and control. A workflow is a directed graph, and running a workflow means traversing that graph under a scheduler that understands dependencies.

Once the graph is explicit, the escape hatches become ordinary features:

• Branching is a node with more than one outgoing edge and a predicate deciding which edges fire.
• Parallelism is two nodes with no dependency between them. The scheduler runs them concurrently without anyone writing async code.
• Cycles are edges that point backwards. A reviewer that rejects a draft simply routes to the writer again, with a guard on the cycle count so the loop terminates.
• Fan-in is a node with several incoming edges that waits for all, or any, of them before executing.

None of this is exotic from a systems point of view. Dataflow engines and build systems have executed dependency graphs for decades. What is different in the agent setting is that the nodes are non-deterministic, expensive, and slow, and the graph topology itself may depend on what a node produced. That combination drives the rest of the design.

Non-determinism changes what “done” means#

A build system can assume that running the same node twice with the same inputs gives the same output. An LLM agent cannot. Two consequences follow.

First, the scheduler has to record what each node actually emitted, not just that it finished. Downstream routing decisions depend on content, so the execution trace must be a first-class artefact. This is also what makes debugging possible: when a workflow goes wrong, you replay the graph with the recorded outputs and inspect the exact edge where it diverged from what you expected.

Second, validation belongs at the edges. A chain tends to validate at the end, when the final answer is checked against a schema or a rubric. In a graph, each edge can carry a contract: the type of data it expects, a schema the payload must satisfy, or a predicate over the payload. A node that produces malformed output fails at the edge, before its output contaminates the rest of the run. In practice this catches a large fraction of agent failures early and locally.

Dynamic topology without losing structure#

The hardest design decision was how much to let the graph change while it runs. A fully static graph is easy to reason about but cannot express “spawn one worker per sub-question the planner produced.” A fully dynamic graph, where any node can add arbitrary nodes and edges, is just the chain-with-escape-hatches problem again.

The compromise that worked is bounded expansion. Nodes can be declared as templates that expand into a fixed pattern at runtime, such as a map over a list the upstream node produced, or a retry sub-graph around a fragile tool. The pattern is fixed at design time. Only its multiplicity is decided at runtime. This keeps the graph inspectable before execution while allowing the shape of the work to follow the shape of the data.

Memory as a node, not a side effect#

In chain-based frameworks memory is usually ambient: a shared dictionary every step reads and writes, or a vector store agents query through a tool. Both approaches hide dependencies. If a downstream agent’s behaviour depends on what an upstream agent wrote into memory, that dependency should be visible in the graph.

Treating memory stores as nodes does exactly that. A write to memory is an edge into the memory node. A read is an edge out of it. The scheduler can now order operations correctly, run reads in parallel safely, and show in the trace precisely which memory state a given agent saw. It also makes it natural to have several memory nodes with different characteristics, such as a fast transient store for the current run and a persistent store that outlives it, without agents needing to know which is which.

What this buys in practice#

The concrete payoff shows up in three places.

Reliability. Edge contracts and bounded cycles turn most failure modes into local, recoverable events rather than end-of-run surprises.

Latency. Independent nodes run concurrently by default. In workflows with any real fan-out, this alone recovers a large share of the wall-clock time a sequential chain wastes.

Legibility. The workflow is a picture. You can draw it, diff it between versions, and point at the node that misbehaved. When a team is iterating on an agent system, this matters more than any individual feature.

Open questions#

Two problems remain unsolved to my satisfaction. The first is cost control under cycles: a loop guarded by a maximum iteration count still burns tokens on every pass, and better stopping criteria that look at whether the loop is making progress are needed. The second is composition. Sub-graphs should be reusable across workflows the way functions are reusable across programs, with clean interfaces, and the type system for edges is not yet rich enough to make that ergonomic.

The broader claim of this post does not depend on either. If the work you want agents to do has branches, independent parts, or feedback, then the structure of that work is a graph. A framework that pretends otherwise will make you write the graph yourself, badly, in the gaps between its abstractions.

The data shape is the whole problem#

A hyperspectral image is a cube: two spatial dimensions and a spectral dimension with a hundred or more narrow bands. The task is usually to assign a land-cover class to every pixel. Two facts about this data dominate every design decision.

First, the spectral dimension carries most of the information. A single pixel’s spectrum is often enough to identify its material, which is why the earliest methods treated each pixel as an independent vector and ignored its neighbours entirely.

Second, labelled pixels are scarce. A benchmark scene may have a few hundred labelled samples per class, and some classes have far fewer. Any model with a large parameter count relative to this will memorise the training pixels and fail on their neighbours.

Pixel-wise multidimensional convolutions#

The first model I built took the spectral-first view seriously. Instead of a 2D network over spatial patches, it applied convolutions along the spectral axis of each pixel, treating the spectrum as a signal with local structure, and combined this with a modest spatial neighbourhood. The multidimensional part refers to running convolutions across both axes with kernels shaped to match: long and thin along the spectrum, small in space.

This outperformed purely spectral classifiers because adjacent bands are highly correlated and a convolution exploits that. It also exposed the next problem. The spectral dimension is redundant, and a network that consumes all bands at full resolution spends most of its capacity on redundancy.

Regularised SVD as a learned front end#

The standard response to spectral redundancy is dimensionality reduction before the network, typically principal component analysis. This works but throws away information the classifier might have wanted, because the reduction is chosen without reference to the labels.

The alternative I tried was to build a singular value decomposition into the model as a differentiable front end, with a regulariser that discourages the network from relying on components with small singular values. The reduction is then shaped by the classification loss rather than fixed in advance, and the regulariser gives the effect of a soft, data-dependent choice of how many components to keep. On small training sets this mattered, because it reduced the parameter count downstream without committing to a dimensionality that might be wrong for a given scene.

Residual attention in both domains#

By 2021 the field had converged on spectral-spatial networks that process both dimensions jointly, and the question became how to weight them. Not every band is equally informative for every class, and not every neighbouring pixel is equally relevant, particularly at class boundaries where a spatial patch straddles two materials.

Attention answers both questions. A spectral attention module produces a weight per band conditioned on the input, suppressing bands that are noisy or uninformative for the current pixel. A spatial attention module produces a weight per neighbour, suppressing pixels that belong to a different class. Wrapping both in residual connections was essential: on small datasets, an attention module that can fail gracefully to the identity mapping trains far more reliably than one that must produce useful weights from the start. The resulting spectral-spatial residual attention network was the first model in this sequence that felt robust across scenes rather than tuned to one.

Dynamic kernels#

Fixed convolutional kernels assume that the right receptive field is the same everywhere in the image. In a hyperspectral scene it is not. Homogeneous regions benefit from large spatial context; boundaries need small kernels that do not mix classes. The dynamic kernel network addressed this by predicting kernel parameters from the local input, so the effective receptive field adapts to the content. This is attention in a different guise, applied to the filter rather than to the features, and it produced the cleanest class boundaries of anything in the sequence.

Energy-based generative models with morphological attention#

Every model above is discriminative: it learns a boundary between classes and nothing about the classes themselves. On tiny training sets this is wasteful. The unlabelled pixels, which vastly outnumber the labelled ones, carry information about the structure of the data that a discriminative model cannot use.

An energy-based model learns a scalar energy function that is low on the data manifold and high elsewhere. Trained on all pixels, labelled or not, it captures the structure of the scene. Combined with a classifier that shares its features, it acts as a powerful regulariser: the classifier is pushed to make decisions that are consistent with where the data actually lies, not just with the few labelled points.

Morphological attention is the spatial component of this framework. Land-cover classes have characteristic shapes, such as roads that are thin and linear and fields that are compact, and morphological operators from classical image processing capture exactly these properties. Using them to drive attention weights gives the model a shape prior that a learned spatial attention module struggles to recover from a few hundred samples. Unifying the energy-based objective with morphological attention gave a framework that handled both the scarce-label problem and the boundary problem at once.

What carried over#

Three principles survived every iteration.

Match the inductive bias to the data shape. Every gain came from encoding something known about hyperspectral data, whether spectral locality, redundancy, class shape, or the abundance of unlabelled pixels, into the model rather than hoping the model would learn it.

Attention is a weighting problem. Spectral, spatial, kernel, or morphological, every attention mechanism here answers the same question: which parts of the input should influence this decision? Framing it that way makes it obvious where the next one belongs.

Small data punishes ambition. The models that worked were the ones with a graceful fallback, whether a residual identity path, a regularised reduction, or a generative prior, when the labelled data was insufficient to specify the answer.

Those principles are the reason this line of work connects to what I do now. Graph-structured power system data and hyperspectral cubes look nothing alike, but the discipline of reading the data shape before choosing the model is the same.

The problem as it presents itself#

A grid-tied inverter produces a handful of measured signals: phase currents, DC-link voltage, sometimes switch-level measurements. An open-switch fault in one of the power transistors distorts these signals in a characteristic way. The diagnostic task is to detect that a fault has occurred and to localise it to the responsible switch, from a short window of measurements, under noise and under operating conditions that were not all present in the training data.

Two things make this hard. The distortions for different switch locations can be similar in the time domain and separate more cleanly in frequency content. And the training data is almost always imbalanced, because healthy operation dominates any realistic recording and some faults are rare.

Building the graph#

The first design decision is what a node is. The natural choice is one node per measured channel. The second decision is what an edge is, and here the physical topology gives you a starting point: two channels are connected if the components they measure are electrically adjacent. That gives a spatial graph whose structure is fixed by the circuit.

Adjacency alone is not enough, though. Two channels that are electrically distant can be strongly coupled during a fault because the fault current finds a path through both. So the spatial graph is usually augmented with learned or correlation-based edges, weighted by how strongly the channels co-vary in the current window. The result is a graph whose backbone is physical and whose fine structure is data-dependent.

Why a second graph#

If you build one graph and one attention network over it, the model does well on faults that manifest as time-domain relationships between channels and poorly on faults whose signature lives in frequency content. Concatenating spectral features onto each node’s input vector helps a little, but the attention weights, which decide which neighbours matter, are still driven predominantly by time-domain similarity.

The fix that worked was to give the spectral view its own graph. Transform each channel into a frequency representation, build a spectral graph whose edges reflect similarity in spectral content, and run a separate graph attention branch over it. The two branches see the same physical system through different lenses and learn different neighbourhood weightings.

Attention, and what it learns#

Graph attention is the right aggregation operator here for a specific reason: the importance of a neighbour depends on the fault. Under a fault on one leg of the inverter, the channels on that leg should dominate the representation of their neighbours. Under a different fault, a different set should dominate. A fixed aggregation, such as mean pooling over neighbours, cannot express this. Attention learns a per-edge weight conditioned on the current node states, so the effective graph reshapes itself around the fault.

Inspecting the learned attention weights is also the most useful diagnostic tool during development. When the spatial branch places high weight on the channels physically nearest the faulted switch, and the spectral branch places high weight on channels sharing the fault’s harmonic signature, the model is learning what you hoped. When attention collapses to near-uniform weights, something upstream, usually normalisation or the graph construction, is wrong.

Fusing the branches without letting one dominate#

Late fusion, where each branch produces a graph-level embedding and the embeddings are concatenated before the classifier, is simple and works. The failure mode to watch is one branch dominating because its embedding has larger scale or because the optimiser finds it easier to fit. Two mitigations helped: normalising each branch embedding before concatenation, and adding a light auxiliary classification loss on each branch alone so neither is allowed to become a passenger.

Imbalance is a loss problem, not an architecture problem#

It is tempting to attribute poor recall on rare faults to model capacity and respond with a bigger network. In my experience the gains come almost entirely from the loss and the sampling. Class-weighted cross-entropy or a focal loss, combined with balanced mini-batch sampling, moved recall on minority faults far more than any architectural change. The architecture determines whether the model can separate the classes. The loss determines whether it bothers to.

Robustness under noise#

Measurement noise is the other practical adversary. Injecting noise during training at a range of signal-to-noise ratios is the obvious defence, and it works, but the graph structure itself contributes something. Because a node’s representation is an attention-weighted aggregate over its neighbours, independent noise on individual channels is partially averaged out before it reaches the classifier. This is one of the concrete reasons a graph model degrades more gracefully than a per-channel model when the test conditions are noisier than the training conditions.

Connections to unsupervised detection#

Everything above assumes labelled faults. In earlier work on power converters the setting was unsupervised: learn what healthy operation looks like and flag departures from it. The two settings are complementary. An unsupervised detector answers “is something wrong?” without needing examples of every fault; a supervised graph model answers “what and where?” for the faults it has seen. A deployed system wants both, with the detector as the first stage and the classifier as the second.

Where this is going#

The natural extension is from a single inverter to a network of them, and then to the distribution grid they feed. The graph gets larger and more heterogeneous, with nodes of different types and edges that represent different physical relationships. Attention still applies, but the graph construction becomes the dominant design problem, and combining topological reasoning with language-model-based decision support for operators is the direction I am currently working on.

Why raw waveforms are a poor input#

A three-phase fault window is a small multivariate time series: six channels, a few hundred samples. The obvious approach is a one-dimensional convolutional network over this sequence. It works, in the sense that it trains and reaches reasonable accuracy on the data it was trained on. It also generalises poorly, and the reason is structural.

The discriminative information in a fault is relational. It lies in how the phases move relative to each other, how the magnitude of one channel compares with another at the same instant, and how these relationships evolve over a few cycles. A 1D convolution has a receptive field along time within a channel, and it mixes channels only through the depth of its filters. It has to learn the cross-channel relationships that the physics makes primary, and it learns them in a way that is fragile to changes in fault inception angle, fault resistance, and load.

Time-series imaging#

Time-series imaging is a family of transforms that convert a sequence into a two-dimensional array where the relational structure is explicit. The one that proved most useful here encodes, for every pair of time steps, a measure of the relationship between the signal values at those steps. The result is a square image whose texture reflects the signal’s temporal dynamics: periodic behaviour shows up as regular patterns, a fault’s onset shows up as a sharp change in texture, and the type of fault shapes the pattern that follows.

Applying this per channel and stacking the results gives an image with one plane per phase. Now a two-dimensional convolutional network sees exactly what it is good at: local textures and their spatial arrangement. The relational information that a 1D network had to reconstruct is present in the pixel values from the start.

The practical benefits are twofold. Accuracy on held-out fault conditions improves because the representation is less sensitive to the nuisance variables that dominate the raw waveform. And the model becomes easier to inspect, because the regions of the image the network attends to correspond to identifiable intervals of the fault.

Where self-attention belongs#

Convolutions are local. A fault’s signature is often not: the relationship between the pre-fault steady state at the start of the window and the post-fault behaviour at the end carries information a small kernel cannot see. Stacking more layers widens the receptive field but dilutes it.

Self-attention over the feature map addresses this directly. Each spatial position computes weights over every other position and aggregates accordingly, so a pixel representing the fault onset can attend to pixels representing the steady state several cycles earlier. In the network I ended up with, a single self-attention block placed after the convolutional stem and before the classifier was enough. Adding more attention increased cost without improving accuracy. The lesson generalises: attention is a tool for the long-range part of the problem, and most of the problem is local.

Localisation as a second head#

Classifying the fault type is only half of what a protection system needs. The other half is distance to fault. The two tasks share almost all of their features, so a single network with two heads, one for the categorical fault type and one for a continuous distance estimate, is both more efficient and more accurate than two separate models. The multi-task loss needs balancing, and the distance head benefits from a robust regression loss because a handful of badly estimated cases otherwise dominate the gradient.

Generative models for the unsupervised case#

There is a second line of attack that avoids the need for labelled faults altogether: learn a probabilistic model of healthy operation and treat faults as low-likelihood events. A wavelet decomposition as the front end, giving a multi-resolution view of the signal, followed by a probabilistic generative network trained on healthy windows only, gives a detector that flags any departure from normal without having seen an example of it. It cannot say which fault occurred, but it catches faults the supervised model was never trained on, which in a safety-critical setting is not a small advantage. The two approaches belong together, with the generative detector as a first stage and the imaging-plus-attention classifier as the second.

What I would do differently now#

Three things. First, I would spend more effort on simulating a wider distribution of operating conditions, because every generalisation failure I encountered traced back to a nuisance variable that was under-represented in training. Second, I would evaluate on measured data from a physical system earlier, since simulated waveforms are cleaner than reality in ways that flatter the model. Third, I would treat the transmission network as a graph from the start. A single line is a special case; the interesting problems, and the ones I am now working on, involve faults propagating through interconnected networks where topology is the primary structure and the per-line signal processing described here is one node’s worth of the picture.