The phrase “LLM-controlled drone” produces two reactions in roughly equal measure: excitement about natural-language mission specification, and alarm about a probabilistic text generator being anywhere near a flight controller. Both reactions are correct, and reconciling them is the design problem I have been working on at MTSU. This post lays out the architecture we have converged on and the reasoning behind where the boundaries sit.
Three time scales
An autonomous drone makes decisions at three very different rates.
The control loop runs at hundreds of hertz. It stabilises attitude, tracks velocity setpoints, and reacts to disturbances. It is deterministic, well understood, and must never wait on anything slow.
The navigation layer runs at a few hertz. It fuses perception into a local map, plans collision-free paths to a waypoint, and issues setpoints to the control loop. It is real-time but tolerates some latency.
The mission layer runs on the scale of seconds to minutes. It decides what the drone should be trying to do: which area to survey next, whether an observed object warrants a closer look, when to abort and return. It is where the ambiguity lives.
A language model is slow, expensive, and non-deterministic. It belongs at exactly one of these levels, the mission layer, and the entire architecture follows from keeping it there.
What the LLM owns
At the mission layer the LLM does three jobs.
Interpreting intent. A mission arrives as natural language: “survey the east field, prioritise anything that looks like standing water, come back before the battery drops below a safe margin.” The model turns this into a structured mission plan: an ordered set of goals, each with a success criterion and a priority.
Replanning on new information. Perception produces events the mission did not anticipate. An area is obstructed, an object of interest appears, weather changes. The model receives a compact description of the event and the current plan, and emits a revised plan. This is where language models earn their place: they handle open-ended situations that no finite state machine anticipated, and they do it with a plan that a human can read and check.
Explaining decisions. Every replanning step produces a rationale alongside the plan. In a research setting this is how we debug the system. In any deployed setting it is how an operator would decide whether to trust it.
What the LLM never touches
The model never emits a velocity, an attitude, or a motor command. It never decides whether a path is collision-free. It never overrides a safety limit. These are the domain of the navigation and control layers, which are conventional, tested, and deterministic.
The interface between the mission layer and the layers below is a small, typed vocabulary of goals: fly to a waypoint, orbit a point at a radius, hold position, return to launch. The LLM composes missions out of this vocabulary and nothing else. If it emits something outside the vocabulary, the plan is rejected and the model is asked again. This is the same discipline as validating at the edges in an agent graph: constrain the output to a schema, check it before it acts, and fail locally.
Perception in, language out
The mission layer needs to know what the drone sees, but feeding raw sensor data into a language model is both wasteful and unreliable. Instead, the perception stack produces symbolic summaries: detected objects with classes, positions, and confidences; a coarse occupancy description of the surroundings; battery and link state. The LLM reasons over this summary.
The consequence is that the perception stack, not the language model, determines the ceiling on what the system can respond to. If perception cannot represent standing water, no mission phrasing will make the drone find it. This is a feature, not a limitation. It keeps the question “what can this system perceive?” answerable, which a vision-language model reasoning over raw pixels would not.
Latency and the fallback plan
A replanning call may take seconds. The drone cannot hover in indecision for that long, and it certainly cannot do so during an emergency. Two mechanisms address this.
First, the current plan always remains valid until a new one replaces it. The navigation layer keeps executing the last accepted goal while the model thinks. Second, a small set of safety behaviours is hard-wired below the mission layer and triggers on sensor state alone: low battery, lost link, geofence violation. These preempt the mission layer entirely. The language model can propose returning to launch; it cannot prevent it.
Simulation first
All of this is developed in simulation, using ROS with Gazebo for the vehicle and environment and Open-RMF for scenarios involving more than one vehicle. Simulation is not just cheaper than flight testing; it is where the mission layer can be exercised against thousands of scenario variations to find the situations where the model produces an inappropriate plan. Those failures are then turned into validation rules at the interface, so that the same class of bad plan is rejected before it reaches the navigation layer in the future.
Open problems
The interesting unsolved questions are about trust calibration. When should the system ask the operator rather than replan on its own? How should the model’s confidence in its interpretation of an ambiguous mission be surfaced? And, as the mission vocabulary grows to cover more capable behaviours, how do we keep the guarantee that every composition of vocabulary items is safe? Those are the questions the next phase of this work is aimed at.
The overall lesson so far is unglamorous. The language model is the least important component for making the drone fly and the most important component for making it useful. Keeping those two facts separate, architecturally, is what makes the system work.