Imagine a tabletop exercise designed to test an AI tool. A severe storm has damaged roads and interrupted power. Reports are arriving from different places at different times. Someone asks an AI tool for a quick situation summary, and it produces a clean, confident paragraph.
Then a participant notices that one of its road-status details came from an older exercise inject. The current inject says something else.
That is a hypothetical scene, but it captures the question I would want answered before using AI in an actual response: when the information changes, will the tool help a team notice the change, or make an outdated answer sound settled?
A useful answer is not necessarily a reliable one
AI can be useful for sorting a long stream of reports, surfacing inconsistencies, and helping people find the item they need. A fluent summary, though, can also hide a missing source or blur the difference between confirmed information and an early report. NIST’s Generative AI Profile describes the risk of systems presenting false information confidently. In emergency management, that is a reason to test the workflow, not just admire the speed of the answer.
An exercise gives us a safer place to do that. FEMA’s Homeland Security Exercise and Evaluation Program provides a structure for setting objectives, observing performance, and turning findings into improvements. Its Exercise Evaluation Guides connect observations to objectives and critical tasks. An AI tool can be evaluated in that same disciplined spirit.
Test an AI tool with a job you can actually score
“See if AI helps” is too vague. Give it a bounded task: summarize the last 30 minutes of fictional situation reports for a planning team. Then decide in advance what success means.
For example, evaluators could ask:
- Did it distinguish a confirmed road closure from an unverified report?
- Did it show where each consequential statement came from and when that source was updated?
- Did it flag two reports that disagreed instead of quietly choosing one?
- Did it make uncertainty visible when the answer was incomplete?
- Did a human reviewer catch and correct mistakes before the summary reached a decision maker?
Those are proposed exercise measures, not a universal AI scorecard. The right measures depend on the tool’s assigned job and the consequences of an error. NIST’s AI Risk Management Framework calls for testing AI before deployment, measuring performance against relevant benchmarks, and defining human oversight.
I would also compare the AI-assisted team with the team’s usual process. Did the tool save time? Did it introduce extra checking work? Did it cause anyone to miss a conflicting report? Speed matters, but a faster path to a wrong answer is not an improvement.
Make the exercise uncomfortable on purpose
A tidy packet of complete, consistent reports will tell us very little. Real information arrives late, gets corrected, and sometimes contradicts itself. The exercise should include stale timestamps, missing fields, duplicate reports, and a plausible but false claim. It should test whether participants can trace a claim back to its source and decide what still needs confirmation.
The team should also rehearse what happens when the tool cannot provide a dependable answer. Who stops the AI-assisted workflow? Where does the issue get recorded? Which established process takes over? NIST’s framework emphasizes human roles and continued monitoring because performance can change with the context in which a system is used.
This is also where the exercise can reveal a human problem. If a polished answer appears on a screen during a busy shift, will anyone challenge it? The best tool design will not help much if the team has no time, authority, or habit of checking it.
Turn the surprise into an improvement
An exercise should end with more than “the AI got one wrong.” FEMA’s improvement-planning guidance links exercise findings to corrective actions. For an AI pilot, that might mean changing the prompt or source feed, requiring timestamps in every summary, assigning a reviewer, narrowing the use case, or deciding the tool is not ready for that task.
I am interested in what AI can add to emergency management. I am more interested in knowing exactly where it fails before the stakes are real. An exercise can give us that evidence, and it gives people a chance to practice the most important part of the workflow: noticing when the machine’s confident answer deserves a second look.
What would you put into an exercise to test an AI tool before your team relied on it?