See how Material uses a hypothesis-driven Debug Agent to automate technical triage and reclaim engineering time.
TL;DR: We built an internal Debug Agent to automate the investigation of alerts and customer bugs using hypothesis-driven orchestration. By combining infrastructure-specific skills with deterministic scripts, we’ve significantly reduced manual toil while maintaining security and reliability.
At Material we continuously evolve how we harness agentic tooling, and our internal software engineering practices are no exception. Our experiments with internal workflows aim to reduce toil and benefit from the powerful associative capacity of these tools. One way we're doing this today is our "Debug Agent," an internal suite of skills and an orchestrator that cooperate to investigate internal alerts and customer-reported bugs.
What is it?
Our Debug Agent comprises three portions:
- An Orchestrator that investigates a problem and runs a five phase process to gather context, form a hypothesis (or a few), validate those against data, and then summarize and post this information for an engineer to pick up and review.
- Infrastructure-specific skills to securely pull and parse data from varying external sources: logs, metrics, as well as recent deploys and alerts.
- Domain-specific skills that have to do with our implementation, such as querying a data store used for bookkeeping the state of emails, investigating a file's progress through a processing pipeline, or routing specific categories of issues to the relevant data source.
What qualities make it work?
Hypothesis-driven investigation
When the Orchestrator is asked to investigate an issue (either via a comment in the issue tracking tool or from an agentic coding session), its first task is to gather all the relevant context from the issue itself, like any linked pull requests, logs, or commentary and discussion. This is made easier by connecting MCPs that link data from SaaS apps we use to run our business to collate information for analysis.
Once gathered, the agent is prompted to make 1-3 hypotheses about the cause of an issue (e.g. attachments for a particular user are not being scanned for malicious content in a timely manner).
Each hypothesis block must state:
- The problem
- e.g. Nodes are running out of memory and crashing, resulting in attachment processing not completing
- Evidence to support
- e.g. The exception tracker shows a stack trace with an OOM error
- What's needed next to prove/disprove this hypothesis
- Log aggregation to find a pattern
- Log tracing to find a deeper cause or cause(s)
- Constraining variables to understand what conditions produce this issue (e.g. the release version or the shape of the data)
After returning this set of hypotheses to the user, they may pick which to investigate. The suite of investigation tools (both infrastructure and domain specific) run for different types of problems. For example, our skill has a table of contents (continually evolving) for the agent to reference. Here's a sample of some we use regularly:
Composable skills and, importantly, scripts
One quality that makes this tool so successful and high-leverage for our team is that it is highly composable. The Debug Agent orchestrator can pull skills that are relevant without bloating its context with those that are not. And users prompting from other entry points can benefit from using skills relevant to their specific investigations, which may not originate with the Debug Agent.
Another way we've found to make this process more effective and secure is to use agentic coding to write deterministic scripting tools for repetitive tasks that both agents and engineers alike can call directly.
Things like…
- pulling logs from a large data set in a repeatable format with authentication, performance, data privacy guardrails, and cost awareness built in automatically
- complex find and replace operations during larger-scale refactors via the use of an AST
- fetching recent build info
- finding the canonical infrastructure name of a project based on its common name
- querying a production data store in a read-only mode with allow-listed attributes
- and more!
Two major concerns with agentic workflows are constrained when developing scripts with their help instead: token use (subsequent script runs don't require tokens to load context on the problem again) and security (boundaries are maintained and can be tested against known-correct results). Results are deterministic too. This leads to a less expensive, more reliable process that may have been prohibitively expensive to build prior to the improvements we've seen in agentic coding since the end of 2025.
Two nice touches: time awareness and issue commentary
The orchestrator agent has the ability to perform investigations in relative time thanks to a skill that pulls the current local time and performs conversions, reducing cognitive load for the human user. This allows the user to prompt things like "about an hour ago, michelle@coolcompany's inbound attachment of an invoice from anya@othercoolcompany failed to complete processing for data loss prevention; why did this occur?"
And finally, the Debug Agent orchestrator will summarize its findings in a comment on the original issue in its tracker (should it have been invoked this way). When added, it is tagged [debug-agent] so that it is easy in the future to aggregate all commentary of this type or search for it in a long issue with a lot of back-and-forth discussion.
Continuous improvement
Skills, like traditional code, are ever-evolving and tracked in git alongside the project they pertain to in order to retain a history of improvements (or fixes for ergonomics issues discovered in use). Any team member can notice something a skill does incorrectly, doesn't do at all, or could do better and contribute from their unique workflow's perspective, and changes are reviewed just like changes to the code. This way we can continue to grow and tend to our skills collection, and enjoy the compounding benefits of time saved as our collective experiences enrich it.
By automating the initial triage and context gathering for complex technical issues, the Debug Agent has become an indispensable member of our engineering team, significantly reclaiming time that would otherwise be spent on manual investigation. We believe these patterns of composable, hypothesis-driven agents are broadly applicable across many domains. We encourage you to look at your own repetitive workflows and consider how agentic tooling can help your team move faster and with greater confidence.
