Building Trustworthy ML at Material Security

When my son was younger, I was the cubmaster for his cub scout pack. Part of that job was teaching the kids the Scout Law, a series of 12 virtues that we hope the scouts embody throughout their lives.

Trustworthiness is the first virtue in Scout Law. That placement isn’t accidental: being trustworthy is at the core of the ethical and moral compass we should all strive to follow. It is foundational to building strong and supportive relationships, effective leadership, and effecting a functional society.

It is also a cornerstone of good business. For all the redlines and negotiation and binding contracts, we remain human. The decisions we make about who we work with are strongly influenced by trust. And the most damaging incidents are those caused by people, businesses, and systems that don’t live up to the trust placed in them.

Cybersecurity is built on trust. Our industry is designed, on a certain level, to detect and protect against those that aren’t trustworthy. We find the malicious emails, the insider threats, the account takeovers, the business email compromises. By exposing and stopping these and countless other threats, we help organizations remain trustworthy to their customers, employees, and shareholders.

There are particular challenges around deploying machine learning and artificial intelligence within cybersecurity. These tools have tremendous potential to detect threats, spot patterns, and uncover anomalous activity that no other method could easily find. It’s not surprising that nearly every cybersecurity company is using them (or will be soon). 

But ensuring that the models we use are trustworthy is critical. There are legitimate concerns about how ML models are developed and used. In this post, we’re going to look at some of those challenges, and how Material addresses them.

The challenges of trusting ML and AI

As AI has taken over the tech and cybersecurity world, we’re seeing tangible reasons for skepticism.

• Data privacy and security - Protecting a company’s data means that we have access to that data. To do our job as a cybersecurity company well, we must sometimes leverage that data, including specific PII. For example, if we want to know how much an employee interacts with a specific email sender, we must know both the sender and recipient emails. Ensuring the data security of the models themselves and the systems feeding them is paramount.
• Bias - ML and AI depend on existing data. Any bias that is baked into that data can and often will manifest in any model predictions that spring from it. Safeguards must be put into place to prevent unequal protection for different subgroups.
• Lack of human control -  Many ML/AI systems do not have a human in the loop to ensure that the quality of the models does not significantly degrade. This can mean consistently tracking the performance of the model or giving the end user the ability to give feedback on the output of the model.  
• Lack of transparency - ML and AI often depend on hundreds, thousands, or even millions of signals to make their predictions. This can make it difficult to provide a compelling explanation as to why, for example, a message was identified as malicious. It leads to the perception that the predictions spring from a black box, and can’t be interpreted.

Building trustworthy AI

For people to trust the use of AI and ML models, each of the above concerns must be addressed. At Material, we believe there are four key elements that are critical to building trustworthy models (these are adapted from Varshnay, 2022): Integrity, Transparency, Alignment, and Mastery.

Together, these components address the concerns.

Integrity

Integrity deals with the security, safety, fairness, and broad applicability of AI and ML models, and is central to the privacy and security risk concerns.

Models should perform at high levels across time, across different contexts, and for different customers. This can be a very difficult task, as the inputs and the context across environments will obviously be widely different. In cybersecurity, different industries receive different attack patterns and volume depending on the nature of their business.

Additionally, models must place a premium on the safety and security of the data used to build and train those models. As the use of AI has exploded, we’re seeing this threat in real time. Models must not lead to data leakage directly, and the output of the model must not enable the data to be reconstructed.

Material tests our models extensively and label cases from all our customers.  We don’t move models to production until consistent performance is demonstrated across a range of environments and contexts and when we do find that performance is weak for a specific customer we work to improve the performance of that model for that company.

Transparency

A common frustration with the predictions and detections that come from AI is their opacity. People can see the results, but they don’t see how those results came to be. Unlike a simple rule (say, an email detection rule that simple asks “is there a request for money in this message?”), models ingest huge numbers of features and signals that can interact in complicated and obscure ways. Providing a clear, cogent explanation for why a detection was made can be challenging.

Transparency is communicating the how, what, and why of any models put into production as clearly and concisely as possible. In the context of security models, often the most important is the what–the nature of the detection. The output must be clear about:

• the determination (is it malicious or not)
• what is included (is it a single message, or a case compromised of any copies of the message sent to multiple people)
• and what was done about it (was it remediated, or is a response still needed)

The what provides the immediate need-to-know information, but obviously it can’t stop there. The reasons why the model made the detection are important as well. That reasoning provides key information that an analyst may need in case further investigation is needed, and can include aspects of the message, sender, attachments, URLs, and much more.

Finally, in order to be deeply transparent, the process of building the model itself must be communicated–the how. That can include features put into the model, including any that were built using models or AI, the technique being used (regression, decision trees, neural networks), and discussion of the advantages and disadvantages of those techniques. 

This blog series will continue to answer exactly those questions as part of our effort to meet this component.

Alignment

A well-made, high-quality model isn’t worth much if it doesn’t actually solve the problems it was built for. This is where the principle of alignment comes into play: ensuring that output of the model is in sync with the needs of the end users (the customers).

This is often complex and nuanced, particularly within cybersecurity–and a misaligned model can have a number of negative effects. No two organizations are the same, nor are any two security teams.

As a very simple example, take a system that automatically deletes a message that the model determines was highly likely to be malicious. The model operates with high recall and high precision, and almost never makes a mistake.

For a company in an unregulated industry with a small security team with no bandwidth to investigate and remediate malicious emails, this could be great. But for a company with compliance requirements to retain all messages, this could cause significant operational and financial damage.

In this case, for the latter company, even though the model worked exactly as designed, because it was misaligned with their needs, it put them at risk. 

This is why Material has been very careful about the models we build, and give our customers many avenues for feedback about the output of the models. This means that we are able to tune our models to what is required by our customers, and ensure our models remain aligned.

Mastery

Mastery deals with the human element behind the AI, and addresses the bias and lack of human control concerns. ML models and AI are enormously complex to build and maintain. Even the smallest changes can result in a model weakening at best, and rendering it useless at worst. The mastery of the engineers and data scientists behind the models is critical.

Take, for example, the phenomenon of feature drift. For most models, the statistical distribution of the independent variables (e.g., features) shift over time. This could be for a variety of reasons in different contexts.

In phishing detection, it’s simply a fact of life, because attackers are highly motivated to evolve their strategies to evade detection. When feature drift happens, however, the power of the model wanes. It’s imperative to have machine learning engineers who know how to measure feature drift for variables that have different scales and distributions, who understand when the drift becomes concerning and who can track that, and who can ultimately effectively apply that knowledge and decide which features need to be retrained, which should be discarded, and when new ones should be added.

There are dozens of aspects of ML and AI like feature drift. In order to effectively build models that do what they promise to do, you must have years of training and experience. In order words, you must have mastery. 

Mastery addresses a number of concerns about ML and AI. A master machine learning engineer is the human that controls the system, and ensures its impact and minimizes its risk. They understand how to protect sensitive data and ensure that it does not leak from the system and cannot be reconstituted from the output of the model. They understand how to test for bias and minimize the impact within the detection system. 

Material’s Data Science team have a combined 50 years of experience in machine learning engineering, data science, and data engineering, including building and managing systems and models at some of the biggest names in tech. We work side by side with our threat research team who combined have more than two decades in threat research and detection.

Conclusion

The challenges facing machine learning in security are real, but they aren't insurmountable. They simply demand a more deliberate and principled approach. Our commitment is to ground every model we build in four key pillars: the Integrity of our systems, the Transparency of our detections, the Alignment of our solutions with your specific operational needs, and the Mastery of our engineers behind it all.

This framework is more than a checklist. It's how we ensure our technology serves its one true purpose: to protect your organization without introducing new risks or operational headaches. Trust, as the Scout Law teaches, is foundational. It’s not something we ask for lightly, but something we work to earn with every threat we stop and every model we ship.

‍