Go back

New in Material: Enhanced Structured Email Search for More Powerful & Precise Results

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

Engineering
September 10, 2024
3m read
3m read
3m listen
3m watch
3m watch
New in Material: Enhanced Structured Email Search for More Powerful & Precise ResultsNew in Material: Enhanced Structured Email Search for More Powerful & Precise Results
speakers
speakers
speakers
authors
Logan Carmody
participants
No items found.
share

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

What’s New

  • Learn as You Search: The typeahead and compilable language provide a seamless learning experience, ensuring you become a search expert in no time.
  • Find Information Faster: The precise syntax and intelligent suggestions mean less time spent searching and more time focused on what matters.
  • Execute Complex Queries: With support for nested statements and boolean ORs, you can perform sophisticated searches that were previously impossible.

Building on the Existing Search Experience

Part of Material’s core offering is a normalized, provider-agnostic view into email. When we originally architected our email product, part of the core experience we designed for was a flexible, fast, cross-platform message search; we wanted the search syntax to be easy to use and onboard onto and feel familiar.

We settled on a free-text experience, where admins were free to type any content into search over their email corpus; we later added operators to make the search a bit more controlled and powerful. We implemented this experience by writing our own query parser for our syntax and operators. Once we had a parsed representation, we would compile this into a SQL query over our BigQuery data warehouse.

This approach has worked well for our customers, and provided a flexible way to search the Material Data Platform. However as time has gone on, we’ve gathered feedback from our customers to the following extent:

  • Ambiguity in Search Intent: The free-text format often made it hard to discern what users were looking for, which sometimes led to irrelevant search results.
  • Unclear Operators: While we provided operators to refine searches, many users found it unclear how these operators functioned or whether they were being applied correctly, leading to confusion and inefficiency. For example, a misspelling in an operator would still result in a successful search, but just searching for the entire string.
  • Limited Boolean Functionality: The old search only supported simple boolean AND operations, restricting the ability to create complex queries with nested statements or boolean ORs.

These customer pain points were coincidentally translating into engineering pain points; the confusion with operators would often lead customers to run multiple searches with wider time windows. Since BigQuery’s cost model is billed on bytes-scanned, rather than time on a compute instance, this often resulted in unnecessarily high cloud costs for the search experience which could’ve been avoided if the product guided users to better understand  our search syntax / behavior.

Introducing Our New Structured Search

With these challenges in mind, we've developed a new structured search system that enhances usability and effectiveness. Here’s how it transforms your search experience:

  • Compilable Language for Accuracy: By using a compilable language, our new search ensures that every query you enter is validated and executed correctly. This feature reduces errors and enhances the reliability of your searches, making it easier to find exactly what you need.
  • Structured and Intuitive: Our new search is structured, using a syntax similar to our advanced file search feature. This structured approach allows for a more targeted and powerful search, reducing ambiguity and improving accuracy.
  • Typeahead Suggestions: One of the standout features of our new search is the intelligent typeahead functionality. As you type, it guides you with suggestions, teaching you how to use the search effectively. This not only speeds up onboarding onto the search experience, but also ensures that your queries are correctly formatted and optimized.

From the engineering side, we were also able to greatly simplify our stack. We defined a language grammar and then used the open-source ANTLR library to code-generate a lexer and parser for us. Now our search code walks the compiled Abstract Syntax Tree and in turn, compiles it to BigQuery SQL. This cleaned up a ton of parsing and edge-case code and allowed for a lot easier testing on our side.

At Material, we take pride in creating products that solve the real-world problems our customers are facing. Our goal is to shift as much of the burden of security operations onto our products as possible, and we’re dedicated to listening to our customers to ensure our offerings evolve with their needs. We’re excited for you to try our new email search and see the difference it makes in your day-to-day workflow. As always, we’re here to support you, so please don’t hesitate to reach out with any questions or feedback.

Happy searching!

Related posts

Our blog is your destination for expert insights, practical tips, and the latest news in technology. Stay informed with our regular updates and in-depth articles. Join the conversation and enhance your understanding of the tech landscape.

blog post

Hack Week 2024 Recap

Hack Week at Material Security is our annual week-long Hackathon where everyone works with whomever they want on whatever they want—without any predefined processes, meetings, or team structures.

Material Team
2
m read
Read post
Podcast

Hack Week 2024 Recap

Hack Week at Material Security is our annual week-long Hackathon where everyone works with whomever they want on whatever they want—without any predefined processes, meetings, or team structures.

2
m listen
Listen to episode
Video

Hack Week 2024 Recap

Hack Week at Material Security is our annual week-long Hackathon where everyone works with whomever they want on whatever they want—without any predefined processes, meetings, or team structures.

2
m watch
Watch video
Downloads

Hack Week 2024 Recap

Hack Week at Material Security is our annual week-long Hackathon where everyone works with whomever they want on whatever they want—without any predefined processes, meetings, or team structures.

2
m listen
Watch video
Webinar

Hack Week 2024 Recap

Hack Week at Material Security is our annual week-long Hackathon where everyone works with whomever they want on whatever they want—without any predefined processes, meetings, or team structures.

2
m listen
Listen episode
blog post

CISO Series Interview with David Spark

“Sky’s the limit with how you can transform your email”

Material Team
4
m read
Read post
Podcast

CISO Series Interview with David Spark

“Sky’s the limit with how you can transform your email”

Chris Long
4
m listen
Listen to episode
Video

CISO Series Interview with David Spark

“Sky’s the limit with how you can transform your email”

Chris Long
4
m watch
Watch video
Downloads

CISO Series Interview with David Spark

“Sky’s the limit with how you can transform your email”

Chris Long
4
m listen
Watch video
Webinar

CISO Series Interview with David Spark

“Sky’s the limit with how you can transform your email”

Chris Long
4
m listen
Listen episode
blog post

TLDR Sec Demo Video Deep Dive with Clint Gibler

Watch a deep dive Material demo with Clint Gibler from TLDR Sec.

45
m read
Read post
Podcast

TLDR Sec Demo Video Deep Dive with Clint Gibler

Watch a deep dive Material demo with Clint Gibler from TLDR Sec.

Max Pollard
45
m listen
Listen to episode
Video

TLDR Sec Demo Video Deep Dive with Clint Gibler

Watch a deep dive Material demo with Clint Gibler from TLDR Sec.

Max Pollard
45
m watch
Watch video
Downloads

TLDR Sec Demo Video Deep Dive with Clint Gibler

Watch a deep dive Material demo with Clint Gibler from TLDR Sec.

Max Pollard
45
m listen
Watch video
Webinar

TLDR Sec Demo Video Deep Dive with Clint Gibler

Watch a deep dive Material demo with Clint Gibler from TLDR Sec.

Max Pollard
45
m listen
Listen episode
blog post

New in Material: Enhanced Structured Email Search for More Powerful & Precise Results

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

Logan Carmody
3
m read
Read post
Podcast

New in Material: Enhanced Structured Email Search for More Powerful & Precise Results

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

3
m listen
Listen to episode
Video

New in Material: Enhanced Structured Email Search for More Powerful & Precise Results

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

3
m watch
Watch video
Downloads

New in Material: Enhanced Structured Email Search for More Powerful & Precise Results

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

3
m listen
Watch video
Webinar

New in Material: Enhanced Structured Email Search for More Powerful & Precise Results

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

3
m listen
Listen episode
Privacy Preference Center

By clicking “Accept”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.