New in Material: Enhanced Structured Email Search for More Powerful & Precise Results

We're thrilled to announce a significant upgrade to our email search functionality featuring a more powerful query experience and faster results.

What’s New

• Learn as You Search: The typeahead and compilable language provide a seamless learning experience, ensuring you become a search expert in no time.
  
• Find Information Faster: The precise syntax and intelligent suggestions mean less time spent searching and more time focused on what matters.
  
• Execute Complex Queries: With support for nested statements and boolean ORs, you can perform sophisticated searches that were previously impossible.

Building on the Existing Search Experience

Part of Material’s core offering is a normalized, provider-agnostic view into email. When we originally architected our email product, part of the core experience we designed for was a flexible, fast, cross-platform message search; we wanted the search syntax to be easy to use and onboard onto and feel familiar.

We settled on a free-text experience, where admins were free to type any content into search over their email corpus; we later added operators to make the search a bit more controlled and powerful. We implemented this experience by writing our own query parser for our syntax and operators. Once we had a parsed representation, we would compile this into a SQL query over our BigQuery data warehouse.

This approach has worked well for our customers, and provided a flexible way to search the Material Data Platform. However as time has gone on, we’ve gathered feedback from our customers to the following extent:

• Ambiguity in Search Intent: The free-text format often made it hard to discern what users were looking for, which sometimes led to irrelevant search results.‍
• Unclear Operators: While we provided operators to refine searches, many users found it unclear how these operators functioned or whether they were being applied correctly, leading to confusion and inefficiency. For example, a misspelling in an operator would still result in a successful search, but just searching for the entire string.‍
• Limited Boolean Functionality: The old search only supported simple boolean AND operations, restricting the ability to create complex queries with nested statements or boolean ORs.

These customer pain points were coincidentally translating into engineering pain points; the confusion with operators would often lead customers to run multiple searches with wider time windows. Since BigQuery’s cost model is billed on bytes-scanned, rather than time on a compute instance, this often resulted in unnecessarily high cloud costs for the search experience which could’ve been avoided if the product guided users to better understand  our search syntax / behavior.

Introducing Our New Structured Search

With these challenges in mind, we've developed a new structured search system that enhances usability and effectiveness. Here’s how it transforms your search experience:

• Compilable Language for Accuracy: By using a compilable language, our new search ensures that every query you enter is validated and executed correctly. This feature reduces errors and enhances the reliability of your searches, making it easier to find exactly what you need.‍
• Structured and Intuitive: Our new search is structured, using a syntax similar to our advanced file search feature. This structured approach allows for a more targeted and powerful search, reducing ambiguity and improving accuracy.‍
• Typeahead Suggestions: One of the standout features of our new search is the intelligent typeahead functionality. As you type, it guides you with suggestions, teaching you how to use the search effectively. This not only speeds up onboarding onto the search experience, but also ensures that your queries are correctly formatted and optimized.

From the engineering side, we were also able to greatly simplify our stack. We defined a language grammar and then used the open-source ANTLR library to code-generate a lexer and parser for us. Now our search code walks the compiled Abstract Syntax Tree and in turn, compiles it to BigQuery SQL. This cleaned up a ton of parsing and edge-case code and allowed for a lot easier testing on our side.

At Material, we take pride in creating products that solve the real-world problems our customers are facing. Our goal is to shift as much of the burden of security operations onto our products as possible, and we’re dedicated to listening to our customers to ensure our offerings evolve with their needs. We’re excited for you to try our new email search and see the difference it makes in your day-to-day workflow. As always, we’re here to support you, so please don’t hesitate to reach out with any questions or feedback.

Happy searching!