From pipe dreams to reality: Announcing Elastic’s piped query language, ES|QL

We listened and are proud to introduce Elasticsearch Query Language (ES|QL), our new innovative piped query language — a single unified method and language to interact with data in Elasticsearch while removing the costly need to transfer it to external systems for specialized processing. Unlike other languages Elastic has adopted over the years like Query DSL, ES|QL is designed and purpose built from the ground up to greatly simplify data investigations and be accessible for beginners while being powerful for experts. 

ES|QL example command:

Connecting the dots of an unfolding attack or navigating through observability data requires you to filter, search, transform, and aggregate across an extraordinary amount of data. ES|QL delivers this functionality from a single query.

Powered by a robust query engine, ES|QL offers advanced search capabilities with concurrent processing, enabling users to seamlessly query across diverse data sources and structure.

There is no translation or transpliations to Query DSL; instead, each query in ES|QL is initially broken down, interpreted for its meaning, validated for accuracy, and then enhanced for best performance. Then a process is laid out for executing the query across various nodes within the cluster. The target nodes handle the query, making on-the-fly adjustments to the execution plan using the framework provided by ES|QL. The result is lightning fast queries that you get out of the box. As an example, view the nightly benchmarks for comparison.

Elastic’s solutions — Search, Observability, and Security — all benefit from features and innovations that are delivered within Elasticsearch and Kibana®. ES|QL fundamentally changes the experience of using these solutions and provides a simple but powerful data investigation workflow.

ES|QL fundamentally changes how analysts pursue threats and strengthens detection. Built in answer to rich community input, it unleashes the power of piped queries at the speed of Elasticsearch, enhancing the SIEM, endpoint security, and cloud security capabilities of Elastic Security.

• Search quickly and iteratively: Following the breadcrumbs of an emerging threat requires quick action and a language that delivers an iterative workflow.
• Enrich results with context: ES|QL enables analysts to correlate suspicious IP addresses with known threat intelligence databases, providing immediate clarity on potential threats.
• Transform data: ES|QL empowers users to manipulate their data by defining new fields or parsing non-normalized data, ensuring data clarity and relevance.
• Aggregate data: Results can be consolidated and aggregated, paving the way for deeper analysis and insight extraction.

Elastic is the only search platform to pair the efficiency of a schema-on-write architecture with the iterative search experience of a schema-on-read piped query language. With incredibly fast search — and query output in full sight — analysts can draw closer to their target with each successive pipe.

ES|QL also enhances Elastic Security’s powerful detection engine. To reduce alarm fatigue, improve alert relevance, and provide another avenue for behavioral detection, organizations can incorporate aggregated values within detection rules. With inline evaluation, practitioners can iteratively develop and hone ES|QL-based rules. Queries are formatted in plaintext, simplifying collaboration and supporting detection-as-code.

SREs using Elastic Observability can leverage ES|QL to analyze logs, metrics, traces, and profiling data, enabling them to pinpoint performance bottlenecks and system issues with a single query. SREs gain the following advantages when managing high dimensionality and high cardinality data with ES|QL in Elastic Observability:

• Remove signal noise: With ES|QL alerting, enhance detection precision by focusing on significant trends rather than individual incidents, minimizing false alarms, and delivering actionable notifications. SREs can manage these alerts through the Elastic API and integrate them into DevOps processes.
• Enhanced analysis with insights: ES|QL can process diverse observability data, including application, infrastructure, business data, and more, regardless of the source and structure. ES|QL can easily enrich the data with additional fields and context, allowing the creation of visualizations for dashboards or issue analysis with a single query.
• Reduced mean time to resolution: ES|QL, when combined with Elastic Observability's AIOps and AI Assistant, enhances detection accuracy by identifying trends, isolating incidents, and reducing false positives. This improvement in context facilitates troubleshooting and the quick pinpointing and resolution of issues.

ES|QL in Elastic Observability not only enhances an SRE's ability to manage the customer experience, an organization's revenue, and SLOs more effectively but also facilitates collaboration with developers and DevOps by providing contextualized aggregated data.

With ES|QL, you can retrieve, aggregate, calculate, and transform data in a single query. It boasts key features, such as the ability to define fields at the time of query, perform data enrichment lookups, and process queries concurrently. Understand and explore your data with ES|QL in various ways. From utilizing clients for direct API/code integration to visualizing results directly from one screen, ES|QL streamlines your data investigations, ensuring you get the most out of your data sets with ease and simplicity. 

ES|QL’s design focus is evident in its capability to reduce code complexity, ultimately leading to cost and time savings. By facilitating the reuse of query results in subsequent searches, ES|QL minimizes computational overhead, eliminating the need for convoluted scripts and redundant queries. ES|QL is not just an API but a simple and powerful way to transform your approach to search.

The future of data exploration and manipulation is here. Elastic invites security analysts, SREs, and developers to experience this transformative language firsthand and unlock new horizons in their data tasks. Learn more about the possibilities with ES|QL, or start your free trial now in technical preview.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.