Introduction

Azure Data Explorer (ADX) in an interactive, fully managed Exploratory Data Analysis (EDA) platform hosted on the Microsoft Azure cloud. ADX enables analysts to onboard datasets natively into ADX, from object storage such as Blob and ADLSv2, select databases, and Delta Lake tables. From there, analysts can further transform data and/or analyze and visualize data using Kusto Query Language (KQL) similar to Azure Log Analytics or Microsoft Sentinel.

ADX is the closest tool that Azure has to Amazon Athena to support performant ad-hoc search against data in object storage. It made all the sense in the world to support Azure Data Explorer as it is the closest platform to be able to search data natively stored in Azure object storage without too much automation or orchestration required.

In this short blog you will learn about use cases for Azure Data Explorer and what teams can accomplish with integrating it into the Query Federated Search platform.

Why Azure Data Explorer?

As mentioned in the introduction, ADX is an exploratory data analysis multi-tool where you can transform data, onboard external sources, ingest data directly into ADX, and visualize or search it all. It’s powered by KQL which is a slick and strongly featured query language, more the better if you already have expertise with Sentinel or Azure Log Analytics. The ability to bridge across to data stored in SQL databases, in Azure Storage Accounts, and even S3 or Azure Event Hub can prove useful for security and other IT operations teams.

When it comes to both ease of data ingestion and searching, ADX offers the better approach (in my opinion) than Azure Log Analytics (ALA) and Azure Blob (or other Azure Storage Accounts). Writing into ALA is a bit contrived, requiring generating HMACs and defining helper functions to load data, Azure Blob has a few different mechanisms such as using PyArrow File Systems or other 3rd party Saas data mobility tools. When it comes to querying remotely, ALA relies on its own SDK (as does ADX), but Blob requires the usage of DuckDB or otherwise moving the data.

If you already have data stored in Blob, and you wanted to move it to ALA to expose it to search via Sentinel or otherwise, the mechanisms are equally contrived requiring a slew of Azure services and custom orchestration. ADX offers a much more straightforward mechanism for ingestion, as well as support for more data formats, allowing you to flex your data engineering muscles and write Parquet data into ADX tables for instance. For the data in Blob and other services, onboarding them as external tables is straightforward and as long as the data isn’t too “chopped” (as the kids say) you can transform it in place to make it easier to query.

If you were starting a greenfields approach to building a Security Data Operations (SecDataOps) program on Azure and picking a core service, ADX makes all the sense in the world to go with. Moreso if you were already familiar with KQL and didn’t want to use SQL in Azure Databricks or another SQL-based service in Azure. If you were multi-cloud, you can also bring in existing data from S3 or push data in Azure Event Hub to also onboard it quicker into ADX to query it.

So, what do you get with onboarding it into Query?

Bringing ADX to the SOC

ADX is a dynamic schema Connector, which means that you can store nearly any type of data in both internal and external tables within ADX and onboard it into Query using our Configure Schema no-code wizard. Configure Schema allows you to map your data, no matter the downstream format, into our Query Data Model – based on the Open Cybersecurity Schema Framework (OCSF).

If you were otherwise not familiar with KQL, or even if you are, Query handles query translation of your search intent (which is also expressed in OCSF) back into KQL. Additionally, Query has its own proprietary query planning and execution engine which allows you to parallelize search against a handful up to hundreds of internal and external tables. This does not use expensive join operations which can increase query times, reduce query performance, and require in-depth knowledge of the schemas.

Since everything is normalized into OCSF, if you had multiple tables in ADX with IP addresses, you can use the IP Address Entity to search across all of these tables regardless of what the actual column names are and the type of table it is. So if you had Blob external tables, tables built from S3 ingestion, and native tables that contained EDR logs, network activity, authentication logs, authorization logs, and asset information you can collate all of the results from a single search.

Additionally, Query has several different AI Agents and CoPilots you can use to facilitate search into ADX tables (and any other Connectors) using human language. For instance, you can use our Vulnerability Intelligence Agent to search your ADX tables for vulnerabilities, automatically enrich the data, and pivot to our Asset Info or Detection Triage Agent to provide more context and remediation data.

What Now?

If you want to get started, check out our docs and then reach out to the team to get a demo or a tenant if you are an ADX user.

Do you have another query engine like Star Rocks or TrinoDB you’d rather use? We are more than happy to partner with you and build it!

SecDataOps Savages are standing by.

Stay Dangerous