tenzir

Tenzir Documentation Map

The security data pipeline platform.

How to use this skill

Navigate the documentation based on the type of question:

Question type	Where to look
"How do I…" tasks	Guides — step-by-step instructions organized by task
Operator or function syntax	Operator Index or Function Index, then the specific page
Integration setup (Splunk, Kafka, S3…)	Integrations — per-product setup and pipeline examples
Concepts (nodes, pipelines, deployment)	Explanations — architecture and design
Learning from scratch	Tutorials — guided lessons
TQL language rules	Language, Expressions, Statements

Always read the relevant page before answering. Prefer TQL examples from the documentation over inventing syntax.

Beyond the docs

This skill bundles only the documentation. For anything else about Tenzir, fetch these live pages on demand instead of answering from memory:

Topic	Where to look
Release notes, "when was X added/changed"	https://tenzir.com/changelog.md
Blog posts and announcements	https://tenzir.com/blog.md
Solution use cases	https://tenzir.com/solutions.md
Company, product, events, everything else	https://tenzir.com/llms.txt — the full site index with per-page Markdown links

Answer patterns

Operator syntax question — "How does where work?" → Read where, explain the syntax, show the doc's TQL examples.

Integration question — "How do I send data to Splunk?" → Read Splunk, provide the pipeline example from the page.

Task question — "How do I parse syslog?" → Read Parse delimited text and read_syslog. Combine the guide's approach with the operator reference.

Guides

Guides are practical step-by-step explanation to help you achieve a specific goal.

Get Started

Quickstart

Drowning in logs, alerts, and rigid tools?

Overview

This guide shows you how to install the Tenzir CLI to run pipelines locally or deploy a persistent node.

Create account

Run pipelines

You can run a pipeline via the platform, on the command line using the tenzir binary, or as code via the configuration file.

Manage a pipeline

This guide shows you how to control pipeline lifecycles through the app or API.

Setup

Overview

Build your own AI Workbench by bringing an AI agent and configuring it with Tenzir’s agent skills.

Use agent skills

Overview

The Tenzir Node is the vehicle to run pipelines.

Provision a node

Size a node

Deploy a node

Configure a node

Configure TLS

Start the API

Tune performance

Overview

The Tenzir Platform acts as a fleet management control plane for Tenzir Nodes.

Deploy on AWS

Choose a scenario

Configure reverse proxy

Configure internal services

Configure identity provider

Configure database

Configure blob storage

Configure secret store

Run the platform

Manage organizations

This guide shows you how to create, configure, and delete organizations in the Tenzir Platform.

Manage organization members

This guide shows you how to invite people to your organization, manage existing members, and understand the role-based permission model.

Manage organization workspaces

This guide shows you how to create, view, and delete workspaces that belong to an organization.

Configure workspaces

Workspaces in the platform logically group nodes, secrets, and dashboards.

Configure dashboards

You can pre-define dashboards for your static workspaces.

Use ephemeral nodes

An ephemeral node is ideal for temporary or auto-scaling deployments.

Collect

Overview

Read events and bytes from files, APIs, brokers, stores, and networks.

Read and watch files

Fetch via HTTP and APIs

Read from message brokers

Read from data stores

Get data from the network

Parse

Overview

Turn byte streams and text fields into structured events.

Parse delimited text

Parse binary data

Parse string fields

Clean up values

Shape

Overview

Prepare event fields and records for downstream processing.

Transform values

Manipulate strings

Normalize event timestamps

Repair out-of-order events

Calculate with time

Shape lists

Shape records

Reshape complex data

Convert data formats

Normalize

Overview

Map source-specific events into a shared schema.

Map to ASIM

Map to CIM

Map to ECS

Map to OCSF

Map to UDM

Optimize

Overview

Reduce data volume and processing work in a pipeline.

Filter and select data

Slice and sample data

Deduplicate events

Protect

Overview

Limit sensitive-data exposure while preserving useful structure.

Mask sensitive data

Encrypt sensitive data

Enrich

Overview

Add context that the original event does not contain.

Use lookup tables

Enrich with asset inventory

Enrich with threat intel

Enrich events with AI

Analyze

Overview

Summarize event streams and compare current activity with prior behavior.

Aggregate event streams

Shape aggregation results

Window event streams

Calculate aggregate statistics

Find unseen and rare values

Set adaptive thresholds from a baseline

Identify distribution drift

Find cardinality spikes

Detect

Overview

Match security activity and model the resulting findings.

Match events with TQL

Model detections in OCSF

Detect over time windows

Baseline behavior from stored events

Detect periodic behavior

Create multi-stage detectors

Execute Sigma rules

Check Sigma rule compatibility with OCSF

Scan bytes with YARA

Store

Overview

Persist events for later search, replay, and analysis.

Write partitioned files

Import into a node

Show available schemas

Transform data at rest

Search

Overview

Query stored events and continue processing the results.

Export from a node

Route

Overview

Deliver event streams to the systems that need them.

Send to destinations

Expose data as a server

Split and merge streams

Fan out with subpipelines

Load-balance pipelines

Replay

Overview

Run stored events through live pipelines again.

Replay historical events

Operate

Overview

Inspect node, pipeline, and host behavior.

Inspect a node

Collect metrics

Overview

Diagnose compilation issues, pipeline failures, resource growth, and node connectivity.

Debug field values

Inspect the compilation output

Investigate slow pipelines

Investigate memory growth

Investigate storage growth

Reconnect an offline node

Build

Install a package

Write your own package

Create a package

This guide shows you how to create a package from scratch.

Test packages

This guide shows you how to add tests to your package.

Add operators

This guide shows you how to create user-defined operators (UDOs) for your package.

Add pipelines

This guide shows you how to add deployable pipelines to your package.

Add contexts

This guide shows you how to add enrichment contexts to your package.

Add constants

This guide shows you how to define package-wide constants in a constants.tql file and reference them as pkg::$name from the package’s own operators and pipelines, as well as from any pipeline that uses the package.

Configure inputs

This guide shows you how to make packages configurable with inputs.

Maintain a changelog

This guide shows you how to manage changelog entries and publish releases with tenzir-ship.

Publish a package

This guide shows you how to publish your package.

Run tests

This guide shows you how to run existing integration tests with the tenzir-test framework.

Write tests

This guide shows you how to create integration tests with the tenzir-test framework.

Run fixtures

This guide shows you how to start fixtures in standalone mode without running tests.

Create fixtures

This guide shows you how to create a fixture, wire it into the test harness, and use it from a test.

Add custom runners

Runners tell tenzir-test how to execute a discovered file.

Configure project hooks

This guide shows you how to configure tenzir-test project hooks for setup and cleanup tasks that belong next to your tests.

Contribute

Code of Conduct

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Git and GitHub Workflow

The following diagram visualizes our branching model:

Security Policy

Security is a serious matter for us.

Setup syntax highlighting

Install TQL language extensions for syntax highlighting in your editor

Build from source

Tenzir uses CMake as build system with a C++23 compiler.

Other

Work with time

We split this guide into focused tasks.

Tutorials

Tutorials are learning-oriented lessons that take you through a series of steps to complete a project.

Fundamentals

Learn the data lifecycle

Security data arrives as somebody else’s text and has to end up as something a detection, a dashboard, or an auditor can use.

Learn idiomatic TQL

This tutorial teaches you to write TQL that is clear, efficient, and maintainable.

Development

Onboard a data source

Onboarding a data source means turning one product’s output into events that every later stage of the data lifecycle can use.

Add enrichment contexts

A context is state, not code.

Analytics

Plot data with charts

In this tutorial, you will learn how to use pipelines to plot data as charts.

Build an OCSF lake with ClickHouse

Store OCSF events in ClickHouse, from a table that accepts anything to a tuned lake with typed columns, skip indexes, codecs, and per-class retention.

Explanations

Explanations are big-picture explanations of higher-level Tenzir concepts.

Architecture

Deployment

This page explains Tenzir’s deployment architecture, which separates data processing from management through a layered design.

Pipeline

A Tenzir pipeline is a chain of operators that represents a dataflow.

Node

A node is a running process that manages and executes pipelines.

Platform

The platform provides fleet management for nodes.

Language

Understand TQL's dataflow model, streaming and batch semantics, and support for events with different schemas.

Executor

Understand how Tenzir compiles TQL into an execution plan and runs operators with fused handoffs, channels, subpipelines, and parallel jobs.

Data model

Understand how Tenzir stores events in columnar batches, keeps events of different shapes and types side by side, and selects rows with bitmaps instead of slicing batches apart.

Concepts

Configuration

This page explains how to configure the Tenzir CLI and Node.

Secrets

Operators and functions accept secrets as arguments for sensitive values, such as authentication tokens, passwords, keys, or even URLs.

Normalization

Why Tenzir maps every source to OCSF first and translates from there

Enrichment

Enrichment means adding contextual data to events.

Detections

Understand how Tenzir matches, correlates, and models security evidence as detection results

Packages

This page explains how packages bundle pipelines, operators, contexts, and examples into a deployable unit.

Encryption

Encryption turns a value into ciphertext that only the holders of a key can turn back into the original.

Help

Glossary

This page defines the terms that name things you work with in Tenzir: the components, the entities a pipeline manipulates, and the units we ship.

Reference

The reference has nitty-gritty technical descriptions of how Tenzir works.

Language (TQL)

Type System

This page explains TQL’s type system, which provides strong typing with automatic inference.

Expressions

Expressions form the computational core of TQL.

Statements

TQL programs are a sequence of statements.

Programs

TQL programs compose statements into complete data processing workflows that can execute.

Overview

Tenzir runs part of a pipeline where the data lives.

Components

Configuration

The below example configuration ships with every Tenzir package.

Helm chart

The Tenzir Helm chart deploys one or more tenzir-node instances on a Kubernetes cluster.

Configuration

These settings configure the Tenzir Platform.

Command line interface

The Tenzir Platform command-line interface (CLI) allows you to interact with the Tenzir Platform from the command line to manage organizations, workspaces, and nodes.

Tools

Test Framework

The tenzir-test harness discovers and runs integration tests for pipelines, fixtures, and custom runners.

Ship Framework

tenzir-ship ships as a Python package that requires Python 3.12 or later.

Security

AWS Authentication

Tenzir’s AWS operators authenticate with AWS using the AWS SDK’s default credential chain, an OIDC web identity token, or static credentials.

Azure Authentication

Tenzir’s Azure operators authenticate with Microsoft Entra ID using the Azure SDK’s default credential chain, a shared account key, an application client secret, or workload identity federation.

API

Overview

This reference is generated from the OpenAPI specification.

Overview

This reference is generated from the OpenAPI specification.

Check node health

post/ping

List Nodes

post/user/list-nodes

Create Node

post/user/create-node

Rename Node

post/user/rename-node

Delete Node

post/user/delete-node

Get Node Token

post/user/get-node-token

Generate Client Config

post/user/generate-client-config

Generate Download Url

post/user/generate-download-url

Create Demo Node

post/user/create-demo-node

Retire Demo Node

post/user/retire-demo-node

Proxy

post/user/proxy

Transparent Proxy

get/user/node-proxy/{tenantid}/{nodeid}/{httppath}

Transparent Proxy

post/user/node-proxy/{tenantid}/{nodeid}/{httppath}

Transparent Proxy

put/user/node-proxy/{tenantid}/{nodeid}/{httppath}

Transparent Proxy

delete/user/node-proxy/{tenantid}/{nodeid}/{httppath}

Transparent Proxy

patch/user/node-proxy/{tenantid}/{nodeid}/{httppath}

Transparent Proxy

head/user/node-proxy/{tenantid}/{nodeid}/{httppath}

Transparent Proxy

options/user/node-proxy/{tenantid}/{nodeid}/{httppath}

Proxy Cached

post/user/proxy-cached

Read pipeline output

post/serve

Store

post/user/dashboard/store

Get

post/user/dashboard/get

Read multiple pipeline outputs

post/serve-multi

List

post/user/dashboard/list

Delete

post/user/dashboard/delete

Add

post/user/alert/add

Create a new pipeline

post/pipeline/create

Delete an existing pipeline

post/pipeline/delete

List

post/user/alert/list

Delete

post/user/alert/delete

Launch a new pipeline

post/pipeline/launch

List pipelines

post/pipeline/list

Reset the TTL of an existing pipeline

post/pipeline/reset-ttl

Update pipeline state

post/pipeline/update

Add External Store

post/user/secrets/add-external-store

Set Default Store

post/user/secrets/select-store

Delete External Store

post/user/secrets/delete-external-store

List Stores

post/user/secrets/list-stores

Add

post/user/secrets/add

Update

post/user/secrets/update

Remove

post/user/secrets/remove

List

post/user/secrets/list

Delete Tenant

post/user/delete-tenant

Update Tenant Name

post/user/rename-tenant

Switch Tenant

post/user/switch-tenant

Health

get/user/health

Get Login Info

post/user/get-login-info

Get Webapp Key

get/admin/webapp-key

Create Tenant

post/admin/create-tenant

Generate User Key

post/admin/generate-user-key

Spawn Node

post/admin/spawn-node

List Tenants

get/admin/global-tenant-list

Delete Tenant

post/admin/force-delete-tenant

Check Connectivity

post/admin/check-connectivy

Update Tenant Owner

post/admin/update-tenant

Add Auth Function

post/admin/add-auth-function

Delete Auth Function

post/admin/delete-auth-function

Operators

Operators

Tenzir comes with a wide range of built-in pipeline operators.

Functions

Functions

Functions appear in expressions and take positional and/or named arguments, producing a value as a result of their computation.

Node Index

• Configuration
• Helm chart

Overview Index

• ClickHouse
• DuckDB
• KQL
• Microsoft SQL Server
• MySQL
• SentinelOne

Platform Index

• Command line interface
• Configuration

Indexes

For the complete operator listing by category, read Operator Index.

For the complete function listing by category, read Function Index.

Integrations

alphaMountain integration

Enrich DNS, web, and network telemetry with real-time threat intelligence.

Amazon integration

Stream events through MSK, store and replay them in S3, ship them to CloudWatch or Amazon Security Lake in OCSF, and pull messages from SQS, all with first-class IAM integration.

Amazon CloudWatch Logs integration

Centralized logging for AWS resources, applications, and custom logs.

Amazon Kinesis integration

Ingest and process real-time data streams from Amazon Kinesis.

Amazon MSK integration

Publish and subscribe to Amazon's managed Kafka service.

Amazon S3 integration

Store, retrieve, and manage data objects in a scalable and durable cloud storage service.

Amazon Security Lake integration

Send OCSF events to Amazon Security Lake.

Amazon SQS integration

Send, receive, and manage messages between distributed applications.

AMQP integration

Send and receive messages between systems using a standardized, protocol-based approach.

Anthropic Claude Code integration

Turn coding agent activity into OCSF security telemetry: commands, files, tools, and who approved them.

Apache Avro integration

Decode schema-driven Apache Avro binary data from streams, files, and message brokers.

Apache Iceberg integration

An open table format for building lakehouses on object storage, with atomic commits, hidden partitioning, and schema evolution.

ArcSight integration

OpenText ArcSight is a SIEM and log management ecosystem.

AWS Glue integration

Write Iceberg tables to the AWS Glue Data Catalog and query them with Athena.

Azure Blob Storage integration

Store, retrieve, and manage unstructured data at scale.

Azure Event Hubs integration

Real-time data streaming platform and event ingestion service.

Azure Log Analytics integration

Query workspace data and send events to Azure Monitor Logs with Tenzir.

Cisco integration

Connect the Cisco security portfolio: firewall and IPS events from ASA and Firepower, DNS-layer telemetry from Umbrella, plus Duo, Secure Email, and Secure Endpoint.

ClickHouse integration

Send structured events to ClickHouse tables.

Cloud Lakehouse integration

Write Iceberg tables to Google's Lakehouse for Apache Iceberg (formerly BigLake) and query them with BigQuery.

CrowdStrike integration

Provides real-time event data including process, file, and network activity.

DuckDB integration

Read from and write to DuckDB database files with an embedded DuckDB engine.

Elasticsearch integration

Index, search, and analyze data in a distributed and scalable manner.

File integration

Read from and write to files.

Fluent Bit integration

Collect, process, and forward logs and metrics from various sources to many sinks.

FTP integration

Transfer files between a client and a server using a reliable, connection-based protocol.

Google integration

Connect Google Cloud, Google SecOps, and Google Workspace: stream through Pub/Sub, store in Cloud Storage, read Cloud Logging, write Cloud Monitoring metrics, and ship UDM events to SecOps.

Google Cloud Logging integration

Send events to Google's log management solution

Google Cloud Monitoring integration

Send node metrics, gauges, counters, and histograms to Google Cloud Monitoring as user-defined metrics.

Google Cloud Pub/Sub integration

Send, receive, and process messages between decoupled applications and services.

Google Cloud Storage integration

Store, retrieve, and manage data objects across a scalable, cloud-based environment.

Google SecOps integration

Send events to Google SecOps

Graylog integration

Collect, index, and analyze log data for monitoring and troubleshooting.

HTTP integration

Transfer and receive data between clients and servers using a request-response protocol.

Hydrolix integration

Stream security telemetry into Hydrolix tables and query it back.

IBM QRadar integration

Send security events to IBM Security QRadar for SIEM analytics and incident investigation.

Kafka integration

Publish, subscribe, store, and process streams of records in a distributed messaging system.

Kunai integration

Monitors Linux hosts with eBPF and records process, file, and network activity.

Microsoft integration

Connect Azure, Microsoft 365, and Windows: stream through Event Hubs, store in Blob Storage, ingest into Sentinel, and parse the full Windows telemetry surface.

Microsoft 365 / Office 365 integration

Collect unified audit logs for activity across Exchange, SharePoint, Microsoft Entra ID, Teams, and other Microsoft 365 services.

Microsoft Defender integration

Detect, prevent, and respond to security threats across endpoints and networks.

Microsoft Graph integration

Microsoft Graph is the unified API for Microsoft 365, Microsoft Entra ID, and other Microsoft cloud services.

Microsoft Sentinel integration

Prepare and route security telemetry for Microsoft's cloud-native SIEM.

Microsoft SQL Server integration

Provides audit logs for database activity, logins, and errors.

Microsoft Windows Event Collector integration

Receives events from a Windows Event Collector through the shipping agent of your choice.

Microsoft Windows Event Forwarding integration

Acts as the collector for Windows Event Forwarding, without a Windows server or shipping agent.

Microsoft Windows Event Logs integration

Collects Security, System, Application, and other critical OS logs.

Microsoft Windows EVTX Files integration

Parses exported Windows Event Log files in the binary EVTX format for forensic analysis.

MySQL integration

Connects to MySQL over the network using the MySQL wire protocol.

nano integration

Run a lake-native SIEM on ClickHouse for search, detections, cases, and AI-assisted triage.

NATS integration

NATS is a messaging system for services, edge deployments, and cloud-native applications.

NetFlow integration

Collect and process NetFlow v5, NetFlow v9, and IPFIX flow telemetry.

Network Interface Card integration

Acquire packets from the network and process them with pipelines.

NXLog integration

Ships Windows Event Logs and other telemetry from the NXLog Agent to Tenzir.

OpenAI Codex integration

Turn coding agent activity into OCSF security telemetry: commands, files, sandbox verdicts, and model traffic.

OpenSearch integration

Search, analyze, and visualize data with a distributed, open-source platform.

OpenTelemetry integration

Receive OpenTelemetry logs, metrics, and traces over OTLP/HTTP or OTLP/gRPC.

OpenWEC integration

Receives the Windows events that the open-source Windows Event Collector for Linux gathers.

PowerShell Script Block Logging integration

Captures deobfuscated PowerShell commands, critical for detecting fileless malware and living-off-the-land attacks.

Prometheus integration

Send metric events to Prometheus-compatible Remote Write receivers, including Prometheus, Grafana Mimir, Cortex, Thanos Receive, and VictoriaMetrics.

RELP integration

Receive Syslog messages with application-level acknowledgements, retransmission, and optional TLS.

rsyslog integration

Forward Syslog to Tenzir with RELP acknowledgements, retries, queues, and optional mutual TLS.

SentinelOne Data Lake integration

Send events to SentinelOne Singularity™ Data Lake

Sigma integration

Run stock Sigma detection rules on streaming events, natively on OCSF.

Snowflake integration

Store, analyze, and share structured and semi-structured data with a cloud-based warehouse.

Splunk integration

Collect, index, and analyze machine-generated data for monitoring, searching, and troubleshooting.

Suricata integration

Detect, log, and analyze network traffic for intrusion detection, prevention, and monitoring.

Syslog integration

Send and receive Syslog over UDP, TCP, and RELP.

Sysmon integration

Advanced system monitor that logs detailed host activity for threat hunting.

TCP integration

Establish, maintain, and terminate reliable, connection-oriented communication over IP networks.

UDP integration

Send and receive lightweight, connectionless datagrams over IP networks.

Velociraptor integration

Collect, monitor, and analyze endpoint data for digital forensics and incident response.

Wazuh integration

Collects security telemetry and runs detections.

Winlogbeat integration

Ships Windows Event Logs from Elastic's Winlogbeat agent to Tenzir.

YARA integration

Scan files and byte streams with YARA-X rules directly in the pipeline.

Zeek integration

Inspect, log, and analyze network traffic for security and operational insights.

ZeroMQ integration

Send, receive, and route messages asynchronously over various transport protocols.

Zscaler integration

Logs all web, firewall, and DNS traffic for users and locations.