Introducing Nessi: The Delta-Native Data Quality Engine

Nessi Team

Introducing Nessi.dev: The Delta-Native Data Quality Engine

In today’s data-driven world, maintaining data quality isn’t just a luxury—it’s a necessity. Modern data teams face unprecedented challenges with schema drift, broken pipelines, and silent anomalies, particularly when working with Delta Lake. These issues aren’t just inconvenient; they can lead to costly mistakes and lost opportunities.

The Challenge

Modern data teams are drowning in a sea of data quality issues:

  • Schema Drift: Unexpected changes in data structure that break downstream processes
  • Pipeline Failures: Silent breakdowns that go unnoticed until it’s too late
  • Data Anomalies: Subtle irregularities that compromise data integrity

And when you’re working with Delta Lake, these challenges become even more pronounced.

Enter Nessi.dev

What we needed wasn’t another dashboard or generic data quality tool. We needed a CLI-first, fast, focused engine that actually understands how Delta Lake works. That’s why we built Nessi.dev.

What is Nessi.dev?

Nessi.dev is a Delta-native data quality and observability engine built for data engineers. It runs locally or in CI/CD, works across cloud storage layers (S3, Azure Blob, GCS), and gives you real insight into your tables — without requiring Spark, JVM, or endless configuration.

With One Command, You Can:

  • Profile Delta tables with stats, nulls, histograms, uniqueness, and type checks
  • Validate schema consistency and version history
  • Detect anomalies (outliers, spikes, trend changes, oscillations)
  • Run custom rules (YAML, SQL, or Python)
  • Generate clean HTML/PDF reports with quality scoring
  • Set up alerts via email, Slack, or Prometheus
  • Integrate with dbt, Airflow, GitHub Actions, or any CI/CD system

Why Nessi.dev Exists

Most data quality tools fall into one of two categories:

  1. Dashboard-heavy platforms: Pretty charts, but disconnected from how engineers actually work
  2. Rigid rule engines: Limited insight, no profiling, and zero context

What Makes Nessi Different:

  • Built in Go for performance, simplicity, and portability
  • Works directly on Delta Lake log files (no black box)
  • CLI-first, DevOps-friendly, and easy to embed into any pipeline
  • Optional Python extensions if you want ML-based anomaly detection

Quick Start: Profile Check

Try this simple command to get started:

1

nessi profile s3://my-datalake/users --report

You’ll instantly get:

  • Statistical summary (min/max/null/unique per column)
  • Partition layout and optimization hints
  • Distribution histograms
  • HTML report with drill-downs
  • Quality scorecard

Need Validation?

1

nessi validate --rules rules.yaml s3://my-datalake/users

Supports predefined rules (null, range, regex, enum) and custom logic — via YAML, SQL, or Python.

Key Features

Nessi.dev isn’t just fast. It’s deeply integrated with how real data teams work:

✅ Delta Lake version control
✅ Schema diff and rollback up to 30 days
✅ Anomaly detection using trend deviation, z-score, IQR
✅ Lineage-aware RCA (Root Cause Analysis)
✅ Freshness + SLA checks with alert routing
✅ Grafana + Prometheus integration
✅ Cross-cloud support (AWS, Azure, GCP)
✅ Security-first: RBAC, JWT auth, audit logs, rate limiting

And it’s all containerized with Docker — no Spark, no JVM, no bloat.

Integration Ecosystem

Nessi works seamlessly with your existing tools:

  • dbt: Run Nessi checks on models, tags, and downstream dependencies
  • Airflow: Trigger profiling and validation tasks as operators
  • GitHub Actions: Fail builds if tables drift or rules fail
  • Grafana: Visualize table health trends in real time
  • Slack/Email/Teams: Get alerted before downstream consumers complain

Roadmap

Here’s what’s coming next:

  • Freshness SLA dashboard
  • Public rule and plugin hub
  • Unity Catalog support
  • More RCA automation
  • Community-contributed report templates

Ready to Try It?

No signup. No credit card. Just visibility.

1

docker run nessi/nessi:latest profile s3://my-datalake/users

Or explore the docs and live examples at nessi.dev

Nessi.dev is here to make your data lake less of a mystery — and your pipelines a lot more reliable.

Let me know what you break, what you love, and what you want next.