Skip to content

Kitsune

Kitsune: type-safe concurrent pipelines for Go

Type-safe, concurrent data pipelines for Go.
Compose functions into stages: channels, goroutines, backpressure, and error routing are handled for you.

Get started   Operator catalog 

raw      := kitsune.From(kafkaStream)
orders   := kitsune.Map(raw, parseOrder)
enriched := kitsune.Map(orders, enrichWithCustomer, kitsune.Concurrency(20))
batched  := kitsune.Batch(enriched, 500, kitsune.BatchTimeout(2*time.Second))
err      := batched.ForEach(bulkInsert, kitsune.Concurrency(4)).Run(ctx)

go get github.com/zenbaku/go-kitsune

What Kitsune handles for you

  • Automatic backpressure

    Bounded channels between every stage. A slow consumer blocks its upstream: no unbounded queuing, no dropped items.

  • Compile-time type safety

    Pipeline[T] carries its element type through the graph. Every stage transition is checked at compile time via Go generics.

  • Per-stage concurrency

    Add Concurrency(20) to any stage to spin up parallel workers. Preserve arrival order with Ordered().

  • Error routing

    Per-stage OnError: Skip, Retry with exponential backoff, RetryThen, Return. Use MapResult to route errored items to a separate pipeline. Errors are values, not panics.

  • Observability

    MetricsHook, LogHook (structured slog), and a live inspector dashboard. OTel, Prometheus, and Datadog via tails.

  • 27 integrations

    Kafka, NATS, RabbitMQ, Postgres, Redis, S3, MongoDB, ClickHouse, SQS, Kinesis, Pub/Sub, and more. Each a separate module via tails.

All features →

Live inspector

Kitsune Inspector: live pipeline view

Add one line to any running pipeline to open a real-time web dashboard with a live DAG, per-stage metrics, and stop/restart controls. See the inspector guide →

Where to go next

  • Getting Started

    Mental model, first pipeline, concurrency, error handling, branching, and testing in ~10 minutes.

  • Operator Catalog

    Every source, transform, terminal, and option: with exact signatures and descriptions.

  • Tails

    Connect pipelines to Kafka, Redis, S3, Postgres, NATS, SQS, MongoDB, and 15+ more systems.

  • Tuning Guide

    Buffer sizing, concurrency settings, batching strategies, and GC-pressure trade-offs.

  • Concurrency Guide

    When to reach for Concurrency(n), Ordered(), MapWithKey, or Partition: decision flowchart and worked examples.

  • Examples

    20 runnable examples with full source code and Go Playground links.