Tail Interface Contract

This document defines the conventions that all tail packages in tails/ must follow. It serves as the template for new tails and the audit baseline for existing ones.

Overview

A tail is an adapter that bridges kitsune pipelines to an external system: a message broker, database, HTTP server, object store, file format, or observability backend. Tails live in tails/<kname> as independent Go modules and are versioned separately from the core library. You only import the tails you use; each brings its own dependency graph.

Naming

Primary verbs for new tails:

• Consume(client, target, unmarshal, opts...) returns *Pipeline[T] (stream from an external system into the pipeline).
• Produce(client, target, marshal, opts...) returns func(context.Context, T) error for use with ForEach (consume pipeline items and write them out).

Domain-idiomatic alternatives accepted when standard in the ecosystem:

State-backend constructors (NewStore, NewCache, NewDedupSet) follow the core interface conventions and are not source/sink verbs.

Parameter order

Read sources: (client/conn, target, unmarshal, opts...)

The "target" is the topic, subject, channel, table, collection, bucket, or URL. When the client already binds a target (e.g. kafka.Reader has a configured topic, gRPC stream is already open), omit the target argument.

Write sinks: (client/conn, target, marshal, opts...)

Same rule: drop target when the client binds it.

Connection lifecycle

Tails never create, open, or close connections, pools, readers, writers, or subscriptions. The caller constructs these objects, passes them in, and is responsible for closing them.

Every tail's package godoc must state this explicitly. The canonical phrasing is:

The caller owns the [ClientType]: configure [credentials/settings] yourself. Kitsune will never create or close [connections/clients].

Individual function godocs use: "The [conn/client/stream] is not closed when the pipeline ends; the caller owns it."

Error propagation

• Unmarshal/marshal errors terminate the pipeline with that error (fail-fast on malformed data).
• Connection/broker errors terminate the pipeline.
• Errors from user-supplied transform functions are governed by OnError on the consuming stage (Map, ForEach, etc.), not by the tail itself.

Delivery semantics

Every tail must declare in its package godoc one of the following:

At-least-once: items may redeliver on reconnect or pipeline restart. This is the default for message brokers that support acks (Kafka after commit, NATS JetStream after ack, SQS after delete, AMQP after ack, Pulsar after ack, Pub/Sub after Ack).

At-most-once: items may be lost on pipeline crash (no ack mechanism; e.g. NATS core, MQTT, Redis pub/sub, WebSocket, PostgreSQL LISTEN/NOTIFY).

Exactly-once-effective: achievable only with idempotent sinks or external deduplication. Document where this applies (e.g. using Distinct/Dedupe with kredis.NewDedupSet).

For sinks, document whether the sink acks synchronously (ForEach returns only after the item is confirmed by the remote) or asynchronously (e.g. kjetstream.PublishAsync batches futures; the caller must call flush after Run).

Delivery semantics do not apply to: - Hook tails (kdatadog, kotel, kprometheus): these implement hooks.Hook and record metrics; they are not source/sink adapters. - State backends (kredis.NewStore, NewCache, NewDedupSet): these are not message-passing constructs.

Package godoc template

Every tail's package comment (top of the main .go file) must include:

1. One-sentence description.
2. Caller-owns-connection statement.
3. Minimal worked example block (source and sink if both exist).
4. Delivery semantics declaration (or a note that it does not apply).

Tail matrix