Introduction

The Pipeline Framework is a powerful tool for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, and deploy pipeline steps.

Visual Pipeline Designer

The Pipeline Framework includes a visual canvas designer at https://app.pipelineframework.org that allows you to create and configure your pipelines using an intuitive drag-and-drop interface. Simply design your pipeline visually, click "Download Application", and you'll get a complete ZIP file with all the generated source code - no command-line tools needed!

Key Features

• Reactive Programming: Built on top of Mutiny for non-blocking operations
• Immutable Architecture (by default): Persistence is append-first and configurable; updates are explicit and opt-in
• Visual Design Canvas: Create and configure pipelines with the visual designer at https://app.pipelineframework.org
• Annotation-Based Configuration: Simplifies adapter generation with @PipelineStep
• gRPC & REST Flexibility: Automatic adapter generation for fast gRPC or easy REST integration
• Multiple Processing Patterns: OneToOne, OneToMany, ManyToOne, ManyToMany, SideEffect and blocking variants
• Health Monitoring: Built-in health check capabilities
• Multiple Persistence Models: Choose from reactive or virtual thread-based persistence
• Modular Design: Clear separation between runtime and deployment components
• Test Integration: Built-in support for unit and integration tests with Testcontainers

Framework Overview

For complete documentation of the framework architecture, implementation details, and reference implementations, see the complete documentation files in the main repository:

• Reference Implementation - Complete implementation guide with examples
• Canvas Designer Guide - Complete Canvas usage guide
• Java-Centered Types - Comprehensive Java-first approach with automatic protobuf mapping

How It Works

The framework lets you define pipeline steps as simple classes annotated with @PipelineStep. At build time, it generates the adapters and infrastructure, keeping services clean and focused on business logic.

Build Fast

• Quick Start: Create a pipeline via the Canvas and run it locally
• Business Value: Speed, ROI, and portability
• Canvas Designer Guide: Complete guide to the visual designer

Design

• Application Structure: Modular layout and service boundaries
• Common Module Structure: Shared domain types and mappers
• Expansion and Reduction: Cardinality explained for imperative developers

Build

• Pipeline Compilation: Build-time generation flow
• Configuration Reference: Build-time and runtime settings
• Dependency Management: Manage build-time and runtime deps
• Best Practices: Operational and design guidance

Develop

• @PipelineStep Annotation: Annotation contract and parameters
• Code a Step: Implement a step and its mappers
• Using Plugins: Apply plugins to pipelines
• Mappers and DTOs: Type conversions across layers
• Dependency Management: Manage build-time and runtime deps
• Upgrade Guide: Version changes and migrations
• Orchestrator Runtime: Coordinate pipeline execution

Observe

• Observability Overview: Metrics, tracing, logs, and security notes
• Metrics: Instrumentation and dashboards
• Tracing: Distributed tracing and context propagation
• Logging: Structured logging and levels
• Health Checks: Liveness and readiness
• Alerting: Alerts and noise reduction
• Security Notes: Protect telemetry data
• Error Handling & DLQ: Failure handling patterns
• Caching: Cache policies, backends, and invalidation

Extend

• Writing a Plugin: Create plugins and aspects
• Orchestrator Extensions: Customize orchestration flows
• Reactive Service Extensions: Wrap or adapt process() behavior
• Client Step Extensions: Customize client-side calls
• REST Resource Extensions: Extend generated REST resources

Evolve

• Functional Architecture: Core concepts and architectural patterns
• Annotation Processor Architecture: Build-time IR, bindings, and renderers
• Plugins Architecture: Cross-cutting behavior model
• Aspect Semantics: Aspect expansion rules
• Aspect Ordering: Ordering guarantees and constraints
• Aspect Warnings: Known limitations and caveats
• Reference Implementation: End-to-end example and rationale
• Template Generator (Reference): Automation/CI usage
• Publishing: Release and publishing workflow
• CI Guidelines: Build validation and automation
• Testing Guidelines: Coverage and test strategy
• Gotchas & Pitfalls: Known sharp edges
• Proto Descriptor Integration: Descriptor generation and troubleshooting

This approach reduces boilerplate and keeps pipeline code consistent and portable.