The Pipeline FrameworkReactive Pipeline Processing

Introduction ​

The Pipeline Framework is a powerful tool for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, and deploy pipeline steps.

Key Features ​

• Reactive Programming: Built on top of Mutiny for non-blocking operations
• Immutable Architecture: No database updates during pipeline execution - only appends/preserves, ensuring data integrity
• Annotation-Based Configuration: Simplifies adapter generation with @PipelineStep
• gRPC & REST Flexibility: Automatic adapter generation for fast gRPC or easy REST integration
• Multiple Processing Patterns: OneToOne, OneToMany, ManyToOne, ManyToMany, SideEffect and blocking variants
• Health Monitoring: Built-in health check capabilities
• Multiple Persistence Models: Choose from reactive or virtual thread-based persistence
• Modular Design: Clear separation between runtime and deployment components
• Auto-Generation: Generates necessary infrastructure at build time
• Observability: Built-in metrics, tracing, and logging support
• Error Handling: Comprehensive error handling with DLQ support
• Backpressure Management: Reactive processing with configurable backpressure strategies

Getting Started ​

New to The Pipeline Framework? Start with our Quick Start guide to learn the basics using the visual Canvas designer.

Guides ​

To get started with The Pipeline Framework, explore these guides:

Build Fast ​

• Quick Start: Create a pipeline with the Canvas
• Business Value: Speed, ROI, and portability
• Canvas Designer Guide: Deep dive into the visual designer

Design ​

• Application Structure: Structuring pipeline applications
• Common Module Structure: Shared components and type mappings
• Expansion and Reduction: Cardinality explained for imperative developers

Build ​

• Pipeline Compilation: How build-time generation works
• Configuration Reference: Build-time and runtime settings
• Dependency Management: Manage build-time and runtime deps
• Best Practices: Operational and design guidance

Develop ​

• @PipelineStep Annotation: Annotation contract and parameters
• Code a Step: Implement a step and its mappers
• Using Plugins: Apply plugins to pipelines
• Mappers and DTOs: Type conversions and mappings
• Dependency Management: Manage build-time and runtime deps
• Upgrade Guide: Version changes and migrations
• Orchestrator Runtime: Coordinate pipeline execution

Observe ​

• Observability Overview: Metrics, tracing, logs, and security notes
• Metrics: Instrumentation and dashboards
• Tracing: Distributed tracing and context propagation
• Logging: Structured logging and levels
• Health Checks: Liveness and readiness
• Alerting: Alerts and noise reduction
• Security Notes: Protect telemetry data
• Error Handling & DLQ: Managing errors and dead letter queues

Extend ​

• Writing a Plugin: Build your own aspects
• Orchestrator Extensions: Customize orchestration flows
• Reactive Service Extensions: Wrap or adapt process() behavior
• Client Step Extensions: Customize client-side calls
• REST Resource Extensions: Extend generated REST resources

Evolve ​

• Functional Architecture: Architectural decisions and trade-offs
• Annotation Processor Architecture: Build-time IR, bindings, and renderers
• Plugins Architecture: Cross-cutting behavior model
• Aspect Semantics: Aspect expansion rules
• Aspect Ordering: Ordering guarantees and constraints
• Aspect Warnings: Known limitations and caveats
• Reference Implementation: End-to-end example and rationale
• Template Generator (Reference): Automation/CI usage
• Publishing: Release and publishing workflow
• CI Guidelines: Build validation and automation
• Testing Guidelines: Coverage and test strategy
• Gotchas & Pitfalls: Known sharp edges
• Proto Descriptor Integration: Descriptor generation and troubleshooting

This approach reduces boilerplate code and ensures consistency across your pipeline steps.