Advanced data processing and orchestration
- Last UpdatedSep 29, 2026
- 2 minute read
A command pipeline is a series of commands that pull inputs and push outputs. An output is optional, but most pipelines must output something to be useful. The following concepts are used.
The following table lists the pipeline concepts.
|
Concepts |
Description |
|---|---|
|
Commands
|
A command is a source or and filter. A source pulls input, generates records and outputs to the next buffer. A filter performs some operation (for example, project data or write to sink) and passes output to the next buffer. |
|
Endpoints
|
An endpoint represents either a source input or sink (target) output – for example, a database or set of files. |
|
Buffers
|
A buffer is used to manage flow of data between commands and always feeds a filter (never a reader). |
A pipeline is a set of readers that pumps registers and records into the pipeline and a sequence of commands (operations) to apply to the records as they flow through the pipeline.
These concepts are then combined to form a pipeline, which uses sized buffers to control flow, as shown in the following diagram:
