# Data pipelines

This reference covers the usage of data pipelines in TypeQL. Data pipeline read and modify data in stages.

## [](#pipeline_kinds)Pipeline kinds

*   A **write pipeline** may comprise stages that modify data in the database (such as `insert` or `delete`).
    
*   A **read pipeline** only comprises stages that read data from the database but do not modify it.
    

## [](#combining_stages)Combining stages

Each stage receives the stream of rows output by the previous stage, and only the variables in those rows are available to it. Stages may be chained in any order permitted by their kind, and a stage kind may appear more than once — for instance, a `match` may follow a `reduce`:

```typeql
match
  $user isa user;
  friendship($user, $friend);
reduce $friend-count = count groupby $user;
match
  $friend-count >= 2;
  $user has username $username;
select $username, $friend-count;
```

Here the second `match` sees only `$user` and `$friend-count`, the output of the `reduce`. See [Using reduced values in later stages](reduce/index.md#_using_reduced_values_in_later_stages).

## [](#_data_manipulation_stages)Data manipulation stages

[Match stage](match/index.md)

Used to match existing data instances in the database.

[Fetch stage](fetch/index.md)

Used to format the output of a data pipeline into JSON.

[Insert stage](insert/index.md)

Used to add new data instances into the database.

[Delete stage](delete/index.md)

Used to remove existing data instances from the database.

[Update stage](update/index.md)

Used to modify existing data instances in the database.

[Put stage](put/index.md)

Used to add new data instances to the database if they do not already exist.

## [](#_stream_manipulation_stages)Stream manipulation stages

[Select operator](select/index.md)

Used to keep specified variables for each element of the data stream and remove the rest.

[Require operator](require/index.md)

Used to remove elements from the data stream that do not contain specified optional variables.

[Distinct operator](distinct/index.md)

Used to remove duplicate elements from the data stream.

[Sort operator](sort/index.md)

Used to order the elements of the data stream based on the value of specified variables.

[Limit operator](limit/index.md)

Used to keep a specified number of elements of the data stream and remove the rest.

[Offset operator](offset/index.md)

Used to remove a specified number of elements from the data stream.

[Reduce operator](reduce/index.md)

Used to perform reduction operations on the data stream according to specified groupings.

## [](#_preamble_stage)Preamble stage

[Preamble](with/index.md)

Used to define functions on an ad-hoc basis for use in the data pipeline.

## [](#_query_terminator)Query terminator

[End clause](end/index.md)

When writing multiple queries together (for example in TypeQL console scripts), used to explicitly mark the end of a query.

[Redefine](../schema/redefine/index.md) [Match](match/index.md)

[Edit on GitHub](https://github.com/typedb/typedb-docs/edit/3.x-development/typeql-reference/modules/ROOT/pages/pipelines/index.adoc) Edit this page on GitHub.