Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/connection-integration/data-integration/intro.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,12 @@ Apache Doris supports integration with a wide range of data ecosystem tools. You
link="./cloudcanal"
/>

<GettingStartedCard
title="OLake"
description="Use OLake Go to replicate operational databases into Apache Iceberg tables with CDC, then query them from Doris through an Iceberg Catalog"
link="./olake"
/>

<GettingStartedCard
title="DataX Doriswriter"
icon="/images/next/connection-integration/data-integration/logos/datax.jpg"
Expand Down
26 changes: 26 additions & 0 deletions docs/connection-integration/data-integration/olake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
{
"title": "OLake",
"language": "en",
"description": "OLake Go is an open-source EL (Extract-Load) platform that replicates operational databases into Apache Iceberg tables or Parquet files on Amazon S3. Apache Doris can query the Iceberg tables that OLake Go produces through an Iceberg Catalog."
}
---

[OLake Go](https://olake.io/) is a high-performance, open-source EL (Extract-Load) platform that bridges operational databases and open lakehouse storage, writing Apache Iceberg tables or Parquet files on Amazon S3. It supports Incremental Sync, Change Data Capture (CDC), and stateful, resumable syncs, and provides a web UI and CLI.

Sources include relational databases such as PostgreSQL and MySQL, document stores such as MongoDB, as well as Kafka and S3. Data is written to Apache Iceberg tables on object storage and registered in an Iceberg catalog such as AWS Glue, Hive Metastore, or a REST catalog. See the [OLake documentation](https://olake.io/docs/) for the full list of supported sources and catalogs.

## Apache Doris Integration

Apache Doris reads Iceberg tables through an Iceberg Catalog. To query data replicated by OLake Go in Doris, run OLake Go with one of the supported Iceberg catalogs, then create an Iceberg catalog in Doris that points to it. Doris reads the replicated Iceberg tables directly from object storage without moving or duplicating data.

- Doris 3.x and 4.0 read the positional and equality delete files that OLake Go writes for CDC workloads.
- Deletion vectors require Apache Doris 4.1.0 or later.

For the full Doris setup, including supported catalogs and delete formats, see the [OLake Go + Apache Doris integration guide](https://olake.io/docs/integrations/doris/).

## Learn More

- [OLake official website](https://olake.io/)
- [OLake Go GitHub repository](https://github.com/datazip-inc/olake-go)
- [OLake documentation](https://olake.io/docs/)
4 changes: 4 additions & 0 deletions docs/lakehouse/best-practices/doris-iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ Query OK, 4 rows affected (0.25 sec)
{'status':'COMMITTED', 'txnId':'10088'}
```

:::tip
To continuously replicate operational databases (for example PostgreSQL or MySQL) into Iceberg tables queried by Doris, you can use [OLake Go](https://olake.io/), an open-source EL platform that writes to Apache Iceberg with CDC support. See the [OLake Go + Apache Doris integration guide](https://olake.io/docs/integrations/doris/).
:::

### 05 Data Query

- Simple query
Expand Down
4 changes: 4 additions & 0 deletions docs/lakehouse/catalogs/iceberg-catalog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ Users can access Iceberg tables using Hive Metastore as metadata through the Hiv
| Data Integration | Read Iceberg data and write it to internal Doris tables, or perform ZeroETL operations using the Doris computing engine. |
| Data Write-back | Process data from any Doris-supported data source and write it back to Iceberg table storage. |

:::tip
To continuously replicate operational databases into Iceberg tables that Doris queries, you can use [OLake Go](https://olake.io/), an open-source EL platform that writes to Apache Iceberg with CDC support. See the [OLake Go + Apache Doris integration guide](https://olake.io/docs/integrations/doris/).
:::

## Configuring Catalog

### Syntax
Expand Down
1 change: 1 addition & 0 deletions sidebars.ts
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,7 @@ const sidebars: SidebarsConfig = {
'connection-integration/data-integration/logstash',
'connection-integration/data-integration/loongcollector',
'connection-integration/data-integration/metabase',
'connection-integration/data-integration/olake',
'connection-integration/data-integration/opentelemetry',
'connection-integration/data-integration/powerbi',
'connection-integration/data-integration/quickbi',
Expand Down