Azure Data Lake as a source and as the delivery layer of your data warehouse
Data from Azure Data Lake
Azure Data Lake Storage Gen2 plays two roles in Yres. As a source, you connect a Data Lake through the Azure Blob Storage source: account name, container (also called a filesystem in a Data Lake) and a SAS token with read and list rights. The wizard has no separate Azure Data Lake option. As a destination, Yres can write every loaded table to your environment's Data Lake as Parquet, next to the Azure SQL database: one file per run holding only the changed rows.
At a glance
- Data Lake as a source
- Add it as an Azure Blob Storage source, with a SAS token on the account's blob address. No separate option in the wizard
- What makes it a Data Lake
- An Azure Storage account with hierarchical namespace enabled; without that setting it is an ordinary Blob account
- Input
- Storage account name, container or filesystem, and a SAS token with at least read and list rights
- Data Lake as a destination
- Optional: also write loaded data as Parquet. Since version 1.56 as an append-only change feed
- Contents of the change feed
- One file per run with only changed rows, each marked as insert, update or delete. A run without changes writes no file
- Source of truth
- The history in the Azure SQL database remains leading; the files in the Data Lake are derived from it
- Where it runs
- On the Azure Data Factory cloud runtime. The SAS address is kept in the Key Vault of your own environment, never in the web application
What you arrange yourself
- 1Check in the Azure Portal that the storage account is a Data Lake: Settings → Configuration → Hierarchical namespace must be Enabled.
- 2Look up the name of the container (the filesystem) that holds your files under Containers.
- 3Create a SAS token under Security + networking → Shared access signature with the Blob service, read and list rights and an expiry date, and save it immediately.
- 4Add a source of type Azure Blob Storage in Yres with the account name, container name and SAS token.
When you need this — and when you don't
A standalone connector is enough if…
If your team already works entirely in a lakehouse with notebooks and Delta tables and has no need for a SQL data warehouse, Yres adds little for reading that lake.
Yres adds value if…
Yres earns its place in two situations. Other systems drop files into the Data Lake and you want them as tables with history in Power BI. Or you want to load your ERP and SaaS sources once and have them available both in SQL and in Parquet, for a bronze, silver and gold layout for instance.
Frequently asked questions: Azure Data Lake
Why is Azure Data Lake not in the list of sources in Yres?
Because a Data Lake Gen2 can also be read through the storage account's blob address. You therefore add the lake as an Azure Blob Storage source, with the same three details: account name, container and SAS token. The storage Yres itself uses to write Parquet is a fixed, internal connection and is separate from the source you add.
What exactly does Yres write to the Data Lake?
Since version 1.56, one Parquet file per run per table containing only that run's changes. Every row carries a marker for insert, update or delete, plus a hash of the key, a hash of the row and the moment the version took effect. The files are organised in folders per source, schema, table, year and month.
Can I use the Parquet files to update Delta tables?
Yes, that is what the design is aimed at. Because each file holds only changes, including explicit deletes, you can use it directly as input for a MERGE into a Delta table. From all files together you derive both the current state and the full history.
I was already reading the old files in the Data Lake. Does anything change?
Yes. Before version 1.56 every run wrote out the entire staging table. The new change feed writes to a different path and each file contains changes only. The old files stay where they are; nothing is moved or cleaned up. Reports or notebooks that relied on a full snapshot per file need to be adjusted.
Full technical description in the knowledge base →·Last reviewed:
Data warehouse automation that runs in your own Azure
Built for organisations on Azure and Power BI. Yres connects your sources, keeps the history and promotes changes in a controlled way, without manual work.