What is data warehouse automation?
Data warehouse automation is software that builds and maintains a data warehouse from metadata: you describe which sources and tables to load and how, and the tool generates the pipelines, the tables, the history and the monitoring. It replaces writing and maintaining load scripts by hand. The aim is not less data warehouse but less manual work: new sources in days rather than weeks, and an environment that behaves the same everywhere.
Last updated:
What exactly gets automated?
The repetitive part of data warehousing: fetching data, writing it, keeping history, handling errors and documenting all of it. That work is nearly identical for every table, which is exactly why it can be generated from metadata. What is not automated is the thinking: which figures the organisation needs and what a term like 'revenue' or 'active customer' means.
| Task | By hand | With automation |
|---|---|---|
| Connecting a source | Program connection, authentication and paging per source | Pick the source from a list, enter credentials |
| Loading tables | Write a pipeline and a load script per table | Tick the table, choose a load type; the pipeline is generated |
| Fetching only changes | Maintain and test a watermark per table | Point at a delta column |
| Keeping history | Build validity-period logic yourself (SCD2) | A standard part of the load process |
| Errors and recovery | Your own logging, your own restart procedures | Central logging, recover or roll back per load |
| Development, test, production | Move scripts across by hand | Changes travel through the environments as a package |
| Documentation and lineage | Falls behind as soon as someone is in a hurry | Follows from the same metadata, so always current |
When does it pay off, and when not?
Automation pays off once the number of sources and tables exceeds what one person can maintain by hand, or once the organisation has to be able to rely on the figures. In practice that point sits around three sources: from there on, maintaining separate connections costs more than building them did.
It does not pay off if you have one source and a handful of reports. A direct connection from Power BI is cheaper and quick enough. Nor does it pay off if your data warehouse consists mainly of unique, complex calculations: you will keep writing those yourself, with or without a tool.
- Yes: three or more sources, or one source with many tables and changes
- Yes: you need to look back at the state on an earlier date
- Yes: several report builders must work from the same figures
- Yes: the BI team spends more time on connections than on analysis
- Yes: an ERP migration is coming and reporting has to keep running
- No: one source, a few reports, no need for history
Which kinds of tools are there?
The market falls into five groups. They solve different problems; most confusion arises because all of them get called 'data integration' or 'ETL'. A point-by-point comparison of Yres, TimeXtender, AnalyticsCreator and building it yourself is on the comparison page, listed under 'Further reading' below.
| Kind | What it does | Examples |
|---|---|---|
| Data warehouse automation | Generates the whole data warehouse from metadata: loading, history, models, monitoring | AnalyticsCreator, TimeXtender, WhereScape, Yres |
| Data Vault automation | As above, but tied to the Data Vault 2.0 modelling method | VaultSpeed |
| Data ingestion (ELT) | Copies source data to a destination with ready-made connectors; the data warehouse on top you model and run yourself, with your own or additional transformation tools | Fivetran, Airbyte |
| Data platform | Storage, compute and tooling; you build on it yourself or with a tool on top | Microsoft Fabric, Azure Synapse, Snowflake, Databricks |
| Build it yourself | Not a product but a way of working: your own pipelines and transformations | Azure Data Factory with dbt or SQL |
Build it yourself or use a tool?
A tool's licence is rarely the largest cost; engineers' hours are. Building it yourself with Azure Data Factory and dbt or SQL carries no licence fee and gives complete freedom. Against that, every source, every history table and every recovery procedure costs engineering time, at build time and again whenever a source API changes.
So calculate the total cost over a few years: build hours, maintenance hours, the cost of standstill when the one person who understands it leaves, and the Azure bill. For technically strong teams with few sources, building it yourself can be the best choice. For organisations without a data platform team of their own, it rarely is.
With any tool, check what remains when you stop. Some tools run your data warehouse on their own execution layer that has to keep running; others place ordinary components in your own cloud environment that you can carry on with independently after you leave. Ask the vendor literally: what do I get when I cancel, and does loading keep running?
What should you look for when choosing?
Start from your sources and your people, not from the feature list. A tool that does not handle your ERP well, or that needs a specialist you do not have, solves nothing.
- Sources: is there a dedicated connection for your ERP, or do you assemble one yourself through a generic REST connector?
- Where the data lives: in your own cloud environment or at the vendor?
- Target platform: does it fit what you already have (Azure SQL, Fabric, Snowflake)?
- Modelling: do you want the tool to generate the data model for you, or would you rather build it yourself on a layer with full history?
- Lineage: at object level or at column level?
- Price: fixed and known upfront, or on request and volume-dependent?
- Dependency: what keeps working if you cancel the contract?
- Skills: can your current team work with it, or is a certified partner required?
Where does Yres fit in this picture?
Yres is a data warehouse automation platform for Microsoft Azure, made by Plainwater in the Netherlands. It generates Azure Data Factory pipelines from metadata and loads the data into an Azure SQL database in the customer's own Azure environment, with full history. What Yres produces are ordinary Azure components in your subscription. Your data and your history live there and remain yours. If you stop using Yres, your environment simply keeps running and your own data engineers can continue developing on it.
Yres deliberately chooses one fixed, proven way of loading and keeping history. It has dedicated connections for AFAS and Exact Online (official partner of both), reaches SAP through OData, and connects any other system through REST or OData. The price is fixed and public.
What Yres deliberately does not do is generate the data model for you. Modelling depends strongly on your own preferences and experience, and changes along with what your reporting tools need. So Yres delivers the layer with full history; the model on top, whether Kimball, Data Vault or something of your own, you build yourself with views and procedures that travel to production through changes. You lose no freedom, but you get no tooling for it. Beyond that, Yres does not generate a semantic model for Power BI, lineage is available at object level rather than column level, and it targets Azure SQL as its platform; Microsoft Fabric as a target platform is planned for the end of 2027. If that is decisive for you, a broader suite is the better fit.
Frequently asked questions
What is the difference between ETL and data warehouse automation?
ETL is the act: extracting, transforming and loading data. Data warehouse automation is a way of not programming that act per table but having it generated from metadata, including history, error handling and documentation. An ETL tool helps you build pipelines; an automation tool builds them for you.
Do I still need a data engineer if I automate?
Less, but not none. Connecting sources and loading tables no longer takes programming. Settling definitions, building the reporting layer and keeping an eye on cost remain skilled work. In practice the BI team's time shifts from connections to analysis.
Does Microsoft Fabric make data warehouse automation redundant?
No. Fabric is a platform: it provides storage, compute and a growing set of building blocks, such as incremental copy and database mirroring. The whole of pipelines, history and monitoring you still assemble and maintain yourself. Automation and Fabric complement each other; the real question is whether you want to maintain that construction yourself.
How long does it take to set up an automated data warehouse?
That depends mostly on the number of sources and on how quickly access to them is arranged. The technical part, setting up the environment and loading the first sources, is a matter of days with automation. The lead time sits in agreeing definitions and obtaining credentials from the administrators of the source systems.
Am I locked in to the vendor?
That differs per tool and is one of the most important questions to ask. Ask what keeps working if you cancel. Separate your data from the loading. If the data warehouse sits in your own cloud environment, the data, the history and the reports on top of it remain yours, even after you cancel. Whether new data keeps loading differs per vendor; ask. With Yres your environment simply keeps running after you leave, including loading new data, and your own data engineers can continue developing on it.
Further reading
- Yres and Microsoft Fabric: competitor or combination?
- A data foundation for AI: build freely on data that is right
- Yres compared with TimeXtender, AnalyticsCreator and building it yourself
- All connectors: AFAS, Exact Online, SAP and more
- What Yres can do: features
- Pricing
- Knowledge base: the seven load types
- Knowledge base: keeping history (SCD2)
- Knowledge base: the data flow from source to report
- Customer stories
