Role
Data engineer
How quickly do you find out that a source has changed structure - from an alert, or from an analyst asking why the report is empty? That is the whole difference in this role.
Intro call
Four questions we would start with
How quickly do you find out about a change in a source?
If it is only when the pipeline stops or the data comes out empty, the response starts with an explanation rather than with a fix.
How do you know the output data is correct?
If the only test is that the pipeline ran, then correctness is an assumption.
What happens when a pipeline does not run on time?
Whether the recipients know about it before they open the report, or after.
What do you not know about a pipeline until it stops working?
Usually: what uses it and who would notice if it disappeared.
We take over the monitoring and the documenting, you keep the design of the pipelines
Detecting changes in sources, data quality checks, pipeline monitoring and documentation can be taken off the desk. Designing the processing and decisions about the data model stay with the engineer.
- A change in a source is a signal, not an outage. - A new column, a changed type or a missing field report themselves before they break the processing.
- Data quality is measured, not assumed. - Completeness, ranges and rules are checked on every run, and a deviation reaches the engineer before it reaches a report.
- It is known what uses what. - The dependencies between pipelines, tables and reports are visible, so the effect of a change can be judged before it is made.
Scope
What can be taken off data engineering
Twenty example tasks. Each one has a human approval point. Below is a selection, and the scope we fit to your data architecture.
Sources and their changes
- A register of sources with the owner and the structure
- Detecting a change in a source's structure
- A signal when a field has stopped being populated
- Comparing the volume with previous runs
Processing
- A draft transformation for repetitive patterns (assessment: the engineer)
- Pointing out pipelines with similar logic
- Reprocessing a range after a fix
- A register of runs with the time and the result
Data quality
- Quality rules at the input and the output
- A signal when a value departs from the expected range
- Pointing out rejected records with the reason
- Quality summarised by source (assessment: the engineer)
Pipeline monitoring
- Pipeline status with delivery deadlines
- A signal when a pipeline is late against what was agreed with the recipient
- Notifying recipients about a delay
- Failures summarised by cause
Data documentation
- The lineage of the data in an output table
- The dependencies between pipelines, tables and reports
- Pointing out tables and pipelines that are not used
- Field descriptions generated from the transformations (verification: the engineer)
A day's work
Today
- You learn about a change in a source when the pipeline falls over
- You check data quality when somebody reports an odd result
- A recipient discovers a pipeline is late by opening the report
- You know the dependencies between pipelines from memory
- Documentation exists for some of the pipelines
- You do not know which tables and pipelines are no longer used
With AI assistance
- A change in a source's structure reports itself before the failure
- The quality rules run on every run
- Recipients are told about a delay before they open the report
- The dependencies are visible, so the effect of a change can be judged
- The field descriptions are generated from the transformations, you verify them
- Unused pipelines and tables are pointed out
- Data on time
fewer reports opened on stale data
a delay is signalled to the recipients
- Detecting source changes
a change is a signal rather than an outage
source structures are compared on every run
- Output data quality
fewer wrong numbers reaching a decision
the rules run before publication
- A pipeline you can see into
it is known where a number in a report comes from
lineage and dependencies are recorded
We start with detecting changes in sources, because they cause most of the failures.
- Step 1
A register of sources and change detection
The structure, the owner, a signal on change and on fields that have stopped being populated.
- Step 2
Quality rules
Completeness, ranges and business rules at the input and the output.
- Step 3
Monitoring and notifying recipients
Statuses, delivery deadlines, information about a delay.
- Step 4
Lineage and dependencies
What is built from what and what uses what. Last, because it needs the pipelines to be in order.
Common questions
Questions that come up at the first meeting
How is this different from a data analyst?
The analyst answers questions with the data. The engineer is accountable for that data existing at all, on time and in good shape. One role works on the result, the other on the road to it.
Are fewer engineers enough for the pipelines?
The number of pipelines grows with every new source and report, and data teams rarely grow at that pace - hence the backlog and the pipelines nobody understands. Taking over the monitoring and the documentation replaces nobody; it makes it possible to maintain what already runs without giving up on building new things.
Does this fall under the AI Act?
Building data pipelines is not a high-risk use. If the pipelines cover personal data, the data protection rules apply - minimisation, a legal basis and retention - and it is worth having the register of sources show which pipelines concern it.
Still have a question that is not here? Book a call
Next step
How many times last quarter did a pipeline fall over because of a change in a source?
We will show you what working with data looks like when a change in a source is a signal and quality is checked before publication.