We'll start with a statement that might sting a little: if your data platform still needs heroics to deliver, govern, or just keep running, it's not a platform.
Technical product owners, data engineers, data scientists — you know exactly what this looks like. The "can we trust this dataset?" conversation you've had four times this quarter. Cost spikes that nobody can explain. AI experiments that look great in a notebook and fall apart the moment someone tries to put them in production.
We work daily across Microsoft Fabric, Databricks, and Snowflake. This blog (and our upcoming webinar) is not a feature comparison. It's not a "one platform for everyone" pitch either. Different constraints demand different designs. But here's the thing: the hard problems — trust, cost, change management — aren't platform-specific. And the patterns behind successful platforms? Remarkably consistent.
Most platform failures aren't dramatic. They're slow. A shortcut here, an exception there, a "temporary" pipeline that quietly becomes mission critical. Six months later you've got a Lakehouse full of tables, three different ways to calculate revenue, and a governance story that lives in a slide deck somewhere and zero assets being governed according the design.
This is what we see across platforms when teams try to scale from "first use case" to "enterprise backbone". If you recognise two or three of these, you'll understand why we're running this session.
Decision drivers that engineers can actually implement
"Be agile" is not a decision driver. Neither is "avoid lock-in" — not unless you've defined what portability actually means for your workloads. Good drivers translate into things you can enforce: acceptable latency, retention policies, data residency, encryption requirements, compliance frameworks like GDPR and the AI Act, allowed languages and runtimes, and the unit of ownership (domain, product, team). The test is simple: if you can't encode it into templates, policies, and CI checks, it will eventually degrade into tribal knowledge. And tribal knowledge doesn't scale.
Architecture that doesn't punish change
Lakehouse or warehouse, centralised or federated, medallion layers within each — pick your patterns, but make sure they're operable and actually mapped to how your organisation works. The red flags are always the same: no clear layering, inconsistent naming, transformations scattered across notebooks and ad-hoc SQL, environment drift between dev and prod. Or sometimes just massive overkill for what the organisation actually needs right now.
Good platforms standardise the boring stuff. Raw/bronze, clean/silver, curated/gold. Transformations that are modular, version-controlled, and idempotent — safe to re-run without side effects. Lineage that's visible across the full path. And something that often gets overlooked: developer experience. If the inner loop — local dev, testing, CI feedback — is slow or painful, engineers will find ways around the platform instead of building on it. That's not a people problem. That's a platform problem.
Trust, governance, and security by design (not by committee)
If your data quality checks aren't in the pipeline, they're not going to happen. And if getting access means "ask the admin and wait", your self-service story is already dead.
The practical baseline looks like this: dataset ownership, data contracts, automated tests, lineage, observability (pipeline monitoring, freshness SLOs, alerting), and access control that actually scales. What do we mean by data contract? It's the schema, quality expectations, SLAs, and ownership of a shared dataset, and it lives in version control — not in a wiki page that nobody updates. For access control, that means role-based and attribute-based access, least privilege, separation of duties. And governance shouldn't be aspirational. Make it executable: policies-as-code, automated data quality checks, PII scanning.... If it's not automated, it's not governance — it's good intentions.
AI readiness is a data problem before it's a model problem
AI teams don't need "more data". What they need is data that's predictable, well-described, legally usable, and semantically stable. They need reproducible feature pipelines, experiment tracking, and an actual path to production — what people call MLOps, though the label matters less than having one.
With the EU AI Act and data sovereignty now firmly in scope, there's more to think about: where do data and model artefacts live? Who can access them? How do you monitor usage and drift over time? If you can't answer "where did this training data come from?" with confidence, you're not ready. Full stop.
Cost control is an engineering responsibility
Consumption-based pricing is unforgiving. Your platform should make waste hard, not easy: right-sized compute, workload isolation, guardrails, and cost visibility per domain or data product. And that starts with something basic — choosing the right compute model per workload. Serverless versus provisioned, auto-suspend policies, capacity reservations. The default configuration is rarely the cheapest one.
The typical failure mode? "Shared everything" plus no ownership. That turns every cost conversation into politics. The fix is an operating model backed by actual technical controls: budgets, quotas, alerts, resource tagging, and standardised deployment patterns. Make cost somebody's problem, and give them the levers to manage it.
There's a thread running through all five challenges, and it's ownership. Who owns this dataset? Who defined the contract? Who's accountable when the cost spikes or the quality drops?
When you start treating data as a product — with an owner, known consumers, an SLA, and a lifecycle — these challenges become manageable. Not solved, manageable. Because ownership doesn't fix things directly; it creates accountability where decisions actually need to happen. Without it, every problem drifts into nobody's responsibility.
Now, this does assume domain teams have the engineering capability to own their data. That's a maturity step in itself, and not every organisation is there yet. But you don't have to go all-in on day one. Start with your highest-value shared datasets. Once the conversation shifts from "who broke this?" to "who's accountable for this?", you're moving in the right direction.
A mid-sized European organisation — regulated, multiple business units — had built a lakehouse-style environment that worked fine for the first few domains. Then two years in, things started to slow down. Every new source needed bespoke work. Incidents crept up. And nobody could confidently explain why numbers looked different across dashboards.
The turning point wasn't a re-platform. It was something much less glamorous: the team agreed on a small set of decision drivers (latency tiers, retention, residency, ownership) and then actually enforced them. Standard raw/clean/curated layers. Contracts for shared datasets. Automated tests in every pipeline. A release process that treated data changes like software changes.
Onboarding became template-driven: new sources followed the same naming, folder structure, and orchestration conventions. No more copy/paste pipelines. What used to take weeks dropped to days.
Trust became visible: key datasets got an owner, a contract, tests, lineage. Access moved from "email the admin" to governed self-service.
Promotion became boring (in a good way): dev/test/prod separation, approvals, reproducible deployments. Fewer "works on my workspace" surprises.
Cost discussions became actionable: tagging and budget guardrails made chargeback per domain possible. Runaway workloads became easy to spot. Attribution went from quarterly guesswork to weekly dashboards.
The hardest part, honestly, wasn't the tooling. It was getting domain teams to accept ownership of their data. Standardisation felt like overhead — until the first team onboarded a source in two days instead of two weeks. After that, it was pull, not push.
Different platform, same lesson: the teams that move fastest aren't the ones with the most features. They're the ones with the clearest rules and the most automation around them.
Want to stress-test whether your platform behaves like a product — or more like a pile of assets that grew organically? These are the questions we'd start with:
Can a team onboard a new source in days, using templates, without copy-pasting last year's notebook spaghetti?
Can you explain a dataset end-to-end — owner, contract, tests, lineage, access — without opening three different wikis?
Can a data scientist get governed access to the right data without a week of ticket ping-pong?
Can you promote changes safely through dev/test/prod with CI/CD, approvals, and reproducible deployments?
Can you attribute cost to a domain or data product, and actually take action when it drifts?
Do you know right now — not after a report next month — which pipelines are late or which datasets are stale?
In our experience, teams don't need more opinions. They need a shared set of principles and patterns that survive platform choice — and survive org change, too. That's what we'll cover in our live session The Data Platform Challenge: Making the Right Choices Beyond Technology, with three Cegeka experts who each go deep on a different platform but answer from one shared engineering viewpoint.
Bring your toughest questions. We’ll answer them.
Join our webinar The Data Platform Webinar to know more! Bring your toughest questions. We’ll answer them.