06.11.26

Shortlisting a Databricks Engineer for a Private Healthcare Azure Migration

Overview

When a Manchester-based data consultancy approached us about resourcing a Databricks-to-Unity-Catalog migration for a private healthcare client, the brief looked, on paper, similar to the financial services mandates that make up a large share of our contract desk. It wasn't. We shortlisted O'Brian Richards, a data engineer working full-time in a private healthcare company's Azure Databricks environment, as he began exploring a move from permanent employment into contracting. The screening conversation surfaced a candidate with hands-on migration experience across the exact architecture the client needed — but it also forced us to apply a different set of technical and sector checks than we'd use for a bank or an insurer. The result was a validated shortlist candidate whose migration experience matched the brief point-for-point, built on a screening approach that treats healthcare data sensitivity as a first-order concern rather than a footnote.

The Challenge

Healthcare migrations carry a different risk profile to financial services ones, even when the underlying technology stack looks identical. A Databricks-Unity-Catalog project for a bank raises questions about regulatory reporting and transaction integrity; the same project for a private healthcare provider raises questions about patient data handling, consumption-layer retraining for clinical and operational users, and the practical reality of running a complex migration on a small tech team. Our job was to find out whether O'Brian's background held up against that specific risk profile — not just whether he knew Databricks.

O'Brian was a strong candidate on paper: full-time data engineer and integration engineer at a private healthcare company, working mainly within the Azure Databricks ecosystem, with a CV sourced via CV Library that aligned closely with client requirements for upcoming data engineering roles. He described his current employer's project as "near the end of a migration project, which is just moving away from our legacy systems" — built around SQL Developer, third-party integration tools like Node-RED, and Snowflake, all being unified into Databricks, alongside a parallel move of the consumption layer from Power BI into ThoughtSpot. That is a materially different challenge from a single-platform lift-and-shift: it's a multi-system consolidation plus a user-facing BI retraining exercise, running in parallel, on what he described candidly as "a small tech team".

The additional wrinkle: O'Brian had never contracted before. He'd always been permanent, though he'd worked alongside contractors at both his current employer and at JD Sports, and had been actively exploring the shift to contracting with several recruiters before we spoke to him. Screening a permanent-to-contract transition candidate for a regulated-sector migration meant validating not just his Databricks depth, but his readiness for the different delivery expectations, pace, and accountability of contract work.

The Approach

Our standard technical screen for Databricks migration roles starts with the architecture itself — what's being moved, from where, to where, and why. For healthcare-sector candidates, we layer in questions that surface how they've handled data flow and system complexity in an environment where the client relationship (clinical operations, patient-linked data, regulatory obligation to the sector) differs from a banking or insurance mandate. The questions aren't about "do you know Databricks" — they're about whether the candidate has actually carried a multifaceted migration end-to-end, including the parts that get hardest in a small team: integration sprawl, consumption-layer retraining, and keeping legacy and new systems running in parallel without disruption to a live operational business.

With O'Brian, that meant asking him to walk through the specifics of what was moving and what it was unifying into. His answer confirmed genuine hands-on depth: "Databricks is the cloud, that's where we're going to do all our ETL transformations, etc., and then create our gold tables to consume. That's obviously underpinned by Azure... everything there, Spark SQL is within there". He was able to describe the legacy environment precisely — SQL Server hosted in Azure — and the peripheral connections still in play during the transition, including BigQuery and Snowflake feeds that the company intended to eventually fold into the unified Azure Databricks environment. That level of specificity, delivered without prompting, is what separates a candidate who has lived inside a migration from one who has merely read about one.

We also probed for transferable migration pattern-matching rather than narrow tool familiarity. O'Brian volunteered that he'd run an analogous migration previously at JD Sports — moving from Oracle into GCP and simultaneously changing the consumption layer from Looker and Tableau toward ThoughtSpot and Power BI. That's a meaningful signal for a Head of Data assessing contract readiness: this wasn't a one-off project exposure, it was a repeatable competency across two different employers and two different cloud ecosystems.

Implementation

The screening call followed a structured arc designed to validate experience, then test sector fit, then assess contract readiness — in that order.

Step one: validate the technical migration claim. We asked O'Brian to describe his current project in detail before asking anything about his CV headlines. His unprompted answer — legacy SQL Developer plus Node-RED plus Snowflake, unifying into Databricks, with a parallel ThoughtSpot consumption-layer migration — gave us confidence the Databricks experience on his CV was substantive, not CV padding.

Step two: confirm sector-specific flow understanding. Rather than asking generic "have you worked in healthcare" questions, we asked him to describe the full tech stack and where data was currently flowing from and to. He confirmed the whole stack sits in Azure, with the SQL Server legacy system itself hosted in Azure, and explained the company's intent to eventually unify the BigQuery and Snowflake connections into Databricks once the migration stabilizes. This told us he understood not just his piece of the pipeline, but the end-state architecture the business was working toward — essential in a regulated environment where data lineage and system consolidation decisions carry compliance weight.

Step three: screen for permanent-to-contract readiness. O'Brian was candid that this was a permanent role, not a contract, and that he'd never contracted before — but had deliberately started exploring the move, driven partly by working alongside contractors throughout his career and partly by conversations already underway with other recruiters. We treated this transparently rather than as a disqualifier: his stated motivation was proactive career development, not a forced exit, which matters when assessing whether a permanent-to-contract candidate will commit to contract-market realities.

Step four: cross-check sector relevance against the brief. We asked directly about financial services exposure, since our specialism sits heavily in contract recruitment for the data and AI market, often with financial-sector end clients. O'Brian's path ran through Lloyds Banking Group as a finance analyst before moving into data analytics, then data science, then engineering — giving him foundational exposure to financial data handling even though his most recent and most relevant migration experience sat in private healthcare. This is where sector-specific screening diverges most sharply from a financial-services-only mandate: a healthcare migration brief doesn't require banking regulation fluency, but a candidate who's touched both sectors brings pattern recognition that a purely single-sector career doesn't.

Results & Impact

The screening call produced a validated, shortlist-ready profile built on verified technical specifics rather than CV claims alone. O'Brian's description of the healthcare company's live migration — legacy SQL Developer and third-party integration tools consolidating into Databricks, with a simultaneous Power BI to ThoughtSpot consumption-layer switch — matched almost exactly the kind of multifaceted, multi-system migration that regulated-sector clients ask us to resource against. His prior JD Sports migration (Oracle to GCP, with a parallel BI tool change) gave us a second, independently verifiable data point confirming this wasn't a one-time project exposure but a repeatable skill.

Critically, the sector-specific screening surfaced a nuance that a generic technical screen would have missed: O'Brian's stack experience was 100% Azure-native, with the healthcare employer intending to consolidate even its BigQuery and Snowflake connections into that single environment — a detail directly relevant to any client running (or planning) a similar Unity Catalog consolidation, where understanding how a candidate has handled multi-source unification in practice is more valuable than a tools checklist.

His transition readiness was also validated rather than assumed. By being transparent that he'd never contracted, combined with a clear, proactive rationale for wanting to move into contracting and demonstrated exposure to working alongside contractors throughout his career, we had the information needed to position him honestly to a client rather than overselling contract-market readiness he hadn't yet proven.

Key Takeaways

Sector-specific screening for healthcare migrations needs to probe system consolidation and consumption-layer complexity, not just platform familiarity. O'Brian's value wasn't "he's used Databricks" — it was that he had lived through a live, multi-system migration (SQL Developer, Node-RED, Snowflake unifying into Databricks) with a simultaneous BI-tool retraining exercise, on a small team, which is precisely the operational pressure a Head of Data running a healthcare migration needs resourced against.

Permanent-to-contract transition candidates require a distinct, honest line of questioning. We didn't treat O'Brian's lack of contracting history as disqualifying — we treated his proactive, considered rationale for the move, combined with sustained exposure to contractors throughout his career, as the signal that mattered.

Cross-sector pattern-matching strengthens a shortlist even when the target sector is narrower. O'Brian's path from Lloyds Banking Group finance analyst into data analytics and engineering gave him a foundation that, while not healthcare-specific, demonstrated the kind of sector-agnostic data flow and regulatory awareness that transfers across regulated verticals.

Unprompted technical specificity is the single best screening signal we have. Across our migration-focused screening calls — whether the end client sits in healthcare, financial services, or elsewhere — the candidates worth shortlisting are the ones who can describe their architecture, their legacy system, and their consumption layer without needing the question reworded.

If you're running a Databricks or Unity Catalog migration in a regulated, data-sensitive environment and need a contractor or permanent engineer screened against your specific sector's risk profile — not a generic Databricks checklist — talk to our data and AI team.