К содержимому

Lead Architect – Databricks & Unity Catalog Governance

XPERTON LLCУдалённо · Remote

The Lead Architect owns the technical approach end to end: discovery across 90+ workspaces, the target governance design for Unity Gateway, the pilot, and the wave plan for full rollout. This person is the client's senior technical counterpart and works directly with platform leadership and the Databricks account team.

Responsibilities

  • Lead discovery of legacy AI usage across all workspaces using system.serving usage tables; classify each workspace by migration complexity (start fresh, simple swap, complex).
  • Design the target Unity Gateway governance model: catalog and schema layout for model services and external providers, grant model (EXECUTE, USE CATALOG/SCHEMA, CREATE SERVICE), budgets, rate limits, guardrails, inference tables, and usage tracking.
  • Map each legacy workload to its target: pay-per-token endpoints to system.ai model services, provisioned throughput to referenced model services, external models (OpenAI, Azure OpenAI, Anthropic) to external model providers.
  • Identify blockers and design workarounds: AI/Vector Search dependencies, Knowledge Assistant and Multi-Agent Supervisor, ai_query limits, custom agents that must move to Databricks Apps.
  • Define the cutover method: per-endpoint traffic drain, zero-traffic validation, and per-workspace enforcement with rollback.
  • Run the 3–5 workspace pilot and turn results into a runbook, wave plan, and Phase 2 estimate.
  • Present findings and recommendations to client platform, security, and engineering leadership; coordinate with the Databricks account team on enforcement settings and feature gaps.
  • Set technical direction for the Platform Engineer and, in Phase 2, a team of 4–5 engineers.

Required qualifications

  • 10+ years in data platform engineering or architecture; 4+ years hands-on with Databricks at enterprise scale (dozens of workspaces).
  • Deep Unity Catalog expertise: metastore design, privilege model, governed tags/ABAC, system tables, cross-workspace governance.
  • Hands-on experience with Databricks Model Serving, Foundation Model APIs, external models, and AI Gateway (legacy) or Unity Gateway.
  • Working knowledge of LLM application patterns: OpenAI-compatible SDKs, ai_query, agents, RAG/vector search, MCP.
  • Experience leading a multi-workspace or multi-tenant platform migration with phased cutover.
  • Strong SQL and Python; able to write the discovery queries and automation personally.
  • Proven client-facing communication with senior engineering and security stakeholders.
  • US-based; able to start by Dec 1, 2026.

Preferred

  • Databricks Certified Data Engineer Professional, Generative AI Engineer Associate, or Platform Architect accreditation.
  • Experience in professional services, financial services, or other environments with confidential client data and audit requirements.
  • Familiarity with AI governance frameworks (NIST AI RMF, EU AI Act) and cost chargeback models for AI usage.
  • Experience on more than one cloud (AWS, Azure, GCP).

Engagement terms: Dec 2026 – June 2027 Remote, US only.

Job Type: Contract

Pay: $70.00 - $100.00 per hour

Work Location: Remote

openqareer не кадровое агентство: вакансия собрана с открытого источника, отклик подаётся на сайте работодателя.