Databricks Administrator

What You'll Do

• Configure, maintain, and administer Azure Databricks workspaces and clusters across development, staging, and production environments. 
• Monitor cluster health, performance metrics, resource utilization, and platform availability. 
• Optimize cluster, SQL warehouse, and workspace configurations for performance and cost efficiency. 
• Manage workspace governance, including job scheduling, notebook organization, audit logging, and operational standards. 
• Administer user access, workspace permissions, SQL warehouse configurations, and group-based security controls. 
• Troubleshoot infrastructure issues, performance bottlenecks, connectivity problems, and workload failures. 
• Support disaster recovery and high-availability planning, validation, and operational readiness for Databricks workloads. 
• Collaborate with data engineering teams to establish Databricks usage standards, support workload reliability, troubleshoot platform issues, optimize Delta Lake performance, and promote governed platform adoption. 
• Administer and enforce security policies, RBAC, encryption standards, and access controls. 
• Maintain documentation and runbooks for operational procedures. 
• Support capacity planning, cost optimization, and platform improvement initiatives. 

Security Governance 

• Maintain security governance standards for Azure Databricks in a financial services environment, including RBAC, least-privilege access, privileged access controls, and group-based permissions. 
• Administer Unity Catalog permissions, metastore access, catalogs, schemas, tables, volumes, and external locations to enforce governed data access. 
• Manage workspace permissions, admin roles, service principals, groups, SCIM provisioning, and Microsoft Entra ID integration to support secure identity governance. 
• Configure and review compute policies, cluster permissions, instance pool access, SQL warehouse permissions, and job ownership to reduce unnecessary privilege and control platform usage. 
• Oversee secrets management, token usage, storage credential access, and Key Vault-backed configurations to protect sensitive financial, customer, and operational data. 
• Coordinate recurring access reviews, access certification, and timely remediation of excessive, inappropriate, or outdated permissions to support internal control requirements. 
• Review Databricks audit logs, security events, access changes, and control findings to identify risks, policy exceptions, segregation-of-duties concerns, and required remediation actions. 
• Support audit and regulatory readiness by maintaining evidence for access controls, configuration standards, policy enforcement, operational procedures, and change management activities. 
• Partner with cybersecurity, compliance, risk, infrastructure, and data governance teams to align Databricks controls with financial services security, privacy, and compliance expectations. 

What You'll Need

• 5+ years of experience administering cloud-based data platforms or analytics environments, with a focus on operational support, governance, and reliability. 
• 3+ years of hands-on experience administering Azure Databricks workspaces, Unity Catalog, clusters, jobs, SQL warehouses, and platform settings. 
• Strong experience managing Databricks users, groups, workspace permissions, admin roles, service principals, SCIM provisioning, and Microsoft Entra ID integration. 
• Experience administering Unity Catalog, including metastores, catalogs, schemas, tables, volumes, permissions, storage credentials, and external locations. 
• Ability to configure and maintain compute policies, cluster permissions, instance pools, SQL warehouses, job ownership, and platform usage controls. 
• Experience monitoring Databricks platform health, cluster performance, job execution, workload failures, audit logs, and resource utilization. 
• Strong troubleshooting skills for Databricks platform issues, Spark workloads, connectivity problems, permission errors, and performance bottlenecks. 
• Working knowledge of Python, SQL, PowerShell, Databricks CLI, REST APIs, and automation practices used to support platform administration. 
• Experience supporting operational controls, access reviews, audit evidence, change management, runbooks, and platform documentation. 
• Knowledge of Databricks security governance, RBAC, least privilege, encryption, secrets management, audit readiness, and secure data access patterns. 
• Understanding of Azure networking and security concepts, including private endpoints, firewalls, network security groups, storage access, Key Vault, and secure connectivity. 
• Experience partnering with data engineering teams to support reliable pipelines, workload troubleshooting, performance tuning, platform standards, and secure Databricks operations.
• Travel requirement: All roles require travel during the first 6 months of employment to attend the BHG Oars in Experience. After the initial 6 months, ongoing travel is expected at approximately ~5% annually for periodic team on-sites, team meetings, and cross-functional collaboration.
• Work Authorization: This role is not eligible for employer-provided immigration sponsorship. To be considered, applicants must already possess valid U.S. work authorization that will not require sponsorship now or in the future.

Technical Skills & Technologies 

• Databricks Platform Administration: Workspaces, workspace settings, admin console, clusters, jobs, notebooks, SQL warehouses, instance pools, compute policies, and audit logs. 
• Unity Catalog Administration: Metastores, catalogs, schemas, tables, volumes, permissions, storage credentials, external locations, and governed data access. 
• Identity and Access Management: Users, groups, service principals, workspace permissions, admin roles, RBAC, SCIM provisioning, Microsoft Entra ID integration, and least-privilege access controls. 
• Security and Governance: Secrets management, token usage, encryption, Key Vault-backed configurations, access reviews, audit evidence, policy enforcement, and regulatory readiness. 
• Monitoring and Troubleshooting: Cluster health, job failures, SQL warehouse performance, Spark workload issues, audit log review, resource utilization, connectivity issues, and performance bottlenecks. 
• Automation and Administration Tools: Databricks CLI, REST APIs, Terraform awareness, Python, PowerShell, SQL, runbooks, operational scripts, and platform documentation. 
• Azure Platform Integration: Azure Storage, Key Vault, private endpoints, virtual networks, firewalls, network security groups, Azure Monitor, Log Analytics, and secure connectivity patterns. 

Preferred Qualifications 

• Databricks Platform Administrator certification or equivalent hands-on Azure Databricks administration experience. 
• Experience administering Unity Catalog, workspace security, compute policies, SQL warehouses, service principals, audit logs, and platform governance controls. 
• Experience supporting Databricks in financial services or another regulated environment with strong audit, access review, and compliance requirements. 
• Experience creating runbooks, operational standards, monitoring procedures, access review processes, and administrator documentation for Databricks environments. 
• Background in database administration, cloud platform administration, or data platform operations supporting enterprise analytics workloads.