Senior DevOps Engineer

We are seeking a highly skilled Senior DevOps Engineer with strong expertise in Databricks platform operations, Infrastructure as Code, and DataOps automation to support enterprise-scale environments within highly regulated industries.This role focuses on automation-first engineering, platform governance, and building secure, scalable data infrastructure.Key ResponsibilitiesCI/CD & AutomationDesign and implement end-to-end CI/CD pipelines using GitHub Actions or Azure DevOps for seamless deployment of data assets.Automate Databricks workspace provisioning and environment setup across Dev, UAT, and Production environments.Automate Databricks workflows and job orchestration.Integrate automated testing (pytest, Spark testing frameworks) into deployment pipelines.Infrastructure as Code (IaC)Develop and maintain Infrastructure as Code using Terraform for scalable and consistent deployments.Implement zero-touch environment provisioning and multi-workspace automation.Establish cluster policies and compute governance standards.Governance, Security & ComplianceEnforce platform security and governance, including Unity Catalog and access controls.Implement fine-grained access control and secure data governance practices.Collaborate with security teams to ensure compliance with enterprise and financial regulations.Configure and manage:Private Link / VNet InjectionIP Access ListsSCIM and Identity Provider (IdP) integrationsObservability & Cost OptimizationImplement observability frameworks for monitoring, alerting, and performance tracking.Monitor platform usage and optimize cloud costs and DBU consumption.Leverage Databricks system tables and audit logs.Build real-time monitoring dashboards and usage guardrails.Experience Requirements3–6+ years of experience in DevOps or Platform Engineering.Minimum 1+ year of hands-on Databricks platform experience in enterprise environments.Proven experience working in large-scale regulated industries (banking, financial services, etc.).Strong exposure to multi-workspace automation and platform standardization.Required CertificationsDatabricks Certified Data Engineer Professional (Mandatory)HashiCorp Certified: Terraform Associate (Required)One of the following cloud certifications:Azure DevOps Engineer ExpertAWS DevOps Engineer – ProfessionalCore Technical SkillsInfrastructure as Code & AutomationAdvanced expertise in Terraform, including Databricks providerExperience with cluster policies and compute governanceCI/CD & DataOpsHands-on experience with Databricks Asset Bundles (DABs)Experience integrating automated testing into pipelinesGovernance & SecurityExperience implementing Unity Catalog and secure access modelsStrong knowledge of secure cloud networking and identity integrationsObservability & Cost ManagementExperience with monitoring frameworks, dashboards, and cost optimization strategies.Preferred QualificationsStrong understanding of Apache Spark internals for performance troubleshooting.Advanced Python scripting skills for automation and tooling.Experience in financial services or regulated environments.Strong understanding of secure cloud architecture and networking.