Lead Data Engineer/architect

Direct message the job poster from Impetus

Manager-Talent Acquisition at Impetus || US Hiring || Big Data|| Cloud-AWS/GCP|| Fulltime/Contracting|| Vendor Management || Product & Services

Job Description

We are hiring an experienced Lead Data Engineer/Architect with good experience with Databricks.

Roles and Responsibilities:

• Overall 10-18 yrs. of Data Engineering experience with Minimum 2+ years of hands on experience in Databricks. Ready to travel Onsite and work from client location.
• Proven experience as a Databricks Architect or similar role with a deep understanding of the Databricks platform and its capabilities.
• Analyze business requirements and translate them into technical specifications for data pipelines, data lakes, and analytical processes on the Databricks platform.
• Design and architect end-to-end data solutions, including data ingestion, storage, transformation, and presentation layers, to meet business needs and performance requirements.
• Lead the setup, configuration, and optimization of Databricks clusters, workspaces, and jobs to ensure the platform operates efficiently and meets performance benchmarks.
• Manage access controls and security configurations to ensure data privacy and compliance.
• Design and implement data integration processes, ETL workflows, and data pipelines to extract, transform, and load data from various sources into the Databricks platform.
• Optimize ETL processes to achieve high data quality and reduce latency.
• Monitor and optimize query performance and overall platform performance to ensure efficient execution of analytical queries and data processing jobs.
• Identify and resolve performance bottlenecks in the Databricks environment.
• Establish and enforce best practices, standards, and guidelines for Databricks development, ensuring data quality, consistency, and maintainability.
• Implement data governance and data lineage processes to ensure data accuracy and traceability.
• Mentor and train team members on Databricks best practices, features, and capabilities.
• Conduct knowledge-sharing sessions and workshops to foster a data-driven culture within the organization.
• Will be responsible for Databricks Practice Technical/Partnership initiatives.
• Build skills in technical areas which support the deployment and integration of Databricks-based solutions to complete customer projects.
• Databricks Certified Data Engineer Associate/Professional Certification (Desirable).

Required Qualifications:

• Bachelor's or Master’s degree in Computer Science, Information Technology, or related field.
• Experience handling structured and unstructured datasets
• Strong proficiency in programming languages like Python, Scala, or SQL.
• Experience with Cloud platforms like AWS, Azure, or Google Cloud, and understanding of cloud-based data storage and computing services.
• Familiarity with big data technologies like Apache Spark, Hadoop, and data lake architectures.
• Develop and maintain data pipelines, ETL workflows, and analytical processes on the Databricks platform.
• Should have good experience in Data Engineering in Databricks Batch process and Streaming
• Should have good experience in creating Workflows & Scheduling the pipelines.
• Should have good exposure on how to make packages or libraries available in DB.
• Familiarity in Databricks default runtimes

Impetus Technologies is a software products and services company focused on creating powerful and intelligent enterprises through deep data awareness, data integration, and advanced data analytics. Our products and services are designed to empower the real-time data-driven enterprise, to help our clients win in the modern world of digital transformation.

Impetus has won several awards and creditable and global organizations have recognized and appreciated our passion for technology and great work culture. Our employees are our greatest assets. We are currently expanding our talent pool; those who’d like to add to their experience of working on cutting-edge technologies in the global IT industry today are welcome to join us.

Seniority level

• Seniority level Mid-Senior level

Employment type

• Employment type Contract

Job function

• Job function Information Technology
• Industries IT Services and IT Consulting, Software Development, and IT System Data Services

Referrals increase your chances of interviewing at Impetus by 2x

Sign in to set job alerts for “Data Engineer” roles. Data Engineer, 2025/2026 Intern Australia & New Zealand

Sydney, New South Wales, Australia 2 weeks ago

Senior Software Engineer (Data Platforms)

Melbourne, Victoria, Australia 2 weeks ago

Australia A$170,000.00-A$180,000.00 1 week ago

Data Engineer/Data Analyst with Datastage Software Engineer, 2025/2026 Intern Australia & New Zealand

Sydney, New South Wales, Australia 2 months ago

Melbourne, Victoria, Australia 2 weeks ago

Senior Software Engineer - (Python / Go) - Remote

Sydney, New South Wales, Australia 1 hour ago

Senior Software Engineer - (Python / Go) - Remote

Sydney, New South Wales, Australia 3 weeks ago

Melbourne, Victoria, Australia 9 hours ago

Sydney, New South Wales, Australia A$140,000.00-A$165,000.00 3 weeks ago

Melbourne, Victoria, Australia 1 month ago

Python and Kubernetes Software Engineer - Data, AI/ML & Analytics

Sydney, New South Wales, Australia 3 months ago

Python and Kubernetes Software Engineer - Data, AI/ML & Analytics

Perth, Western Australia, Australia 3 months ago

Senior Software Engineer (Python / Go) - Remote

Sydney, New South Wales, Australia 2 days ago

Senior Software Engineer (Python / Go) - Remote

Sydney, New South Wales, Australia 2 days ago

Sydney, New South Wales, Australia 2 days ago

Python and Kubernetes Software Engineer - Data, AI/ML & Analytics

Adelaide, South Australia, Australia 3 months ago

Victoria, Australia A$180,000.00-A$230,000.00 1 day ago

Artificial Intelligence Developer / Data Scientist

Ballarat, Victoria, Australia 3 hours ago

Python and Kubernetes Software Engineer - Data, Workflows, AI/ML & Analytics

Melbourne, Victoria, Australia 1 month ago

Python and Kubernetes Software Engineer - Data, AI/ML & Analytics

Melbourne, Victoria, Australia 3 months ago

Sydney, New South Wales, Australia 1 day ago

Python and Kubernetes Software Engineer - Data, AI/ML & Analytics

Brisbane, Queensland, Australia 3 months ago

Melbourne, Victoria, Australia 1 week ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

#J-18808-Ljbffr