Role: Lead PySpark Engineer
Job type: Fulltime
Location: Owings Mill, MD
Roles & Responsibilities:
10+ years of experience in big data and distributed computing.
• Very Strong hands-on experience with PySpark, Python, Airflow, DBT & DWH concepts
• Strong Hands on experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.).
• Proficiency in data modeling and ETL workflows.
• Proficiency with workflow schedulers like Airflow
• Hands on experience with AWS cloud-based data platforms.
• Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus.
• Strong problem-solving skills and ability to lead a team
Develop, test and maintain high-quality solutions using PySpark /Python programming language.
· Participate in the entire software development lifecycle, building, testing and delivering high-quality
data pipelines.
· Collaborate with cross-functional teams to identify and solve complex problems.
· Write clean and reusable code that can be easily maintained and scaled.
· Keep up to date with emerging trends and technologies in Python development. Design and develop
business controls using AWS Glue and PySpark
· Integrate data from various sources into Amazon Redshift
· Optimize performance of data processing jobs and fine tune queries
Salary Range: $110,000 - $115,000 a Year