Data Engineer
Tata Consultancy Services
Job Description
AWS Data Engineer – Job Description Mandatory Role: AWS Data Engineer Required Technical Skill Set: Python, PySpark, AWS Glue, AWS Lambda, Amazon S3, Redshift, Athena, EMR, DynamoDB, Kinesis, IAM, CI/CD, Databricks Desired Experience Range: 6–8 Years Location of Requirement: Pan India Desired Competencies Must-Have Strong proficiency in Python and PySpark for large‑scale distributed data processing. Experience building data ingestion pipelines using AWS Glue, Kinesis, Lambda, API-based ingestion. Experience with Glue ETL, Spark on EMR, or Databricks on AWS.
Expertise with Amazon S3, Redshift, DynamoDB, Athena. Strong understanding of ETL/ELT, dimensional modeling, data lakes, and warehouse architecture. Good-to-Have Experience with DevOps & Automation using Git, AWS CodePipeline, CloudWatch, CI/CD.
Performance tuning of Spark jobs, Glue jobs, and Redshift queries. Knowledge of data governance, quality checks, and cataloging using Glue Catalog or Lake Formation. Key Responsibilities Design, develop, and maintain ETL/ELT pipelines using AWS Glue, Lambda, Spark, and Databricks.
Build scalable data lake and data warehouse solutions using S3, Redshift, and Athena. Orchestrate workflows using Glue Workflows, Step Functions, Airflow, or Databricks Workflows. Integrate AWS services such as S3, Glue, Lambda, Redshift, Kinesis, and DynamoDB.
Optimize pipelines for performance, reliability, and cost. Implement data quality checks, cataloging, lineage, and security best practices. Collaborate with business and analytics teams to translate data requirements. er