About the Company
We are seeking a dynamic professional for the Data Engineering role. The ideal candidate will be responsible for designing, developing, and maintaining the data architecture, infrastructure, and pipelines that enable the organization to collect, store, process, and analyze large volumes of data. This role is critical in ensuring data availability, quality, and accessibility to support data-driven decision-making.
About the Role
The Data Engineering role involves various key responsibilities that contribute to the overall data strategy of the organization.
Responsibilities
- Data Pipeline Development:
- Design, develop, and maintain data pipelines to ingest, process, and transform data from various sources.
- Implement data integration solutions connecting disparate systems, including databases, APIs, and third-party sources.
- Data Storage and Warehousing:
- Build and manage data storage solutions such as data lakes, data warehouses, and NoSQL databases.
- Optimize data storage for performance, scalability, and cost-efficiency.
- Data Quality and Governance:
- Define and enforce data quality standards, implementing validation and cleansing processes.
- Collaborate with data analysts and data scientists to ensure data consistency and accuracy.
- ETL (Extract, Transform, Load):
- Design and implement ETL workflows to transform raw data into structured, usable formats.
- Monitor, debug, and ensure smooth execution of ETL processes.
- Data Security and Compliance:
- Apply data security protocols and access controls to safeguard sensitive information.
- Ensure adherence to data privacy regulations and standards (e.g., GDPR, HIPAA).
- Performance Tuning:
- Optimize data pipelines and query performance.
- Identify and resolve processing bottlenecks and latency issues.
- Data Documentation:
- Maintain detailed documentation for data pipelines, schemas, and data dictionaries.
- Create and update metadata and data lineage documentation.
- Scalability and Reliability:
- Design scalable and resilient data architectures to support growing data volumes and business needs.
- Implement backup and disaster recovery strategies.
- Collaboration:
- Work closely with cross-functional teams including data scientists, analysts, and business stakeholders to gather data requirements and deliver appropriate solutions.
- Offer technical support and guidance to data users.
- Continuous Learning:
- Stay abreast of the latest tools, technologies, and trends in data engineering.
- Evaluate and adopt new technologies to improve data processing capabilities.
Qualifications
- Minimum 9 years of experience in data engineering or a related role.
- Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
Required Skills
- Expertise in ETL development, data integration, and pipeline orchestration tools (e.g., Apache NiFi, Apache Airflow).
- Strong understanding of SQL and NoSQL databases, data warehousing, and data modeling.
- Familiarity with big data frameworks (e.g., Hadoop, Spark) and cloud platforms (e.g., AWS, Azure, GCP).
- Proficient in programming languages like Python, SQL, Java, Scala, R, and/or Go.
- Hands-on experience with version control systems (e.g., Git) and data versioning techniques.
- Strong problem-solving, analytical, and troubleshooting skills.
- Excellent communication and collaboration abilities.
- Knowledge of data security practices and regulatory compliance standards.
Preferred Skills
- Experience with cloud data warehousing solutions (e.g., Snowflake, Redshift).
- Familiarity with data visualization tools (e.g., Tableau, Power BI).