Role Summary:

We are seeking a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data platforms on AWS. The role involves building robust ETL/ELT workflows, managing data ingestion from multiple sources, ensuring data quality, and supporting analytics and machine learning use cases. The engineer will work closely with stakeholders, project managers, and vendors to deliver high-quality data solutions in a cloud-based environment.

Key Responsibilities:

🔹 Data Engineering & Pipeline Development

• Design, develop, and maintain end-to-end batch and near real-time data pipelines on AWS Data Platform.
• Build and manage ETL/ELT workflows using AWS services such as AWS Glue, S3, Redshift, Athena, EMR, DBT, and orchestration tools like Airflow.
• Implement data ingestion pipelines from multiple sources including databases, APIs, flat files, and streaming/event systems.

🔹 Data Processing & Transformation

• Develop data transformation logic using SQL, Python, and PySpark.
• Perform data cleansing, enrichment, aggregation, and standardization across datasets.
• Design and maintain data lake architecture including raw, curated, and processed data layers.

🔹 Data Quality & Monitoring

• Implement data validation, reconciliation, and data quality checks.
• Build monitoring and alerting systems to detect pipeline failures and anomalies.
• Ensure reliability, scalability, and performance of data workflows.

🔹 DataOps / DevOps Practices

• Follow CI/CD practices, version control, and automated testing for data pipelines.
• Support release management, deployment processes, and operational readiness.
• Maintain production support for data pipelines and workflows.

🔹 Collaboration & Stakeholder Engagement

• Work closely with data analysts, data scientists, and business stakeholders to enable analytics and ML use cases.
• Collaborate with project managers to define deliverables and timelines.
• Engage with third-party vendors, track deliverables, and ensure acceptance criteria are met.

🔹 Documentation & Governance

• Create and maintain technical documentation including:
  • Data flow diagrams
  • Data mappings
  • Job schedules
  • Runbooks and operational procedures
• Maintain a centralized repository of project artefacts.

🔹 Production Support & Optimization

• Support production deployments and resolve incidents.
• Perform performance tuning and optimization of data pipelines.
• Participate in defect triage and issue resolution.

Required Skills & Experience:

• Strong experience in AWS data services (Glue, S3, Redshift, Athena, EMR).
• Proficiency in SQL, Python, and PySpark.
• Experience with ETL/ELT pipeline development and data lake architecture.
• Hands-on experience with Airflow or similar orchestration tools.
• Knowledge of CI/CD, Git, and DataOps practices.
• Experience in data quality frameworks and monitoring tools.
• Strong problem-solving and analytical skills.

Preferred Skills:

• Experience with DBT (Data Build Tool).
• Knowledge of real-time streaming data pipelines.
• Exposure to machine learning data preparation pipelines.
• Experience working in staff augmentation / consulting environments.
• Strong documentation and stakeholder communication skills.