Implement and test data pipelines from variety of systems (batch and real time).
Collaborate with the development team and end customers.
Mentor data engineering interns
Advise the team on SQL optimization, performance tune when required
Troubleshoot existing datasets, broken pipelines, data integrity issues
Ensure quality of data through tested deliverables and auditing
Understand team, data and software architecture; actively seeks knowledge and solutions
Bachelors in Computer Science, Engineering, Statistics, Mathematics or related field
1-3 years of experience in data engineering / business intelligence space
Curious, self-motivated & a self-starter with a can do attitude’. Comfortable working in fast paced dynamic environment.
Good understanding of ETL concepts and explosure to large-scale, complex datasets proessing using traditional or map reduce batch mechanism.
Proficient in writing performant SQL working with large data volumes; ability to understand and action query plans
Experience with at least one of scripting languages(e.g., UNIX Shell scripting, Python, Perl, Ruby).
Exposure to AWS stack / cloud computing
Clear thinker with superb problem-solving skills to prioritize and stay focused on big needle movers
Master’s degree in Computer Science, Information Systems, Mathematics or related discipline
Good data modelling skills with solid knowledge of various industry standards such as dimensional modelling, star schemas etc
Strong knowledge of Python / Spark / PySpark
Strong analytical skills; ability to present complex datasets in visual form
Strong understanding of ETL techniques and best practices to handle extremely large volume of data
Experience with AWS using S3, EC2, Redshift, Aurora, Lambda, QuickSight, etc.
Experience in data ingestions techniques for batch and stream processing using AWS Batch, AWS Kinesis, AWS Data Pipeline
Experience in AWS Big Data technologies such EMR, Glue, Athena, Redshift Spectrum
Ability to handle multiple, competing priorities in a fast-paced environment
Work well in teams, respecting ideas from teammates, business partners, and technical experts
Strong ownership and drive to get things done
AWS certifications or other related professional technical certifications