Highlights
Join our team as a Data Engineer and collaborate on robust data pipelines. Essential skills in SQL, Python, and AWS platforms.
Description
We are seeking a talented and experienced Data Engineer to join our dynamic team in the Pharmaceutical & Life Sciences industry. The ideal candidate will have a strong background in building and optimizing data infrastructure on AWS and Databricks platforms. As part of this role, you will collaborate with cross-functional teams to design and implement robust data pipelines that efficiently extract, transform, and load data into our data lake and warehouse.
Your responsibilities will include:
- Building and optimizing data pipelines, data warehouses, and data lakes on AWS and Databricks platforms
- Managing and maintaining the AWS and Databricks environments
- Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring
- Maintaining system uptime and optimal performance
- Working closely with cross-functional teams to understand business requirements and translating them into technical solutions
- Exploring and implementing new tools and technologies to enhance ETL platform performance
Must-Have Skills:
- Proficient in SQL for extracting, transforming, and analyzing complex datasets from both relational and columnar data stores. Proven ability to optimize query performance on big data platforms.
- Proficient in leveraging Python, PySpark, and Airflow to build scalable and efficient data ingestion, transformation, and loading processes.
- Ability to learn new technologies quickly. Strong problem-solving and analytical skills. Excellent communication and teamwork skills.
Good-to-Have Skills:
- Experience with SQL/NOSQL database, vector database for large language models
- Experience with data modeling and performance tuning for both OLAP and OLTP databases
- Experience with Apache Spark, Apache Airflow
- Experience with software engineering best-practices, including version control (Git, Subversion), CI/CD (Jenkins, Maven), automated unit testing, and DevOps
- Experience with AWS, GCP or Azure cloud services
We are committed to providing reasonable accommodation for individuals with disabilities. Please contact us if you require assistance.
Skills
