About the role:
As a Data Engineer III, you will be responsible for designing, maintaining, and optimizing large-scale automated ELT processes. Working actively with analysts and customer teams specializing in enterprise data warehousing, you will leverage industry-standard data orchestration tools as well as in-house proprietary scheduling and automation tools to create efficient and reliable ELT jobs which support the product offerings and data warehousing needs for the customers. As we incorporate more cloud technologies into our processes, you will be at the forefront of exploring and defining best practices and helping us transition our products to be more scalable.
This role is not sponsor able
What you will take on:
- Taking end-to-end ownership of data pipelines and custom solutions for our clients
- Coordinating with the systems, core, CX, and analytics teams to build and maintain data products and custom solutions for our clients
- Designing and writing automated scripts to preprocess terabytes of data from our partners/clients
- Designing and writing new enterprise-scale ELT/ETL workflows from scratch in Python using Airflow, Docker, AWS, etc.
- Modifying/redesigning legacy ELT/ETL processes to leverage cutting-edge open source and proprietary technologies
- Ensuring quality, reliability, and uptime for critical automated processes
- Migrating our products and processes into the cloud while drastically reducing our in-house data center footprint
What you already have:
Required Skills:
- At least 3-5 years in a Data Engineering role/similar programming in Python with exposure to Docker
- Ability to work independently
- Proficient in SQL
- Prior experience managing a small team in an Engineering role.
- Exposure to ETL/ELT pipeline automation
- Knowledge of Apache Airflow
- Exposure to basic database concepts
Preferred Skills:
- Good understanding of Data Engineering, NoSQL databases and database design, distributed systems and/or information retrieval
- Work with SaaS products
- DBA experience
- Ability to plan and collect requirements for projects, and interact with the analyst and data science teams
Education:
- STEM Bachelor’s required, a graduate degree is a big plus