- Develop reusable, metadata-driven data pipelines
- Develop and maintain the shared functions organized as libraries
- Optimize data storage using partitioning and Databricks Delta native features
- Automate data platform-related processes, including the set up of a metastore, compute clusters and policies
- Write unit tests, perform code reviews to ensure code quality, and proactively resolve performance or quality issues in ETL processes and reporting queries
- Collaborate with Product Owners to gather functional requirements and develop mapping rules for data from different sources
- Cooperate with infrastructure engineering team to set up cloud resources
- Contribute to data platform wiki / documentation
- Initiates and implements improvements to the data platform architecture
- Expert-level programming experience in Python/PySpark, SQL
- Proficient in dealing with large and complex datasets
- Experience with building robust data pipelines using Databricks Spark
- Knowledge of stream processing challenges and familiarity with Spark Structured Streaming
- Experience building and optimizing data models
- Experience developing CI/CD pipelines
- Experience with Agile Software Development methodologies (Scrum)
- Additionally, it is not required but would be helpful if the candidate has:
- Knowledge of Azure cloud native solutions (Azure Event Hubs, Azure Data Factory, Azure Function App, Azure Container Instances)
- Familiarity with DWH concepts and dimensional data modeling (Kimball, Inmon, Data Vault)
- Experience building event sourcing solutions
- Competitive benefits package, including health care, life insurance and Multisport,
- Challenging projects.
- Spacious, modern, and fully equipped office space in the heart of Krakow.
- Flexibility and possibility to work remotely.
- Superior co-working and personal development experience.
Senior Data Engineer - Kraków, Polska - First Advantage
Opis
Who You Are:You are self-motivated and ready to "roll up your sleeves." While you are an independent contributor, you are also collaborative. You can spearhead a project and see it through from start to completion.
As a team player, you navigate cross-functional teams and work well with team members in other business units and departments toward a common goal.
An Innovator — you see gaps in current processes or workflows as an opportunity to improve and try something new.
A lifelong learner and always seeking out opportunities to learn and upskill, you understand the importance of thorough and secure screenings and are interested in the Human Capital sector and the confluence of people, process, and technology.
What You'll Do
We are looking for a Senior Data Engineer with strong experience in PySpark, to help us build our data lakehouse in Azure cloud. Specifically, the person in this role will focus on and need the below skills to be successful.
Responsibilities:
What You Need to be Successful:
Why First Advantage is Your Next Big Career Move
First Advantage is going through a technology transformation We are looking for experts who are excited to work with advanced technologies and provide best-in-class user experiences, drive the development and deployment of scalable solutions, and smoothly guide our agile teams and clients through meaningful changes as we continue to expand our impact.
Additional benefits offered to our eligible people include: