We are looking for Data Engineer to work with mission critical systems accounting for around 150 million Euros in advertising revenue. Advertising Solutions is a relatively new area in Media organisation which houses engineering teams for our back-office systems used by various sales organisations at Schibsted. These systems include Rose which is used to book advertising campaigns and Vantage which provides campaign reporting.
We are now looking to establish a team to own and operate the Advertising API that underpins these products, along with others within Schibsted. You will be off to a running start and will have to learn the ropes of existing systems with the help of our established teams, plan and execute on the hand-over from the current team in London. Expect initial travels to London and/or hosting the London team members locally.
Once you learn the system you and your teammates will continuously work on its technical evolution, scaling and simplification. You will be expected to be an active participant when deciding how to implement new features together with the neighbouring teams that depend on you for their work.
We handle more than 250000 campaigns, 100000 advertisers, and more than 140 publishers across 20 different countries. About 1.5 TB of data is processed every day using more than 100 Spark jobs.
Our data pipeline is built on top of AWS EMR, Spark, Yarn, Airflow and microservices based on Twitter Finatra framework. Apache Avro and Parquet are used for data serialization and schema definition/evolution. We don’t expect you to have experience with all the technologies that we use but it would be good if you know at least some of them or have worked with similar ones.
SKILLS & REQUIREMENTS
- You should be deeply interested in data processing and data storage in general. You need to be well-versed in the area of databases – mainly SQL (Postgresql in particular), optimization of them, schema design and indexes.
- Perhaps you enjoy writing Spark jobs as well.
- You should have an understanding of such Big Data concepts as map reduce, CAP theorem and big table.
- Experience with JVM based languages is a big plus for you as we write our Spark jobs in Scala.
- The team is expected to be self-sufficient and as such you must be interested in more than just writing code.
- You will have to check third-party documentation, support users, debug, maintain and operate the system as well as gaining knowledge in the ad-tech domain.
- Full professional proficiency in English is required.