The Data Integration Engineer’s responsibilities include providing data for other teams to analyze and build their systems upon. The Data Integration team are plumbers, building a company-wide pipeline of clean, usable data for others to use. We typically don’t analyze the data, but instead we make the data available to others. Your job, if you choose to join us, is to help us build a real-time data platform that connects our applications and makes available a data stream of potentially anything happening in our business.
- Implement near real-time ETL-like processes from hundreds of applications and data sources using the Apache Kafka ecosystem of technologies.
- Designing, developing, testing and tuning a large-scale ‘stream data platform’ for connecting systems across our business in a decoupled manner.
- Deliver data in near real-time from transactional data stores into analytical data stores.
- R&D ways to acquire data and suggest new uses for that data.
- “Stream processing.” Enable applications to react to, process and transform streams of data between business domains.
- “Data Integration.” Capture application events and data store changes and pipe to other interested systems.
- Bachelor degree in Computer Science or related field, or equivalent experience.
- 4 – 7 years of related experience in a technical environment.
- Proficient in a functional programming language such as Scala, Clojure, Erlang, Elixir, or F# (we’re using Scala).
- Familiarity with JVM languages such as Java, Scala, or Clojure.
- Familiarity with C# and the .Net framework is helpful. While we don’t use it day to day, most of our systems run on Windows and .Net.
- Comfortable working in both Linux and Windows environments. Our systems all run on Linux, but we interact with many systems running on Windows servers.
- Shell scripting & common Linux tool skills.
- Experience with build tools such as Maven, sbt, or rake.
- Knowledge of distributed systems.
- Knowledge of, or experience with, Kafka a plus.
- Knowledge of Event-Driven/Reactive systems.
- You dig:
- Stream processing tools (Kafka Streams, Storm, Spark, Flink, Google Cloud DataFlow etc.)
- SQL-based technologies (SQL Server, MySQL, PostgreSQL, etc.)
- NoSQL technologies (Cassandra, MongoDB, Redis, HBase, etc.)
- Server automation tools (Ansible, Chef, Puppet, Vagrant, etc.)
- Distributed Source Control (Mercurial, Git)
- The Cloud (Azure, Amazon AWS)
- The ELK Stack (Elasticsearch, Logstash, Kibana)
- Experience with DevOps practices like Continuous Integration, Continuous Deployment, Build Automation, Server automation and Test Driven Development.
As you’ve probably guessed, OverDrive is a place that values individuality and variety. We don’t want you to be like everyone else, we don’t even want you to be like us—we want you to be like you! So, if you’re interested in joining the OverDrive team, email us your resume at firstname.lastname@example.org and tell us what inspires you about OverDrive and why you think you are perfect for our team.
OverDrive values diversity and is proud to be an equal opportunity employer.