Part III of our series on data science and engineering. We make some design decisions about how we'll use Kafka then take to the air to build a producer in Scala. We set up sbt, demonstrate 'one click' deploy to Kubernetes. (How cool does that make a four-line shell script sound?) Then we fix all the things that we've broken and talk a bit about Scala and some of its pitfalls.
Our blog series on data science and engineering is using Kafka as a distributed data plane for our platform. But Kafka often gets misunderstood. Don't know your partitions from your replications? Read on to learn some of the most common misunderstandings people have about Kafka.
Part II of our series on data science and engineering. As threatened, we finally set up Kubernetes as a development environment, add some (probably paltry) security and deploy etcd, Kafka and Fluentd.
Part I of our series on data science and engineering. Inexplicably, the introductory post did not scare you off. You still want to read about distributed systems, data engineering, and machine learning. Abandon hope all ye who enter here - in this post we will discuss what we want from a big data platform, some of the hidden (or assumed) requirements, and why pulling tweets off Twitter might be trickier than it sounds.
A gripping introduction to the world of data science, engineering, and distributed systems - and the bitter struggles of making them work.