Architecture posts

• DS III - Making a Data Plane Fly.

  September 10, 2019 •

  • Data Engineering tutorial series
  • Docker
  • Kubernetes
  • etcd
  • Fluentd
  • Kafka
  • Architecture
  • Platform
  • Scala

  Part III of our series on data science and engineering. We make some design decisions about how we'll use Kafka then take to the air to build a producer in Scala. We set up sbt, demonstrate 'one click' deploy to Kubernetes. (How cool does that make a four-line shell script sound?) Then we fix all the things that we've broken and talk a bit about Scala and some of its pitfalls.

• Six Ways Your Kafka Design will Fail.

  September 6, 2019 •

  • Kafka
  • Architecture
  • Platform
  • Schema Registry
  • Avro

  Our blog series on data science and engineering is using Kafka as a distributed data plane for our platform. But Kafka often gets misunderstood. Don't know your partitions from your replications? Read on to learn some of the most common misunderstandings people have about Kafka.