Summary

Building a data pipeline that is reliable and flexible is a difficult task, especially when you have a small team. Astronomer is a platform that lets you skip straight to processing your valuable business data. Ry Walker, the CEO of Astronomer, explains how the company got started, how the platform works, and their commitment to open source.

Preamble

• Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure

• When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.dataengineeringpodcast.com/linode?utm_source=rss&utm_medium=rss) and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.

• Go to dataengineeringpodcast.com) to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.

• You can help support the show by checking out the Patreon page) which is linked from the site.

• To help other people find the show you can leave a review on iTunes), or Google Play Music), and tell your friends and co-workers

• This is your host Tobias Macey and today I’m interviewing Ry Walker, CEO of Astronomer, the platform for data engineering.

Interview

• Introduction

• How did you first get involved in the area of data management?

• What is Astronomer and how did it get started?

• Regulatory challenges of processing other people’s data

• What does your data pipelining architecture look like?

• What are the most challenging aspects of building a general purpose data management environment?

• What are some of the most significant sources of technical debt in your platform?

• Can you share some of the failures that you have encountered while architecting or building your platform and company and how you overcame them?

• There are certain areas of the overall data engineering workflow that are well defined and have numerous tools to choose from. What are some of the unsolved problems in data management?

• What are some of the most interesting or unexpected uses of your platform that you are aware of?

Contact Information

• Email)

• @rywalker) on Twitter

Links

• Astronomer)

• Kiss Metrics)

• Segment)

• Marketing tools chart)

• Clickstream)

• HIPAA)

• FERPA)

• PCI)

• Mesos)

• Mesos DC/OS)

• Airflow)

• SSIS)

• Marathon)

• Prometheus)

• Grafana)

• Terraform)

• Kafka)

• Spark)

• ELK Stack)

• React)

• GraphQL)

• PostGreSQL)

• MongoDB)

• Ceph)

• Druid)

• Aries)

• Vault)

• Adapter Pattern)

• Docker)

• Kinesis)

• API Gateway)

• Kong)

• AWS Lambda)

• Flink)

• Redshift)

• NOAA)

• Informatica)

• SnapLogic)

• Meteor)

The intro and outro music is from The Hug) by The Freak Fandango Orchestra) / CC BY-SA)

Support Data Engineering Podcast)