On November 9th I had the privilege of speaking at DataEngConf under the “Hero Engineering” track. My talk was titled “The Highs and Lows of Building an AdTech Data Pipeline” and I covered our evolution from a dead simple, sampled approach that had nothing to do with big data to the latest version which is leveraging a variety of modern open source data technologies.

I spoke about the motivation, challenges, and lessons learned during each iteration and ended the talk with the top 3 lessons learned across the various iterations of the pipeline. If you’re interested in the details you can grab the slides as either Google Slides or as PowerPoint from the DataEngConf site. Note that there was also a recording made but I’m still waiting for it to be processed and uploaded it to YouTube and will share that when it’s available.