#236: Building ML Products at Compare the Market, with Conor O'Neill, the Head of Data Science at Compare The Market

Data Futurology - Leadership And Strategy in Artificial Intelligence, Machine Learning, Data Science

This week on the Data Futurology podcast, we have an in-depth conversation with Conor O’Neill, the Head of Data Science at Compare The Market exploring his career journey and current role leveraging data and innovating with machine learning.

When O’Neill landed at Compare The Market, he quickly found himself in a senior data role within an organisation that needed to both transform and mature its approach to data. On the podcast, O’Neill walks through the various stages of transformation, and getting the rest of the organisation aligned with that vision.

He also shares some use cases that Compare The Market is effectively leveraging data for, as well as how they have been building ML products. He explains how he involves data scientists in this process and offers advice on building ML as a product when it comes to planning, delivery and infrastructure.

Finally, O’Neill shares some thoughts on the difference between a data scientist’s role and that of a senior manager, and how this shifts the perspective and how a data professional will look at projects. He then rounds out the conversation with some thoughts about where data science is heading as a profession.

For anyone interested in data science, O’Neill’s unconventional journey into and through the profession is both interesting and inspiring. Enjoy the show!

Connect with Conor: https://www.linkedin.com/in/conoroneill1/

Thank you to our sponsor, Talent Insights Group!

Join us for our next events Advancing AI and Data Engineering Sydney (5-7 September):

https://www.datafuturology.com/events

Join our Slack Community: https://join.slack.com/t/datafuturologycircle/shared_invite/zt-z19cq4eq-ET6O49o2uySgvQWjM6a5ng

What we discussed

2:26: Felipe introduces Conor O’Neill.

3:23: O’Neill shares his journey from astrophysics to data science.

6:49: In astrophysics, the data sets that scientists work on are massive. O’Neill shares some insights about how he managed data in that role.

8:40: O’Neill shares his journey at Compare The Market so far.

12:04: O’Neill shares some information about a current data project that he and his team are working on.

18:08: Compare The Market had to do significant foundational work in transformation. O’Neill shares insights into that process.

21:18: O’Neill shares his experience in getting the Compare The Market organisation aligned behind their data vision.

25:12: O’Neill explains the value of having data scientists involved at the earliest stages of transformation design.

28:44: O’Neill describes his experience in moving from a data scientist role to heading a team, and the differences between these roles.

32:56: O’Neill explains some of the thinking that goes into reusing data projects, as well as how they decide the projects to not follow through.

34:04: Getting a model in front of the end users and driving adoption is a critical step – O’Neill explains how he has approached it for Compare The Market.

37:54: O’Neill overviews the various consumers of the work done by the data team, and how the data team needs to think about each of them.

40:51: Tips and guidance for creating ML as a product to be consumed internally

45:48: O’Neill shares some thoughts on how the data science industry is evolving.

Key Quotes

• “We’ve been on a transformational journey now for a little over a year, and that’s been really good. We’ve been migrating off our legacy on-prem stack to Databricks. We’ve also been focused on getting the right people, and then also establishing a process, because if you just change the tool, you haven't fixed the issues, typically.”

• “You don't want your control group to be too large and you then miss opportunities. But you also don't want it to be so small that you don't get sufficient data. That's where the algorithm behind our recommendation system controls that, to optimise according to our confidence that we are or are not exceeding the required threshold, and adjust the weighting of the control group accordingly.”