#249: Three Humans and an AI at Marketing Analytics Summit

The Analytics Power Hour

How good are humans at distinguishing between human-generated thoughts and AI-generated…thoughts? Could doing an extremely unscientific exploration of the question also generate some useful discussion? We decided to dig in and find out with a show recorded in front of a live audience at Marketing Analytics Summit in Phoenix! With Michael in the role of Peter Sagal, Julie, Tim, and Val went head-to-GPU by answering a range of analytics-oriented questions. Two co-hosts delivered their own answers, and one co-host delivered ChatGPT’s, and the audience had to figure out which was which. Plus, a bit of audience Q&A, which included Michael channeling his inner Charlie Day! This episode also features the walk-on music that was written and performed live by Josh Silverbauer (no relation to Josh Crowhurst, the producer of this very podcast who also wrote and recorded the show’s standard intro music; what is it about guys named Josh?!).

Photo by Nader Abushhab on Unsplash

Episode Transcript

0:00:05.8 Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally. And sometimes with explicit language.

0:00:14.1 Jim Sterne: May have your kind attention. How many people… Show of hands, how many people in the room? Thank you. Have not. Some over here too. How many people in the room have not listened to the Analytics Power Hour before? You are in for a treat and there is a backlog of almost 250 shows you can listen to. Now I’d like very much to introduce them, but they’re really seriously good at introducing themselves. So.

[music]

0:01:28.8 Michael Helbling: Welcome Everyone to the Analytics Power Hour Live, recording live at Marketing Analytics Summit where the temperature is just a cool, calm, collected, 108 degrees Fahrenheit. But we’re cool in here and we’re excited to spend a little time with all of you today. And without further ado, let me introduce my co-hosts today on the show. All right, down here we have Julie Hoyer, analytics manager at Further.

0:02:04.3 Julie Hoyer: Hi everyone.

0:02:05.5 MH: Next to her is Tim Wilson, head of solutions at Facts and Feelings.

0:02:13.7 Tim Wilson: That’s good.

0:02:14.1 MH: I knew I gotta screw it too.

0:02:14.1 JH: Almost forgot it.

[overlapping conversation]

0:02:16.7 TW: One more. There’s two down more to go.

0:02:17.2 MH: And then there’s Val Kroll, who is the head of delivery at Facts and Feelings.

0:02:27.6 MH: And I’m Michael Helbling and I’m the managing partner of Stacked Analytics. All right, let’s get into it. So you might be wondering, what do they have in store? Well, we don’t know. [laughter] It’s gonna be a little bit of a back and forth. And we thought in honor of what was sort of a recurring theme during this conference, this sort of whole thing with AI. What could we do with AI? And we thought, let’s see how well we know what’s AI and what’s not. So there’s going to be a quiz. And what I’m gonna do is I have asked some questions. We’ve asked some questions. My co-hosts have prepared some very brilliant answers. Your job is to figure out which one of their answers is generated by AI.

0:03:12.7 TW: Wait, wait, don’t tell me.

0:03:14.6 MH: It’s kind of a wait, wait don’t tell me situation.

0:03:15.4 TW: Wait, wait, don’t tell me.

0:03:17.4 MH: All right, so without further ado, should we get started?

0:03:19.7 Val Kroll: Yes.

0:03:21.2 MH: There are prizes for correct answers, however, I forgot to pack them. So I need you to come up after the show and I will get your address and I’ll even ship it internationally to make up for my lapse in memory. All right, if only there was an AI agent for that. [laughter] All right, let’s get into it. So we’ll get, we’ll start with an easy one. It doesn’t matter how we answer the questions. All right, here’s the question. What are the most overused words in analytics in 2024? And anybody? We can just go.

0:03:57.9 JH: Oh, we just dump it in. Okay.

[laughter]

0:04:00.4 JH: I’ll go. All right. Insights, benchmark, directional, month over month and Gen AI.

0:04:06.9 TW: I think synergy, disruptive, big data, paradigm, scalable.

0:04:16.6 JH: We should also mention, we haven’t seen each other’s answers, so…

0:04:18.9 TW: We have not.

0:04:19.3 JH: I’m trying to think which one [laughter] as we’re going through this. So my answers were I have to unlock my phone ’cause I went off script here. Insights, AI, automated, real time and directional.

0:04:33.6 MH: All right, so here’s how this works. Somebody have a have a guess, we guess. Okay. I think Josh, you had your hand up first.

0:04:40.7 Josh: Tim’s.

0:04:41.2 MH: Tim’s you sir, are correct. Those are AI driven responses. So we got you an easy one to start. Easy one to start. Great job.

0:04:51.2 JH: You didn’t sell it. You didn’t sell it. You’re like synergy…

0:04:51.3 TW: I didn’t sell it.

0:04:53.2 MH: You’re gotta sell it, Tim.

0:04:55.0 TW: Yeah. Come on. I didn’t have insight. There was exactly zero overlap.

0:04:58.7 JH: I was gonna say Tim without insights doesn’t.

0:05:01.6 TW: Zero. Yeah, zero overlap.

0:05:02.8 MH: You could tell Tim’s heart wasn’t in it, right. Yeah, that’s what it was.

0:05:06.0 JH: Do better Tim.

0:05:07.6 MH: All right well, this one Tim actually has a much better response for this question. I’m sure. What makes Tim Wilson the quintessential analyst?

[laughter]

0:05:18.7 TW: Yeah. I’ll start with this one.

0:05:20.8 MH: Yeah, go ahead Tim.

0:05:21.6 TW: Absolutely nothing. It’s a who rather than a what and the who is fucking Michael Helbling who is the quintessential needler of me in this regard. [laughter] Did I sell that better? I tell you that ChatGPT is amazing.

0:05:39.0 MH: I think ChatGPT could have come up with that probably. I don’t know.

0:05:42.3 JH: That Tim sass. All right, so mine is Timothy Robert Wilson, the former Australian politician [laughter] served as the chair of the standing committee on economics from 2018 to 2021. He’s the quintessential analyst because he represents the most perfect and typical example of an analyst.

[laughter]

0:06:04.9 MH: All right. Al right, that’s a good question. Oh, let’s wait to hear them all first. We don’t know.

0:06:08.9 VK: All right, here’s mine. His knack for asking the hard and important questions around data and the business needs the skill of smelling BS five steps ahead because he’s been there, done that in his career and he knows what path to fight like hell to avoid with a stakeholder. And his thoughtfulness in how he approaches conversations with those he mentors and consults. Some may even call it empathy.

0:06:31.3 TW: So we know it wasn’t mine but you both could be garbage.

0:06:33.0 MH: All right, so who’s got a guess? Okay, way in the back.

0:06:37.4 Speaker 7: I think it was Val.

0:06:39.9 MH: Okay, he thinks it was Val. Well, this one’s a little bit of a trick. None of them were generated by ChatGPT. Val just tricked you.

0:06:50.2 JH: Her Googling skills.

0:06:50.7 MH: We didn’t do that very much, but there are some false positives in there. But, yeah, when I looked at those answers and I saw Val’s, I was like, well, I can’t replace that with ChatGPT, that was too good.

0:07:02.3 TW: So we had to keep it.

0:07:03.2 MH: And obviously if I forced Tim to read something positive about himself, he’d probably just leave.

0:07:08.3 TW: I would just storm out.

0:07:08.5 JH: Just combust.

0:07:13.6 MH: It’s time to step away from the show for a quick word about Piwik Pro. Tim, tell us about it.

0:07:19.9 TW: Well, Piwik Pro has really exploded in popularity and keeps adding new functionality.

0:07:24.9 MH: They sure have. They’ve got an easy-to-use interface, a full set of features with capabilities like custom reports, enhanced e-commerce tracking, and a customer data platform.

0:07:36.0 TW: We love running Piwik Pro’s free plan on the podcast website. But they also have a paid plan that adds scale and some additional features.

0:07:43.7 MH: Yeah, head over to piwik.pro and check them out for yourself. You can get started with their free plan. That’s piwik.pro. And now let’s get back to the show.

0:07:57.4 MH: All right. So okay, we’ve got… We’re going to go deeper. We’ve got to go deeper and more innovative, more creative. You have recently been appointed as the headmaster of LogWarts, the school for analysts with magical powers. Your first task as headmaster is to fill three teaching vacancies. Anyone you invite to teach will happily accept. Which three people, living or dead, will you appoint to these vacancies, and what classes will they teach?

0:08:21.3 VK: I’ll go first on this one. The three professors I would hire to teach magical powers in the realm of analytics are, number one, Houdini, the art of disappearing when someone drops by your desk with a quick question, 201. Number two, Einstein, time is an illusion, how to get date formats to work for you, 302. And three, Marie Kondo, cleaning your data and feeling joy, 400.

0:08:51.8 TW: Nice.

0:08:53.9 JH: My turn.

0:08:55.7 MH: Sure.

0:08:56.7 JH: All right, my three teachers and their positions. The first would be Hans Rosling, class data visualisation magic. He’s a Swedish physician, by the way, and statistician, was renowned for his ability to transform complex data into compelling visual stories. Then we have Grace Hopper, and the class would be programming potions and spells. She was a pioneering computer scientist and Navy rear admiral, called the queen of code, and she would help students master the art of writing clean and efficient code and understanding underlying principles of computer science. And then third would be Florence Nightingale, statistical healing, and she’s the founder of modern nursing.

0:09:39.3 TW: Plus more data visualisation.

0:09:41.3 MH: Nice.

0:09:41.5 TW: From Florence Nightingale, right?

0:09:43.6 MH: All right, Tim, what about you?

0:09:46.6 TW: Well, anyone who puts me in a position to influence the minds of tomorrow should have 10 points deducted from their house. But notwithstanding all that, I’ll go with dead giveaway. Number one, Cassie Kozyrkov to teach the dark arts of decision making. Katy Milkman to teach choiceology and cognition in muggles. And Cole Nussbaumer Knaflic to teach data visualisation spells and formulas.

0:10:13.1 MH: All right, these are good answers. All right, this is a little more complex. Oh, okay, I saw this hand first.

0:10:17.7 Speaker 8: Julie’s.

0:10:18.4 Speaker 9: Julie.

0:10:22.4 MH: That is correct.

0:10:23.6 JH: It was so long and wordy. I’m like, can I take this more human? We’re like, no, that’s not what you’re allowed to do. I was like, that’s so awkward.

0:10:31.6 VK: Not the rules. Not the rules.

0:10:32.2 TW: And I gave it prompts. I told it to try to be succinct and informative.

[overlapping conversation]

0:10:34.4 TW: Wait a minute. It was Grace Hopper…

0:10:34.5 JH: I know, Michael was insulted when I told him these are really long.

0:10:34.6 TW: Grace Hopper, Florence Nightingale, and who was the first one?

0:10:37.7 JH: Hans Rosling.

0:10:38.3 TW: Oh, Hans Rosling.

0:10:38.9 MH: I thought they guessed the answer.

0:10:40.5 TW: Except they… It referenced him…

0:10:41.2 JH: Good people, just long.

0:10:41.8 TW: But there was a… Because it referenced him in the present, and he died three or four years ago, five years ago. And then it referenced him in the past.

0:10:48.5 JH: Well, it said was renowned. I might have messed up.

0:10:51.2 TW: Oh, okay. I’m curious, like, show of hands, how many people know Hans Rosling?

0:10:55.7 MH: Oh, yeah.

0:11:00.1 TW: He was the most, yeah, the fun. That was a…

0:11:00.8 MH: Best TED Talks ever.

0:11:01.4 TW: That was like 400 people raised their hands? Just for the listening audience and all.

0:11:07.2 MH: Pretty much everybody.

0:11:08.7 JH: Roughly 400.

0:11:08.8 VK: The whole stadium.

0:11:11.3 TW: What’s going on in section H?

0:11:16.8 MH: All right. Okay. Next question. We’re going to get into some more practical stuff, stuff that could be helpful, but still, there might be an AI answer in there. So this is the question. What are the top two pitfalls you would warn a new analyst about?

0:11:32.2 TW: I’ll go. So many possibilities, but I’ll go with number one, diving into the data prematurely. And number two, believing and acting as though the data provides objective truths.

0:11:46.5 JH: All right. I’ll go next. My two pitfalls that I chose were, one, do not let anyone convince you that building dashboards is your biggest contribution to the business. They will try. Two, beware of being sent to chase down evidence in the data just to confirm someone’s biased opinion.

0:12:05.5 S8: That’s very good.

0:12:07.0 VK: All right. And my two pitfalls for new analysts are, one, tool obsession. Don’t get too caught up with the latest tools. Understand the problem first. Tools are just there to help you solve it. And then the second one is ignoring data quality. Always check your data for accuracy and completeness before jumping into analysis. Bad data equals bad insights.

0:12:28.2 MH: Hmm. Tricky. All right. Who thinks they’ve got a response or an answer? Oh. All right.

0:12:37.5 JH: We did good on that one.

0:12:37.6 MH: I’m going to go with this one right back here and we’ll…

0:12:42.3 S7: Val.

0:12:42.4 MH: Val.

0:12:43.3 VK: Ah, I thought I sold it.

0:12:44.8 TW: You didn’t sell it. Val you didn’t sell it.

0:12:44.8 MH: That was good. But that’s correct. Nice job. Very nice.

0:12:52.1 TW: What were your…

0:12:52.2 MH: Oh yeah. Val, why don’t you say what your answers were? ’cause I thought your answer was also really great.

0:12:56.0 VK: It was. Even though there are beautifully designed documents that outline the process for a business partner to make requests of the analytics team that is rarely to be followed and is likely that you’ll wish you were pulled in earlier and you’ll have to deliver something anyways. And the second is that date will never be perfect and you don’t need to find, and you’ll need to find the delicate balance of diminishing returns.

0:13:16.9 MH: Nice.

0:13:17.2 TW: That was so long. I would’ve thought that was from ChatGPT.

0:13:19.6 JH: I know.

0:13:19.6 VK: See. How intentional. I was trying to fake you out.

0:13:22.3 JH: Val’s up here trying to like really fake everybody out.

0:13:25.1 MH: Yeah. That one’s sort of like trying to make it hard for people to guess the answers. That’s okay. Al right. This one should be pretty easy. I think. [laughter] Write a poem about analytics.

0:13:41.6 TW: Who wants to start?

0:13:41.9 VK: All right, I’ll start. So you may feel adrift when your line see a lift since you can’t be too sure movement’s truly improvement. So what shall you do when leaders knocking yoo-hoo, well, first I’ll tell you, surely don’t cry boo-hoo. Because there exists a team that is full of dreams that understand variation and don’t joke about causation. Why? I’m talking about analysts. The best ones to manage it. Trust them with your thoughts so their efforts won’t be fraught and you will have delight when they bring you true insights to light.

0:14:18.4 S8: There you go.

0:14:20.6 TW: Even Christopher Barry is kind of stumped by that one.

0:14:23.6 JH: Yeah. Look at his face.

0:14:27.1 TW: The look on his face is priceless.

0:14:30.4 JH: You go next.

0:14:31.0 TW: Oh me? I have a little, briefer one, missed targets are red exceeded targets are green deuteranopia colour blindness makes that distinction unseen.

0:14:43.9 MH: Oh.

0:14:48.4 VK: All right. Dashboard modules turn red, trend lines are blue. This wasn’t written by AI. I think it would be better, don’t you?

0:15:01.0 MH: All right. Who’s got an, somebody’s already got their hands up.

0:15:07.2 S?: I’ll go with Tim’s for sure.

0:15:09.2 S1: Tim, are you sure? Final answer. All right. Tim, do you wanna reveal that.

0:15:17.2 TW: It was not me.

0:15:18.1 MH: It was not Tim.

0:15:18.8 TW: I mean that was mine for real.

0:15:20.2 MH: Yeah, that was Tim. Tim did write that.

0:15:22.5 TW: It was me. So you’re wrong. Is that…

0:15:26.8 JH: To put it nicely.

0:15:27.9 TW: I mean, I still get a point if we’re following wait, wait, don’t tell me rules.

0:15:31.0 MH: Yeah, that’s right.

0:15:31.7 VK: There’s a score?

0:15:34.5 MH: Yeah. It’s like the, we keep score, but the rules don’t matter. The score matter, don’t worry. All right. Anybody else wanna make a guess? Oh, Brian.

0:15:43.9 Brian: I’ll go with Julie.

0:15:45.1 MH: Julie is also a good guess, but I have something important to tell you. You’re incorrect.

0:15:50.2 JH: Sadly I wrote the poem myself.

0:15:52.7 MH: Guess what?

0:15:54.4 TW: I was guessing.

0:15:56.2 MH: All of them were written by our hosts. None of them are written by ChatGPT. Those were human poems. Folks. Go figure it out. There’s so much creativity on this stage.

0:16:05.1 JH: I was so bad. Tim thought it was AI.

0:16:10.2 TW: That was a lot of work.

0:16:11.4 MH: I’ll give you a one hint that is your last false positive. So no more of those to worry about, I think. But anyways, I thought a poem would be good because we all expect ChatGPT to do poems, and be observable in that regard. But…

0:16:25.5 TW: I will say on that one, I thought I was gonna come up with something that rhymed with Phoenix, which I have not resorted to AI, but it’s like Xanax, Kleenex. Somebody said Kleenex.

0:16:41.3 JH: If only we could have used AI to help with our answers. Yeah.

0:16:44.1 MH: So what we’re learning crowdsourcing trumps AI, at least for now. All right, here we go. Let’s do another question. See, you are the co-host of the Analytics Power Hour podcast, and you have to list the top two reasons that you spend your time working on the show. What are those two reasons?

0:17:05.3 JH: All right, I’ll go first. Number one, the vibrant discussions that I always laugh and learn from. And two, it’s an honour to work alongside such intelligent, fun, loving professionals who can push themselves to be the best they can be.

0:17:17.7 TW: Oh yeah, really.

0:17:21.5 JH: That’s a lot of feelings for this one.

0:17:26.1 TW: Sharing knowledge. I love diving into analytics topics and sharing insights with our listeners. It’s like a fun ongoing conversation where we all get to learn something new. And building community, the podcast helps create a sense of community among analytics professionals. It’s rewarding to connect with people, share experiences, and sometimes even laugh at our shared data woes. And hey, it’s all a great excuse to geek out with my co-hosts.

0:17:52.1 VK: All right, my two reasons. One, to learn from people smarter, more experienced and in different roles than me. And two, to get involved and get back to the analytics community.

0:18:03.0 MH: Nice. All right. Who’s got a guess? Okay. Yeah.

0:18:06.9 Drake: I’m sorry. I dunno. Your name in the middle.

0:18:11.5 TW: Tim [laughter] Yeah.

0:18:12.5 Drake: [0:18:12.5] ____.

0:18:13.0 MH: Tim. That is correct. Because you can kind of tell when Tim’s reading it, right? It’s different. Yeah.

0:18:19.9 TW: My actual second reason was because Michael keeps telling me it’s a lot easier than blogging.

0:18:24.9 MH: That’s right.

[laughter]

0:18:25.8 TW: Which is kind of grounded in how this whole thing got started.

0:18:29.1 MH: I realise I didn’t read ChatGPT’s poem. Do you want me to read it?

0:18:32.2 TW: Oh.

0:18:32.5 JH: Oh.

0:18:32.8 VK: Yeah.

0:18:33.2 JH: Yeah.

0:18:33.6 MH: So we’ll see. You compare. Remember the other poems, right?

0:18:35.9 TW: Do we have time or is it like an epic.

0:18:37.2 MH: No, no. I told it to keep it short.

0:18:39.2 JH: Nothing’s longer than the mine, I guess.

0:18:41.3 MH: I said, Hey, you’re gonna be presenting live in front of a smart audience of analysts. So keep your answers succinct and informal, but maybe slightly humourous. This is what it came up with.

0:18:51.5 TW: Well, you could have given us that instruction.

0:18:55.9 MH: Well, I could Tim, but you wouldn’t follow it. All right.

0:19:00.7 JH: No, that’s true.

0:19:01.0 MH: There once was a data filled plot where insights were carefully sought with numbers aligned. The truth we did find in analytics knowledge is caught. It’s not bad.

0:19:14.1 VK: See, it was better than mine.

0:19:17.5 TW: But not Julie’s.

0:19:18.8 VK: Yeah, no.

0:19:19.0 TW: Her’s was great.

0:19:19.6 JH: Epic.

0:19:20.5 MH: Julie’s yours was better. All right, now we’re gonna get into something a little esoteric or a little harder for maybe a ChatGP to answer. I think maybe. All right, so what is your favourite episode of the Analytics Power Hour podcast and why…

0:19:38.8 JH: Okay, I’ll go.

0:19:40.9 TW: Okay.

0:19:41.6 JH: Tim took a deep breath. He was pausing. Looking if I would go first. Okay. My favourite episode of the Analytics Power Hour is Operationalising a Culture of Experimentation with Lukas Vermeer. It’s packed with practical tips on how companies can embrace and scale a culture of testing and learning, making it super useful and engaging.

0:20:01.7 VK: Want me to go next?

0:20:01.8 MH: Sure.

0:20:05.9 VK: Episode 199. Media Measurement Revisited Matched Markets, Media Mix and More with John Wallace. In this episode, the host delve into the complexities of multitouch attribution, media mix modeling and understanding matched market testing. Understanding the individual strength of those methodologies helps me in my role as an analyst.

0:20:25.7 TW: So there’s likely some recency bias going on, but the one I keep finding myself referring to is number 240, Asking Better Questions with Taylor Buonocore-Guthrie actually wasn’t on the show, but I used the on a scale technique, she explained at least once a week. Also, that show wound up being a behind the scenes technical nightmare, which was no one’s fault, but that one’s got kind of a special place in, for me because the hosts and the guest and Josh Crowhurst rallied to make the final product kind of hide all of that. Way to go ChatGPT getting me really messed up.

0:21:02.8 VK: It knew.

0:21:02.9 MH: ChatGPT knows about Josh.

0:21:03.9 VK: It knew… All knowing.

0:21:05.1 MH: All right, who’s think they know… Who in the back? Val.

0:21:12.4 VK: It was not.

0:21:15.1 MH: Oh, it was not Val’s answer.

0:21:15.4 VK: But I used Delve intentionally. I heard you say Jim’s…

0:21:19.2 Brian: We were told… There would be no more false positive.

0:21:21.4 VK: I just used… I used the word…

0:21:21.5 MH: Well, there’s not, there is actually a ChatGPT response. Yeah.

0:21:25.2 S?: Is it Julie?

0:21:26.4 MH: It is Julie’s. Nice job.

0:21:28.4 JH: Nice.

0:21:29.4 MH: All right. Julie, do you wanna say what your actual favourite episode is?

0:21:34.7 JH: Sure. Maybe some recency bias also, but episode 237, Crossing the Chasm from the Data to the Meaningful Outcomes with Kathleen Maley is one of my favourites. Just a small tidbit that she talks about that I love and talk about constantly is why she tells her analysts to never say yes when being asked to just pull data. But also why you can never say no. So that’s a good one. Take a listen.

0:21:57.6 VK: It’s a really good one.

0:21:58.3 TW: And I will say, I was validated today when we were having a chat and Eli had mentioned the Taylor Buonocore-Guthrie episode is one that he’d listened to multiple times. So I was like, we wrote these a couple days ago, so I wasn’t just playing to the crowd but…

0:22:13.9 MH: That’s right.

0:22:14.0 JH: That one was for you.

0:22:17.3 MH: Nice. All right. This one’s practical. Should be insightful. You are an experienced analyst what is the value of starting an analysis with a hypothesis statement?

0:22:31.7 VK: I’ll go first on this one. Starting an analysis with a hypothesis statement is arguably one of the most important steps of the analytical process. The value is that it puts parameters around the question at hand and sets up everyone to take action once it’s been validated.

0:22:48.6 TW: Okay I’ll go. Starting with a hypothesis helps keeps your analysis structured, relevant and efficient.

0:22:57.0 JH: All right. Starting with a hypothesis statement in your analysis highlights what assumptions exist by stating what is believed to be true, which gives clearer direction on what to look for in the data. It helps keep efforts focused once you’re deep into an analysis and it helps with communication of your findings back to the stakeholder.

0:23:12.5 VK: Lots of hands on this one.

0:23:15.8 MH: Oh okay a lot of People willing to take a risk. Who, I wanna pick someone I haven’t called on. I haven’t called on you, Drake. Go.

0:23:19.8 Drake: I’m guessing Val.

0:23:22.1 MH: Val.

0:23:22.8 VK: It was not.

0:23:23.3 MH: Oh, you got Val, got you again.

0:23:23.5 VK: See my strategy now is to undersell.

0:23:26.4 MH: Gotta wake up pretty early in the morning. All right. Who else I wanna pick on, I wanna pick someone who hasn’t gone. If they, if I… If they raise their hand, but if they don’t, then I’ll pick you.

0:23:35.2 JH: You could win up a prize.

0:23:37.4 MH: Go ahead.

0:23:39.2 S?: All three.

0:23:39.3 MH: All three. Oh, well, I mean, blanket coverage, one of them is generated by ChatGPT, but it’s not all three. Only one. Ooh Jim.

0:23:48.7 Jim: It’s Julie.

0:23:49.7 MH: Wow, look at that. So my prompt engineering skills are strong.

0:23:55.3 TW: Sarah’s like, I got it.

0:23:55.7 MH: Tim was the ChatGPT response. Yeah. So there you go. All right. Tim, what was your actual answer as well? ’cause I think yours was also really great.

0:24:05.6 TW: It overlapped definitely partly with Julie. So we rarely receive requests in the form of a hypothesis. So converting the request to a hypothesis does two things. One, it forces us to slow down and think a little bit, which two tends to surface assumptions and fuzziness in the requests that really need to be worked through before diving into the analysis itself. I don’t think I can answer a question about hypotheses in…

0:24:26.3 JH: Not use assumptions.

0:24:28.6 TW: Well, not in like less than 12 words. That was the most… Like ChatGPT was like, oh yeah. Duh. I’m like…

0:24:34.4 VK: I feel like I underst…

0:24:35.2 TW: There’s a little bit more to it.

0:24:35.3 VK: I feel like I understood the assignment.

0:24:36.5 MH: That’s right… That’s why I gave you that answer, Tim. I knew it would wrinkle.

0:24:40.8 JH: I feel like you didn’t really understand the assignment.

0:24:41.5 TW: Okay.

0:24:43.9 JH: ’cause you like, you were like Tim Wilson flare all over these questions. There’s like, you know, we kinda like pare it down ’cause that wouldn’t have been my perfect answer, but I was trying to go.

0:24:54.9 MH: Well what would, what would be Val? What would be your perfect answer…

0:24:58.4 TW: Get away… I think we were given.

0:24:58.6 VK: Come up off with off the…

0:24:58.7 MH: Yeah you can go off the cuff.

0:25:00.3 TW: I think we were given incomplete, inaccurate prompts. It wasn’t in the prompt. I just followed the assignment.

0:25:03.7 VK: You didn’t take into a hypothesis and asked about the assumptions. [laughter]

0:25:06.0 TW: I did.

0:25:06.3 MH: Humans am I right? [laughter] All right. Do you wanna talk about it a little bit more though?

0:25:13.8 VK: No, no, no. We’re good.

0:25:14.8 TW: No, she just wanted to give me shit.

0:25:16.5 VK: Yep. Yeah. 100%.

0:25:19.2 MH: Okay Well that’s fine. All right, let’s jump to another question. Can you give me one good tip for doing analytics well that demonstrates that you are a human?

0:25:35.6 TW: Okay. Okay. [laughter] Always be learning. It doesn’t really matter what or how, as long as you’re finding a path that is interesting to you and challenging to you, for one, that will make for a more fulfilling life, but it will also make you a better analyst.

0:25:54.9 JH: All right. I’ll go. My one tip would be talk with your stakeholders live and ask questions to gain more context around where their ask is truly coming from and what they’re trying to achieve.

0:26:05.5 MH: Right.

0:26:08.2 VK: All right. Here’s a good tip for doing analytics well. Always keep asking why. Dig into your data like a detective solving a mystery. If sales are down, don’t just note it. Ask why? Keep asking why until you hit the root cause. Because you remember curiosity didn’t just kill the cat, it also found the insights. [laughter]

0:26:26.0 MH: All right, right here.

0:26:31.0 S?: Is it Tim?

0:26:33.5 MH: Is it Tim?

0:26:34.8 TW: That was me, legit human.

0:26:37.3 MH: Oh, in the back.

0:26:37.7 S?: Is it Val?

0:26:38.7 VK: Yeah!

0:26:39.6 MH: It was Val this time!

0:26:41.9 JH: That was good. You sold it well, I liked that. I really was convinced that was Val’s, like, prompt engineering of herself, like, trying to play AI.

0:26:51.7 TW: I was reading it thinking, I kinda saw this.

0:26:53.0 MH: Well that was what was…

0:26:53.2 TW: And there might be a suck it Val on that one!

0:26:58.0 MH: That’s what worked out well this time, because people were primed to be like, well, wait a second.

0:27:03.2 JH: No one knew about the Val now no.

0:27:04.9 MH: That’s right.

0:27:05.8 TW: Try working with her.

0:27:07.1 MH: Val’s always trying to trick you.

0:27:10.9 MH: All right, well.

0:27:11.0 VK: I just got serious.

0:27:11.9 MH: There’s facts and then there’s feelings am I right? . All right, Val, what was your real answer though?

0:27:19.1 VK: Oh.

0:27:19.2 MH: We can try to do that.

0:27:20.3 VK: It’s okay to admit when you don’t know the answer to something and you need to do some research before responding. That one I couldn’t help but just be real talk about.

0:27:26.5 JH: That was good.

0:27:27.7 MH: That was good.

0:27:28.2 JH: That’s a good advice.

0:27:28.9 VK: It’s okay.

0:27:30.4 MH: That was awesome. All right. Well, this is going pretty good.

0:27:33.6 TW: Although, kind of, not to tie it back to Jim’s, like, asking the ChatGPT or what else I need or whatever the platform is, what else it needs to know, like, that I feel like there’s a little bit of a tie in there. If you’re a human, okay, never mind. We’re losing the thread.

0:27:52.5 VK: I followed. I know where you’re going with that.

0:27:55.9 MH: It’s all good. All right. We’ve got another question. What are two deeply ingrained habits of our industry that are problematic to being able to deliver value?

0:28:09.4 JH: All right, I’ll go first. All right. Number one, dashboard dependency. Dashboards are great, but don’t stop there. Obviously, dig deeper to find real insights. And two is trend chasing. Stay updated, but don’t forget the basics. Focus on fundamental skills and critical thinking over the latest buzzwords.

0:28:31.9 TW: You were gonna your phone.

0:28:32.0 VK: All right, I’ll go. I’m ready.

0:28:32.1 TW: I was like you were like all right, I’ll go. You already had like inhaled.

0:28:33.5 VK: Lip rearing.

0:28:34.0 TW: And had your phone unlocked so.

0:28:36.0 VK: All right. Number one, thinking you can schedule insight delivery on a monthly basis. And two, democratizing data via dashboards is possible.

0:28:49.0 TW: Okay. One, we have fetishised the collection of data to the point that we chase more data and more granular data and cleaner data to the detriment of actually doing things with the data. Two, we jump too quickly to can I deliver the data asked for rather than pausing and asking am I really, really clear on the underlying business need and business thinking that drove the request in the first place.

0:29:12.6 MH: These are good questions. All right. Joe.

0:29:17.4 Joe: Julie.

0:29:20.1 MH: Julie.

0:29:20.2 JH: It was I.

0:29:21.1 MH: It was Julie. All right. So yeah, sometimes ChatGPT is hard to pick out and sometimes it’s kind of easy. It’s kind of interesting to see where it does or doesn’t kind of rise to the surface. What no, no I’m saying it’s…

0:29:38.1 TW: Joe said don’t finish my accomplishment.

0:29:41.5 MH: I’m saying some of them are harder than others, Joe. Julie, what was your real answer?

0:29:48.2 JH: My real answers. Okay. Number one for me was constantly looking backwards at historical data and trying to determine why a metric moved instead of determining what they need to learn from the data moving forward to take action and make change. That was number one. Number two was mistaking correlation for causation.

0:30:06.0 MH: I like that. All right. These are awesome. Thank you all for putting up with this exciting test of AI. Thanks all of you for like figuring out the AI agent in the midst. We still have some time. So we’d like to open it up to some Q&A and it doesn’t have to be about ChatGPT or any of the questions. It could be about anything. But if you have a question you’d like to ask, we’ll come around with a microphone and please do use the microphone because we are recording this so we can play it live. No stress. No, don’t worry.

0:30:41.6 TW: Yeah I think Jim’s trying to determine if there’s another channel that actually we they can record on.

0:30:46.3 MH: Well we can just take this one. All right. Does anyone have a question?

0:30:52.8 John Lovett: Hello. Hello. This is John Lovett and I’ve got a question. What the heck does rock flag mean?

0:30:58.0 MH: Oh, that’s a good question.

0:31:00.6 VK: Yes… Let’s talk about this.

0:31:01.7 MH: All right. So there’s a show on television called It’s Always Sunny in Philadelphia. And on that show, this is nothing to do with the Analyst Power Hour, but this is where it comes from. On that show, there was a sketch where one of the characters was getting really patriotic and fired up about America. And he made up a little song about how he was getting fired up. And it went something like I’m gonna get fired up and then going to get my truck, going to run out of the truck to the USA… And at the end, he goes, a rock flag and eagle.

0:31:31.3 TW: Can you do a little bit more.

0:31:32.8 JH: Yeah. I was like.

0:31:33.2 MH: No that was all I remember. And so what had had happened was we sort of just laughed at that at one point and then threw it into a show at the end. And then it became a thing that happened every time after that because of our deep love for the city of Philadelphia. And the show, It’s Always Sunny in Philadelphia, which I don’t know. We actually love it. But that is where it came from. Rock flag and eagle.

0:32:03.5 TW: So, Michael, you need to hold on a second ’cause we’re gonna have… I think we’re are we gonna be you and three. Yeah.

0:32:10.2 MH: You’re on the podium.

0:32:12.1 TW: Okay.

0:32:14.1 S?: I mean…

0:32:14.6 JH: And that also we should say is like the hot potato.

0:32:16.2 VK: Yes.

0:32:16.4 JH: So whoever starts the show prep doc assigns someone else to do it at the end of the show because you can’t tell the guests we’re about to do it. Right. You tell that them.

0:32:23.3 MH: Oh yeah that’s right. We keep it a secret from every guest. So they don’t know we’re doing it until we actually do it.

0:32:28.1 JH: So if like the show ends…

0:32:29.9 TW: So we’re having done the first like 180 of them, I’m like, this is so awkward. And I know it’s fine. And then we started shuffling it around where somebody else had to do it. And the first time everyone had to do it, they’re like, we just had like a great conversation with this person. And I just need to blurt out.

0:32:44.6 MH: Belt it out.

0:32:45.9 TW: Something like a moron. Like, what are you doing? I’m like, yeah.

0:32:52.4 MH: You know you just have a culture, folks. That’s what this is. Norms.

0:32:57.8 Alex: Sure. Hi, Alex. From the great city of Philadelphia. Go birds. Val, one of your answers towards the end, you could tell wasn’t AI generated. The dead giveaway is that it described a problem that’s like ripped from daily life and it was something to the effect of I’m gonna butcher it. But working with stakeholders in senior leadership to let them know that insights can’t be delivered on a weekly schedule. It’s not insights o’clock.

0:33:25.5 VK: Insights o’clock.

0:33:25.6 Alex: How do you… What… Do you have any tips for, and this might just be a stodgy problem, but having that conversation and communicating it and setting expectations in a better way?

0:33:35.2 VK: That is a great question. And I actually think this was one of the ones that we addressed on the one of our most recent episodes where some of our listeners called in. Do you wanna talk a little bit about that, Tim? ‘Cause, I’m probably gonna end up paraphrasing a lot of what you said. ‘Cause you know, I agree.

0:33:48.5 TW: But I’ve blacked out and have no idea what I said. [laughter] So [laughter], I can… I mean, I…

0:33:55.0 VK: Go for it.

0:33:55.0 TW: Go for it?

0:33:55.6 VK: Yeah. [laughter]

0:33:56.1 TW: Well, I mean, to me, the part of what we’re trying to do is kind of frame, help them understand and kind of talk them through that there’s two things they’re doing when there’s the weekly report or the monthly meeting. And one of it is the where are we, how are we executing, how are we delivering against whatever goals we set? It’s just an objective, quantitative view of how are we doing. And in our world that’s performance measurement and saying, that’s good and that’s valuable, and we will definitely do that. There are no insights in that. The insights come from, and actually, it was in one of your answers here are more of the forward looking of like, what do we need to know? And let’s look forward. So then the next time that we meet, we’re answering your questions or responding to hypotheses. So trying to split it into two pieces so that they get the piece, they see those as being two separate things.

0:34:53.9 TW: The tricky part is like the first time, like once, if you’ve got the monthly meeting or the biweekly meeting, you have to have that one where you get them to look forward. And then you can be in a cycle where you’re not necessarily delivering monthly insights, but you are coming back to them with stuff. Whether you had to run a test or do some analysis, you’re coming back and answering questions. And then maybe some of those are insights, but I don’t know the years that I lived in the agency world where it was, we’ll just do whatever we’re gonna do, and then at the end of the period, the analyst find some stuff in the data. And it took relationship, communication, being patient to kind of affect that sort of shift. And then everybody was happy as could be, but it’s painful to get there.

0:35:42.3 VK: And the other thing that I’ll say that, like, the part that you can schedule.

0:35:45.2 TW: You’re gonna say the thing that I actually said on that show…

0:35:46.8 VK: No, that last part. No.

0:35:48.2 TW: Okay.

0:35:49.0 VK: That wasn’t, no, that’s exactly what I wanted you to say. And I knew you would do better than I could do it justice, but the part that you can schedule is checking in on your progress against the previously established goals. How are we pacing? And that part you can come back to on a weekly cadence if they want to biweekly, monthly. And that can be the starting point for that conversation. So I think that that’s, but yeah, I love the relationship building part of it.

0:36:09.3 JH: One last thing too is, Tim, you talk about this all the time, it’s kind of like keeping the word insights. Just don’t use it everywhere for everything. Don’t call every thing you’re gonna put in your slides and insight. To his point, just talk about, how are we going against our goals? Like, where are we at? That’s not an insight, it’s just kind of a fact. And then at least keep insight special as you’re trying to work with them and do the change management of like, what should you be expecting in those monthly reports every time.

0:36:37.9 S?: Question over here.

0:36:39.6 MH: Yeah. Question over here.

0:36:40.8 Thomas: Hello Thomas [0:36:41.8] ____ MashMetrics. So I know you guys have all worked for several, you know, different sites agencies, large, small, medium, maybe just explain some of the differences between those experiences. [laughter]

0:36:58.2 TW: Good. Okay. Julie hasn’t. [laughter]?

0:37:02.0 VK: Okay.

0:37:02.1 TW: I just realized I looked it. Sorry.

0:37:02.0 MH: Yeah. Julie, do you wanna start?

0:37:03.6 JH: It started smaller. It’s gotten bigger. Okay.

0:37:06.2 TW: It’s.

0:37:06.2 VK: There you go. That’s a perspective.

0:37:08.8 MH: There’s a correlation between size and fun sometimes.

0:37:11.8 JH: And I mean, maybe.

0:37:15.4 TW: ’cause Michael, you were at big and then you were at small and the small grow or medium to grow into big…

0:37:19.1 MH: A little bit of that. Other than that, like, I also think it was one of the speakers today, Mohammed was talking about some of his experiences and I definitely got some flashbacks to big agency life.

0:37:29.5 JH: He slacked us. He’s like, PTSD [laughter]

0:37:32.5 MH: He was like, yep. Oh yeah. I was like, that’s not my experience now in a smaller company. I don’t know. What about you guys?

0:37:41.0 TW: I mean, I think the larger agencies you wind up in the challenge, and actually Mohammad’s talked about having like an account executive who was like really solid and let’s just say hypothetically, not from my personal experience. But at larger agencies you can wind up with the bigger it gets, the more kind of overhead there is and the more kind of expectations if the accounts are bigger… I feel like I’ve theoretically experienced being a little more whipsawed by the, the case of like, “Oh, you’re the analyst and let me tell you what I told the client you would do.” And it’s too late to shift that. Whereas the smaller agencies, I think there’s a lot more direct connection in the sales process and in the expectation setting and in the relationship. And there are kind of more barriers removed. And I think there are agencies and consultancies that have managed that really well, but I think there are also ones that just as they’re scaling they wind up inserting kind of some middleware in the client relationship that makes it really, really tough as an analyst.

0:38:48.2 VK: I’ll say one other difference for me is when I was at the smaller agencies and organisations, I felt very connected to the business. Like I kind of knew everyone and I knew who to go to and I understood like how we were performing month over month. I felt very connected to that. Whereas in the larger organisations, [laughter] felt kind of like lost. But they also could like take those bumps a little bit better, and so if I didn’t really enjoy a client or I really wanted, I felt like I’d outgrown it, I could like make a request and four weeks later, like my entire workload changed. And so I would say that there’s like pros and cons of all of it.

0:39:21.2 JH: One internal thing, less on the client side when coming into what was search discovery and it was a little smaller. It felt really nice on all my project, different teams that I had, it was very consistent and you kind of naturally picked up like, this is the way we do things. And it’s been interesting through the years to watch as we’ve gotten a lot larger as a company. And you get new people coming in, you get a lot of great new ideas, but it becomes a much more formal process that you have to set out and say, this is the way we do things. And you have to be better at communicating and training and kind of getting everybody on that same page of like, this is how we deliver, this is the way we run things. This is what this deliverable looks like, this is what we call things. So that’s been interesting to experience as well.

0:40:04.0 TW: Oh, I’m gonna do one add on to that. That’s a, it’s a great point that I think when you’re, when an agency is growing up by itself, it can figure that out. But you just spark the, when you bring in… Due to growth, bringing in people at different levels who come in with the way they’ve worked, what they’ve done, they’re being hired for their experience and their expertise and then trying to manage how that gets kind of folded in. And there is a tendency with all of us to wanna keep doing things the way we were doing and especially if you’ve been hired in and people are excited like, oh, you’re the hot new analyst coming in, you get that signal coming in and that makes you think and I need to put my imprimatur on this agency which really means I’m gonna go shake up the process they have without necessarily thinking through why they have the process and way of doing things and that does… That causes, I think, a lot of friction.

0:41:00.2 JH: Yeah and pressure.

0:41:00.8 TW: And doesn’t tend to get addressed head on. It just kind of happens and people get frustrated.

0:41:03.2 VK: That’s a good point.

0:41:04.6 MH: All right, I’m gonna ask one last question before we wrap up and it’s to all my panelists and I’ll try to answer to you. What’s one idea, insight, interesting thing you’ve taken from the conference this week?

0:41:19.3 JH: Okay, I’ll go first, It was actually the opening talk that Jim did about AI. That’s why I went first. Just the way you talked about thinking about using generative AI and you stated it’s not good at facts, use it to help you be creative and think outside the box. I love the prompts that you put up as examples of better ways to get it to kind of like tell you what else to ask it or different ways to think, especially the one too when you have a presentation and kind of outlining like if these three people were to look at my presentation and I left the room, what would they say about it? So I love those. I can’t wait to try them.

0:42:00.4 TW: This is 100% the thing where I’m gonna have like three or four other ones that I think of right after this.

0:42:05.4 MH: You can see the clock right there, Tim.

0:42:06.9 TW: That’s cool.

0:42:09.3 VK: I didn’t even notice exactly he said that.

0:42:12.2 TW: It’s like stall stall.

0:42:12.3 JH: This is like when you put the podcast on 1.5 speed. Okay. Tim go.

0:42:13.5 TW: Go go go. Well, because the thing was we were trying to recall later and this was actually not in a session and there were great sessions where I’m gonna and so Jim’s definitely, but Christopher, what was your line about? Help me out. The…

0:42:28.9 VK: Organisations…

0:42:31.4 TW: Organizations.

0:42:32.4 VK: Optimise.

0:42:33.7 TW: Optimise. You have to go back to it. They optimize to the least…

0:42:39.1 JH: Not have the conversation.

0:42:40.9 TW: Yeah. Christopher, what’s the line?

0:42:42.9 Christopher: Every organisation.

0:42:44.4 TW: Wait, hold on.

0:42:45.0 JH: Hold it.

0:42:45.2 TW: Wait a minute.

0:42:45.7 VK: Mic him.

0:42:46.6 JH: Mic him.

0:42:46.7 TW: Yeah Mic him… Mic him up.

0:42:49.3 Christopher: Every organisation optimises for avoiding the conversations they need to have.

0:42:57.1 JH: That’s it.

0:42:57.2 VK: Amen. So good.

0:42:58.8 TW: So.

0:43:00.6 VK: I like that.

0:43:00.7 TW: And there they were absolutely great sessions. That was just an easy sound bite.

0:43:04.0 VK: That was a good one. All right. So mine is gonna be a huge shout out to Barb and Claire, their presentation that was fucking phenomenal. It was really good and I had never I’m totally okay with saying I did not know what Cappy was before the presentation and I actually mispronounced it during their introduction. I’m like copy or cap. And like your presentation just walked through like it spoke to me as a beginner of that topic. But there was like 15 people who rushed them afterwards.

0:43:34.0 TW: Could you say what the topic is? What.

0:43:36.2 VK: Oh, sorry. It was demystifying and talking about laying some of the groundwork and some of the the reasons why you would investigate or see if it’s a good fit for you to deploy Cappy. Is that a good kind of.

0:43:49.6 Speaker 18: First party data strategies?

0:43:52.4 VK: First party data strategies And so.

0:43:54.3 S1: Yes.

0:43:54.4 VK: And so yeah, it was an excellent topic. And I think that those 15 people who rushed them after their presentation to ask them more questions. So really really well done.

0:44:03.0 S1: Thank you Val.

0:44:07.5 TW: Nice.

0:44:08.3 MH: And my, one of my favourites this week was probably the presentation by Martin Broadhurst, about how analysts can use a generative AI. Yeah he just walked into the room.

0:44:17.4 TW: He literally just walked into the room.

0:44:18.5 MH: It’s good I feel good about that.

0:44:21.0 TW: Michael was like, whoever I make eye contact with.

0:44:23.7 MH: That’s right.

0:44:23.8 TW: I’m gonna say that was my the best line.

0:44:25.4 MH: It’s your talk. That’s who it’s gonna be. Well, no, it was I thought was just well researched and well reasoned in terms of where we can as analysts lean into AI and where we can’t and I felt like that was very practical for me. So very lovely. All right. Well, thank you to all my co-hosts, Julie, Val, Tim, and also a huge shout out to our executive producer, Josh Crowhurst, who makes all this magic happen behind the scenes. You see what kind of crazy operation we’re running here and We’re doing this in person. And I just want to say thank you all for having us. It’s been a delight to be here at Marketing Analytics Summit this week and to share knowledge and be with our community. So thank you all very much.

0:45:10.3 Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at @Analytics Hour, on the web @analyticshour.io, our LinkedIn group and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst.

0:45:28.0 Charles Barkley: So show smart guys want to fit in so they made up a term called analytics. Analytics don’t work.

0:45:35.2 Kamala Harris: I love Venn diagrams. It’s just something about those three circles and the analysis about where there is the intersection, right?

0:45:42.8 MH: This is your moment.

0:45:46.5 MH: Yeah, yeah. Don’t worry about dead air. We’ll fix it in post.

0:45:50.7 JH: We usually say sorry, Josh, about 16 times in an episode. So which you never hear in the final edit.

0:46:00.4 MH: Rock flag and I am not an AI.

The post #249: Three Humans and an AI at Marketing Analytics Summit appeared first on The Analytics Power Hour: Data and Analytics Podcast.