The New Metric That Can Predict Product Sales

[00:00:02] Influencers on TikTok aren't just entertainers anymore. They're also sellers, blending engagement with product promotion. But marketers face a big question. Does high engagement on these influencer ads actually lead to increased sales? This is where a new study published in the Journal of Marketing Science steps in.

[00:00:23] The authors developed a new metric called the Product Engagement Score, which measures how well a product is integrated into a video and predicts its sales impact. They've tested this with thousands of short form videos linked to real sales data. With me today is one of the co-authors, Jeremy Yang, who teaches business administration at Harvard, and he joins me from his office in Cambridge, Massachusetts. Welcome. Thank you, Todd.

[00:00:48] Before I begin, I should mention that we are recording this mid-January. This is after TikTok was in front of the Supreme Court, but before we got a ruling on that. So as we record this today, the Chinese app Red Note is gaining in popularity, and there are rumors that, God help us all, Elon Musk is interested in buying the app. So just as a precursor, people of the future, that's the world where we sit as we record this right now.

[00:01:15] So, Dr. Yang, I guess I should ask as a precursor to our discussion, do you think your findings, which we'll talk about in a moment, do you think they would be similar across other short form video platforms like Instagram Reels or YouTube Shorts? Yeah, absolutely. Okay. All right. Well, let's continue then. Tell me about this PE score you developed and how does it work?

[00:01:34] Yeah. So I think, Todd, what you started was really the motivation for the study, which is, you know, when we look at these short videos on TikTok, but also across social platforms, people typically think about engagement as the North Star for how well the video is performing, right? Because it can be challenging to connect the video with sales data.

[00:01:58] So very often you see marketers using engagements such as number of likes, comments, and shares as indicator or proxy, noisy proxy, I would say, for how well the video will perform in driving sales. So starting from there, we use data to test if that hypothesis is actually true.

[00:02:16] It turns out in our data set, also it's also collaborated with, the result is verified by a few previous studies that it is not the case that engagement always leads to higher sales. So that's why we want to dive deeper into if the overall engagement is not a good indicator of sales, what exactly about the video that can tell us about the sales impact?

[00:02:43] You know, you're right. That has been the North Star, right? And, you know, when you go on these influencer marketing platforms and you're trying to select a creator, usually the score that they give you, I mean, they give you two things in most cases. They give you the number of followers, but they also give you engagement rate, you know, because that's sort of the has been, like you say, the proxy. Did that surprise you? It was a bit surprising before we think deeply about this topic.

[00:03:09] But after we find the result, we report in the paper. If you think about it, it's also not that surprising after the fact, right? Because, you know, in many cases, what makes the video engaging actually have nothing to do with the product itself. So that's why you see tons of engagement. But we call those engagement ineffective engagement. Because if the goal is to drive sales, it's not enough just to have the video being very engaging, but for reasons unrelated to the product.

[00:03:40] Right. That makes sense. Can we talk about your data set that you had? These were videos that you correlated to real sales data. What was the video source? What was the sales data? Walk us through where your data came from. So in order to test our key hypothesis in the paper, which is product engagement score, help predict sales lift. We need to combine two data sources. So one is the video data set that also contains engagement.

[00:04:09] So we got this from the Chinese version of TikTok or Doin. And then for the other part of the data, which is sales, because at the moment when the paper was written, so the project started in 2019. So at that time in the Chinese market, what happens is when the advertiser put an ad on Doin or Chinese TikTok to advertise for the product,

[00:04:37] the platform does not host the product. So they would link the product to external website, usually e-commerce aggregator, something like Taobao or Alibaba or JD.com. Right? Right.

[00:04:52] So in order to test our hypothesis, what we did was essentially linking the influencer ads on Doin or TikTok with the corresponding sales of the product being advertised from Taobao.com so that we can actually look at what's the sales impact after the video is posted. You know, I've seen a lot of sort of studies that have studied TikTok actually do use Doin data, the Chinese app.

[00:05:21] I'm not sure whether it's just much easier to get a hold of or whether, you know, I'm not quite sure why. Do you have a sense of why it's easier to use Doin data? Yeah. So I think the key reason is e-commerce on Doin is much more advanced and mature than e-commerce on TikTok. Right?

[00:05:41] So TikTok recently, they started to really prioritize their e-commerce in the international market. So they call this TikTok shop. Right? So before, I think it really started probably the last year. Before that, really it's not an option to buy stuff on TikTok. It's purely entertainment platform.

[00:06:04] But if you look at Doin, it's a much developed e-commerce ecosystem, I would say. So you would very easily be seeing product ads that contains a link to purchase. Very much like what you would see on e-commerce platform. Is there anything different about engagement from Chinese consumers on that app versus engagement from quote unquote Western nations on the TikTok app?

[00:06:34] It's a good question. So in the paper, I think the good thing about our method is that we don't decide a priori what elements in the video are driving engagement. We let the data tell us which part of video are engaging.

[00:06:51] And the way of doing that is essentially we have a lot of data on Doin video ads and their corresponding engagement measured by number of likes, comments, and shares generated on that video. Right? So we didn't do analysis with international TikTok data or data on other platforms.

[00:07:16] So I cannot tell you the result if what makes, what kind of content would engage Chinese consumers would be the same type of content that would engage international consumers. I'm not even thinking content though. I'm thinking like just sort of the mechanics of it. Like do Chinese consumers tend to have a higher click-through rate? Are they more likely to hit the heart button? Or maybe there's not been a study on that. I don't know. Not to my knowledge.

[00:07:45] Not to my knowledge. But you can very easily compare that if you have both data from Doin and TikTok. All right. Back to your score, the PE score. Yes. Can this score that you've developed, can it predict sales before a video is even posted? Yes. Yes, exactly. And that's the whole intention of the score, right? Because it's already too late after you post a video. So our intention is to compute a score prior to the release in an ad.

[00:08:14] Think about you're the marketer and then you might have a few different options of what kind of ad you might post on the platform. And then you can use the algorithm to generate the score. And you can use the score to select the winner, right? The best performing video to post. So exactly. We want to do this prior to the release of the video. And can it be used to help or can marketers use it to select the right influencer? Or is this just useful after the video is made?

[00:08:40] The algorithm is currently designed to take a video as the input. So the score is video specific. So even for the same influencer, say the influencer has 10 different videos, the scores will be different for each video. Right. However, that being said, you can still compute a score at the influencer level if you were to just aggregate the scores for different videos posted by the same influencer. So can I just sort of rephrase that and see if I have this right?

[00:09:10] So if I am trying to select, if I have access to this PE algorithm, I could go on to a TikTok account holder or, you know, reels or whatever, apply this score, generate an average PE score, and then use that. Theoretically anyway, that average PE score per influencer would reflect their likely sales potential. Is that fair? That's exactly right. Yeah.

[00:09:36] Your study highlighted that human presence and certain emotions can boost engagement. Can you share some examples of what worked best? So one key component in our algorithm is the engagement heat map, which is to use the algorithm to output a heat map overlaid on the original video that would highlight the key moment and regions in the video that are more engaging than other parts of the video.

[00:10:04] And then once we have that, we can try to ask the algorithm. Okay. So what objects, what kind of activity are there driving the engagement?

[00:10:26] So it's hard to, there are wide range of activities because as you can imagine, there are all sorts of content on TikTok. So that's why we stay with, I guess, these more generic terms to capture. Because I think overall, the thing that drives engagement in our data is really what we call these, you know, high intensity, high energy type of things.

[00:10:54] So like dancing, for example, that's the first thing people would think about when they think about TikTok video. So that would be one good example. And is it as simple as saying if they are dancing, the PE score is higher, therefore there is more likelihood of sales? It can't be that direct or simple. I'm glad you asked. That's a great question. So when we're thinking about engagement, we talk about engagement heat map. I didn't mention the product at all, right? So what's driving engagement? That's just one part of the equation. The other part is the product, right?

[00:11:23] So it is indeed not the case that, you know, if you just dance, you know, you have high engagement, you necessarily would have high product engagement score. That's because we have to consider product and engagement together. It's not enough just to make the video engaging, right? Back to our motivating example of, you know, many videos that are being engaging but not driving sales.

[00:11:47] So the key really is that you would want to feature the product in the most engaging parts of the video. So it's not enough just to be engaging. You have to be engaging in a way that introduce and present the product. You know, I don't want to pick it apart too much, but let me see. What does your gut tell you? I don't think you studied this level, but what does your gut tell you on something like this? 60-second video.

[00:12:14] You have the influencer be fairly, you know, light, I don't know, lightweight engaging. But then when the product, when they hold up the product, when they go for the jugular, then, you know, they turn on the energy. Is it as easy as that? Can we manipulate it that much? So in the video, we want the high engagement part to overlap with the part where the product pairs. So that's really the key and intuition behind the algorithm.

[00:12:40] So if you can find a way to make the product pairs engaging, then based on our algorithm, you would have a higher PE score. Does your heat map include things like facial expressions? It does. Oh, okay. And what did you learn from that? So facial expression. So we try to analyze the emotion of the face of the influencer.

[00:13:03] So what we find is happy, I think happy and sad faces tend to generate more engagement. Happy or sad. Happy or sad. Yeah. So is it just a sort of extreme emotions? Is it doesn't really matter what emotion, it's just recognizable? Or is it literally happy or sad and no other emotions? Yeah, we did test other strong emotions like anger, right? Anger, disgust.

[00:13:30] We didn't see those emotions generating higher engagement. It wasn't moving the needle on it. Yeah. The sad one surprises me. Did that surprise you? It was surprising. Because typically people tend to think about, you know, you have to be very positive being an influencer, promoting a product. But I guess sadness is a way to also captivate attention. People might be curious why you're so sad.

[00:13:55] And you mentioned that anger and other emotions didn't necessarily move the needle positively. Was it that it just didn't move the needle at all? Or did it have a negative effect on sales? So based on our result, it didn't move the needle at all. In either direction. Not in either direction. Yeah. Do you think that TikTok's format, you know, the short videos, fast-paced videos, makes it harder to integrate products effectively?

[00:14:21] I think it probably depends on what type of product. If it's a product that people really need to, for example, laptop or headphone, right? There's some specifications you might need to look into. Like how big is the screen? What is the resolution? What is the storage? How fast is the CPU? And stuff like that, right? So basically, you need to communicate a lot of information in that ad. And consumers need to process and think about this information.

[00:14:51] So if that's the case, then probably it's not very well suited for a very fast-paced video ad. Versus where, you know, for products, people don't really need to deliberate that much before making a purchase. Something very fast and quick. Then it's suited. Yeah. Can we get our hands on this PE calculation? Yeah. I know your paper is open source. Like your paper is published in its entirety. It's not just a...

[00:15:19] And we have a link to it I'll mention at the end. But is that in the paper? Can we just rip that off and start using it? Yes, you can. But with a caveat. So we have a licensing agreement with a startup that is productizing the algorithm. So you can totally access and use the code, but only for academic purpose. Oh. Well, that's not much fun. What did you have in mind? For profit purposes. For profit purposes.

[00:15:50] Yeah. I mean, I suppose that is sort of culturally appropriate in the academic world. But does it frustrate you at all? I mean, I certainly know as a marketer and someone who's married to a scientist that these things are done in sort of theoretical levels. But then don't get into the sort of the practical application. That gap must be frustrating for you as a scientist. Yeah, absolutely. Absolutely.

[00:16:16] So we definitely want to study something that is managerially relevant, right? We don't want to lock ourselves in the office just right on the whiteboard. So, you know, exactly. So overall, it's my intention to really close that gap. So finding ways to find practical use cases of the topics we study. Yeah. So this example would be one good example. Yeah. All right.

[00:16:44] So bottom line here, for marketers who are working with influencers, what should they be doing differently based on your findings? I would say two things. Right. So think about the process of marketers working with influencers. I would say there are two big steps. So one is selection. Right. Finding the right influencer to work with. The second thing is after you've found influencers you want to work with, then deciding what's the optimal content to post out there. Right.

[00:17:13] So for the first step, I would come back to Todd, the earlier question you asked, which is, can we compute something like a score for an influencer? And indeed, you propose the way to do that. So if you compute a P score for all the videos the influencer has posted in the past, then you can have an influencer level score. Right.

[00:17:35] And then marketers can use the score to select which influencer has the best, I guess, conversion potential. Right. So that's the first step.

[00:17:48] And after the marketer has found the right influencer to work with, you know, when the influencer is thinking about what kind of content to post, the brand, the advertiser can use the algorithm to essentially evaluate the sales potential of this video.

[00:18:11] And then if they're not happy with the result, they can tell the influencer, hey, based on our evaluation, it's not likely that this video is going to drive a lot of sales. So maybe we can tweak this and that part a bit and see, you know, maybe it will have a better score. Something like that. So it's like running it through a fortune machine, fortune telling machine. Yeah, that's right. You had co-authors on your paper. Who were they? Yes, I have two co-authors. One is Jun Jun Zhang, who is a professor at MIT.

[00:18:41] She is also my dissertation advisor. So this paper is one chapter in my thesis when I was at MIT. Oh, OK. And then the other person is Yu Han Zhang, who was at that point a visiting student at MIT. She was doing her PhD at Tsinghua in Beijing, China. And then now I think she is a professor at the Harbing Technology Institute. Well, the paper is very interesting.

[00:19:10] I think it gives us a lot to think about, especially as TikTok evolves. You know, as you and I sit here, as I mentioned at the start, we don't know which way it ended up evolving between when we're recording this and when this runs. But it certainly will be interesting. Jeremy, thank you. Thank you, Todd. Jeremy Yang is an assistant professor of business administration at Harvard University. His paper is called Engagement That Sells Influencer Video Advertising on TikTok. As I mentioned, the full paper is available for free online.

[00:19:39] That's sometimes a bit unusual. You can find it at the shortcut that we created, which is b.link slash ttstudy. I'm Todd Maffin. Thank you for listening. See you Friday for our wrap up of the week's news in digital marketing.