Problem statement: We need to figure out if something has gone wrong that caused engagement to drop.

Problem understanding: Let’s see some example clarification questions we could ask on this problem:

• Q: Is it rational to assume that we measure user engagement through some version of Daily Active Users, or should I assume we use something a bit more sophisticated such as the number of posts, likes, etc?

• A: You can assume that we track we use a weighted sum of posts, replies, likes, and views.

• Q: Measuring this daily might be a very high variance, should I assume that we smooth this over a specific window, say weekly?

• A: Yes, you can assume that we monitor weekly averages.

• Q: How much did this metric move?

• A: Let’s assume that the metric moved 3 standard deviations

• Q: Is this metric really important?

• A: Internally, we know that a drop of 1 standard deviation of engagement maps to a 2% drop in Ads revenue.

We can keep asking questions but let’s assume that we now have a good understanding of what the interviewer is asking: We need to figure out what went wrong given that 3 standard deviations of change are costing a ton of revenue to the platform.

Metric definition: For this problem, we will use the engagement metric that the interviewer has given us.

Population: Now we can start digging into specifics:

• Q: Is this drop in engagement homogeneous across all users or does it affect any particular population in particular?
• A: Let’s assume that we don’t have any specific information on the type of users that it affects yet. How would you go about it?
• Q: We could break down the population into different cohorts, e.g., new users, tenured users, users across different geographic areas, and different age groups, and see whether this change originates from a particular cohort. Should we assume that indeed one of these cohorts is the problem?
• A: Let’s assume that you run the above analysis, and you find that the drop can be explained only through a drop in new user engagement.

Troubleshoot: We already know that the problem is with new users. The engagement equation involves posts, likes, and responses.

Q: Is the drop in all three components of the engagement? Or is it coming from only some of them?

A: Let’s assume that after looking into the data, we see that the number of posts of new users did not go down, however, there is a drop in likes and responses.

Q: If this is the case, then did something change in the way that new users interact with existing threads?

A: How would you test this?

Q: I would look into any recent product changes that affected the ability of new users to like or respond. One quick and easy test is to see whether the number of likes/responses of new users went to zero.

A: Let’s assume that it is not exactly zero.

Q: How about the correlation between posts and likes/responses?

A: After looking into that, we see that likes and responses are coming only from new users who have made new posts. What does this tell you?

Q: Maybe there was a recent code push that had a bug and restricted the ability of new users to like/respond unless they were first posting a new thread?

A: Exactly!