• Problem statement: We need to figure out if something has gone wrong that caused engagement to drop.

• Problem understanding: Let’s see some example clarification questions we could ask on this problem:

  • Q: Is it rational to assume that we measure user engagement through some version of Daily Active Users, or should I assume we use something a bit more sophisticated such as the number of posts, likes, etc?

  • A: You can assume that we track we use a weighted sum of posts, replies, likes, and views.

  • Q: Measuring this daily might be a very high variance, should I assume that we smooth this over a specific window, say weekly?

  • A: Yes, you can assume that we monitor weekly averages.

  • Q: How much did this metric move?

  • A: Let’s assume that the metric moved 3 standard deviations

  • Q: Is this metric really important?

  • A: Internally, we know that a drop of 1 standard deviation of engagement maps to a 2% drop in Ads revenue.

  We can keep asking questions but let’s assume that we now have a good understanding of what the interviewer is asking: We need to figure out what went wrong given that 3 standard deviations of change are costing a ton of revenue to the platform.

• Metric definition: For this problem, we will use the engagement metric that the interviewer has given us.

• Population: Now we can start digging into specifics:

  • Q: Is this drop in engagement homogeneous across all users or does it affect any particular population in particular?
  • A: Let’s assume that we don’t have any specific information on the type of users that it affects yet. How would you go about it?
  • Q: We could break down the population into different cohorts, e.g., new users, tenured users, users across different geographic areas, and different age groups, and see whether this change originates from a particular cohort. Should we assume that indeed one of these cohorts is the problem?
  • A: Let’s assume that you run the above analysis, and you find that the drop can be explained only through a drop in new user engagement.

• Troubleshoot: We already know that the problem is with new users. The engagement equation involves posts, likes, and responses.

• Q: Is the drop in all three components of the engagement? Or is it coming from only some of them?

• A: Let’s assume that after looking into the data, we see that the number of posts of new users did not go down, however, there is a drop in likes and responses.

• Q: If this is the case, then did something change in the way that new users interact with existing threads?

• A: How would you test this?

• Q: I would look into any recent product changes that affected the ability of new users to like or respond. One quick and easy test is to see whether the number of likes/responses of new users went to zero.

• A: Let’s assume that it is not exactly zero.

• Q: How about the correlation between posts and likes/responses?

• A: After looking into that, we see that likes and responses are coming only from new users who have made new posts. What does this tell you?

• Q: Maybe there was a recent code push that had a bug and restricted the ability of new users to like/respond unless they were first posting a new thread?

• A: Exactly!

We'll tackle this issue by methodically exploring potential causes:

1. Identify the Problem: We need to determine if something specific caused the decline in user engagement.

2. Clarify the Problem: By asking targeted questions, we aim to understand the situation better:

   • Q: Is user engagement measured by Daily Active Users, or is it a more detailed metric? A: It's a composite metric, a weighted combination of posts, replies, likes, and views.

   • Q: Are these metrics averaged weekly to smooth out daily variance? A: Yes, we analyze them on a weekly basis.

   • Q: What magnitude of change are we observing? A: A change of 3 standard deviations is noted.

   • Q: Is this change significant? A: A 1 standard deviation drop in engagement can lead to a 2% loss in ad revenue.

   With this information, we understand the task: diagnosing a revenue-impacting decline of 3 standard deviations in engagement.

3. Engagement Metric: We'll use the provided engagement metric to assess the situation.

4. Analyze Population: Let's delve into demographic impacts:

   • Q: Is the engagement drop uniform among all users? A: Without specific user data, how would you proceed?

   • Q: By analyzing user cohorts (e.g., new vs. tenured users), can we identify a particular issue? A: Our analysis shows a decline is confined to new user engagement.

5. Troubleshoot: Focus on new user activities - posts, likes, responses.

   • Q: Is the decline across all engagement components? A: Rather, the issue is with likes and responses, not new posts.

   • Q: How do new users interact with existing threads? A: How would you examine this?

   • Q: Consider recent platform changes affecting new user interaction. A: Verify by checking if likes/responses reached zero.

   • Q: Explore posts-likes/response correlation. A: It indicates only new posters are liking/responding. Insights?

   • Q: Could a recent bug limit new users' interactions unless posting first? A: Precisely!