Documents edited by AI

Probability theory Easy

Assume that 1 in every 100 documents is written by AI. We have a predictive model, that has 96% accuracy in predicting a true positive. The model makes a False Positive prediction in 10% of the human-written documents. If a document is labeled as written by AI, what is the likelihood that it is indeed written by AI?

To answer this question, we will apply the Bayes’ rule. From the description, we know that:

\[\begin{align} \Pr(\text{Written by AI}) &= 1/100 \\ \Pr(\text{Predicted Positive | Written by AI}) &= 96/100 \\ \Pr(\text{Predicted Positive | Written by humans}) &= 10/100 \end{align}\]

We are looking for the probability of \(\Pr(\text{Written by AI | Predicted Positive})\). From Bayes’ rule, we know that:

\[ \Pr(\text{Written by AI | Predicted Positive}) = \frac{\Pr(\text{Predicted Positive | Written by AI}) \Pr(\text{Written by AI})}{\Pr(\text{Predicted Positive})} \]

From the above, we only need to estimate the denominator:

\[ \begin{align} \Pr(\text{Predicted Positive} &= \Pr(\text{Predicted Positive | Written by AI}) * \Pr(\text{ Written by AI}) \\ &+ \Pr(\text{Predicted Positive | Written by humans}) * \Pr(\text{ Written by humans}) \\ &= 0.96 * 0.01 + 0.1 * 0.99 = 0.1086 \end{align} \]

Plugging the numbers we get:

\[ \Pr(\text{Written by AI | Predicted Positive}) = \frac{(0.96 * 0.01)}{0.1086} = 0.088 \]

The Bayes’ rule gives counter-intuitive results when there is an imbalance in the input distribution. In this case, the fact that AI prevalence is so low (1%) updates the posterior probability to be really low, despite the high recall (True Positive Rate) of the model (96%).

This is an AI-enhanced solution that took as input the original solution.

To determine this, we apply Bayes’ theorem. Based on the information provided:

\[\begin{align} \Pr(\text{AI-written}) & = 1/100 \\ \Pr(\text{Predicted as AI | AI-written}) & = 96/100 \\ \Pr(\text{Predicted as AI | Human-written}) & = 10/100 \end{align}\]

We seek \(\Pr(\text{AI-written | Predicted as AI})\). By employing Bayes’ theorem:

\[ \Pr(\text{AI-written | Predicted as AI}) = \frac{\Pr(\text{Predicted as AI | AI-written}) \cdot \Pr(\text{AI-written})}{\Pr(\text{Predicted as AI})} \]

Now, let's estimate the denominator:

\[ \begin{align} \Pr(\text{Predicted as AI}) &= \Pr(\text{Predicted as AI | AI-written}) \cdot \Pr(\text{AI-written}) \\ &+ \Pr(\text{Predicted as AI | Human-written}) \cdot \Pr(\text{Human-written}) \\ &= 0.96 \times 0.01 + 0.10 \times 0.99 = 0.1086 \end{align} \]

Substituting the values, we have:

\[ \Pr(\text{AI-written | Predicted as AI}) = \frac{0.96 \times 0.01}{0.1086} \approx 0.088 \]

The application of Bayes’ theorem often leads to surprisingly low probabilities when there's an imbalance in data, as evident here with the sparse prevalence of AI-authored documents (1%). This significantly lowers our posterior probability despite having a high true positive rate of 96%.

Bayes rule, Conditional probability

Monty Hall Medium (Bayes rule, Conditional independence, Prior evidence)
Unfair coin probability Easy (Bayes rule, Conditional probability)
Two fair die rolls Easy (Independence, CDF, PMF)
Largest number rolled Medium (Counting, Permutations, Repetition)
Sample digits 1-10 Medium (Sample from samples, Uniform)

How did you find the solution?
Correct
Needs more details
Incorrect
Did not read

Did you find this question to be:
Easy
Medium
Hard

Overall, do you feel this is a good interview question?

Did you see this question in an interview?

Thank you, we have received your feedback.

Documents edited by AI Sudden drop in user engagement Trailing by two: should we go for two or three?

Topics

Companies

Similar questions

Provide feedback

Documents edited by AI