Linear regression likelihood function
L1 (Lasso) vs. L2 (Ridge) regularization
Multiclass evaluation metrics
L1 (Lasso) vs. L2 (Ridge) regularization
Multiclass evaluation metrics
Machine Learning
Medium
Seen in real interview
Derive the likelihood of a linear regression model \(y=WX + \epsilon\) if the error distribution is normally distributed around zero (\(\epsilon \sim N(0,\sigma^2)\)).
The likelihood function will be (assume \(N\) observations):
\[ \begin{align} L &= \prod_i^N p(Y_i | X_i; W) = \prod_i^N N(Y_i | WX; \sigma^2) = \prod_i^N \frac{1}{2\pi \sigma^2} \exp(-\frac{1}{2\sigma^2}(y-\mu)^2) \\ logL = LL &= N \log(\frac{1}{2\pi \sigma^2}) - N \frac{1}{2\sigma^2} \sum_i^N (y-\mu)^2) \end{align} \]
We can replace \(\mu = WX\) since the error has zero mean. Typically we want to minimize the negative log-likelihood, hence:
\[ -LL = NLL = N \frac{1}{2\sigma^2} \sum_i^N (y-WX)^2) + \text{constant} \propto \sum_i^N (y-WX)^2) \]
Linear regression, Likelihood function, Formula derivation
- Bias-variance equation Easy (Bias-variance tradeoff, Formula derivation)
- Range of R2 when combining regressions Medium (Linear regression, Goodness of fit, R-squared, Correlation)
- Prove that a median minizes MAE Hard (MAE, Median, Formula derivation, Proof)
- Exponentially weighted moving average Medium (Exponentially weighted moving average, Formula derivation, Proof)
- Linear regression with stochastic gradient descent (formula derivation) Medium (Linear regression, Stochastic gradient descent, Formula derivation)
Thank you, we have received your feedback.