Jtdylktuy

I know that for categorical features we just calculate the prior and likelihood probability assuming conditional independence between the features.

How does it work for continuous variables? How can we calculate likelihood probability for continuous variable?

The difference boils down to "how we define $P(x_i|C_k)$?", where $x_i$ is a single feature, and $C_k$ is a class from a total of $K$ classes.

Discrete

In discrete case, $P(x_i|C_k)$ is represented by a table as follows:

x_i   P(x_i|C_k)

a     0.5

b     0.2

c     0.3

We have one of these tables for each feature-class pair $(i, k)$.

Lets denote $i$-th feature of data point $n$ as $x_{n,i}$. Each row of this table can be estimated using

$$hat{P}(x_i=a|C_k) = frac{sum_{n:n in C_k}{mathbb{1}_{x_{n,i}=a}}}{N_k}$$

which divides the number of samples that have i-th feature equal to $a$ by total number of samples in class $C_k$ (of course from the training set).

Also, check out pseudocount that avoids zero estimations.

Continuous

In continuous case, we either discretize the continuous interval of $x_i$ into bins ${b_1,..,b_m}$ and proceed the same as discrete case, or we assume a function like Gaussian (or any other one), as follows:

$$P(x_i|C_k)=frac{1}{sqrt{2pi}sigma_{i,k}}e^{-(x_i-mu_{i,k})^2/2sigma_{i,k}^2}$$

This way, for each feature-class pair $(i, k)$, $P(x_i|C_k)$ is represented with two parameters ${mu_{i,k}, sigma_{i, k}}$ instead of a table in discrete case. The estimation of the parameters is the same as fitting a Gaussian distribution to one dimensional data, that is:

$$hat{mu}_{i,k} = frac{sum_{n:n in C_k}{x_{n,i}}}{N_k}, hat{sigma}^2_{i, k} = frac{sum_{n:n in C_k}{(x_{n,i} - hat{mu}_{i,k})^2}}{N_k-1}$$

Instead of Gaussian, we can opt for a more complex function, even a neural network. In that case, we should look for a technique to fit the function to data just like what we did with Gaussian.

The rest is independent of feature type

Representing $P(C_k)$ is the same for both discrete and continuous cases and is estimated as $hat{P}(C_k)=N_k/N$.

Finally, the classifier is $$C(x_n) = underset{k in {1,..,K}}{mbox{argmax }}hat{P}(C_k)prod_{i}hat{P}(x_i=x_{n,i}|C_k)$$

Or equivalently using log-probabilities,

$$C(x_n) = underset{k in {1,..,K}}{mbox{argmax }}mbox{log}hat{P}(C_k)+sum_{i}mbox{log}hat{P}(x_i=x_{n,i}|C_k)$$