{"id":43,"date":"2013-09-07T18:19:10","date_gmt":"2013-09-08T00:19:10","guid":{"rendered":"http:\/\/inductivebias.com\/Blog\/?p=43"},"modified":"2015-10-10T23:59:58","modified_gmt":"2015-10-11T05:59:58","slug":"naive-bayes","status":"publish","type":"post","link":"http:\/\/inductivebias.com\/Blog\/naive-bayes\/","title":{"rendered":"Naive Bayes"},"content":{"rendered":"
\"Normal<\/a>Assume a distribution and conditional independence, calculate means and standard deviations, and use it to make predictions. It’s about the simplest thing that qualifies as machine learning. I’m not really a fan, but we’ve got to start somewhere, right?<\/strong><\/span><\/div>\n
Today we’re talking about the Naive Bayes Classifier. Not to be confused with the everything else<\/em> that has Bayes’ name on it.\u00a0 There’s a whole branch of Bayesian statistics. It’s all very interesting, but today we’re just talking about the parts it takes to build the most basic of classifiers, why it’s done the way it is and when it’s likely to work.<\/p>\n

Building the Classifier<\/span><\/h2>\n

The Naive Bayes classifier is a generative model (as opposed to a discriminative model), which means it works by building an approximate statistical model, and then using that model to calculate probabilities which are then used to make predictions. What makes the Naive Bayes model so naive is that all the inputs are assumed to be independent of each other (i.e. knowing one input doesn’t give you any information about any of the others), and typically the distributions are assumed to be a fixed shape.\u00a0 With these assumptions the “training” portion of the algorithm just consists of calculating the means and standard deviations for each input separately for each class.<\/p>\n

For the sake of brevity I’m going to assume you can figure out how to calculate variances. If not there’s a nice Wikipedia article<\/a> devoted to it. The important part here is that you calculate a separate distribution for each class, so that we can compare their likelihoods on new inputs. For my example I use the binary version of the algorithm, but there’s nothing stopping you from making a version that works on an arbitrary number of classes.<\/p>\n