Inductive bias is the set of assumptions a learner uses to predict results given inputs it has not yet encountered. It’s also the name of this blog. Let’s talk about swans.

Let’s suppose you don’t know anything about swans. You decide to visit Swan River in Australia. You see a swan for the first time, and that swan is black.  What color will the next Swan you see be? Black seems like a pretty good guess. It’s not a logical guess, but given what we know, black is a better guess than anything else. We could add extra assumptions to make it logical. How about “All swans are the same color”? If all swans are the same color, and there exists a black swan then it follows logically that all swans are black. This sort of added, often implicit, assumption about the nature of your problem is what’s known as inductive bias.

Rara avis in terris nigroque simillima cygno translates roughly to “a rare bird in the lands, and very like a black swan”.  It’s a satirical way of saying something doesn’t exist. Up until about 1700 black swans were used as a metaphor in Europe for things that aren’t real, but  I bet you didn’t need me to tell you that all swans aren’t actually black. In fact, you’re a swan expert now because you decided to travel the world recording thousands of swan sightings. So, I saw a swan recently. What color do you think it was? Well, being an expert in all things swan, you might ask: “Where did you see it?”.  I saw it in Toronto, and based on that you could tell me that it was probably white.  However, even having seen thousands of swans you can’t logically conclude the color of a new one without some added inductive bias, and using the exact same data you could get different results by using different biases.

Maybe you’ve assumed a new swan sighting will be the same color as the most common color of the 5 nearest previous sightings. That would be a k-nearest neighbor classifier. Maybe you’ve assumed each swan color has a normal distribution about a certain latitude. That would be a Naive Bayes classifier.  Maybe you’ve assumed the swan color distribution can be represented using 20 properly scaled and rotated  sigmoidal functions over location.  That could be handled by building an artificial neural network. Which one of these is the “right” answer? Well, It depends.

Generally speaking, the closer your assumptions are to reality the better your results will be.  Additionally, more and stronger assumptions tend to make solving problems easier. If an assumption is correct then it’s always helpful to use it. However, it’s also common to assume things that obviously aren’t true because they’re practical.  Every machine learning algorithm has some sort of inductive bias, and it’s those underlying assumptions you may not even realize you’re making that determine how well your algorithms are going to work in practice.

Inductive Bias is the set of assumptions a learner uses to predict results given inputs it has not yet encountered. This is a blog about machine learning, computer vision, artificial intelligence, mathematics, and computer science. It’s a blog about practical ways of doing things that aren’t technically possible, but mostly it’s a blog about how to write better AI.