Home » Data Science Resources » Bias and Variance TradeOff

Bias and Variance TradeOff

Introduction:-

While designing solutions for any business problem with the help of machine learning many challenges are faced like data gathering, cleaning, transformation etc But the most important and critical is prediction errors. Machine learning algorithms aim to learn underlying pattern hidden in the dataset and this can be validated by check performance on the new or test data. Consider a case when solving a classification task, even if we have done a good amount of feature engineering, selection, still the algorithm gives huge misprediction, and the reasons can be that the algorithm selected is not that flexible to get the underlying pattern hidden in the dataset.

BIAS AND VARIANCE TRADEOFF

Generally, the error given by an algorithm is summed up as

ERROR=Bias2 +Variance +Irreducible Error

Let’s understand each component of the error one by one

Bias = This is simplifying assumptions made by the model to make the target function easier to learn.

Variance = Variance is the amount that estimate of the target function will change if the training data has been changed.

Irreducible Error = This error is native to the algorithm and cannot be reduced.

The goal of any supervised machine learning algorithm is to best estimate the mapping function for the output variable (y) given input data X.Mapping function is the hidden pattern in layman terms to understand.

The equation of the best fit line is called a mapping function.

Let’s see which machine learning algorithms have bias and variance

  • Linear algorithms like Linear Regression, Logistic Regression, LDA have high bias making then to learn faster but ultimately low test performance.
  • Algorithms like Decision Tree, KNN, SVM have a low bias.
  • Linear Regression, Logistic Regression and LDA have low variance.
  • Decision Tree, KNN, SVM have high variance.

In short consider bias is an error of training data and variance is an error of test data.

Let’s use train and test error to actually determine the bias and variance of the model.

Case 1 – When train error is 20% and the test error is 19%

Conclusion:- The created model is having high variance and high bias so this is called an under the fitted model.

Case 2 – When the training error is 2% and the test error is 20%

Conclusion:- The model has low bias and high variance. This is the condition of overfitting.

The optimal model should have low training and test error and hence that error is known as generalization error. Ultimately get a model with low bias and low variance.

Model bias and variance can be reduced by hyper tuning the parameters.

RandomForest:- Random Forest is a collection of multiple decision trees coupled parallelly. Decision tree itself has low bias and high variance i.e it will completely fit the training data(low bias ) but if a new test points come the error goes high(high variance). But when multiple decision trees are coupled with row and column sampling the combined variance offered by this collection is low. Hence Random forest is low bias and low variance model.

Boosting algorithms:- The same concept goes with boosting also as they have base learners as DecisionTree only so their aim to reduce the variance. The base learner is weak learners in which the bias is high. Each of these weak learners contributes some vital information for prediction enabling the boosting techniques to produce a strong learner. This stronger learn bring downs the variance.

About DataMites Team

DataMites Team will publish articles on various topics like data science, machine learning, artificial intelligence, deep learning, python programming, statistics, DataMites® press releases and career guidance.

32 comments

  1. It’s really a nice and helpful piece of information. I’m glad that you just shared this useful info with us. Please stay us informed like this. Thanks for sharing.

  2. I appreciate, cause I found exactly what I was looking for. You have ended my four day long hunt! God Bless you man. Have a great day. Bye

  3. Thanks for one’s marvelous posting! I really enjoyed reading it, you can be a great author.I will make certain to bookmark your blog and will come back in the future. I want to encourage continue your great writing, have a nice afternoon!

  4. It is really a nice and helpful piece of info. I am glad that you shared this useful information with us. Please keep us informed like this. Thanks for sharing.

  5. I have recently started a website, the information you provide on this website has helped me tremendously. Thanks for all of your time & work.

  6. You helped me a lot with this post. I love the subject and I hope you continue to write excellent articles like this.

  7. Thanks for your help and for posting this. It’s been wonderful.

  8. Thanks for writing this article

  9. Thank you for writing about this topic. Your post really helped me and I hope it can help others too.

  10. Thank you for sharing this article with me. It helped me a lot and I love it.

  11. You’ve been really helpful to me. Thank you!

  12. Thank you for your articles. They’re very helpful to me. May I ask you a question?

  13. Can you write more about it? Your articles are always helpful to me. Thank you!

  14. You’ve been terrific to me. Thank you!

  15. Thanks for your help and for posting this. It’s been great.

  16. Fantastic goods from you, man. I’ve understand your stuff previous to and you are just extremely excellent. I actually like what you have acquired here, really like what you’re saying and the way in which you say it. You make it enjoyable and you still take care of to keep it wise. I can not wait to read far more from you. This is actually a great site.

  17. That’s what i mean when i say that content is the king!

  18. Can you write more about it? Your articles are always helpful to me. Thank you!

  19. I really enjoyed reading your post and it helped me a lot

  20. Hi there, I found your site via Google at the same time as searching for a similar subject, your website came up, it seems great. I’ve bookmarked it in my google bookmarks.

  21. Thank you for your post. I liked reading it because it addressed my issue. It helped me a lot and I hope it will help others too.

  22. May I request more information on the subject? All of your articles are extremely useful to me. Thank you!

  23. Thank you for writing this post!

  24. I would like to know more about this subject if you don’t mind.

  25. Thank you for writing about this topic. Your post really helped me and I hope it can help others too.

  26. Thanks for posting. I really enjoyed reading it, especially because it addressed my issue. It helped me a lot and I hope it will help others too.

  27. Thank you for your post. I really enjoyed reading it, especially because it addressed my issue. It helped me a lot and I hope it will also help others.

  28. You’ve been a great help to me. Thank you!

  29. I really enjoyed reading your post, especially because it addressed my issue. It helped me a lot and I hope it can help others too.

  30. You’ve been a great help to me. Thank you!

  31. I appreciate, cause I found exactly what I was looking for. You’ve ended my 4 day long hunt! God Bless you man. Have a nice day. Bye

Leave a Reply

Your email address will not be published. Required fields are marked *

*

x

Check Also

What is the Salary for Python Developer in India

What is the Salary for Python Developer in India?

Python is leading the way in programming, which is the future of the planet. Its popularity is increasing tremendously with each passing year. Python is ...

Is Data Science and Artificial Intelligence in Demand in South Africa?

Is Data Science & Artificial Intelligence in Demand in South Africa?

According to the Economic Complexity Index, South Africa was the world’s number 38 economy in terms of GDP (current US$) in 2020, number 36 in ...