Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review

Nwokonkwo Obi Chukwuemeka; John-Otumu Adetokunbo MacGregor; Nnadi Leonard Chukwualuka; Ogene Ferguson

doi:doi:10.11648/j.mlr.20220701.11

| Peer-Reviewed

Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review

Nwokonkwo Obi Chukwuemeka, John-Otumu Adetokunbo MacGregor, Nnadi Leonard Chukwualuka, Ogene Ferguson

Published in Machine Learning Research (Volume 7, Issue 1)

Received: 20 December 2021 Accepted: 8 January 2022 Published: 14 January 2022

Views: Downloads:

Download PDF

Share This Article

Twitter
Linked In
Facebook

Abstract

Automated machine learning (AutoML) models is one of several machine learning algorithms that can be used to automate the solution of real-world problems. It automates the selection, composition, and parameterization processes of the machine learning models in particular. Machine learning could be more user-friendly when it is automated, and it often produces faster, and more accurate results than hand-coded machine learning methods. For more than ten years, AutoML for supervised learning has been the main focus of research under the discipline of artificial intelligence, and significant progress has been made theeafter; consider the usefulness of AutoML methods in the most popular machine learning toolkits, as well as the AutoML mechanisms in large scale platforms such as Microsoft Azure. This paper provides a methodical analysis of the AutoML workflow as well as the state-of-the-art effort in dealing with the challenges involving Combined Algorithm Selection and Hyperparameter Optimization by gathering information about AutoML from several published articles from different online repositories in order to delve more into the methods used in different domains and the level of accuracy obtained. Findings revealed that the next generation of machine learning and artificial intelligence research is focused on automating the other phases of the whole end-to-end machine learning pipeline, from data comprehension to model deployment. With significantly better deep learning algorithms and big datasets, AutoML is predicted to be able to handle most of the data cleaning process in the future. AutoML will evolve into a highly human-competitive system that will change the way we think about data research.

Published in	Machine Learning Research (Volume 7, Issue 1)
DOI	10.11648/j.mlr.20220701.11
Page(s)	1-7
Creative Commons	This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.
Copyright	Copyright © The Author(s), 2024. Published by Science Publishing Group

Keywords

Transfer Learning, Machine Learning, Hyperparameter, Automation, Artificial Intelligence

References

[1]	Chauhan, K., Jani, S., Thakkar, D., Dave, R., Bhatia, J., Tanwar, S., & Obaidat, M. S. (2020). Automated Machine Learning: The New Wave of Machine Learning. (Icimia), 205–212.
[2]	Tuggener, L., Amirian, M., Rombach, K., Stefan, L., Varlet, A., Westermann, C., … Ag, P. (2019). Automated Machine Learning in Practice: State of the Art and Recent Results. 31–36. https://doi.org/10.1109/SDS.2019.00-11.
[3]	Lutkevich, B. (2020). What is automated machine learning (AutoML). Retrieved from https://searchenterpriseai.techtarget.com/definition/automated-machine-learning-AutoML.
[4]	Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., and Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529 (7587), 484–489. https://doi.org/10.1038/nature16961.
[5]	He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, 770–778. https://doi.org/10.1109/CVPR.2016.90.
[6]	Yao, Q., Wang, M., Chen, Y., Dai, W., Li, Y., Tu, W., … Yu, Y. (2018). Taking Human out of Learning Applications: A Survey on Automated Machine Learning. 1–20. Retrieved from http://arxiv.org/abs/1810.13306.
[7]	Mirtaheri, S. M., & Dinc, M. E. (2014). A Brief History of Web Crawlers. (May).
[8]	Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., Blum, M., & Hutter, F. (2019). Auto-sklearn: Efficient and Robust Automated Machine Learning.
[9]	Thornton, C., & Leyton-brown, K. (2013). Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. 847–855.
[10]	Escalante, H. J. (2020). Automated Machine Learning -- a brief review at the end of the early years. 1–17. Retrieved from http://arxiv.org/abs/2008.08516.
[11]	Olson, R. S., Bartley, N., Urbanowicz, R. J., & Moore, J. H. (2016). Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science.
[12]	Elsken, T., Metzen, J. H., & Hutter, F. (2019). Neural Architecture Search: A Survey. 20, 1–21.
[13]	Cai, H., Chen, T., Zhang, W., Yu, Y., & Wang, J. (2017). Efficient Architecture Search by Network Transformation. 2787–2794.
[14]	Donges, N. (2021). What Is Transfer Learning A Simple Guide Built In. Retrieved from https://builtin.com/data-science/transfer-learning.
[15]	Pan, S. J., & Fellow, Q. Y. (2009). A Survey on Transfer Learning. 1–15.
[16]	Kaul, A., Maheshwary, S., & Pudi, V. (2017). AutoLearn - Automated Feature Generation and Selection. https://doi.org/10.1109/ICDM.2017.31.
[17]	Krauß, J., Pacheco, B. M., Zang, H. M., Heinrich, R., Design, C., Stief, P., … Siadat, A. (2020). ScienceDirect ScienceDirect methodology to analyze the functional and physical architecture Automated machine learning for predictive quality in production existing products for an assembly oriented product family identification. Procedia CIRP, 93, 443–448. https://doi.org/10.1016/j.procir.2020.04.039.
[18]	Yang, C., Fan, J., Wu, Z., & Udell, M. (2020). AutoML Pipeline Selection: Efficiently Navigating the Combinatorial Space.
[19]	Swearingen, T., Drevo, W., Cyphers, B., Cuesta-Infante, A., Ross, A., & Veeramachaneni, K. (2017). ATM: A distributed, collaborative, scalable system for automated machine learning. Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017, 2018-January (December), 151–162. https://doi.org/10.1109/BigData.2017.8257923.
[20]	Liang, J., Meyerson, E., Hodjat, B., Fink, D., Mutch, K., & Miikkulainen, R. (2019). Evolutionary neural automl for deep learning. GECCO 2019 - Proceedings of the 2019 Genetic and Evolutionary Computation Conference, 401–409. https://doi.org/10.1145/3321707.3321721.
[21]	Feurer, M., & Hutter, F. (2019). Hyperparameter Optimization. 3–33.
[22]	Imbrea, A. (2021). Automated Machine Learning Techniques for Data Streams. Retrieved from http://arxiv.org/abs/2106.07317.
[23]	Mohr, F., Wever, M., & Hüllermeier, E. (2018). ML-Plan: Automated machine learning via hierarchical planning. Machine Learning. https://doi.org/10.1007/s10994-018-5735-z.
[24]	He, X., Zhao, K., & Chu, X. (2021). AutoML: A survey of the state-of-the-art. Knowledge-Based Systems, 212 (Dl). https://doi.org/10.1016/j.knosys.2020.106622.
[25]	Pan, W., Xiang, E. W., Liu, N. N., & Yang, Q. (2010). Transfer Learning in Collaborative Filtering for Sparsity Reduction. 230–235.
[26]	Lam, H. T., Thiebaut, J., & Sinn, M. (2016). One button machine for automating feature engineering in relational databases.
[27]	Katz, G., & Song, D. (2016). ExploreKit: Automatic Feature Generation and Selection. https://doi.org/10.1109/ICDM.2016.176.
[28]	Nargesian, F., Samulowitz, H., Khurana, U., Khalil, E. B., & Turaga, D. (2017). Learning Feature Engineering for Classification. 2529–2535.
[29]	Garrido-merch, E. C., Tom, F., Hern, D., & Tom, F. (2016). Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints.
[30]	Li, L., Jamieson, K., & Desalvo, G. (2018). Hyperband: A Novel Bandit-Based Approach to. 18, 1–52.
[31]	Baker, B., Gupta, O., and Raskar, N. N. (2018). Accelerating neural architecture search using performance prediction. 6^th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings, 1–19.
[32]	Mendoza, H. (2016). Towards Automatically-Tuned Neural Networks. 1–8.
[33]	Guyon, I., Sun-hosoya, L., Boullé, M., Escalante, H., Liu, Z., Jajetic, D., … Guyon, I. (2018). Analysis of the AutoML Challenge series 2015-2018 To cite this version: HAL Id: hal-01906197 Analysis of the AutoML Challenge series 2015-2018.
[34]	Hutter, F., Hoos, H. H., & Leyton-brown, K. (2011). Sequential Model-Based Optimization for General Algorithm Configuration.
[35]	Jamieson, K., & Talwalkar, A. (2016). Non-stochastic best arm identification and hyperparameter optimization. Proceedings of the 19^th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, 240–248.
[36]	Maclaurin, D., Duvenaud, D., and R. A. (2015). Hyper-Parameter Initialization for Squared Exponential Kernel-based Gaussian Process Regression Nalika Ulapane - Academia. Retrieved from https://www.academia.edu/45378935/Hyper_Parameter_Initialization_for_Squared_Exponential_Kernel_based_Gaussian_Process_Regression.

Cite This Article

Plain Text BibTeX RIS

APA Style

Nwokonkwo Obi Chukwuemeka, John-Otumu Adetokunbo MacGregor, Nnadi Leonard Chukwualuka, Ogene Ferguson. (2022). Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Machine Learning Research, 7(1), 1-7. https://doi.org/10.11648/j.mlr.20220701.11

Copy | Download

ACS Style

Nwokonkwo Obi Chukwuemeka; John-Otumu Adetokunbo MacGregor; Nnadi Leonard Chukwualuka; Ogene Ferguson. Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Mach. Learn. Res. 2022, 7(1), 1-7. doi: 10.11648/j.mlr.20220701.11

Copy | Download

AMA Style

Nwokonkwo Obi Chukwuemeka, John-Otumu Adetokunbo MacGregor, Nnadi Leonard Chukwualuka, Ogene Ferguson. Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Mach Learn Res. 2022;7(1):1-7. doi: 10.11648/j.mlr.20220701.11

Copy | Download

@article{10.11648/j.mlr.20220701.11,
  author = {Nwokonkwo Obi Chukwuemeka and John-Otumu Adetokunbo MacGregor and Nnadi Leonard Chukwualuka and Ogene Ferguson},
  title = {Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review},
  journal = {Machine Learning Research},
  volume = {7},
  number = {1},
  pages = {1-7},
  doi = {10.11648/j.mlr.20220701.11},
  url = {https://doi.org/10.11648/j.mlr.20220701.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.mlr.20220701.11},
  abstract = {Automated machine learning (AutoML) models is one of several machine learning algorithms that can be used to automate the solution of real-world problems. It automates the selection, composition, and parameterization processes of the machine learning models in particular. Machine learning could be more user-friendly when it is automated, and it often produces faster, and more accurate results than hand-coded machine learning methods. For more than ten years, AutoML for supervised learning has been the main focus of research under the discipline of artificial intelligence, and significant progress has been made theeafter; consider the usefulness of AutoML methods in the most popular machine learning toolkits, as well as the AutoML mechanisms in large scale platforms such as Microsoft Azure. This paper provides a methodical analysis of the AutoML workflow as well as the state-of-the-art effort in dealing with the challenges involving Combined Algorithm Selection and Hyperparameter Optimization by gathering information about AutoML from several published articles from different online repositories in order to delve more into the methods used in different domains and the level of accuracy obtained. Findings revealed that the next generation of machine learning and artificial intelligence research is focused on automating the other phases of the whole end-to-end machine learning pipeline, from data comprehension to model deployment. With significantly better deep learning algorithms and big datasets, AutoML is predicted to be able to handle most of the data cleaning process in the future. AutoML will evolve into a highly human-competitive system that will change the way we think about data research.},
 year = {2022}
}

Copy | Download

TY  - JOUR
T1  - Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review
AU  - Nwokonkwo Obi Chukwuemeka
AU  - John-Otumu Adetokunbo MacGregor
AU  - Nnadi Leonard Chukwualuka
AU  - Ogene Ferguson
Y1  - 2022/01/14
PY  - 2022
N1  - https://doi.org/10.11648/j.mlr.20220701.11
DO  - 10.11648/j.mlr.20220701.11
T2  - Machine Learning Research
JF  - Machine Learning Research
JO  - Machine Learning Research
SP  - 1
EP  - 7
PB  - Science Publishing Group
SN  - 2637-5680
UR  - https://doi.org/10.11648/j.mlr.20220701.11
AB  - Automated machine learning (AutoML) models is one of several machine learning algorithms that can be used to automate the solution of real-world problems. It automates the selection, composition, and parameterization processes of the machine learning models in particular. Machine learning could be more user-friendly when it is automated, and it often produces faster, and more accurate results than hand-coded machine learning methods. For more than ten years, AutoML for supervised learning has been the main focus of research under the discipline of artificial intelligence, and significant progress has been made theeafter; consider the usefulness of AutoML methods in the most popular machine learning toolkits, as well as the AutoML mechanisms in large scale platforms such as Microsoft Azure. This paper provides a methodical analysis of the AutoML workflow as well as the state-of-the-art effort in dealing with the challenges involving Combined Algorithm Selection and Hyperparameter Optimization by gathering information about AutoML from several published articles from different online repositories in order to delve more into the methods used in different domains and the level of accuracy obtained. Findings revealed that the next generation of machine learning and artificial intelligence research is focused on automating the other phases of the whole end-to-end machine learning pipeline, from data comprehension to model deployment. With significantly better deep learning algorithms and big datasets, AutoML is predicted to be able to handle most of the data cleaning process in the future. AutoML will evolve into a highly human-competitive system that will change the way we think about data research.
VL  - 7
IS  - 1
ER  -

Copy | Download

Author Information

Nwokonkwo Obi Chukwuemeka

Department of Information Technology, Federal University of Technology, Owerri, Nigeria
John-Otumu Adetokunbo MacGregor

Department of Computer Science, Morgan State University, Baltimore, USA
Nnadi Leonard Chukwualuka

Department of Information Technology, Federal University of Technology, Owerri, Nigeria
Ogene Ferguson

Department of Computer Science & Info Tech, Petroleum Training Institute, Effurun, Nigeria

Download PDF

Sections

Plain Text BibTeX RIS

APA Style

Nwokonkwo Obi Chukwuemeka, John-Otumu Adetokunbo MacGregor, Nnadi Leonard Chukwualuka, Ogene Ferguson. (2022). Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Machine Learning Research, 7(1), 1-7. https://doi.org/10.11648/j.mlr.20220701.11

Copy | Download

ACS Style

Nwokonkwo Obi Chukwuemeka; John-Otumu Adetokunbo MacGregor; Nnadi Leonard Chukwualuka; Ogene Ferguson. Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Mach. Learn. Res. 2022, 7(1), 1-7. doi: 10.11648/j.mlr.20220701.11

Copy | Download

AMA Style

Nwokonkwo Obi Chukwuemeka, John-Otumu Adetokunbo MacGregor, Nnadi Leonard Chukwualuka, Ogene Ferguson. Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Mach Learn Res. 2022;7(1):1-7. doi: 10.11648/j.mlr.20220701.11

Copy | Download

@article{10.11648/j.mlr.20220701.11,
  author = {Nwokonkwo Obi Chukwuemeka and John-Otumu Adetokunbo MacGregor and Nnadi Leonard Chukwualuka and Ogene Ferguson},
  title = {Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review},
  journal = {Machine Learning Research},
  volume = {7},
  number = {1},
  pages = {1-7},
  doi = {10.11648/j.mlr.20220701.11},
  url = {https://doi.org/10.11648/j.mlr.20220701.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.mlr.20220701.11},
  abstract = {Automated machine learning (AutoML) models is one of several machine learning algorithms that can be used to automate the solution of real-world problems. It automates the selection, composition, and parameterization processes of the machine learning models in particular. Machine learning could be more user-friendly when it is automated, and it often produces faster, and more accurate results than hand-coded machine learning methods. For more than ten years, AutoML for supervised learning has been the main focus of research under the discipline of artificial intelligence, and significant progress has been made theeafter; consider the usefulness of AutoML methods in the most popular machine learning toolkits, as well as the AutoML mechanisms in large scale platforms such as Microsoft Azure. This paper provides a methodical analysis of the AutoML workflow as well as the state-of-the-art effort in dealing with the challenges involving Combined Algorithm Selection and Hyperparameter Optimization by gathering information about AutoML from several published articles from different online repositories in order to delve more into the methods used in different domains and the level of accuracy obtained. Findings revealed that the next generation of machine learning and artificial intelligence research is focused on automating the other phases of the whole end-to-end machine learning pipeline, from data comprehension to model deployment. With significantly better deep learning algorithms and big datasets, AutoML is predicted to be able to handle most of the data cleaning process in the future. AutoML will evolve into a highly human-competitive system that will change the way we think about data research.},
 year = {2022}
}

Copy | Download

TY  - JOUR
T1  - Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review
AU  - Nwokonkwo Obi Chukwuemeka
AU  - John-Otumu Adetokunbo MacGregor
AU  - Nnadi Leonard Chukwualuka
AU  - Ogene Ferguson
Y1  - 2022/01/14
PY  - 2022
N1  - https://doi.org/10.11648/j.mlr.20220701.11
DO  - 10.11648/j.mlr.20220701.11
T2  - Machine Learning Research
JF  - Machine Learning Research
JO  - Machine Learning Research
SP  - 1
EP  - 7
PB  - Science Publishing Group
SN  - 2637-5680
UR  - https://doi.org/10.11648/j.mlr.20220701.11
AB  - Automated machine learning (AutoML) models is one of several machine learning algorithms that can be used to automate the solution of real-world problems. It automates the selection, composition, and parameterization processes of the machine learning models in particular. Machine learning could be more user-friendly when it is automated, and it often produces faster, and more accurate results than hand-coded machine learning methods. For more than ten years, AutoML for supervised learning has been the main focus of research under the discipline of artificial intelligence, and significant progress has been made theeafter; consider the usefulness of AutoML methods in the most popular machine learning toolkits, as well as the AutoML mechanisms in large scale platforms such as Microsoft Azure. This paper provides a methodical analysis of the AutoML workflow as well as the state-of-the-art effort in dealing with the challenges involving Combined Algorithm Selection and Hyperparameter Optimization by gathering information about AutoML from several published articles from different online repositories in order to delve more into the methods used in different domains and the level of accuracy obtained. Findings revealed that the next generation of machine learning and artificial intelligence research is focused on automating the other phases of the whole end-to-end machine learning pipeline, from data comprehension to model deployment. With significantly better deep learning algorithms and big datasets, AutoML is predicted to be able to handle most of the data cleaning process in the future. AutoML will evolve into a highly human-competitive system that will change the way we think about data research.
VL  - 7
IS  - 1
ER  -

Copy | Download