# Machine Learning Architectures From a large number of possible ML concepts and architectures, XGBoost \[7[^1]] has been selected from tree boosting ML methods, owing to its proven performance in solving similar problems, see \[8[^2]-9[^3]] and references therein. XGBoost, which a scalable, distributed gradient-boosted decision tree algorithm, was first introduced in 2016 by Tianqi Chen and Carlos Guestrin. The structure of XGBoost includes multiple root nodes, internal nodes, leaf nodes, and branches (Figure 3a). Then, the internal nodes make subsequent decisions, the branch points point directly to the decision to be made, and the leaf nodes represent the prediction results of a single three. Finally, the results of all leaf-pointing nodes are combined to obtain the prediction results of the XGBoost model \[8[^2]]. In the search for the best leaf node segmentation, XGBoost uses the basic exact greedy algorithm and the corresponding approximate algorithm to enumerate all the features to ensure accuracy \[9[^3]]. Many authors have shown that XGBoost is superior to other algorithms \[10[^4]] in handling tabular datasets, such as artificial neural networks (ANN) and support vector regression (SVR), however, in the last decade, deep learning counterparts, such as TabNet \[11[^5]], have emerged expanding the use of high-performance deep learning architectures from images and videos to tabular data. Its main features can be summarized as follows: (i) uses sparse instance-wise feature selection learned from data; (ii) constructs a sequential multi-step architecture, where each step contributes to a portion of the decision based on the selected features; (iii) improves the learning capacity via nonlinear processing of the selected features; and (iv) mimics ensembling via higher dimensions and more steps \[11[^5]]. The TabNet structure has one encoder and one decoder (Figure 3b). The encoder consists of multiple feature transformers and multiple attentive transformers, which obtain the mask matrix of the current step according to the result of the previous step, and tries to make the mask matrix sparse and non-repetitive. Thus, the attentive transformer plays the function of feature selection. The feature transformer layer does the calculation, and processing of the features selected in the current step, and uses the output of the previous step to determine the importance of the data features. The output of each step is used for the final decision by accumulation \[12[^6]].

Figure 3. (a) XGBoost architecture [10]

Figure 3. (b) Tabnet architecture [11]

[^1]: Chen T., Guestrin C.: XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). Association for Computing Machinery, New York, NY, USA, 785–794. 2016. [^2]: Zou, M.; Jiang, W\.-G.; Qin, Q.-H.; Liu, Y.-C.; Li, M.-L. Optimized XGBoost Model with Small Dataset for Predicting Relative Density of Ti-6Al-4V Parts Manufactured by Selective Laser Melting. Materials 2022, 15, 5298. [^3]: S. Yan, D. Chen, S. Wang and S. Liu: Quality prediction method for aluminum alloy ingot based on XGBoost, 2020 Chinese Control and Decision Conference (CCDC), Hefei, China, 2020, pp. 2542-2547, doi: 10.1109/CCDC49329.2020.9164112. [^4]: Deng, H., Zhou, Y., Wang, L. et al. Ensemble learning for the early prediction of neonatal jaundice with genetic features. BMC Med Inform Decis Mak 21, 338 (2021). [^5]: Arik O.S., Pfister T.: TabNet: Attentive Interpretable Tabular Learning, Cornel University Archive (2020) [^6]: Sun J., Yang F.: Multi-factor Investment Model Based on TabNet, J. Phys. (2022) Conf. Ser. 2171 012057 --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.totalmateria.com/predictor/predictor-2-whitepaper/development-methodology/machine-learning-architectures.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.