Machine Learning Classification Decision Tree
Prepared by Dr Wayne Bovey
Objective
According to Wikipedia, decision trees are a supervised machine learning approach commonly used in predictive modelling to draw conclusions about a set of observations. Decision tree models where the target variable is discrete are called classification trees, and trees, where the target variable is continuous (typically numbers), are called regression trees. Decision trees visually and explicitly represent decisions for decision-making.
This vignette demonstrates a classification decision tree using machine learning to model and predict client satisfaction with service quality. Service quality is critical for business success. It is essential for attracting new customers and retaining existing customers. The data set consists of observations from 424 clients in a business-to-business (B2B) relationship. Data was gathered on a custom-designed survey instrument based on the SERVQUAL theoretical framework. SERVQUAL consists of five dimensions described as follows:
- Tangibles – the appearance of physical facilities, equipment, personnel and communication materials
- Assurance – knowledge and ability to inspire trust and confidence
- Reliability – ability to perform service dependably and accurately
- Responsiveness – willingness to help and provide prompt service
- Empathy – providing caring and individualised attention.
In addition to the above specific dimensions, measured each client’s overall satisfaction with service quality.
Workflow
The raw data set was wrangled and tidied before processing. Conducted a brief exploratory analysis comprising a statistical summary, distribution visualisation and correlation analysis to understand the variables.
The classification decision tree model was developed and fitted on the training data using a workflow that considered resampling methods, feature engineering and hyperparameter optimisation. Reviewed the results of the training model and identified important predictor variables associated with a client’s overall satisfaction with service quality.
The trained model was then applied to the unseen test data to predict the target or outcome variable. Evaluated performance and metrics of the model on the test data visually with ROC curves and confusion matrix heatmap.
Results
1. Explore data
Commenced by reviewing the proposed outcome variable for the classification tree. The existing seven-point Likert scale measuring overall satisfaction was collapsed into a three-level categorical scale (dissatisfied, neutral, satisfied). Tables 1 and 2 summarise the transformation of the outcome variable from a seven-point Likert scale to a three-level categorical variable.
| Table 1 Original Likert scale | |
| Overall satisfaction | Freq |
|---|---|
| Very Dissatisfied | 3 |
| Dissatisfied | 15 |
| Partially Dissatisfied | 16 |
| Neutral | 29 |
| Partially Satisfied | 39 |
| Satisfied | 245 |
| Very Satisfied | 77 |
| Table 2 Categorical scale for classification | |
| Overall satisfaction | Freq |
|---|---|
| dissatisfied | 34 |
| neutral | 29 |
| satisfied | 361 |
Table 3 is a statistical summary for each of the five explanatory variables.
| Table 3 Statistical summary of explanatory variables | ||||||||||||
| variable | n | mean | sd | median | trimmed | mad | min | max | range | skew | kurtosis | se |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| tangibles | 424 | 5.92 | 0.90 | 6.00 | 6.05 | 0.74 | 2.00 | 7.00 | 5.00 | −1.61 | 3.51 | 0.04 |
| assurance | 424 | 5.52 | 1.07 | 6.00 | 5.61 | 0.74 | 2.00 | 7.00 | 5.00 | −0.85 | 0.09 | 0.05 |
| reliability | 424 | 5.43 | 1.32 | 6.00 | 5.60 | 0.74 | 1.00 | 7.00 | 6.00 | −1.13 | 0.60 | 0.06 |
| responsiveness | 424 | 5.29 | 1.37 | 6.00 | 5.44 | 1.48 | 1.00 | 7.00 | 6.00 | −0.90 | 0.23 | 0.07 |
| empathy | 424 | 5.39 | 1.17 | 5.50 | 5.47 | 1.48 | 1.00 | 7.00 | 6.00 | −0.78 | 0.71 | 0.06 |
Chart 1 combination violin box plots show the distribution of five explanatory variables. The box plots show favourable levels of service quality for each explanatory variable.
Chart 2 pairs plot compares the relationship between the five numeric explanatory variables and the multilevel outcome variable, overall satisfaction.
2. Classification Tree
2.1 Train model
2.1.1 Build model
Commenced building the model by randomly splitting the data into a training and testing set at a 3:1 ratio using stratified sampling. Stratified sampling allocates approximately equal proportions of observations across the range of values for the outcome variable to balance the training and testing sets. Resampled data in the training set using 10-fold cross-validation repeated five times. The recipe for the classification tree was a standard formula, with no additional feature engineering. Hyperparameters were tuned to default settings initially, then optimised using regular grid search. Added these components to a workflow for execution.
The workflow was implemented using parallel processing. During this process, tuned the hyperparameters. Table 4 shows a short list of parameter tuning combinations.
| Table 4 Short-list of parameter tuning combinations | ||||||||
| cost_complexity | tree_depth | min_n | .metric | .estimator | mean | n | std_err | .config |
|---|---|---|---|---|---|---|---|---|
| 1.000000e-10 | 8 | 21 | accuracy | multiclass | 0.8855 | 50 | 0.0084 | Preprocessor1_Model13 |
| 1.000000e-10 | 15 | 21 | accuracy | multiclass | 0.8855 | 50 | 0.0084 | Preprocessor1_Model16 |
| 3.162278e-06 | 8 | 21 | accuracy | multiclass | 0.8855 | 50 | 0.0084 | Preprocessor1_Model14 |
| 3.162278e-06 | 15 | 21 | accuracy | multiclass | 0.8855 | 50 | 0.0084 | Preprocessor1_Model17 |
| 1.000000e-10 | 8 | 40 | accuracy | multiclass | 0.8730 | 50 | 0.0087 | Preprocessor1_Model22 |
| 1.000000e-10 | 8 | 21 | roc_auc | hand_till | 0.7738 | 50 | 0.0180 | Preprocessor1_Model13 |
| 1.000000e-10 | 15 | 21 | roc_auc | hand_till | 0.7738 | 50 | 0.0180 | Preprocessor1_Model16 |
| 3.162278e-06 | 8 | 21 | roc_auc | hand_till | 0.7738 | 50 | 0.0180 | Preprocessor1_Model14 |
| 3.162278e-06 | 15 | 21 | roc_auc | hand_till | 0.7738 | 50 | 0.0180 | Preprocessor1_Model17 |
| 1.000000e-10 | 8 | 40 | roc_auc | hand_till | 0.7671 | 50 | 0.0162 | Preprocessor1_Model22 |
Model 13 in Table 4 has the best parameter tuning combination with the equal highest mean score for accuracy and roc_auc with equal lowest cost-complexity, minimum tree depth and a minimum number of nodes. Model 13 parameters have been extracted in Table 5.
| Table 5 Optimal model parameter tuning combination | |||
| cost_complexity | tree_depth | min_n | .config |
|---|---|---|---|
| 1e-10 | 8 | 21 | Preprocessor1_Model13 |
Chart 3 visualises Tables 4 and 5 by comparing parameter tuning combinations. Model 13, with a tree depth of 8, and minimal node size of 21, share the equal highest mean accuracy and mean roc_auc with a tree depth of 15, as shown in Table 3. Consequently, the green line for tree depth 8 is plotted beneath tree depth 15 and is not visible in Chart 3.
With parameter tuning complete, the workflow and model were finalised for fitting.
2.1.2 Fit and review model
The tuned model was fitted on the training set resulting in the classification decision tree shown in Chart 4.
Chart 5 illustrates the importance of predictor variables on the target variable, overall satisfaction. Reliability and responsiveness were calculated as the most important variables explaining overall client satisfaction.
2.2 Test model
2.2.1 Predict on test data
The model fitted on the training data was then applied to the unseen testing data to predict the outcome variable. Table 6 shows actual overall satisfaction and predicted overall satisfaction (.pred_class) in a small sample of observations extracted from the testing data.
| Table 6 Sample of outcome variable predictions in test data | ||||||
| tangibles | assurance | reliability | responsiveness | empathy | overall_satisfaction | .pred_class |
|---|---|---|---|---|---|---|
| 4.5 | 4.0 | 6.0 | 5.5 | 4.5 | satisfied | satisfied |
| 6.0 | 6.0 | 5.5 | 6.0 | 6.0 | satisfied | satisfied |
| 5.0 | 5.0 | 5.0 | 5.0 | 5.0 | satisfied | satisfied |
| 5.5 | 4.0 | 5.0 | 4.0 | 4.5 | neutral | satisfied |
| 6.5 | 4.0 | 6.0 | 5.0 | 5.5 | satisfied | satisfied |
| 6.0 | 5.0 | 2.0 | 1.5 | 4.5 | dissatisfied | dissatisfied |
| 6.5 | 6.5 | 6.5 | 6.0 | 6.5 | satisfied | satisfied |
| 6.0 | 3.5 | 2.0 | 2.5 | 4.5 | dissatisfied | dissatisfied |
| 6.0 | 6.0 | 5.0 | 4.5 | 5.5 | satisfied | satisfied |
| 6.0 | 3.5 | 3.5 | 3.0 | 4.0 | dissatisfied | dissatisfied |
| 5.5 | 5.0 | 3.5 | 4.0 | 4.5 | satisfied | satisfied |
| 4.0 | 4.0 | 4.0 | 4.0 | 4.0 | neutral | neutral |
| 7.0 | 6.0 | 6.5 | 4.5 | 5.0 | satisfied | satisfied |
| 6.0 | 6.0 | 6.0 | 6.0 | 4.0 | satisfied | satisfied |
| 6.5 | 6.5 | 6.5 | 7.0 | 6.5 | satisfied | satisfied |
2.2.2 Review model on test data
Table 7 summarises key classification metrics for test set predictions.
| Table 7 Classification metrics for the test set | |||
| .metric | .estimator | .estimate | .config |
|---|---|---|---|
| accuracy | multiclass | 0.9528 | Preprocessor1_Model1 |
| roc_auc | hand_till | 0.7803 | Preprocessor1_Model1 |
The ROC curve (receiver operating characteristic curve) plots the true positive rate (sensitivity) against the false positive rate (specificity) at all classification thresholds. AUC (area under the curve) measures the entire two-dimensional area underneath the ROC curve. Chart 6 shows ROC curves for each of the three outcome classes. The ROC curve for dissatisfied is almost perfect, with the model strongly predicting dissatisfaction. However, the model was not as accurate in predicting neutral observations, which impacted both neutral and satisfied ROC curves in Chart 6. The combined multiclass ROC-AUC on the test data in Table 7 was 0.7803.
Chart 7, the confusion matrix heatmap, summarises predictions by categorising and comparing the predictions against the true responses of the three outcome classes. The confusion matrix calculates model accuracy at 95.3 per cent (true positive, true neutral, and true negative). As mentioned above, Chart 7 shows the model accurately predicted satisfaction and dissatisfaction; however, it was partially accurate in predicting neutral satisfaction levels.
Reference:
Data was gathered using a custom-designed survey instrument based on the SERVQUAL theoretical framework. The SERVQUAL methodology is documented in Delivering Quality Service: Balancing Customer Perceptions and Expectations by Zeithaml, Parasuraman and Berry.
Session information and package update
## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.4.0 (2024-04-24 ucrt)
## os Windows 11 x64 (build 22631)
## system x86_64, mingw32
## ui RTerm
## language (EN)
## collate English_Australia.utf8
## ctype English_Australia.utf8
## tz Australia/Brisbane
## date 2024-07-30
## pandoc 3.1.11 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date (UTC) lib source
## backports 1.5.0 2024-05-23 [1] CRAN (R 4.4.0)
## broom * 1.0.6 2024-05-17 [1] CRAN (R 4.4.0)
## bslib 0.7.0 2024-03-29 [1] CRAN (R 4.4.0)
## cachem 1.1.0 2024-05-16 [1] CRAN (R 4.4.0)
## checkmate 2.3.1 2023-12-04 [1] CRAN (R 4.4.0)
## class 7.3-22 2023-05-03 [2] CRAN (R 4.4.0)
## cli 3.6.3 2024-06-21 [1] CRAN (R 4.4.1)
## codetools 0.2-20 2024-03-31 [2] CRAN (R 4.4.0)
## colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.4.1)
## cvms * 1.6.1 2024-02-27 [1] CRAN (R 4.4.0)
## data.table * 1.15.4 2024-03-30 [1] CRAN (R 4.4.0)
## devtools 2.4.5 2022-10-11 [1] CRAN (R 4.4.0)
## dials * 1.2.1 2024-02-22 [1] CRAN (R 4.4.0)
## DiceDesign 1.10 2023-12-07 [1] CRAN (R 4.4.0)
## digest 0.6.36 2024-06-23 [1] CRAN (R 4.4.1)
## doParallel * 1.0.17 2022-02-07 [1] CRAN (R 4.4.0)
## dplyr * 1.1.4 2023-11-17 [1] CRAN (R 4.4.0)
## ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.4.0)
## evaluate 0.24.0 2024-06-10 [1] CRAN (R 4.4.0)
## fansi 1.0.6 2023-12-08 [1] CRAN (R 4.4.0)
## farver 2.1.2 2024-05-13 [1] CRAN (R 4.4.0)
## fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0)
## fontawesome 0.5.2 2023-08-19 [1] CRAN (R 4.4.0)
## forcats * 1.0.0 2023-01-29 [1] CRAN (R 4.4.0)
## foreach * 1.5.2 2022-02-02 [1] CRAN (R 4.4.0)
## fs 1.6.4 2024-04-25 [1] CRAN (R 4.4.0)
## furrr 0.3.1 2022-08-15 [1] CRAN (R 4.4.0)
## future 1.33.2 2024-03-26 [1] CRAN (R 4.4.0)
## future.apply 1.11.2 2024-03-28 [1] CRAN (R 4.4.0)
## generics 0.1.3 2022-07-05 [1] CRAN (R 4.4.0)
## GGally * 2.2.1 2024-02-14 [1] CRAN (R 4.4.0)
## ggplot2 * 3.5.1 2024-04-23 [1] CRAN (R 4.4.0)
## ggstats 0.6.0 2024-04-05 [1] CRAN (R 4.4.0)
## globals 0.16.3 2024-03-08 [1] CRAN (R 4.4.0)
## glue 1.7.0 2024-01-09 [1] CRAN (R 4.4.0)
## gower 1.0.1 2022-12-22 [1] CRAN (R 4.4.0)
## GPfit 1.0-8 2019-02-08 [1] CRAN (R 4.4.0)
## gt * 0.11.0 2024-07-09 [1] CRAN (R 4.4.1)
## gtable 0.3.5 2024-04-22 [1] CRAN (R 4.4.0)
## gtExtras * 0.5.0 2023-09-15 [1] CRAN (R 4.4.0)
## hardhat 1.4.0 2024-06-02 [1] CRAN (R 4.4.0)
## here * 1.0.1 2020-12-13 [1] CRAN (R 4.4.0)
## highr 0.11 2024-05-26 [1] CRAN (R 4.4.0)
## hms 1.1.3 2023-03-21 [1] CRAN (R 4.4.0)
## htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0)
## htmlwidgets 1.6.4 2023-12-06 [1] CRAN (R 4.4.0)
## httpuv 1.6.15 2024-03-26 [1] CRAN (R 4.4.0)
## infer * 1.0.7 2024-03-25 [1] CRAN (R 4.4.0)
## ipred 0.9-15 2024-07-18 [1] CRAN (R 4.4.1)
## iterators * 1.0.14 2022-02-05 [1] CRAN (R 4.4.0)
## jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.4.0)
## jsonlite 1.8.8 2023-12-04 [1] CRAN (R 4.4.0)
## knitr 1.48 2024-07-07 [1] CRAN (R 4.4.1)
## labeling 0.4.3 2023-08-29 [1] CRAN (R 4.4.0)
## later 1.3.2 2023-12-06 [1] CRAN (R 4.4.0)
## lattice 0.22-6 2024-03-20 [2] CRAN (R 4.4.0)
## lava 1.8.0 2024-03-05 [1] CRAN (R 4.4.0)
## lhs 1.2.0 2024-06-30 [1] CRAN (R 4.4.1)
## lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0)
## listenv 0.9.1 2024-01-29 [1] CRAN (R 4.4.0)
## lubridate * 1.9.3 2023-09-27 [1] CRAN (R 4.4.0)
## magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.4.0)
## MASS 7.3-60.2 2024-04-24 [2] local
## Matrix 1.7-0 2024-03-22 [2] CRAN (R 4.4.0)
## memoise 2.0.1 2021-11-26 [1] CRAN (R 4.4.0)
## mime 0.12 2021-09-28 [1] CRAN (R 4.4.0)
## miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.4.0)
## mnormt 2.1.1 2022-09-26 [1] CRAN (R 4.4.0)
## modeldata * 1.4.0 2024-06-19 [1] CRAN (R 4.4.1)
## munsell 0.5.1 2024-04-01 [1] CRAN (R 4.4.0)
## nlme 3.1-164 2023-11-27 [2] CRAN (R 4.4.0)
## nnet 7.3-19 2023-05-03 [2] CRAN (R 4.4.0)
## paletteer 1.6.0 2024-01-21 [1] CRAN (R 4.4.0)
## parallelly 1.37.1 2024-02-29 [1] CRAN (R 4.4.0)
## parsnip * 1.2.1 2024-03-22 [1] CRAN (R 4.4.0)
## pillar 1.9.0 2023-03-22 [1] CRAN (R 4.4.0)
## pkgbuild 1.4.4 2024-03-17 [1] CRAN (R 4.4.0)
## pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.4.0)
## pkgload 1.4.0 2024-06-28 [1] CRAN (R 4.4.1)
## plyr 1.8.9 2023-10-02 [1] CRAN (R 4.4.0)
## prodlim 2024.06.25 2024-06-24 [1] CRAN (R 4.4.1)
## profvis 0.3.8 2023-05-02 [1] CRAN (R 4.4.0)
## promises 1.3.0 2024-04-05 [1] CRAN (R 4.4.0)
## psych * 2.4.6.26 2024-06-27 [1] CRAN (R 4.4.1)
## purrr * 1.0.2 2023-08-10 [1] CRAN (R 4.4.0)
## R6 2.5.1 2021-08-19 [1] CRAN (R 4.4.0)
## RColorBrewer 1.1-3 2022-04-03 [1] CRAN (R 4.4.0)
## Rcpp 1.0.13 2024-07-17 [1] CRAN (R 4.4.1)
## readr * 2.1.5 2024-01-10 [1] CRAN (R 4.4.0)
## recipes * 1.1.0 2024-07-04 [1] CRAN (R 4.4.1)
## rematch2 2.1.2 2020-05-01 [1] CRAN (R 4.4.0)
## remotes 2.5.0 2024-03-17 [1] CRAN (R 4.4.0)
## rlang 1.1.4 2024-06-04 [1] CRAN (R 4.4.0)
## rmarkdown 2.27 2024-05-17 [1] CRAN (R 4.4.0)
## rpart * 4.1.23 2023-12-05 [2] CRAN (R 4.4.0)
## rpart.plot * 3.1.2 2024-02-26 [1] CRAN (R 4.4.0)
## rprojroot 2.0.4 2023-11-05 [1] CRAN (R 4.4.0)
## rsample * 1.2.1 2024-03-25 [1] CRAN (R 4.4.0)
## rstudioapi 0.16.0 2024-03-24 [1] CRAN (R 4.4.0)
## sass 0.4.9 2024-03-15 [1] CRAN (R 4.4.0)
## scales * 1.3.0 2023-11-28 [1] CRAN (R 4.4.0)
## sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0)
## shiny 1.8.1.1 2024-04-02 [1] CRAN (R 4.4.0)
## stringi 1.8.4 2024-05-06 [1] CRAN (R 4.4.0)
## stringr * 1.5.1 2023-11-14 [1] CRAN (R 4.4.0)
## survival 3.5-8 2024-02-14 [2] CRAN (R 4.4.0)
## tibble * 3.2.1 2023-03-20 [1] CRAN (R 4.4.0)
## tidymodels * 1.2.0 2024-03-25 [1] CRAN (R 4.4.0)
## tidyr * 1.3.1 2024-01-24 [1] CRAN (R 4.4.0)
## tidyselect 1.2.1 2024-03-11 [1] CRAN (R 4.4.0)
## tidyverse * 2.0.0 2023-02-22 [1] CRAN (R 4.4.0)
## timechange 0.3.0 2024-01-18 [1] CRAN (R 4.4.0)
## timeDate 4032.109 2023-12-14 [1] CRAN (R 4.4.0)
## tune * 1.2.1 2024-04-18 [1] CRAN (R 4.4.0)
## tzdb 0.4.0 2023-05-12 [1] CRAN (R 4.4.0)
## urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.4.0)
## usethis 2.2.3 2024-02-19 [1] CRAN (R 4.4.0)
## utf8 1.2.4 2023-10-22 [1] CRAN (R 4.4.0)
## vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.4.0)
## vip * 0.4.1 2023-08-21 [1] CRAN (R 4.4.0)
## withr 3.0.0 2024-01-16 [1] CRAN (R 4.4.0)
## workflows * 1.1.4 2024-02-19 [1] CRAN (R 4.4.0)
## workflowsets * 1.1.0 2024-03-21 [1] CRAN (R 4.4.0)
## xfun 0.46 2024-07-18 [1] CRAN (R 4.4.1)
## xml2 1.3.6 2023-12-04 [1] CRAN (R 4.4.0)
## xtable 1.8-4 2019-04-21 [1] CRAN (R 4.4.0)
## yaml 2.3.9 2024-07-05 [1] CRAN (R 4.4.1)
## yardstick * 1.3.1 2024-03-21 [1] CRAN (R 4.4.0)
##
## [1] C:/Users/wayne/AppData/Local/R/win-library/4.4
## [2] C:/Program Files/R/R-4.4.0/library
##
## ──────────────────────────────────────────────────────────────────────────────