Hypothesis Testing - Categorical Variables
Prepared by Dr Wayne Bovey
Objective
According to Wikipedia, statistical hypothesis testing is a method of statistical inference used to determine whether the data sufficiently supports a particular hypothesis.
This vignette presents three examples of hypothesis testing between categorical variables using chi-square. The dataset comprised 616 respondents from 10 public and private sector organisations.
The tests focus on whether or not a relationship exists between:
- job role and education level
- gender and education level
- gender and job role.
Workflow
Tidied the raw dataset and derived hypotheses for testing. The null hypothesis (H0) implies that the variables are independent. In contrast, the alternative hypothesis (Ha) implies that the categorical variables are associated or related. A p-value of <0.01 was chosen for this vignette to determine statistical significance.
Calculated a contingency table and visualised a proportional stacked bar chart. The chi-square test examines whether rows and columns of a contingency table are associated and statistically significant. The following table, originally sourced from the Journal of Clinical and Preventive Cardiology, provides a straightforward interpretation of p-values for testing.
To conclude each test, a post hoc pairwise analysis is performed between the levels of each categorical variable to understand any relationship. Visualised the association between factor levels by plotting the standardised residuals. In these charts, positive residuals are blue. Positive values specify positive associations or dependencies between the corresponding row and column variables. Negative residuals are red and imply a negative association between the corresponding row and column variables. A table at the end of the vignette summarises testing results.
Results
Test 1: Job role and education
Hypothesis
H0: Job role and level of education are independent;
there is no association between these two variables.
Ha: Job role and level of education are associated;
a relationship exists between these two variables.
Table 1 calculates the cross-tabulation frequency of education level within each job role.
Table 1 Job role and education contingency table | ||||
|---|---|---|---|---|
Job role | High school | Certificate/Diploma | Undergraduate degree | Postgraduate degree |
Employee | 176 | 103 | 62 | 12 |
Supervisor | 46 | 24 | 10 | 11 |
Middle management | 45 | 22 | 29 | 25 |
Executive/Senior management | 6 | 3 | 11 | 20 |
N = 605 | ||||
Charts 1 and 2 visualise the frequency and proportion of education level within each job role, respectively.
Table 2 summarises the chi-square test result, followed by visualisation in Chart 3.
Table 2 Chi-square statistic for job role and education | ||
|---|---|---|
statistic | chisq_df | p_value |
110.7415 | 9 | 1.039e-19 |
Significance level: p < 0.01 | ||
Table 3 summarises the result of post hoc analysis based on residuals using the Bonferroni method with adjusted p-values to assist interpretation. Chart 4 visualises this result.
Table 3 Chi-square post hoc analysis for job role and education | |||||
|---|---|---|---|---|---|
Dimension | Value | High school | Certificate/Diploma | Undergraduate degree | Postgraduate degree |
Employee | Residuals | 2.769707 | 2.7213657 | -0.7110449 | -7.2261618 |
Employee | p values | 0.089771 | 0.1040200 | 1.0000000 | 0.0000000 |
Supervisor | Residuals | 1.128393 | 0.2981899 | -2.0047169 | 0.2779398 |
Supervisor | p values | 1.000000 | 1.0000000 | 0.7198930 | 1.0000000 |
Middle management | Residuals | -1.960824 | -1.9684577 | 1.7271464 | 3.6684360 |
Middle management | p values | 0.798393 | 0.7842460 | 1.0000000 | 0.0039050 |
Executive/Senior management | Residuals | -3.961874 | -2.6593262 | 1.5144337 | 8.0312789 |
Executive/Senior management | p values | 0.001190 | 0.1252750 | 1.0000000 | 0.0000000 |
Significance level: p < 0.01 | |||||
The above post hoc analysis shows executive/senior management and middle management generally held postgraduate qualifications compared to employees who generally did not.
Output 1 interprets the strength of the association between job role and education by calculating Cramér’s V.
Output 1
Cramer's V (adj.) | 95% CI | Interpretation
-------------------------------------------------
0.24 | [0.19, 1.00] | small
- One-sided CIs: upper bound fixed at [1.00].
- Interpretation rule: lovakov2021Result
Since the p-value of 1.039e-19 is less than the significance level of 0.01, the null hypothesis is rejected. A small relationship exists between job role and education.
Test 2: Gender and education
Hypothesis
H0: Gender and education level are independent;
there is no association between these two variables.
Ha: Gender and education level are associated; a
relationship exists between these two variables.
Table 4 calculates the cross-tabulation frequency of education level by gender.
Table 4 Gender and education contingency table | ||||
|---|---|---|---|---|
Gender | High school | Certificate/Diploma | Undergraduate degree | Postgraduate degree |
Female | 147 | 63 | 54 | 27 |
Male | 115 | 80 | 55 | 39 |
N = 580 | ||||
Charts 5 and 6 visualise the frequency and proportion of education level by gender. There was a close to even frequency of gender distribution in this study, with 291 females and 289 males.
Table 5 summarises the chi-square test result, followed by visualisation in Chart 7.
Table 5 Chi-square statistic for gender and education | ||
|---|---|---|
statistic | chisq_df | p_value |
8.113568 | 3 | 0.04372198 |
Significance level: p < 0.01 | ||
Table 6 summarises the result of post hoc analysis based on residuals using the Bonferroni method with adjusted p-values assisting interpretation. Chart 8 visualises this result.
Table 6 Chi-square post hoc analysis for gender and education | |||||
|---|---|---|---|---|---|
Dimension | Value | High school | Certificate/Diploma | Undergraduate degree | Postgraduate degree |
Female | Residuals | 2.594565 | -1.685290 | -0.1462405 | -1.598836 |
Female | p values | 0.075768 | 0.735461 | 1.0000000 | 0.878857 |
Male | Residuals | -2.594565 | 1.685290 | 0.1462405 | 1.598836 |
Male | p values | 0.075768 | 0.735461 | 1.0000000 | 0.878857 |
Significance level: p < 0.01 | |||||
This above post hoc analysis supports the chi-square results, with p-values across all levels in Table 6 not meeting the statistical significance level of 0.01.
Output 2 interprets the level of association between gender and education by calculating Cramér’s V.
Output 2
Cramer's V (adj.) | 95% CI | Interpretation
-------------------------------------------------
0.09 | [0.00, 1.00] | very small
- One-sided CIs: upper bound fixed at [1.00].
- Interpretation rule: lovakov2021Result
The p-value of 0.04372 is higher than the significance level of 0.01; therefore fails to reject the null hypothesis. There is very small or insufficient evidence to support an association between gender and level of education.
Test 3: Gender and job role
Hypothesis
H0: Gender and job role are independent; there is no
association between these two variables.
Ha: Gender and job role are associated; a
relationship exists between these two variables.
Table 7 calculates the cross-tabulation frequency of job role by gender.
Table 7 Gender and job role contingency table | ||||
|---|---|---|---|---|
Gender | Employee | Supervisor | Middle management | Executive/Senior management |
Female | 196 | 32 | 59 | 5 |
Male | 143 | 54 | 57 | 34 |
N = 580 | ||||
Charts 9 and 10 visualise the frequency and proportion of job role by gender.
Table 8 summarises the chi-square test result, followed by visualisation in Chart 11.
Table 8 Chi-square statistic for gender and job role | ||
|---|---|---|
statistic | chisq_df | p_value |
35.48673 | 3 | 9.614e-08 |
Significance level: p < 0.01 | ||
Table 9 summarises the result of post hoc analysis based on residuals using the Bonferroni method with adjusted p-values assisting interpretation. Chart 12 visualises this result.
Table 9 Chi-square post hoc analysis for gender and job role | |||||
|---|---|---|---|---|---|
Dimension | Value | Employee | Supervisor | Middle management | Executive/Senior management |
Female | Residuals | 4.268732 | -2.639900 | 0.1245712 | -4.852894 |
Female | p values | 0.000157 | 0.066344 | 1.0000000 | 0.000010 |
Male | Residuals | -4.268732 | 2.639900 | -0.1245712 | 4.852894 |
Male | p values | 0.000157 | 0.066344 | 1.0000000 | 0.000010 |
Significance level: p < 0.01 | |||||
The post hoc analysis confirmed that men were more likely to hold executive/senior management roles than women, who were more likely to be employees.
Output 3 interprets the strength of the association between job role and gender by calculating Cramér’s V.
Output 3
Cramer's V (adj.) | 95% CI | Interpretation
-------------------------------------------------
0.24 | [0.15, 1.00] | small
- One-sided CIs: upper bound fixed at [1.00].
- Interpretation rule: lovakov2021Result
Since the p-value of 9.614e-08 is less than the significance level of 0.01, the null hypothesis is rejected. A small association exists between gender and job role.
Summary
Table 10 summarises the testing results. Variables are listed with respective p-values, significance and outcomes from testing.
Table 10 Summary of categorical variable hypothesis testing | |||
|---|---|---|---|
variables | p_value | significance | H0 |
job role and education | 1.039e-19 | small | reject |
gender and education | 0.04372 | very small | fail to reject |
gender and job role | 9.614e-08 | small | reject |
Significance level: p < 0.01 | |||
Session information and package update
## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.4.0 (2024-04-24 ucrt)
## os Windows 11 x64 (build 22631)
## system x86_64, mingw32
## ui RTerm
## language (EN)
## collate English_Australia.utf8
## ctype English_Australia.utf8
## tz Australia/Brisbane
## date 2024-07-29
## pandoc 3.1.11 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date (UTC) lib source
## askpass 1.2.0 2023-09-03 [1] CRAN (R 4.4.0)
## backports 1.5.0 2024-05-23 [1] CRAN (R 4.4.0)
## bayestestR 0.14.0 2024-07-24 [1] CRAN (R 4.4.1)
## broom 1.0.6 2024-05-17 [1] CRAN (R 4.4.0)
## bslib 0.7.0 2024-03-29 [1] CRAN (R 4.4.0)
## cachem 1.1.0 2024-05-16 [1] CRAN (R 4.4.0)
## chisq.posthoc.test * 0.1.2 2019-10-25 [1] CRAN (R 4.4.0)
## cli 3.6.3 2024-06-21 [1] CRAN (R 4.4.1)
## coda 0.19-4.1 2024-01-31 [1] CRAN (R 4.4.0)
## codetools 0.2-20 2024-03-31 [2] CRAN (R 4.4.0)
## colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.4.1)
## corrplot * 0.92 2021-11-18 [1] CRAN (R 4.4.0)
## crayon 1.5.3 2024-06-20 [1] CRAN (R 4.4.1)
## crul 1.5.0 2024-07-19 [1] CRAN (R 4.4.1)
## curl 5.2.1 2024-03-01 [1] CRAN (R 4.4.0)
## data.table * 1.15.4 2024-03-30 [1] CRAN (R 4.4.0)
## datawizard 0.12.2 2024-07-21 [1] CRAN (R 4.4.1)
## devtools 2.4.5 2022-10-11 [1] CRAN (R 4.4.0)
## digest 0.6.36 2024-06-23 [1] CRAN (R 4.4.1)
## dplyr * 1.1.4 2023-11-17 [1] CRAN (R 4.4.0)
## effectsize * 0.8.9 2024-07-03 [1] CRAN (R 4.4.1)
## ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.4.0)
## emmeans 1.10.3 2024-07-01 [1] CRAN (R 4.4.1)
## estimability 1.5.1 2024-05-12 [1] CRAN (R 4.4.0)
## evaluate 0.24.0 2024-06-10 [1] CRAN (R 4.4.0)
## fansi 1.0.6 2023-12-08 [1] CRAN (R 4.4.0)
## farver 2.1.2 2024-05-13 [1] CRAN (R 4.4.0)
## fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0)
## flextable * 0.9.6 2024-05-05 [1] CRAN (R 4.4.0)
## fontBitstreamVera 0.1.1 2017-02-01 [1] CRAN (R 4.4.0)
## fontLiberation 0.1.0 2016-10-15 [1] CRAN (R 4.4.0)
## fontquiver 0.2.1 2017-02-01 [1] CRAN (R 4.4.0)
## forcats * 1.0.0 2023-01-29 [1] CRAN (R 4.4.0)
## fs 1.6.4 2024-04-25 [1] CRAN (R 4.4.0)
## gdtools 0.3.7 2024-03-05 [1] CRAN (R 4.4.0)
## generics 0.1.3 2022-07-05 [1] CRAN (R 4.4.0)
## gfonts 0.2.0 2023-01-08 [1] CRAN (R 4.4.0)
## ggplot2 * 3.5.1 2024-04-23 [1] CRAN (R 4.4.0)
## glue 1.7.0 2024-01-09 [1] CRAN (R 4.4.0)
## gtable 0.3.5 2024-04-22 [1] CRAN (R 4.4.0)
## here * 1.0.1 2020-12-13 [1] CRAN (R 4.4.0)
## highr 0.11 2024-05-26 [1] CRAN (R 4.4.0)
## hms 1.1.3 2023-03-21 [1] CRAN (R 4.4.0)
## htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0)
## htmlwidgets 1.6.4 2023-12-06 [1] CRAN (R 4.4.0)
## httpcode 0.3.0 2020-04-10 [1] CRAN (R 4.4.0)
## httpuv 1.6.15 2024-03-26 [1] CRAN (R 4.4.0)
## infer * 1.0.7 2024-03-25 [1] CRAN (R 4.4.0)
## insight 0.20.2 2024-07-13 [1] CRAN (R 4.4.0)
## jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.4.0)
## jsonlite 1.8.8 2023-12-04 [1] CRAN (R 4.4.0)
## knitr 1.48 2024-07-07 [1] CRAN (R 4.4.1)
## labeling 0.4.3 2023-08-29 [1] CRAN (R 4.4.0)
## later 1.3.2 2023-12-06 [1] CRAN (R 4.4.0)
## lattice 0.22-6 2024-03-20 [2] CRAN (R 4.4.0)
## lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0)
## lubridate * 1.9.3 2023-09-27 [1] CRAN (R 4.4.0)
## magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.4.0)
## MASS 7.3-60.2 2024-04-24 [2] local
## Matrix 1.7-0 2024-03-22 [2] CRAN (R 4.4.0)
## memoise 2.0.1 2021-11-26 [1] CRAN (R 4.4.0)
## mime 0.12 2021-09-28 [1] CRAN (R 4.4.0)
## miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.4.0)
## multcomp 1.4-26 2024-07-18 [1] CRAN (R 4.4.1)
## munsell 0.5.1 2024-04-01 [1] CRAN (R 4.4.0)
## mvtnorm 1.2-5 2024-05-21 [1] CRAN (R 4.4.0)
## officer 0.6.6 2024-05-05 [1] CRAN (R 4.4.0)
## openssl 2.2.0 2024-05-16 [1] CRAN (R 4.4.0)
## parameters 0.22.1 2024-07-21 [1] CRAN (R 4.4.1)
## pillar 1.9.0 2023-03-22 [1] CRAN (R 4.4.0)
## pkgbuild 1.4.4 2024-03-17 [1] CRAN (R 4.4.0)
## pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.4.0)
## pkgload 1.4.0 2024-06-28 [1] CRAN (R 4.4.1)
## profvis 0.3.8 2023-05-02 [1] CRAN (R 4.4.0)
## promises 1.3.0 2024-04-05 [1] CRAN (R 4.4.0)
## purrr * 1.0.2 2023-08-10 [1] CRAN (R 4.4.0)
## R6 2.5.1 2021-08-19 [1] CRAN (R 4.4.0)
## ragg 1.3.2 2024-05-15 [1] CRAN (R 4.4.0)
## Rcpp 1.0.13 2024-07-17 [1] CRAN (R 4.4.1)
## readr * 2.1.5 2024-01-10 [1] CRAN (R 4.4.0)
## remotes 2.5.0 2024-03-17 [1] CRAN (R 4.4.0)
## rlang 1.1.4 2024-06-04 [1] CRAN (R 4.4.0)
## rmarkdown 2.27 2024-05-17 [1] CRAN (R 4.4.0)
## rprojroot 2.0.4 2023-11-05 [1] CRAN (R 4.4.0)
## rstudioapi 0.16.0 2024-03-24 [1] CRAN (R 4.4.0)
## sandwich 3.1-0 2023-12-11 [1] CRAN (R 4.4.0)
## sass 0.4.9 2024-03-15 [1] CRAN (R 4.4.0)
## scales 1.3.0 2023-11-28 [1] CRAN (R 4.4.0)
## sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0)
## shiny 1.8.1.1 2024-04-02 [1] CRAN (R 4.4.0)
## stringi 1.8.4 2024-05-06 [1] CRAN (R 4.4.0)
## stringr * 1.5.1 2023-11-14 [1] CRAN (R 4.4.0)
## survival 3.5-8 2024-02-14 [2] CRAN (R 4.4.0)
## systemfonts 1.1.0 2024-05-15 [1] CRAN (R 4.4.0)
## textshaping 0.4.0 2024-05-24 [1] CRAN (R 4.4.0)
## TH.data 1.1-2 2023-04-17 [1] CRAN (R 4.4.0)
## tibble * 3.2.1 2023-03-20 [1] CRAN (R 4.4.0)
## tidyr * 1.3.1 2024-01-24 [1] CRAN (R 4.4.0)
## tidyselect 1.2.1 2024-03-11 [1] CRAN (R 4.4.0)
## tidyverse * 2.0.0 2023-02-22 [1] CRAN (R 4.4.0)
## timechange 0.3.0 2024-01-18 [1] CRAN (R 4.4.0)
## tzdb 0.4.0 2023-05-12 [1] CRAN (R 4.4.0)
## urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.4.0)
## usethis 2.2.3 2024-02-19 [1] CRAN (R 4.4.0)
## utf8 1.2.4 2023-10-22 [1] CRAN (R 4.4.0)
## uuid 1.2-0 2024-01-14 [1] CRAN (R 4.4.0)
## vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.4.0)
## withr 3.0.0 2024-01-16 [1] CRAN (R 4.4.0)
## xfun 0.46 2024-07-18 [1] CRAN (R 4.4.1)
## xml2 1.3.6 2023-12-04 [1] CRAN (R 4.4.0)
## xtable 1.8-4 2019-04-21 [1] CRAN (R 4.4.0)
## yaml 2.3.9 2024-07-05 [1] CRAN (R 4.4.1)
## zip 2.3.1 2024-01-27 [1] CRAN (R 4.4.0)
## zoo 1.8-12 2023-04-13 [1] CRAN (R 4.4.0)
##
## [1] C:/Users/wayne/AppData/Local/R/win-library/4.4
## [2] C:/Program Files/R/R-4.4.0/library
##
## ──────────────────────────────────────────────────────────────────────────────