Date of Award
2023
Degree Name
Computer Science
College
College of Engineering and Computer Sciences
Type of Degree
M.S.
Document Type
Thesis
First Advisor
Dr. Jamil Chaudri, Committee Chairperson
Second Advisor
Dr. Husnu Narman
Third Advisor
Dr. Haroon Malik
Abstract
Performance testing generates vast amounts of data, making it challenging for human analysts to process within a reasonable timeframe. Therefore, black-box machine learning models are often used to determine pass/fail status, but these models lack transparency and cannot explain why a test has failed, leading to a time-consuming manual analysis process. To address this issue, this thesis proposes using Explainable Artificial Intelligence (XAI) to improve the trustworthiness of black-box and interpretable models in performance testing. The proposed approach leverages the Shapley Additive Explanation (SHAP) algorithm as a surrogate model to help performance analysts understand the decision-making process of black-box machine learning models. By wrapping SHAP around black-box models, analysts can gain explainability on why a model predicted a test's pass or fail status and identify the relative importance of performance data to machine learning models. The proposed approach was evaluated through several load text experiments on a real testbed, using industry-standard performance benchmarks, manually injecting performance bugs into the system to synthesize ground truth and building machine learning models using a black box learner (Artificial Neural Network) and an interpretable learner (Random Forest) to predict the test's pass or fail status. The results demonstrate that classical performance measures such as precision, recall, F-measure, and accuracy are not sufficient to gauge the reliability and trustworthiness of machine learning models. Instead, the proposed approach stands out by providing the explanations behind the decisions made by learning algorithms and enhancing their trustworthiness. The proposed approach, though evaluated through load testing can be generalized to other domains and require little or no effort to operate.
Subject(s)
Performance – Data processing.
Artificial intelligence – Data processing.
Artificial intelligence – Technological innovations.
Recommended Citation
Shoemaker, Eric, "Leveraging Explainable Artificial Intelligence (XAI) to Understand Performance Deviations in Load Tests of Large Software Systems" (2023). Theses, Dissertations and Capstones. 1822.
https://mds.marshall.edu/etd/1822
