--- ## Overview **EHR Navigation Sequences as Predictors of Resident Performance** is a research study conducted with the UNC School of Information and Library Science and the University of Cincinnati College of Medicine. Electronic health record (EHR) audit logs offer a rich but underused signal into clinical work patterns. This study applies **sequential frequent pattern mining** and machine learning to examine whether the way graduate surgical residents navigate the EHR is predictive of their clinical performance. --- ## 📌 Project Info - **🧑‍💻 My Role:** Lead Researcher / Data Analyst - **🤝 Collaborators:** Danny T.Y. Wu, PhD, MSI (Faculty Advisor); Catherine G. Pratt, MD; Ralph C. Quillin III, MD - **📅 Project Time:** December 2025 – Present - **🛠️ Skills:** `Python`, `Sequential Pattern Mining`, `Frequent N-gram Extraction`, `Leave-One-Out Cross-Validation`, `Feature Selection (Mann–Whitney U)`, `Gradient Boosting`, `XGBoost`, `Statistical Analysis`, `EHR Audit Log Analysis` --- ## Purpose This study uses sequential pattern mining to identify differences in EHR navigation sequences between high- and low-performing residents, and to evaluate whether these patterns can serve as predictors of clinical performance — moving beyond single-point log metrics, which prior work showed were not effective at identifying low-performing residents. --- ## Methods - Analyzed **1.59 million EHR events** from **28 residents** using the Epic system, aggregated into daily navigation sequences where each character represents a discrete EHR section (e.g., Clinical Info, Reporting, Medications). - Extracted **contiguous n-gram patterns** (lengths 3–5) with a 10% minimum support threshold, yielding **1,180 candidate features** out of 7,643 total patterns. - Performed feature selection with **Mann–Whitney U tests** inside a **leave-one-out cross-validation (LOOCV)** framework to identify performance-discriminating patterns while preventing data leakage. - Evaluated **six machine learning models** (LR-L1, Decision Tree, Random Forest, SVM, Gradient Boost, XGBoost) using LOOCV, reporting AUROC, accuracy, sensitivity, and specificity. - Constructed a final feature set of **28 residents × 1,188 variables** (ID + navigation pattern features + outcome), including demographic and behavioral covariates such as gender, PGY group, sequence length statistics, and off-hour/weekend activity. --- ## Key Features - **End-to-End Pipeline**: Built a full pipeline — event preprocessing, encoding, sequence construction, pattern mining, feature construction, and predictive modeling — to go from raw audit logs to a validated predictive model. - **Leak-Free Validation**: In-fold feature selection within LOOCV ensured no data leakage between training and evaluation. - **Interpretable Patterns**: Top predictive patterns (e.g., **EAGA**: Reporting→Clinical Info→Demographics→Clinical Info; **GFIE**: Demographics→Problems→Medications→Reporting) were statistically significant (p < 0.001) and interpretable as navigation behaviors, not just black-box features. --- ## Results **Gradient Boosting** with the top 10 selected features achieved the best performance: **89.3% accuracy** and an **AUROC of 0.857** (92.9% sensitivity, 85.7% specificity), outperforming XGBoost, Decision Tree, and L1-regularized logistic regression. Specific navigation patterns — such as **EAGA** (Reporting→Clinical Info→Demographics→Clinical Info) and **GFIE** (Demographics→Problems→Medications→Reporting) — were significantly more frequent in lower-performing residents (p < 0.001). --- ## Discussion - Contiguous pattern mining is a valid method for predicting resident performance from EHR audit logs. - Lower-performing residents exhibit more **repetitive and fragmented** navigation patterns, while higher-performing residents show more **purposeful and structured** access. - Session length did not differ significantly between groups, suggesting that **navigation quality**, rather than activity volume, is what distinguishes performance. --- ## Technologies Used - **Python**: Core language for data processing, pattern mining, and modeling. - **Sequential/Frequent Pattern Mining**: Contiguous n-gram extraction over per-resident daily EHR navigation sequences. - **scikit-learn / XGBoost**: Model training and evaluation (LR-L1, Decision Tree, Random Forest, SVM, Gradient Boosting, XGBoost). - **Statistical Testing**: Mann–Whitney U tests for feature selection and pattern significance testing. --- ## Challenges and Learnings A key challenge was designing a feature selection process that avoided data leakage while still surfacing meaningful patterns from a high-dimensional, low-sample-size dataset (28 residents, 1,188 variables) — addressed by performing feature selection **within each LOOCV fold** rather than on the full dataset upfront. Translating raw, high-volume audit log events (3.37M → 1.59M after preprocessing) into clinically interpretable navigation patterns also required close collaboration with clinical co-authors to ensure the resulting features made behavioral sense, not just statistical sense. --- ## Conclusion This study demonstrates that sequential EHR behavior provides a predictive and behaviorally meaningful signal, offering a scalable and objective approach for assessing resident performance in surgical training programs. Future work will examine junior and senior residents separately and explore interaction effects between PGY group and key behavioral patterns. --- ## Acknowledgement This project was conducted as part of the School of Information and Library Science at the University of North Carolina at Chapel Hill, in collaboration with the University of Cincinnati College of Medicine. Special thanks to Ms. Roxy Huang and Dr. Suguna Kotte for their support with preliminary data analysis.