Comparing Saudi Students’ Academic Performance Before and After Implementing E-Learning During The Covid-19 Pandemic in the Adult Nursing Course: A Mixed-Methods Study Dr. Sharifah Alsayed1 College of Nursing King Saud bin Abdulaziz University for Health Sciences Dr. Ebtsam Aly Abou Hashish2 College of Nursing K

Introduction

The COVID-19 pandemic has changed education delivery and imposed e-learning as an essential tool. Many institutions have switched to e-learning programs in response to the pandemic to continue with educational plans and to support students during thepandemic1. Similarly, King Saud bin Abdulaziz University for Health Sciences (KSAU-HS) used e-learning as a strategy to keep all students on track with their education plans2.

However, many challenges were faced by the educators and students due to the quick shift to e-learning, and the outcomes of the switch to e-learning still require further evaluation. For example, several studies have reported some challenges associated with e-learning, especially in health education, like bad networks, a lack of clinical experience, lack of motivation to learn, and stress associated with online exams3,4.

Despite these challenges, e-learning is highly adaptable in terms of providing teaching opportunities at any time and from any location1,5. In addition, it is a useful strategy to contain infection rates during the pandemic and maintain social distance as the most effective preventative strategy during the pandemic6.It also supports different teaching methods like small-group discussions, large group teaching, online examinations, and other learning materials5.

As part of Saudi Arabia’s Vision 2030, e-learning should be widely implemented to overcome the high cost of traditional learning and to get some advantages, like a flexible education system7,8. Yet little is known about the impact of shifting to e-learning on students' performance in nursing education, and the challenges associated with e-learning will improve students' experiences and faculty willingness to participate in e-learning programs6.

Background

Before the COVID-19 pandemic, all medical and nursing courses at the study site, King Saud bin Abdulaziz University for Health Sciences (KSAU-HS), were taught on campus. This included lectures in face-to-face classrooms, clinical training in simulation labs, and clinical placements in hospitals, followed by a final assessment and exam on campus.

Likewise in nursing education, there are essential practical and clinical courses taught in the curriculum, like fundamental nursing, adult nursing, pediatric nursing, obstetrics, women's health, etc. Both theoretical and clinical components are included in these courses, where the theoretical part is covered by lectures and the clinical part is covered by Clinical Simulation Lab (CSL) training and hospital placements.

During the implementation of e-learning, all lectures were shifted to virtual lectures through the blackboard, and the physical labs were replaced by 50% virtual clinical sessions through the blackboard and 50% physical lab sessions at CSL because hospital training was stopped during the pandemic. In terms of assessment, all exams were done online except the final exam, which was conducted on campus. But, for clinical assessment, students submitted clinical assignments through Blackboard and performed clinical evaluations in the CSL. The mid Objective Structured Clinical Examination (OSCE) was replaced by an online clinical exam through Blackboard, while the final OSCE was conducted on campus.

Significance of the study

Currently, the emphasis on student success in higher education has shifted to identifying the factors that can be improved. Previous studies revealed that students have varied levels of satisfaction with the online learning experience and agreed that online learning is more flexible, helps them improve their time management skills, and motivates their self-directed learning. Nevertheless, the students faced many challenges with online learning that affected their social, professional, and clinical skills2,3. To the best of the researchers' knowledge, no prior studies have been conducted on a specific clinical nursing courses like adult nursing, which is one of the major courses essential for preparing nursing students with the basic clinical skills required before and during clinical placement and in the hospital. This research project is a part of the medical education program at KSAU-HS, and it’s worth lies in using the mixed-methods approach to have a comprehensive picture of students' experiences of the course before and after switching to online learning. Understanding the online learning experiences of nursing students and how these experiences affect their academic performance allows for the identification of strategies to improve their online experiences and learning satisfaction2,3.

Thus, there is a need to assess students’ experiences regarding E-learning to identify the obstacles that students experience to devise suitable action plans that improve their academic achievement and to evaluate the learning outcomes.

Aim

The purpose of this study was to compare the academic performance of nursing students in an adult nursing course before and after the implementation of e-learning during COVID-19 pandemic.

Methods

Setting and designs

The study was conducted at College of Nursing-Jeddah at King Abdulaziz University for Health Sciences (CON-J, KSAU-HS), on adult nursing course students in Fall 2019-2020 and Fall 2020-2021. This is a sequential exploratory mixed-method study. A causal comparative design was used in the quantitative part, while a phonomyography design was used in the qualitative part9.

Sampling and subjects

A non-probability convenience sampling technique was used in the quantitative part, and a total number of 140 students' academic transcripts were reviewed for students who are enrolled in adult nursing course in Fall 2021 and transcripts were checked to assess if there was a significant difference in students’ academic performance before and after implementing e-learning strategies.

The Openepi website was used as a sample size calculator, using a confidence interval of 95% with a power of 80%. The recommended sample size was calculated as 62. The distribution of the population size for the two selected semesters was 74 students in Fall semester 2019-2020 who completed their adult nursing course in campus and 66 students who shifted to E-learning in Fall 2020-2021.

Data collection and Ethical considerations

Data collection was done after obtaining the Institutional Board Review approval (IBR) from King Abdullah International Medical Research Center (KAIMRC) at KSAU-HS. The IBR number is SP21R/442/11. The study was conducted on the premises of the College of Nursing-Jeddah at KSAU-HS. Informed consent was taken from students who agreed to participate. The consent did not include their names or IDs. Participants were informed with the purpose of the study and their voluntary participation with the right of withdrawal at any time without penalties. The interviews were conducted in a private and quiet place. The privacy of the conversation and the recording was assured as all data was saved anonymously and accessed only by the research team.

Quantitative data were collected from students with the following inclusion criteria: (1) had experience of studying adult nursing course before and after implementing E-learning; (2) regular students; and (3) students varied in academic performance from high to average to low performance. A convenient sample of 140 students were included by comparing their academic transcripts in two semesters before and after implementing e-learning.

For quantitative data, the academic transcripts were reviewed for all students who are enrolled in adult nursing course in Fall semester 2019-2020 and Fall 2021 to assess if there was a significant difference in students’ academic performance before and after implementing e-learning strategies.

Qualitative data were collected through focus group interviews, with a purposeful sample of 12 participants. Two focus group interviews were conducted via the Microsoft team with 6 students in each interview. As data saturation was reached after the two interviews, the interview was stopped at this point. The purpose of these interviews was to explore students’ experiences in taking adult nursing courses before and after implementing E-learning strategies.

Data analysis and management

For quantitative data, SPSS version 27 was used for data entry and analysis. T-tests and analysis of variance were used to investigate any significant differences among the scores before and after implementing E-learning strategies into the course. A Pearson correlation coefficient analysis (r) was used to test the nature of the relationship between study variables. All statistical analysis was done on a significance level of 0.05.

For qualitative data, thematic analysis was used to highlight commonalities and patterns in the student’s responses about their personal experience through coding and categorizing themes. Following transcription, initial line-by-line coding started. This process is called "open coding" or "initial coding."19,22. Then, substantive codes (focused coding) were created to synthesize and explain larger segments of data focused coding. Lastly, themes were developed to link all substantive codes, which helped in explaining data22.

Academic rigor

Trustworthiness was established by examining the consistency of themes in the transcripts and contrasting these observations with the literature. The three coders of this study required a consensus about the observations and interpretation of the data. Data was constantly compared and deliberated to attain concurrence. I ensured that the phrasing of the codes also reflected the actual words of the participants.

Results

I-Quantitative results

Students’ performance on course domains

The academic transcripts showed higher grades in all theoretical domains with e-learning compared to on-campus, with a significant difference (t =-11.6, p<0.001). In contrast, with e-learning, grades were significantly lower in all clinical domains compared to on-campus, with a significant (t = 15.7, p 0.001). All results are presented in tables I, II, and III.

Table I: Course Domain Parts Score
Items	On campusMean (SD)	E-learningMean (SD)	Significance
Total clinical score	30.4 (2.6)	24.3 (1.9)	t= 15.8, p<0.001*
Total theoretical score	45.7 (4.6)	54.7 (4.5)	t= -11.6, p<0.001*
Total course score	76.1 (6.0)	79.0 (5.5)	t= - 2.9, p=0.004*

Note: *p 0.05

Table II: Clinical Domain Parts Score
Items	On campusMean (SD)	E-learningMean (SD)	Significance t p
Clinical performance evaluation	10.8 (1.6)	8.4 (0.5)	t= 11.3, p<0.001*
Mini OSCE	3.7 (0.6)	3.0 (0.9)	t= 4.9, p<0.001*
Professionalism & Attendance	3.3 (0.5)	4.9 (0.5)	t= -17.5, p<0.001*
Final OSCE	12.5 (1.2)	7.2 (1.2)	t= 22.5, p<0.001*
Total clinical score	30.4 (2.6)	24.3 (1.9)	t= 15.7, p<0.001*

Note: *p<0.05

Table III: Theoretical Domain Parts Score
Items	On campusMean (SD)	E-learningMean (SD)	Significance t p
Midterm exam	10.9 (1.5)	10.5 (1.9)	t= 1.5, p=0.117
Assignments	9.5 (1.0)	12.0 (1.2)	t= -13.1, p<0.001*
Quiz	5.9 (1.8)	9.50 (0.8)	t= -15.3, p<0.001*
Final exam	19.2 (3.3)	22.6 (3.1)	t= -6.0, p<0.001*
Total theoretical score	45.7(4.6)	54.7(4.5)	t= -11.6, p<0.001*

Note: * p<0.05

II- Qualitative results

There are mainly two essential themes that have been used to describe the experiences of students in taking adult nursing course before and after the implementation of E-learning strategies , which are; (1) lived on-campus experience and (2) lived e-learning experience with six categories, three under each theme as following; performing actual nursing roles, the richness of hospital experience, the value of direct contact with instructors , e-learning flexibility, insufficient skills, and poor experience but high grades.

1. Lived on-campus experience

Participants have described their general experiences with on-campus learning as a familiar experience where they used to come to college to attend all course-related activities.

Three main categories have been emerged and used to describe the on-campus learning experiences, which are (a) performing actual nursing roles, (b) richness of hospital experience, and (c) value of direct contact with instructors

1. Performing actual nursing roles

This category describes the experience of the students at the clinical simulation lab (CSL). According to them, CSL gives them a chance to practice in a safe environment and to have a hospital-like experience where everything mimics the real hospital setting. Most of them reported that they missed the experience of physical labs in CSL, where they demonstrate all essential competencies while being evaluated by course instructors who continuously provide them with feedback for improvement, which makes them more competent.

Below are some quotations reflecting this.

"Simulation lab is very useful to do hands-on practice and to practice documentation and communication skills, as well as it teaches us to be professional in terms of dress code and general attitude." Participant 2.

In fact, this finding is significantly reflected in students' grades in the observational structured clinical exam (OSCE) during e-learning, where students had lower marks compared to on-campus learning as students were weak in the clinical skills due to lack of lab practice.

1. Richness of hospital experience

This category describes the significance of clinical training in defining and clarifying the actual nursing role and responsibilities.

Participants reported that clinical placements give them the opportunity to play the actual role of the nurse in terms of patients’ assessment, performing care, and evaluating patient’s conditions. Additionally, they attend patients’ endorsement and observe several medical procedures which introduce them to the real clinical environment.

Additionally, some participants mentioned that the complexity of the cases they observe and reviewing patients’ charts make their learning experience richer in terms of comprehensive nursing role which allow them to apply what they have learned in theory to what they perform in the hospital in terms physical assessment, nursing care planning, intervention, and evaluation. Some quotations reflect this like;

"Hospital experience gives value to the learning experience and allows for a full understanding of the actual nursing role." Participant 13

"Clinical rotations were very helpful in giving us the chance to deal and communicate with real patients." Participant 6

Moreover, participants stated that communicating with patients and relatives allows for empathy and involves them in more patients’ aspects of care, which gives them a more comprehensive experience than CSL.

In summary, most of the students performed badly in clinical evaluation and clinical exams because they did not have the chance to deal with real medical cases in the hospital as they missed the chance to practice clinical reasoning and develop critical thinking skills.

1. Value of direct contact with instructors

This category describes students’ experiences of the on-campus lectures and their associated advantages. Most of the participants have described their experiences of attending physical lectures as similar, in that they have both positive and challenging sides. Most of them stated that on-campus lectures allow for direct contact with course instructors, which facilitates discussion and encourages direct questions for more clarification. Furthermore, many participants stated that attending college lectures puts them in a learning mode because it removes them from home and family distractions, increasing their attention and readiness to receive information. In addition, the live discussion through the lectures facilitates critical thinking and reasoning, which are significantly vital for being able to pass the exams and understand the content more deeply. This eventually improves learning experiences and facilitates understanding of the content. Here are some quotations that express these sentiments:

"It is good to ask faculty in person because it puts you in the learning and study mood." Participant 1

"Discussion with faculty is more interesting and opens more discussion." Participant 5

On the other hand, seven participants stated some challenges associated with physical attendance at the college, like long travel time to the college, challenges attending lectures and clinical on time, and lack of time left to work on assignments and study for exams.

In summary, most of the participants declared that attending physically at the college makes them more engaged and motivated because they have access to different learning resources like the library, CSL, and hospital placements. On the other hand, some participants mentioned that there are some challenges associated with on-campus learning, like long travel time to college, lack of time to do assignments and lack of time to study.

1. Lived e-learning experience

The participants have described their experiences with e-learning as an interesting experience that has both negative and positive sides.

Three main categories have emerged and are used to describe e-learning experiences: (1) flexibility of e-learning, (2) insufficient skills, and (3) poor experience but high grades.

1. Flexibility of e- learning

This category describes the high flexibility associated with virtual learning. According to the participants, the flexibility associated with hybrid learning allowed them to continue their study without being affected by COVID situation. In addition, participants stated that shifting all lectures to virtual and reducing physical labs to only 50% was a right decision because it reduced the physical contact with others and relieved them from the stress of being infected during the pandemic.

Most of the participants stated that shifting to virtual lectures has saved their time and gave them more time to study and to work on their assignments which helped them to obtain high marks in the exams and group assignments. In addition, most participants claimed that attending the virtual lectures with having the recorded lectures was very helpful in terms of accessing teaching material and listening back to lectures. This in turn helped the students to not miss the lectures if they could not attend the virtual classes due to any reason.

“We were more comfortable with online lectures because no traveling time to school and because we got more time to study” Participant 8

“Recorded lectures were helpful because we can go back to listen to the lectures again and we have more time to study” Participant 11

In fact, this was significantly reflected in students’ academic performance as most students had high grades in the assignment and theoretical exams during e-learning.

1. Insufficient skills

This category describes students’ experience with virtual labs. According to most of them, virtual lab experience was unhelpful in terms of mastering the required clinical competencies and procedures. They mostly stated that virtual clinical sessions make students more independent and self-learners, however, they believe is not enough in terms of mastering clinical skills. They mentioned that having a virtual lab instead of attending physically in the simulation lab was a very bad experience as they missed most of the clinical practice and skills. Below are some quotations reflecting this.

"Virtual labs do not teach a lot." Participant 3

"Receiving the information from video only it is not helpful in skills." Participant 6

Other students claimed that relying only on watching virtual videos and discussing virtual case studies were inadequate to teach them the real demonstration of the required skills. As well, lack of hospital experience has resulted in weak clinical reasoning and lack of critical thinking skills. Therefore, many participants claimed that they did not perform well in the OSCE and the clinical evaluation because they were unfamiliar with some equipment in the lab and did not have the chance for adequate lab training to be well prepared for real demonstration that is required in OSCE.

In addition, most of the participants reported that they did not have the chance to be introduced to real clinical cases in the hospital which was badly reflected in their performance because they claimed that they became unconfident and incompetent in terms of clinical skills and knowledge. Many other participants mentioned that they have lost the commination skills as well as they lost the ability to analyze the clinical cases in a more comprehensive way which justifies the poor clinical judgment skills at that time.

"There was a lack of clinical and lab experience which affected our performance in OSCE and badly affected our grades." Participant 3

"The absence of clinical rotations affected our clinical reasoning and critical thinking." Participant 4

This was significantly reflected in students’ academic performance in OSCE and clinical evaluation, as most students had lower grades during e-learning compared to their on-campus grades.

1. Poor experience but high grades

This category describes the general experience of the students with e-learning. According to most of the participants, e-learning was described by a number of participants as a poor experience in terms of engagement and subject comprehension; however, they were able to get high grades in exams. A variety of obstacles were associated with virtual learning were mentioned by the participants, including poor attention in virtual lectures; poor internet connection, lack of recourse to provide studying materials like iPad or tablets; technical issues related to the Microsoft team and blackboard use, which caused attendance issues. Below are some quotations reflecting some of these obstacles:

“It is very hard to focus during online classes because of family disturbance”. Participant 11

“Poor internet connection was a real issue for most of us” Participant7

Moreover, taking online exams was expressed by many students as a stressful experience for many reasons, like connection problems, having no chance to revise the answers, and having no chance for clarification if needed. Thus, some students mentioned that they prefer on-campus exams more than virtual ones as it is a totally new experience for them.

"Online exams are harder than physical exams because of the time, and you do not have the chance to revise or review your answers." Participant 1

On the other hand, many students claimed that despite the associated challenges with virtual learning, the experience was good in general in terms of grades. Many participants stated that their grades were higher compared to before COVID because they had more time to study, fewer assignments to do, and because high marks were obtained in the assignments.

"Grades are higher because there is more time to study." Participant 10

"Less assignments makes you focus and do high quality assignments, which raises marks." Participant 6

"We have got more time to study and work on assignments because we stay at home." Participant 3

Surprisingly, five students believed that virtual learning made them more independent and self-learners because it gave them more freedom to choose and plan their study activities.

"Virtual learning facilitates self-learning" Participant 2

This was significantly reflected in students’ academic performance in terms of online exams and assignments, as most of them got high grades.

Generally, most of the participants reported a negative impact of the lack of lab practice and absence of clinical training on their academic performance in the clinical domain.

Discussion

In general, the participants who were enrolled in the adult nursing course on-campus have described the course in the interviews as challenging, and an interesting course that needs time to digest. Furthermore, the course has a high academic workload and requires a lot of work. In addition, they stated that the course gives them comprehensive experience where the clinical part is extremely connected to and built on the theory part; therefore, they must constantly work hard and attend all course activities to be able to connect the lectures with the clinical skills. In another hand, the students who were enrolled in the course during the implementation of E-learning strategies have described the course as very challenging course and hard to digest due to lack of clinical training and practice.

This study reported a significant increase in students’ performance in all theoretical domains with e-learning compared to on-campus learning among undergraduate nursing students in the adult nursing course at KSAU-HS. The participants mostly declared that it was very challenging to shift to e-learning in such a short time, especially with clinical courses like the adult nursing course. Relatively, multiple studies have reported challenges associated with quick shift to e-learning among undergraduate students, especially, medical and nursing education10,24.

Most of the participants reported many advantages associated with e-learning despite some associated challenges like network connections, stress, and lack of clinical experience. This result was parallel with a study that was conducted in 2021, by Tashkandi, 2021 and reported high flexibility level associated with online learning10.

For instance, they reported high flexibility associated with e-learning with adequate time to study and full access to e-learning materials like recorded lectures and clinical videos on blackboard which had a positive impact on their learning and academic performance. In addition, they reported that they performed very well in assignments because they had adequate time to work on them. Similar finding was found in observational study conducted by Slavko et al, 2019, which reported high academic performance associated with e-learning due to availability of online resources and flexibility of learning24.

Nevertheless, many participants reported some challenges associated with e-learning, like online exam-associated stress due to lack of chance for question clarification and less motivation to study due to the feeling of isolation that is associated with home self-study. Similarly, this finding was reported by Genis-Gruber and Weisz (2022), as the study reported that most of the students face difficulties in finding a quiet space for writing the online exam due to the ambient noise from family, flat mates, or neighbors during the exam. In addition, this study reported that the disadvantaged students who come from low-income backgrounds are facing several challenges affecting the access to a stable technology supporting online exam systems13. Interestingly, this finding was significantly reflected in quantitative data as students had low grades in cognitive questions in virtual exams due to misinterpretation of questions and the lack of a chance to review the answers before submission. Similar findings are reported by Zakaryia Almahasees et al 2021, which found that e-learning is highly associated with some issues that negatively affect the academic performance of students. Furthermore, the most important challenges were lack of motivation in e-learning and the nature of technical experience, which has some limitations27.

The students reported lower grades in clinical domains with e-learning approach compared to on-campus learning because on campus training gives the students full access to perform the clinical competencies in the CSL under full supervision from the course instructors, which makes them feel safe and protected to learn and keep motivated. In addition, according to the students, CLS provides them with a safe and comfortable environment to learn where they can practice with no fear of making mistakes and without taking the risk of harming patients. Similar finding was reported by Koukourikos et al 2021, as CLS was described as one of the most successful clinical training tools that offers instructors and students the flexibility and accessibility for unique learning opportunities and provide freedom for students to make mistakes. Additionally, it provides improved opportunities for feedback and evaluation15.

Moreover, students reported that CSL gives them chance to work in teams and experience different nursing roles, which is highly useful in consolidating cognitive knowledge and enhancing their nursing skills. Similar findings were reported by other studies9,18,23.

Virtual labs also found to be a challenge for most of the students as they reported the negative impact of the lack of actual lab practice and absence of clinical training on their clinical performance, which eventually badly influenced their performance in OSCE. In fact, this was compatible with a quantitative study that was done by Priyanka Gautam et al 2020, and results showed a significant decline in students’ grades in all clinical domains like OSCE and clinical evaluation11.

Many students reported low performance in all clinical-related assignments like nursing care plans and case studies, which was reflected in their grades. According to them, this is due to superficial understanding and a lack of simulation sessions that stimulate their critical thinking and clinical judgment. Similarly, a study was conducted by Crowe et al. (2018), revealed significant improvements in nursing confidence and knowledge associated with simulation sessions as well as significant improvement in clinical reasoning among undergraduate students who are involved in simulation 9. In addition, a study that was done in Saudi Arabia reported similar poor clinical performance associated with e-learning due to a lack of clinical experience and a lack of resources that substitute physical training10.

Moreover, most of students stated that the absence of clinical training had a negative impact on their communication skills as usually students have the chance to communicate with patients, relatives, and the multidisciplinary team. It is worth to mention that dealing with non-Arabic speakers at the hospital improves their English language skills and allows students to practice the language more confidently. A similar finding is found in A study argued the significance of communication skills as part of education in nursing programs, as they stated that practicing communication skills during education enhances communication skills with both patients and colleagues6.

What this study adds?

The current study added to what we already know how educational systems dealt with the COVID-19 outbreak as this study presented students' experiences and their personal perspectives on e-learning strategies and how it contributed to or affecting their academic performance. This helps to understand the impact of shifting toward e-learning on students’ academic performance in health education. Also, it helps to know what students want and need and to come up with good ideas for how to use e-learning education.

The current study revealed some issues associated with e-learning in terms of clinical practice that can be addressed in the future by adopting hybrid learning or blended learning to improve clinical skills or developing tools to improve clinical experience and skills using virtual or simulated patients. E-learning can be useful for clinical instruction, especially with the help of virtual reality. Educators need to come up with creative ideas that can be used in the curriculum to meet educational goals16.

Future research could conduct similar studies that include other institutions and compare the results. In addition, it is significant for future studies to focus on the reliability of e-learning by using larger sample sizes and explore if artificial intelligence could be successfully incorporated into e-learning, particularly in medical education.

Conclusion

This study investigated the impact of implementing e-learning strategies on students’ academic performance in an adult nursing course. The academic transcripts showed a significant increase in terms of grades with e-learning compared to on-campus, where clinical performance was poor with e-learning due to lack of clinical practice and the absence of clinical placement. Qualitative findings were parallel with quantitative results as most of participants reported negative impact of the lack of clinical practice on their academic performance in clinical domains. On the other hand, participants reported high flexibility associated with e-learning which contributed to improving their academic scores.

It is highly recommended to adopt hybrid (blended learning) as a learning strategy, especially in delivering nursing curriculum, because it is more flexible and accommodates different learning styles. Relatively, the theoretical part can be conducted virtually to maintain flexibility, while the practicum part can be conducted physically to maintain adequate skills and enhance clinical performance. Furthermore, it is essential to add self-study hours to practical courses to make students more active and responsible learners.

References:

1. Alshammari AF, Adaileh MJ (2018) E-Learning Readiness: A Scale Development in Saudi Higher Education Institutions. Int J Econ Manag Sci 7: 553. doi: 10.4172/2162-6359.1000553.

1. Abou Hashish E A, Baatiah NY, Bashaweeh AH, Kattan AM. The online learning experience and reported headaches associated with screen exposure time among Saudi health sciences students during the COVID-19 pandemic. BMC medical education. 2022 Dec;22(1):1-3.

1. Amir LR, Tanti I, Maharani DA, Wimardhani YS, Julia V, Sulijaya B, et al. Student perspective of classroom and online learning during COVID-19 pandemic in the undergraduate dental study program Universitas Indonesia. BMC Med Educ. 2020;20(1):1–8. https://doi.org/10.1186/s12909-020-02312-0.

1. Aristovnik A, Keržič D, Ravšelj D, Tomaževič N, Umek L. Impacts of the COVID-19 pandemic on life of higher education students: A global perspective. Sustainability. 2020; 12(20): 8438.

1. Amirullah Khan & Bareq Raad Raheem. (2020).The role of e-learning in covid-19 crisis. International Journal of Creative Research Thoughts. 8 (3),9-14.

1. Ahu Genis-Gruber and Gerold Weisz (2022) Challenges of Online Exam Systems in the COVID-19 Pandemic Era: E-Assessment of Perception, Motivation, and Performance. doi: 10.4018/978-1-7998-8661-7.ch005.

1. Chung E, Subramaniam G, Dass LC. Online learning readiness among university students in Malaysia amidst COVID-19. Asian J Univ Educ. 2020a;16(2):46–58. https://doi.org/10.24191/ajue.v16i2.10294.

1. Cook D, Levinson A, Garside S, Dupras D, Erwin P, Montori V. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med. 2010;85:909–22.

1. Crowe S, Ewart L, Derman S. The impact of simulation-based education on nursing confidence, knowledge and patient outcomes on general medicine units. Nurse Education in Practice. 2018 Mar; 29:70–75. doi: 10.1016/j.nepr.2017.11.017.

1. Emad Tashkandi (2021). E-Learning for Undergraduate Medical Students. Open access to scientific and medical research, 23 (12)665—674.

1. Gautam, P. (2020). Advantages and Disadvantages of Online Learning - eLearning Industry. Available online at: https://elearningindustry.com/advantages-and-disadvantages-online-learning (accessed December 1, 2020).

1. Gurukkal, R. (2020). Will COVID 19 turn higher education into another mode? High. Educ. Future 7, 89–96. doi: 10.1177/2347631120931606.

1. Johnson RB, Onwuegbuzie AJ. Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educational Researcher. 2004;33(7):14-26.

1. Kim S. The future of e-learning in medical education: current trend and future opportunity. J Contin Educ Health Prof. 2006;3:3. https://doi.org/10.3352/jeehp.2006.3.3.

1. Koukourikos K, Tsaloglidou A, Kourkouta L, Papathanasiou IV, Iliadis C, Fratzana A, Panagiotou A. Simulation in Clinical Nursing Education. Acta Inform Med. 2021 Mar;29(1):15-20. doi: 10.5455/aim.2021.29.15-20. PMID: 34012208; PMCID: PMC8116070.

1. Muhammad Turki Alshurideh, Barween Al Kurdi, Ahmad Qasim Alhamad, Said A. Salloum (2021). Factors Affecting the Use of Smart Mobile Examination Platforms by Universities' Postgraduate Students during the COVID 19 Pandemic: An Empirical Study. doi:10.3390/informatics8020032 8(32):32.

1. Means B, Toyama Y, Murphy R, Bakia M, Jones K. Evaluating of evidence-based practices in online learning: a meta-analysis and review of online learning studies. Washington, DC: Centre for Technology in Learning, Department of Education; 2010.

1. Norman J. Systematic review of literature on simulation in nursing education. ABNF J. 2012 Spring;23(2):24–28.

1. Polit, D.E. and Beck, C.T. (2006) Essentials of Nursing Research. 6th Edition, Lippincott Williams & Wilkins, Philadelphia.

1. Rieble-Aubourg, S. and A. Viteri (2020), “COVID-19: Are we prepared for online learning?”, CIMA Brief,No. 20, Washington, D.C., Inter-American Development Bank (IDB)[https://publications.iadb.org/publications/english/document/CIMA-Brief-20-COVID-19-Are-We-Prepared-for-Online-Learning.pdf [accessed on: 2 January 2021].

1. Schindler, L. A., Burkholder, G. J., Morad, O. A., & Marsh, C. (2017). Computer-based technology and student engagement: a critical review of the literature. International Journal of Educational Technology in Higher Education, 14(25), 1-28.https://doi.org/10.1186/s41239-017-0063-0.

1. Strauss, A. L., & Corbin, J. M. (Eds.). (1997). Grounded theory in practice. Sage Publications, Inc.

1. Shin S, Park JH, Kim JH. Effectiveness of patient simulation in nursing education: Meta-analysis. Nurse Education Today. 2015 Jan;35(1):176–182. doi: 10.1016/j.nedt.2014.09.009.

1. Slavko Rakić, Marko Pavlović, Selver Softic, Bojan Lalic (2019). An Evaluation of Student Performance at e-Learning Platform. 17th International Conference on Emerging eLearningb Technologies and Applications (ICETA), DOI:10.1109/ICETA48886.2019.9040066.

1. The Joint Commission. “Joint Commission Center for Transforming Healthcare Releases Targeted Solutions Tool for Hand-Off Communications.” Joint Commission Perspectives 32, no. 8 (August 2012): 1, 3. https://pubmed.ncbi.nlm.nih.gov/22928243/. Accessed: January 4, 2022.

1. Tehmina Basit(2003). Manual or electronic? The role of coding in qualitative data analysis. Educational Research, 45(2).

1. Zakaryia Almahasees, Khaled Mohsen, Mohammad Omar Amin (2021). Faculty’s and Students’ Perceptions of Online Learning During COVID-19. doi.org/10.3389/feduc.2021.638470.