**Hadjerrouit, S. (2015). Evaluating Students' Perceptions of SimReal+ in Mathematics Education.**

*Abstract*

This project addresses the issue of adaptability of SimReal+ to mathematics teacher education. SimReal+ is an interactive digital tool for teaching and learning mathematics and physics. It uses a graphic calculator, video lectures, video streaming, and simulations. It also provides exercises and applications in various areas of mathematics and physics at different university and school levels (Brekke & Hogstad, 2011). SimReal+ is a free software tool that is designed with the needs and competencies of engineering students in mind. SimReal+ is available online (http://grimstad.uia.no/perhh/phh/video/video.htm). The research project aims at evaluating students' perceptions of SimReal+ in a course on digital tools in mathematics education with 22 mathematics teacher education students. These students have very different knowledge background both in mathematics and digital tools than the engineering students for who SimReal+ was developed. None of the teacher education students had any prior experience with SimReal+. The research was carried out in two stages. First, teaching and exercise activities over a period of two weeks were performed. The activities included video lectures, simulations of mathematical functions, programming visualizations using ActionScript, and diverse teaching material made available online such as Pythagoras theorem, conic section, trigonometric functions, properties of triangles, cycloid, and 3D figures. Second, at the end of the activities, the students were asked to evaluate SimReal+ using a set of criteria, and mixed data collection and analysis methods, including a survey questionnaire and open-ended questions.

**Report**

**Introduction**

This project addresses the issue of adaptability of SimReal+ to teacher education. SimReal+ is an interactive digital tool for teaching and learning mathematics and physics. It uses a graphic calculator, video lectures, video streaming, and simulations. It also provides exercises and applications in various areas of mathematics and physics at different university and school levels (Brekke & Hogstad, 2011). SimReal+ is a free software tool that is available online (http://grimstad.uia.no/perhh/phh/video/video.htm). The project aims at evaluating students' perceptions of SimReal+ in a course on digital tools in mathematics education with 22 students with very different knowledge background both in mathematics and digital tools. None of the students had any prior experience with SimReal+. The work was carried out in two steps. Firstly, teaching and exercise activities over a period of two weeks were performed. The activities included video lectures, simulations of mathematical functions, programming visualizations using ActionScript, and diverse teaching material made online such as Pythagoras theorem, conic section, trigonometric functions, properties of triangles, cycloid, and 3D figures. Secondly, at the end of the activities, the students were asked to evaluate SimReal+ using a set of criteria, and mixed data collection and analysis methods, including a survey questionnaire and open-ended questions. To measure the students' responses, the survey questionnaire used a five-point Likert scale from 1 to 5, where 1 was coded as the highest and 5 as the lowest (1="Strongly Agree"; 2 = "Agree"; 3 = "Neither Agree or Disagree"; 4 = "Disagree"; 5= "Strongly Disagree"). The average score (MEAN) was calculated, and the responses to open-ended questions were analyzed. The items of the survey were categorized in 5 groups of predefined criteria, including technical usability, pedagogical usability, mathematical content, assessment issues, and teacher education considerations. Open-ended questions allowed the students to express in their own words what they think about different issues related to SimReal+.

**Evaluation Criteria**

The criteria used for evaluating SimReal+ are rooted in four theoretical frameworks used in mathematics education. First, the instrumental approach, and the distinction between artefact and instrument, and how to transform an artifact to an instrument through the processes of instrumental genesis and instrumental orchestration (Rabardel, 1995; Trouche, 2004). Secondly, the anthropological approach, and the Task-Technique-Theory triad to capture the relationships between tool techniques and conceptual understanding, on the one hand, and tool techniques and paper-and-pencil techniques, on the other hand (Chevallard, 1995). Thirdly, the Theory of Didactical Situations, and the tool's potential for providing feedback, which is an essential condition for supporting learning (Brousseau, 1997). The fourth theoretical background is related to usability issues, and research work on evaluating digital tools in mathematics education (Bokhove, & Drijvers, 2010). On the basis of this theoretical background, several criteria can be derived for evaluating SimReal+: Technical usability, pedagogical usability, assessment issues, mathematical content, and teacher education considerations.

Technical usability is a self-evident requirement for any digital tool in education (Nielsen, 1993). Nokelainen (2006) expanded the concept of technical usability to include pedagogical usability criteria, such as learner control, learning activity, collaborative learning, goal orientation, applicability, added value, motivation, previous knowledge, flexibility, and feedback. It is obvious that both aspects of usability are related to each other (Hadjerrouit, 2010). In addition, the assessment dimension has emerged as one of the most important criteria for evaluating the quality of SimReal+. Assessment is associated with the feedback that the tool provides to users, and builds the ground for formative assessment. Another criterion is the inclusion of mathematical content in terms of correctness of mathematical concepts, notations, and symbols (Bokhove, & Drijvers, 2010). Equally important is the congruence between the tool's features and paper-and-pencil techniques to facilitate mathematical reasoning and problem solving. Finally, to address the issue of SimReal+ adaptability to teacher education, it is important to evaluate whether the tool is appropriate for use in teacher education, and whether it takes into account the requirement of adapted education, and enables the teacher to concretize the mathematical subject curriculum*. *

**Results**

The students' responses to the survey questionnaire and open-ended questions are described for the five broad categories that emerged from the evaluation criteria, and for supplementary issues that were addressed by open-ended questions.

*Global Impression*

First, students were asked to express their opinions on SimReal+, and what they globally liked and disliked about the tool. Basically, most respondents reported both positive and negative aspects of SimReal+:

*"It seems like a good program. I like that it is free and easy to use. So you can use it when you want. Sometimes it can be a bit hard to find your way around."*

Some students were more critical as the following comment indicates:

*"I think it is confusing. It has too many options that are poorly linked together. It could learn a lot from "Mastering physics", in terms of more linear, logical structure and better feedback assessment. (…) I really like a teachers' opportunity to program himself and add new graphics."*

*Technical Usability*

Technical usability is a prerequisite for using SimReal+ as a tool for learning and teaching mathematics. Students were globally satisfied with the technical usability in terms of easy-to-start, management facilities, availability of mathematical content and exercises, quality of video streaming and calculator, and accessibility of the tool anytime and anywhere as this students' comment indicates:

*"It is not all that difficult for first timer to use SimReal+ due to the information provided to make mathematics easy"*

In contrast, students were not satisfied with the technical usability in terms of ease-of-use or user friendliness, navigability, response time, as this comment clearly reveals:

*"Everything does not work for me. I got confused. The long lists of lessons/exercises/ simulations in the rightmost window made it hard to focus attention to a specific task. Pre-made simulations and demonstrations worked well for illustration, but the visualizations that the activity should give, e.g. that Pythagoras' theorem is seen to be true, was not clear to me. The intention was clear, but I don't think the user will readily gain anything."*

*Pedagogical Usability*

In terms of pedagogical usability, students agreed that SimReal+ provides various mathematical activities and multiple representation of mathematical content, and that SimReal+ can be used as a lecture and textbook supplement. Concerning motivational issues, many students think that the tool is motivating to do mathematics with, as this comment shows:

*"Especially for mathematics interplay with physics, SimReal+ served as a good motivation and demonstration. However, motivation derives not only from an experience of «wow, this is useful/beautiful», but also from one's own development of capabilities to make mathematics useful/beautiful. This latter element of motivation was not offered by SimReal+, so I only felt the brief sense of wonder, not of achievement and skillfulness"*

Furthermore, many students think that SimReal+ contains multiple levels of difficulty and provides opportunities for the teacher to make individual adjustment if necessary. However, the tool does not easily allow students to customize the tool. Nevertheless, the tool enables students to work at their own pace, which is a motivational factor in keeping students engaged in mathematics, as this comment reveals:

*"I think this is the greatest advantage of SimReal+ the opportunities for adjustment and individualization. The teacher knowing a little programming (as should all upper secondary teachers in my opinion) can rather easily make small addition and alterations to a pre-programmed lesson, so it fits his needs: add a second particle, hiding/showing the kinematic (…) equations, etc."*

Moreover, most students think that SimReal+ does not fully allow to work independently from teacher assistance or fellow students. Also textbooks are still needed when using SimReal+ in classroom:

*"Autonomous learning is not well-supported, only autonomous playing around without goal. And for the purpose of playing with mathematics, there were too few options for the student, (….), and mostly teacher-chosen and implemented possibilities/options."*

Furthermore, most students agreed that SimReal+ is not fully appropriate to use as an alternative to achieve variation in teaching mathematics as this comment clearly shows:

*"The problem as I see it is that there is no clear-cut connection between the variations of a specific topic or notion. For example, changing the parameters in a wave function, (…) and its exact correlation with the graph, (….), but such things are not in any way implemented in SimReal+."*

*Assessment Issues*

According to most students, SimReal+ does neither provide a diagnosis of student's problem solving nor appropriate feedback that is adapted to the students' knowledge level. In addition, SimReal+ does neither build student profiles nor serve up appropriate questions or several questions to the students. Furthermore, SimReal+ does neither have a review mode showing what the student has done wrong or right, nor allow for the use of several question types. The lack of student profiles may prevent students from engaging in authentic learning activities, particularly when they do not know a topic well enough, and SimReal+ cannot provide support. Given the students' perceptions, it is obvious that the role of the teacher is still important in assessing the students' learning of mathematics. The following comment summarizes the limitations of SimReal+ in terms of assessment:

"*SimReal+ is very weak with respect to feedback. I experience no feedback or assessment of any work or exercises."*

*Mathematical Issues*

Many students believed that that SimReal+ has a good quality of mathematical content in terms of mathematical correctness and representation of mathematical properties and operations, e.g., mathematical formulas, functions, graphs, and geometrical figures. A typical comment was:

*"SimReal+ is rich in mathematical content. It gives systematic procedure for solving a particular problem in mathematics. It is able to translate algebraic expressions into graphs or geometric figures, etc. The video and audio visual aspect of the software enhances deeper understanding of mathematical concepts"*

However, the practical applications and exercises are not fully well-designed as this comment indicates:

*"The exercises were too general, not specifically focusing on some mathematical learning object. For example, the most focused exercise was about changing angles into radians and vice versa. The connection was very well visualized on the unit circle, but there weren't added any techniques to do the calculations. (…) The exercises were well-formulated but they were too focused on visualizations done by the program with no connection to paper-pencil work or text book theory"*

Furthermore, SimReal+ is useful when it combines video lessons, simulations, live streaming of lessons, and exercises. It helps to acquire new mathematical knowledge, and in a lesser degree, it is congruent with paper-pencil techniques. SimReal+ also provides opportunity to help students gain knowledge that is otherwise difficult to acquire. In addition, the tool's openness enables students to express mathematical ideas and strategies. The negative side of SimReal+ is that students do not like programming mathematical visualizations and simulations. They prefer focusing on the mathematical part of the program, which is more important than understanding programs written in Action Script as this comment expresses:

*"Maybe for the students, but they don't need all the programming stuff, just an interface where mathematical properties can be approached. (…) For the teacher, I think it is essential that he/she understands the programming enough to make non-trivial changes such as adding a particle or a button with a new function."*

*Adaptability of SimReal+ to Teacher education*

Most students think that SimReal+ could be an appropriate tool in secondary schools, but not in middle or primary schools. SimReal+ also enables teachers to concretize the curriculum. However, when asked whether they will continue using SimReal+ for teaching mathematics, most of them answered negatively as this comment clearly reveals:

*"I will use SimReal+ to further develop an illustration of a phenomenon in my teaching but only for illustration, not interactive student activity."*

*Suggestions for Improvements*

Students were asked to provide suggestions to make SimReal+ more appropriate for use in teacher education. Perhaps, the most interesting suggestion is expressed by the following comment:

*"Focus. It tries to do everything, but really only presents a fragmentary list. Technologically, it needs a more directed structure such that one task leads to the next in a logical way, not just present an array of tasks. (…) Mathematically, it must connect the different settings graphical, algebraic, tabulator, arithmetic, geometric, etc. Assessment-wise, it must offer feedback to the implemented exercises."*

*Flipped Classroom*

The students were asked whether flipped classroom will be of interest and whether new digital technology can give students new possibilities in teaching mathematics.

Most students agreed that flipped classroom is of huge interest, but

"*(…) we need to focus and stop believing that just planning around will result in learning. We need strictly directed lessons that allow for continuous interchanging of visual illustrations, theoretical presentations, exercise solving, etc. In short, digital tools could help us in writing the classroom activities that have traditionally been separated in time and space. (…) Flipped classroom also is very interesting but needs highly motivated students.*

*Programming Visualizations*

The students were asked in what way do they think programming mathematical visualizations and simulations by their own will help them in understanding mathematics if they can use different templates so the concentration can be on mathematics and not on difficult details in the programming process. As mentioned earlier, students prefer focusing more on the mathematical part of the program than learning programming. Most students agreed that templates can help them to concentrate on the mathematical part, but

*"(…), I need very specific templates focusing on illustrating a specific notion, e.g. the kinematic equations of cycloid (….), so I need to know in advance rather precisely which properties I am investigating.(…). *

The project leader is planning to develop such templates so that students can be able by their own to program mathematics (elementary and advanced visualizations and simulations) directly into their own Web pages without any special tool. They just need to write their own mathematical code. Most students like the idea of using templates, as this comment clearly highlights:

*"I think it is great. I think there should be a clear distinction in the programming interface whether you are implementing graphics mathematical properties, exercises and theory (….). Then you can access more readily different settings that a visualization or simulation should offer to a student using it."*

**Conclusions, Limitations and Recommendations**

The number of participants (*N=22*) may not be sufficient to adequately support the generalization of the results. Hence, new cycles of experimentations and evaluations of SimReal+ are warranted to generalize the findings of the present work to ensure more validity and reliability. However, despite the limitations of the study, it has been possible to make some reasonable interpretations of the results and draw some recommendations for using SimReal+ in teacher education. Firstly, the use of SimReal+ indicates that the tool shows potential for teaching mathematics that is suited to the students' knowledge level, although not all criteria are equally met. Secondly, SimReal+ is technically well designed in terms of accessibility and management facilities, which is a pre-requisite for using it in teacher education. Furthermore, SimReal+ covers a wide range of mathematical content with varied levels of difficulty, including school mathematics. Likewise, the content is mathematically correct and reflects the underlying properties of mathematics at different levels. However, mathematical and technical issues in themselves are not sufficient to make SimReal+ become an integral tool in learning and teaching mathematics in a school environment. SimReal+ is not pedagogically well designed to ensure a smooth integration of the tool in teacher education in terms of variation, student autonomy, differentiation, and individualisation, and assessment issues as well. Hence, SimReal+ in its present form is not fully appropriate for use in teacher education, unless didactical functionalities and modalities of using the tool are considered in future versions of the system. Likewise, the user interface must be simplified to make SimReal+ more intuitive and easy to use in teacher education. Summarizing, the use of SimReal+ in teacher education needs to take into account the pedagogical dimension of teaching and learning mathematics in order to adapt the tool to the modalities of using digital tools in teacher education.

**References**

Bokhove, K., & Drijvers, P. (2010). Digital Tools for Algebra Education: Criteria and Evaluation. *International Journal of Mathematics Learning*, *15*, pp. 45-62.

Brekke, M., & Hogstad, P.H. (2010). New Teaching Methods – Using Computer Technology in Physics, Mathematics, and Computer Science. *International Journal of Digital Society* (IJDS), *1*(1), pp. 17-24.

Brousseau, G. (1997). *Theory of Didactical Situations in Mathematics*. Kluwer Academic Publishers

Chevallard, Y. (1985). *La transposition didactique - Du savoir savant au savoir enseigné*. La Pensée sauvage, Grenoble (126 p.). Deuxième édition augmentée 1991.

Hadjerrouit, S. (2010). A Conceptual Framework for Using and Evaluating Web-Based Learning Resources in School Education. *Journal of Information Technology Education 9*, pp. 54-79.

Nielsen, J. (1993). *Usability Engineering*. Boston, MA: Academic Press.

Nokelainen, P. (2006). An Empirical Assessment of Pedagogical Usability Criteria for Digital Learning Material with Elementary School Students. *Educational Technology & Society, 9*(2), pp. 178-197.

Rabardel P. (1995). *Les hommes et les technologies, approche cognitive des instruments contemporains*. Armand Colin.

SimReal+ Web site: http://grimstad.uia.no/perhh/phh/video/video.htm

Trouche, L. (2004). Managing the Complexity of Human/Machine Interactions in Computerized Learning Environments: Guiding Students' Command Process Through Instrumental Orchestrations. *International Journal of Computers for Mathematical learning* 9, pp. 281-307.