Evaluating Telelearning

In the previous chapter we have seen that the structuring of the learning experiences (i.e. telecoaching) is a complex process that comprises almost all levels and aspects of the educational practice. In order to evaluate this telecoaching process in a systematic manner I have chosen for the transformational perspective on education. In this chapter this analytical perspective will be used to specify (a) a general evaluation approach for telelearning processes, and particularly (b) a method for evaluating the telecoaching which has been practised in the TAET-course. I will start with the construction of a more general analytical framework that can be used to evaluate the entire process of telelearning.

In educational research of online courses we can differentiate seven analytical levels which have to be taken into account. Each level has its own peculiarities, potentials and problems. In the absence of a funny acronym, I will call it the SLNM: Seven_Levels_ No_More. In the next table these seven levels of telelearning are presented in a scheme in which the main features of the levels and the corresponding potential problems are outlined.

This SLNM-model can be used for (a) the construction of a theoretical framework and literature study, (b) the development of a questionnaire for TAET students and teachers, and (c) the empirical analysis of TAET course. In this thesis I concentrate primarily on the community and the coaching level. Since coaching is also inextricably bound up with course content and assignments, these levels will also be discussed. Other levels will only be touched on briefly, either due to time constraints or due to a lack of direct connection with the subject of this thesis: What does coaching 'look like' in a virtual learning environment and how can coaches stimulate the development of a vital learning community.

My hypothesis will be that the effectiveness of telecoaching depends primarily - though not exclusively - on the extent in which coaches succeed in moderating collaborative learning processes between and among students (and instructors). I presuppose that the creation of a learning community during a course is one of the most critical success factors of tele-education.

My empirical investigation will be directed by this main question:

what activities and roles of coaches have an impact on the learning process
as well as on the development of a learning community?

This question can be subdivided into these subquestions:

1. What are the activities of coaches that stimulate and moderate self reflective learning practices?
   
   
2. To which extent and in which forms has a learning community been developed in the course?

Below I will indicate what aspects of the different levels I believe are important and can be analysed and what I will leave out due to time constraints (pragmatic reason). For some levels I have worked out some indicators to be measured. Some of the questions that were asked in the questionnaire are inserted behind these indicators.

The infrastructural level is defined by the conditions that facilitate good (fast, reliable etc.) access to personal computer, local networks, and the internet. The infrastructural level doesn't really concern the main theme of my thesis. Yet, some TAET students didn't possess a PC when they started the course and were enabled by the faculty to borrow one. And as we will see in the next chapter, some students had problems getting a reliable connection with the TAET-site.

In the questions on infrastructural level respondents were asked to briefly state the problems they had with the technology (network, computer) and also indicate positive experiences. This was specified in the following questions:

1. Did you have problems with the technology (computer & network)?
2. Did you have problems with your personal computer?
   1. No computer, Computer too slow: underachieving PC
   2. Inadequate Software
3. Did you have problems to get access to the Twente network?
4. Did you have problems with your own internet connection?
5. How did it influence your work rhythm?

Basic Skill Level

The basic skill level is defined by the specific state of the basic computer, network, and internet abilities of students. In the studies I've read so far a recurring problem is pointed out: the lack of basic computer skills and webwisdom, combined with unfamiliarity with the electronic learning environment. Some studies report students' frustrations due to technological problems and suggest an introductory course before they really get started. In such introductory courses students can acquire basic computer skills, learn how to surf on and search the web, and get familiar with the electronic learning environment. We know in advance that quite a few TAET students had problems originating in a lack of computer- or webwisdom, especially with building a website in one of the first courses. They also had to become familiar with the electronic learning environment.

The questions on basic skills can be specified as follows:

1. What was your basic skills level like when you started?
   1. related to computer skills
   2. related to internet skills
   3. related to electronic learning environments
      
      
2. How long did it take you to get familiar with TeleTOP?

Course Content Level

The course content level is defined by the quality and transparency of the material that is used in an online course. Course content relates to:

1. Quality and level of difficulty of the study material.
2. Structuring of the whole course (order of subjects).
3. Relationship between study material and new subject.
4. Logical coherence between subjects.

On all these four items students will be asked some questions.

Assignment Level

The assignment level is defined by the structuring and content of assignments. The assignment level can be questioned and analysed as to:

1. investment of time
2. connection of assignments per subject
3. clarity of assignments
4. scope of assignments: too broad, too narrow (restricting creativity)
5. relevance of assignments, etc.

It will not be easy to answer questions for this level for TAET students, because there were generally many assignments per subject and memory may have faded.

Community Level

The community level is defined by the imagined and experienced community that motivates students and stimulates them to engage in collaborative learning. The danger inherent in distance learning is that the social aspects of learning are getting lost. Social embedding of a learning process is of paramount importance, not only for an efficient learning process, but also for future professional practices in which collaboration is of vital importance. In some studies students' wishes to be part of a learning community is explicitly reported (transformation from outsider to insider) .

Cooperation can be measured for example as to

1. Broadness of participation
2. Intensity of participation

Sources of information for TAET-students are

1. Discussion page
2. Q & A page
3. Workspace
4. Groupmail
5. Personal email between students (this can only be measured by asking the students if and how much email contact they had with fellow students).

Coaching Level

The coaching level is defined by the intensity and temporality of individual and collective return-information. Coaching refers to coaching activities of the instructor. In general we can say that coaching has two basic components: (a) instruction and (b) process control. These components will be subdivided later (e.g. different roles of the teacher: instructor, judge, motivator, provider of examples, advanced organiser, etc.). The instructor is the person to develop and control the quality of the (collaborative) learning environment and learning process and the one who can moderate the social embedding of the learning process.

In the teachers' questionnaire questions concerning available time for coaching and views on coaching will be included (for example: "what has been - or should be - your role in stimulating a collaborative learning process?") . In the students' questionnaire explicit questions about expectations and appreciation will be included.

Evaluation Level

The evaluation level is defined by the intricate system of evaluation instruments and procedures. In my empirical investigation I will not enter into the discussion about course evaluation. My investigation is itself a form of evaluation research.

Yet I will complete my TAET research with conversations with some experts of the Open University [Heerlen] on the issue of coaching strategies and practices. My hypothesis will be that the introduction of tele-education generates an increasing coaching need of students/trainees. A kind of 'revolution of rising coaching expectations'. Telelearners who present a paper or assignment expect an adequate, personal and above all fast reaction. My question to the experts will be how they practically handle this increasing demand of adequate, personal and fast feedback. (a) What problems do they identify: where do things go wrong and why? (b) What are the strategies to solve these problems: how can you realise a good coaching that comes up to the coaching expectations of students in a situation of scarce means?

In my qualitative research project I have drawn upon the following sources of insight and information:

1. A webbased questionnaire for my fellow TAET-students (also for the students that dropped out).
   
   
2. A webbased questionnaire for the TAET-teachers (core courses and optional courses).
   
   
3. My own knowledge of the TAET course stemming from my participation as a student [see A personal view].
   
   
4. Personal email messages from and to students and teachers (during the whole training).
   
   
5. Workspaces on TeleTOP for the different courses.
   
   
6. Question and Answer pages on TeleTOP for the different courses.
   
   
7. Course instructions.
   
   
8. A questionnaire for some experts in the field of coaching at the Open University (Heerlen)
   
   
9. Personal communication with some university teachers who have had experience with coaching in online educational learning processes.
   
   
10. Personal communication with three teachers whom I asked to read part of my thesis (results of the empirical part).

The urls of the questionnaires were sent to my fellow students and teachers in a letter sent by email, explaining the purpose of my research. In the weeks after I was so fortunate to receive quite a number of replies.

Of the 16 students 11 responded. 3 out of 16 students dropped out and although I asked them to respond I received no reply. I did not fill out the questionnaire myself. In the period when the questionnaires came in it turned out that 2 had disappeared somewhere in cyberspace. I had not been clear in my instructions that the questionnaire had to be filled out online. This omission may be due to the fact that I have a fast cable connection and can remain online all the time. Quite a few of my fellow students have a telephone connection and limit their time online for economical reasons. That might have been the problem. One student tried to file it halfway the process of answering the questions. When she tried to retrieve the answers it appeared that they had got lost. I approached the two students whose questionnaire had disappeared and came to an agreement with them. I had an interview with one of them at her home and a telephone interview with the other (who had sent in the questionnaire twice already!). I sent a reminder twice to the students who hadn't responded (4, including the 'dropouts'). I asked them kindly to give me a reason for not filling out the questionnaire if they couldn't or wouldn't respond. From none of them I received an answer.

Of the 19 teachers who taught in the TAET course 16 responded. I sent them a reminder once and also asked them to give me a reason for not filling out the questionnaire if they couldn't or wouldn't respond. The same story counts here: no replies from 2 teachers. I assume no teachers' questionnaires got lost in cyberspace, at least not to my knowledge.

The main problem with the questionnaire was that the senders received no 'copy' of it. This was not possible because the senders were outside of the domain of the server of the University of Amsterdam, where the questionnaires were sent to after having been answered. This is a problem to be avoided in future research.

Connie Menting Amsterdam, Mei, 2000 Last updated: 27th December, 2005