Multimedia Courseware on the WWW

This evaluation study was carried out as part of the Leonardo Project "MediaWeb – Multi Media Courseware on the World Wide Web" (http://learning.ericsson.net/mediaweb/index.html) funded by the EU Commission.

The British Open University is generally accepted to the leading distance teaching institution worldwide and has extensive experience in all aspects of large-scale distance teaching and learning. NKI Distance Education is a specialised institution for distance learning in Norway, and one of the institutions worldwide that first started developing on line teaching. Ericsson Systems Expertise is a training organisation within the Swedish Ericsson Group, the world’s largest supplier of intelligent network products. Ericsson has a long tradition in supplying training based on multi media and programmed learning packages delivered on CD-ROMs and recently started converting and developing training on the Internet. TecMinho is a training organisation for business and industry within the Minho University.

The main aim of the MediaWeb Project was to develop and evaluate the use of multi media in courses on the Internet. This evaluation report constitutes the NKI contribution to the evaluation part of the project.

NKI and the NKI Internet College

NKI is one of the largest non-governmental educational institutions in Scandinavia. The NKI Group is organised as a non-profit foundation comprising NKI Distance Education, The Polytechnic College (DPH), The Business Training Centre (NA) and NKI Publishing House. It has approximately 350 full-time and 700 part-time employees. The group's head office is situated in Oslo, and there are district offices in 15 other towns. Altogether the NKI Group has each year a total of around 6000 full-time and 25000 part-time students.

NKI Distance Education offers approximately 60 programmes and more than 300 courses at secondary and undergraduate level, as well as specialised courses and hobby courses. The school employs some 60 full-time and 400 part-time employees. It has each year around 15 000 active students out of a population of four million Norwegians. It is recognised by the Ministry of Education and receives government grants towards operating costs. During the last 15 years NKI Distance Education has developed from a correspondence school to an institution applying the Internet for delivery of a large number of courses. Still, in the year 2000, paper based learning materials and traditional correspondence courses are important for a majority of the students.

NKI was probably the first European online college, and it has offered distance education online every day since 1987. Few - if any - online colleges in the world has been longer in continuous operation (Paulsen & Rekkedal 2001).

In June 2000, the accumulated number of course enrolment exceeded 10000. At that time the NKI Internet College had about 75 tutors and 2000 active students in 15 countries. At the end of year 2000, the NKI Internet College offers about 30 complete programs and 200 different Norwegian courses online.

The Courses Evaluated

The MediaWeb Project run for two years, 1999 and 2000. The project plan presupposed that the partners should develop and test the use of multimedia in one course each year. NKI chose to develop some multimedia elements within the course Multimedianett (Multi Media Networks) for the first year and within a study programme Arbeidslederskolen (Management programme) for the second year.

Multimedianett is one course within the study programme Informasjonsnett (Information Networks). The course constitutes 2 ‘vektall’ (in the Norwegian university and college system equivalent to 1/10 of a complete full time study year). The course was the final course in the programme, which is equivalent to ½ year full time study. This meant the most of the students taking the course, will have completed a number of courses as distance students on the Internet before embarking on this course. The reason for choosing this course for the project was that the course was under revision when we started the project, and also that we found it specifically suitable because of the relation between the content of the course itself and the project.

Arbeidslederskolen is a complete programme for lower and middle level managers equivalent to ½ year full time study. It consists of 4 courses. As part of the MediaWeb project we developed some multimedia content for the general introduction and for each of the 4 courses.

Multimedia in WWW Courses

An AltaVista Search 1^st February 2001 gave nearly 1.3 million hits, among them a multitude of companies offering hardware, software and services for producing and presenting multimedia on the Internet or through other types of technology. The term "multimedia" is understood as "combined use of more than one medium" (Hofstad et al. 1996). Szuprowich (1995) defines "interactive multimedia communications" as "the synthesis of TV concepts and computer capabilities into an integrated whole. This approach combines TV’s power to grab and hold attention, motivate, excite, and indoctrinate, with the computer network’s ability to randomly access, select, compare, transmit, and display data, information and knowledge in form of text, graphics images, animated sequences, voice and video from a variety of sources…" (Szuprowich 1995, p xiii).

When discussing media for delivering global education Mason (1998) describes three current technologies, text-based systems, audio-based systems and video-based systems as discrete media and shows how the evolution of these previously discrete media proceeds "towards integration – of real time and asynchronous access, of resource material and communication, of text and video, in short, of writing speaking and seeing." (Mason 1998, p.19). While CD-ROMs represent an earlier good example of this integration, Mason (Ibid. p. 28) refers to the Web as "…the most phenomenally successful educational tool to have appeared in a long time. It combines all the media described above: text, text-based interaction, audio and video clips, and with somewhat less robustness, multiway interactive audio and video."

There is no doubt that generally multimedia on the WWW is more expensive to develop than text. Why use it in education? What does add to the learning experience?

Our didactic choices concerning the organisation of the learning experiences and our choices of media and represential forms for presenting content will depend on learning philosophies and views on student learning styles. E.g. traditional computer assisted learning comprises one strand which continues to influence the design of multimedia learning environments. Based on the behaviourist principles of B. F. Skinner, this approach emphasizes a highly formal design process in which the target learning outcomes are broken down into intermediate measurable steps leading to the behavioural objectives. Simpler skills are practiced and mastered before more complex ones. This systematic structuring of the curriculum is matched by a pedagogical emphasis on drill and practice. It is a precise procedural model, which has been very popular with designers of CBT systems.

Seymour Papert advocated a very different approach to computer support for learning. He argued that the child, through interacting with the world, constructs knowledge of the world. This constructivist view of learning and cognitive development provides another important theme in understanding the design of multimedia learning environments. Learners are viewed as active constructors of their knowledge of the world.

One of the strengths of multimedia as a learning tool is its facility to adapt to different types of learners. However, there is considerable controversy over what these different types are and whether teachers and learners should 'play to their strengths' (i.e. use the most compatible approach and medium) or should stretch their present capacities (i.e. to encourage cognitive development by accessing information in unfamiliar modes). The best learning materials usually present information is as many formats as possible in order to provide broad initial appeal and subsequently to reinforce understanding through multiple perspectives. One simple categorisation of learning styles is: visual, auditory, kinaesthetic, or more commonly known as showing, telling and doing. As for modes of accessing information, one well-known categorisation is that between linear and holistic approaches. Some learners prefer material presented linearly, with new knowledge following concepts already mastered; holistic learners prefer materials with little structure which allow them to gain an overall picture first before filling in the pieces of the big picture.

Another important aspect of learning is student activity and interaction. "Most theories of learning suggest that for learning to be effective it needs to be active; in other words the learner must respond in some way to the learning material. It is not enough merely to observe or read; learners have to do something with the learning material. Thus they may need to demonstrate (if only to themselves) that they have understood, or they may need to reprocess new material to incorporate it with existing knowledge, or to apply the knowledge they have acquired successfully to new situations." (Bates, 1991)

Multimedia, especially when delivered via the web, facilitates all three kinds of interaction. Interactivity with multimedia content can take the form of simulations, self-assessment questions, and quizzes with feedback, modelling, tutorial programs and micro-worlds. Interactivity with teachers and other students can be facilitated asynchronously through text based conferencing systems and synchronously through audio, video and chat software. In fact, this distinction between real time and delayed interaction is beginning to break down as technologies which allow real time interaction can be ‘canned’ for access at any time after the event.

Although, there is no doubt that multimedia can increase both learning satisfaction, motivation and learning efficiency, the positive effects of multimedia may have been oversold by some writers. In short, it is assumed that in learning situations, information represented in hypermedia (multiple media in a hypertext structure) may be more easily assimilated by the learner.

The fact is, that the effectiveness of any technology in the learning process is the result of a complex interaction of conditions:

Till now it has been difficult to prove that one technology or one teaching method has any universal advantage to others (see e.g. Clark 1983, 1994; Dubin & Taveggia 1968; Schramm 1979; Keegan & Rekkedal 2000). It is the way in which any learning technology is embedded into the structure of an educational activity that has the most important impact on how successful the learning is likely to be. No matter how excellent the design of any multimedia material is, if the learning context into which it is introduced is not appropriate, it will either be ignored or misunderstood.

NKI Distance Education has delivered online courses since 1987. The online courses were developed from a tradition of mainly print based distance education. The distance courses could to some extent include audio and videotapes, telephone and fax communication - and later video conferencing. But during the early days of online delivery the system was completely text based, not even graphics or pictures could be distributed through the online system. Thus, any use of multimedia was not integrated within the system of computer-mediated communication. It was not until 1996 when the NKI Internet College introduced web-based courses that multimedia became an option.

The main reason for being careful in introducing multimedia content in the online courses have been:

Until recently most students did not have PCs with sound cards and load speakers, and the majority accessed the NKI Internet College via low speed modems from home. In this situation multimedia content would have to be ‘add-ons’ rather than necessary components for learning. Further, we assumed that students not able to access multimedia content could be anxious that they lost learning experiences and thus would have disadvantages relative to other students. As the NKI Internet College has been in continuous operation since its initiation, it has been a priority not to require access to the most recent and advanced technology. Of course, cost-effectiveness considerations has been important when applying text rather that advanced multimedia solutions. Concerning learning philosophy, we have also regarded learning as hard work, studying and interaction rather than passive reception of content. Multimedia both in the form of audio or video-lectures and may other types of content presentation might often stimulate passive listening and viewing more than active studying. We have experienced that students report more learning from reading and discussing than from viewing and listening.

Thus, NKI Internet courses have been mainly text-based both in content on the WWW and in interpersonal communication.

One of the project aims project was to learn from the experiences of developing different kinds of multimedia elements using different kinds of developing software. Another important aim was to assess the student’s views on multimedia in general, possible problems concerning technology and considerations concerning experiences of learning and learning effects.

Thus a wide variety of different multimedia solutions was tried out, such as graphics, pictures, sound, video, animated graphics, and different kinds of integration of representations, such as audio with illustrations, video with illustrations, animated graphics with pictures, sound and text etc. In addition to the multimedia content developed as part of the project, the course Multimedianett also included a number of links to multimedia sites on the Internet.

Programmes for developing multimedia content included Macromedia Flash and Fireworks for picture editing, Adobe Premier for video editing, Sound Forge for audio editing and MPEG Suite Encoder for compressing sound into mp3 files. Real Producer and Real Slideshow were also applied. In addition Dreamweaver was used for integrating text and multimedia for distribution on the WWW. Necessary plug-ins for the students were Flash plug-in, Real Player and mp3 player.

Even after the introduction of multimedia elements in these courses, the courses were mainly based on texts (in printed form and for presentation and communication on the Internet) and students unable to take advantage of the multimedia content could study the courses without difficulties.

Method and Design of the Evaluation Study

The project partners decided to try to apply a common framework for evaluating the courses developed as part of the MediaWeb Project. As the partner institutions, courses and organisational structures are very different, this proved difficult, but the Evaluation Framework (Dias 2000) developed by one of the partners was accepted as basis for all the evaluations to be carried out.

Because of previous experiences concerning low response rates to questionnaires distributed by post, by e-mail or on the WWW, we decided to carry out telephone interviews with the students based on a pre-produced questionnaire with mainly fixed answers for ticking responses. In total we received 33 answers after sending out the questionnaire by e-mail to students who we did not reach by phone. 19 students answered by phone and 14 by returning the filled in questionnaire by e-mail. Response rates are difficult to calculate, as we initially asked only students who had examined the multimedia content to answer. Of the 33 respondents only 19 had really looked at/studied the multimedia elements. As the questionnaire also included evaluation of the courses and views on online study in general, all 33 respondents are included in the material presented. The data were processed using the FOSS statistical package (Amundsen 1993).

Results

12 of the respondents had completed or were studying the course Multimedianett. 10 (84%) said that they had looked at the multimedia content. 21 respondents had completed or were studying Arbeidslederskolen. Of these, only 9 (43%) had looked at the multimedia content. For matter of simplicity in the following the course and programme is called ‘Multimedia’ and ‘Management’. The reason for not having seen the multimedia content is that the multimedia elements for both groups was published as they were produced, and some students had finished before the material was published, while others had just started their studies. As this evaluation study was carried out as one of the defined project activities, it had to be finished before the end of January 2001. The study should be followed up by another evaluation study later.

There is a majority of men taking the courses evaluated, 67 percent of the Multimedia students and 57 percent of the Management students respectively. The students’ age varies from below 24 (only two students) to over 40 years of age. 6 (50%) of the Multimedia students are over 40 years old. The median age (Md) for the Multimedia students is 39 years, and for the Management students 34 years.

Concerning occupation, one third (4 students) of the Multimedia students worked as teachers, while 3 students ticked administration and 3 ticked student. 11 (52%) of the Management students answered administration and 8 (38%) ticked other.

When it comes to previous education, all the Multimedia students, except one, had completed tertiary education before (3 students 1-3 years and 8 students more than 4 years). Although the Management course does not require competence for access to higher education, 17 said that they satisfied entry requirements for higher education (80%), but only 2 students ticked that they had completed any tertiary level studies.

Generally, students in both groups said that the reason for taking this online study was either that they "needed the competence in their present work", or that they took the course to "increase their formal competence".

25 (76%) of the 33 students who responded to the questionnaire said that they chose online education because of the flexibility. 4 students (12%) said that the main reason for choosing the online course was that they were curious about this form of learning. No one ticked that they chose online learning "because there is no local course available".

Nearly all the students said that they had enrolled at their own initiative. 3 of the 21 Management students said that they had enrolled at the initiative of their employer. On the other hand, however, only 11 students (33%) are paying the course themselves, while 16 students (49%) have their course paid by their employer and the rest (6 students/18%) has the course paid by competence development funds or by some combination of work related and own funds.

12 students (38%) had found information about the course or programme on the Internet, while 10 students (31%) had found the information in advertisements, 5 students (16%) had received the information at work, and 5 (16%) ticked other (which seemed to mean that they new NKI and distance education courses generally).

As the Multimedia course was the last (fourth) course in a study programme, most of these students had experience from other courses taken before the course evaluated, while none of the 21 Management students had any previous experience from online learning.

When responding to the questionnaire, the majority of students in both groups (27/82%) had been registered on the course/programme for more than 2 months. Only 6 students (16%) had been registered for less than 2 months. 8 (67%) of the Multimedia students and 2 (10%) of the Management students had completed the course/programme at the time of evaluation. 2 Multimedia students and 4 Management students said that they had yet not submitted any assignments.

We asked the students what type of Internet access they had at the place where the studies "mainly take place". We were actually surprised that as many as 7 (58%) of the Multimedia students and 12 (57%) of the Management students said that they accessed the Internet through ISDN connection at home. 3 (25%) of the Multimedia students and 4 (19%) of the Management students accessed the course material via 56kb telephone modems. 2 students (1 in each group) had broad band connection, and one Multimedia student said that he accessed the course material via broadband, ISDN, ordinary modem and mobile phone dependent on where he was – "I may use my mobile connection when waiting at an airport".

To the question "Has your Internet connection had sufficient capacity and speed?" all the students except 3 Management students claimed to "agree completely" (58%) or "agree" (33%). Of the 3 students, one answered "uncertain" and 2 answered "disagree". Thus, as far as we can judge, the students are generally satisfied with the speed of their Internet access. (Still, there is probably no doubt that a 64k modem is too slow for receiving acceptable video at acceptable speed. Video was used only marginally in the study.)

As the multimedia content in the course and programme evaluated was introduced while these students were already enrolled, and the students differed in how much they had studied, not all respondents had actually seen, used or studied the multimedia elements or content.

10 (83%) of the Multimedia students and 9 (43%) of the Management students said that they had studied the multimedia content in the courses. However, in the following tables the numbers of students may vary somewhat as some students found that they could answer the questions even if they had not spent time studying the actual multimedia content.

Table 1. "I had sufficient competence for downloading software/plug-ins needed".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	11	100	8	67	19	83
Agree			2	17	2	9
Uncertain			2	17	2	9
Disagree
Fully disagree
N	11	100	12	101	23	101

There seems to be no doubt that the majority of the students mean that they have sufficient competence for downloading necessary software. Obviously the Multimedia students should have this competence. The Management students who answered "uncertain" had not seen the multimedia elements.

Table 2. "I had sufficient competence for using the software/plug-ins needed".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	7	64	5	39	12	50
Agree	4	36	6	46	10	42
Uncertain			1	8	1	4
Disagree			1	8	1	4
Fully disagree
N	11	100	13	101	24	100

It seems that some of the students have had some difficulties. The 2 Management students who do not agree, have not seen the multimedia elements. The answers do not tell whether lack of competence is the reason for not having seen the multimedia content.

Table 3. "It was technically uncomplicated to download the multimedia content".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	9	90	5	42	14	64
Agree			6	50	6	27
Uncertain			1	8	1	5
Disagree	1	10			1	5
Fully disagree
N	10	100	12	100	22	101

The Multimedia student who "disagree" and the Management student who is "uncertain" have not seen the multimedia content.

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	5	50	4	44	9	47
Agree	4	40	2	22	6	32
Uncertain	1	10			1	5
Disagree			3	33	3	16
Fully disagree
N	10	100	9	99	19	100

Most of the students found that it was quick to download the multimedia elements in the two courses. 3 Management "disagree" in the statement. 2 of the 3 Management students had actually ISDN access.

The students’ views on the quality and learning effects of multimedia in the courses

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	2	20	4	50	6	33
Agree	3	30	3	38	6	33
Uncertain
Disagree	4	40	1	13	5	28
Fully disagree	1	10			1	6
N	10	100	8	101	18	100

The majority of the students "agree" that the quality of the multimedia elements was high. It is worth noticing that many of the Multimedia students express negative attitudes. This may be caused both by the fact that some of the multimedia content presented was quite simple presentations (video, audio and/or video and audio with text) and that these students, having studied multimedia as a subject, had higher expectations to what multi media in teaching materials actually could have been. The majority of the Management students seem to be satisfied.

Table 5. "The Multimedia content represented an increase in the impression of quality of the course".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	2	20	1	10	3	15
Agree	4	40	5	50	9	45
Uncertain	2	20	3	30	5	25
Disagree	1	10	1	10	2	10
Fully disagree	1	10			1	5
N	10	100	10	100	20	100

There is a somewhat higher tendency that the students see the multimedia content to increase the impression of quality. Probable only text in the courses gives a stereotype impression, while some variation by multimedia gives a higher impression of quality.

Table 6. "The multimedia elements were well integrated into the other content of the course".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	2	20	1	11	3	16
Agree	2	20	2	22	4	21
Uncertain	5	50	5	56	10	53
Disagree	1	10	1	11	2	11
Fully disagree
N	10	100	9	100	19	101

No one "fully disagrees" with the statement saying that the multimedia elements were well integrated into the course. More than half of the students are "uncertain". This result is probably not surprising, as the multimedia elements actually were developed more as add-ons than totally integrated.

Table 7. "The multimedia content represented an increase in quality concerning learning effects."

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	2	20			2	11
Agree	4	40	3	33	7	37
Uncertain	2	20	6	67	8	42
Disagree	1	10			1	5
Fully disagree	1	10			1	5
N	10	100	9	100	19	100

More students "agree" than "disagree" in the statement that the multimedia elements in the courses increase learning effect. However, more than 50 percent of the students are "uncertain" or "disagree".

Table 8. "The multimedia elements made the course more inspiring/made it more motivating to study the course".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	1	10	1	11	2	11
Agree	5	50	2	22	7	37
Uncertain	1	10	3	33	4	21
Disagree	3	30	2	22	5	26
Fully disagree			1	11	1	5
N	10	100	9	99	19	100

There is no general agreement that the multimedia content made the courses more inspiring or motivating.

Table 9. "It is important that courses on the Internet has some multimedia content".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	6	54	6	33	12	41
Agree	5	46	6	33	11	38
Uncertain			4	22	4	14
Disagree			1	6	1	3
Fully disagree			1	6	1	3
N	11	100	18	100	29	99

It is, not surprisingly, a tendency that the Multimedia students find it important that courses on the Internet has some multimedia content. Generally, the students "agree" to the statement – with some few exceptions.

Why is multimedia content important in Internet courses, what is the main purpose?

Table 10. "Multimedia elements may have different aims – which do you find most important?"

	Multimedia		Management		Total
	n	%	n	%	n	%
Make the course more inspiring	2	17	3	17	5	17
Increase learning effectiveness	5	42	6	33	11	37
Give variation	4	33	7	39	11	37
Other	1	8	2	11	3	10
N	12	100	18	100	30	101

The students were asked to tick the most important purpose of multimedia. Those ticking "other" seemed actually to mean all of the given reasons for including multimedia in Internet courses. It might be important to notice that "variation" is ticked as many times as "learning effectiveness"

There is hardly no doubt, that even with the most recent software, developing multimedia content is more costly that developing text material in courses. We asked the students whether they would accept to pay more for courses containing multimedia.

Table 11. "Would you prefer multimedia elements in your Internet courses if…"

	Multimedia		Management		Total
	n	%	n	%	n	%
Course price did not increase	4	40	16	94	20	74
Course price increased with 25%	5	50	1	6	6	22
Course price increased with 50%	1	10			1	4
Course price doubled
N	10	100	17	100	27	100

It seems clear that the Multimedia students are more open to accept a modest increase in the price of courses containing multimedia content. He Management students are nearly unanimously against increase in price for courses including multimedia content. 5 of the 6 Multimedia students and the single Management student who accept a minor price increase have their courses paid by the employer. In the total group there is only one single (Multimedia) student who accepts a minor (25%) course price increase for multimedia content in the courses on the Internet.

This means that any organisation when developing course materials for the Internet has to be aware that the private individual market does not seem to be willing to pay more for courses with multimedia content.

As part of the questionnaire we included some open-ended questions on course quality in general and some questions specifically on multimedia.

From the Multimedia students we received some few suggestions. One student suggested that multimedia should be interactive.

Another student suggested more "Flash elements". Other comments were that "multimedia should be in addition to written texts but not for repetition.

Directly related to the subject, one student suggested that the assignments for submission should require more application of multimedia software in this specific course to develop practical skills rather than theoretical knowledge.

Another comment was that it was a positive experience to see that NKI really used multimedia in its own course, and that the course gave a much better impression of quality after the multimedia content was introduced, but, on the other hand, that much of the multimedia on other sites referred to in the course had higher quality.

One of the Management students suggested that practical examples related to the theoretical content could have been illustrated by multimedia. Another student commented that most of his course work was done off line and that he did not use the Internet pages much, …"but for those specifically interested it is of course more inspiring with more channels of information".

One Multimedia student suggested that the course should include a conference from time to time (probably referring to the possibility of synchronous audio or video conference on the Internet), a solution that for many reasons is not used in NKI online distance courses. Another student also suggested the use of conferencing systems (and is probably not referring to asynchronous text based conferencing, which is an integrated part of the course).

One Multimedia student commented that multimedia to some extent may lead to the student becoming a passive viewer and receiver of information, rather than an active learner. Some others also indicated that multimedia not necessarily constitute any improvements of quality, "… one must not exaggerate – multimedia must have a clear and defined purpose, if not one could better do without."

Another student commented in the same way: "Multimedia must have a clear purpose, it not don’t use it."

One student commented: "It must be directly useful – not just supplement to make the course more fancy."

Other viewpoints on the quality of the Multimedia course and the Management Programme

We used the opportunity to evaluate the course and study programme by some general questions and statements on the students’ experiences.

Generally, we may conclude that the Multimedia students were rather negative both to the course materials and towards support from the tutor in the course. Both concerning agreement or disagreement with statements and answers to he open-ended questions the students indicated that support from the tutor was not up to their expectation. The comments were few, to little help, and feedback was to slow. They also criticised the textbook and the study guide for the course.

The Multimedia course was the final course in the study programme ‘Information Networks" and it seemed clear that the students were generally satisfied with the programme and the other courses, but disappointed about both the materials and teacher support in the Multimedia course.

Some of the tutors were not good and should have been replaced
As a student I wish to know how long time it takes to receive feedback from the tutor, this should have been stated in the catalogue, does it take one week, or two weeks – or? As the subject is developing quickly, I wished that the tutor supplied web pages with updated information.
Better feedback, more helpful comments
Very dissatisfied with the tutor
Too little feedback from the tutor, the tutor must be more willing to answer questions, the tutor seems to have to many students
The tutors should be more active in stimulating discussions with the students

This is the first phase with the revised course, I really doubt that the ‘new’ course is better than the previous one
The text-book was to difficult to understand – exchange the book, it goes too much in details compared with the exam
Choose books that are more suited for self study, the present book is only suited as supplement to lectures and not for distance learning, the study guide does not help much, it gives only reading directions
The text-book was outdated – but except for this course, the courses and the total programme was good, I have no critical comments to the other courses, or the other tutors
American text-books are generally difficult to read, they use too many words to explain, it takes too long to find what is important, the core content

The administrative study manager did an excellent job
As a student resident in Thailand I wish to stress that our administrative study manager did an excellent job, I wish to thank for the support and service attitude that met us, it is really important to have one specific contact in the NKI administration who answers all questions quickly and with consideration
I am very satisfied with NKI, distance learning has worked very well for me
What I learned was good, but the service in the course was not good enough, this relates only to the Multimedia course, not the programme in general
I am very satisfied with the learning results, which mainly are due to my own work more that the tutor and the course materials

The Multimedia students were quite negative in their attitudes to the course, but generally positive to the quality of the total study programme. In the Management case it was the total programme that was subject to evaluation.

We may clearly conclude that the students were generally very positive to the general arrangements, materials, and tutors of the programme. Their open statements are presented below:

I am generally very satisfied
I am very satisfied; it is an excellent way of studying and learning. Especially for me with long working hours
It has been very useful, in the sense that one can apply what one is learning immediately
I am very satisfied with what I have experienced so far; I did not imagine how close and personal the communication could be when learning online
The courses seem to be excellent; everything seems so simple and very well organised and structured; I have known NKI as a recognised distance teaching institution, which has been confirmed
A very fine offer, it is simple for everyone, you can be everywhere, and communicate efficiently to really solve possible difficulties
I am generally satisfied, the everything functions as I expected
The Management programme: very job related, very positive, when I answered assignments I always had to reflect on the job, job, and job. When I prepared for the exam, I had a new experience and found the broadness and got a complete overview

One should strive to get more activity in the student discussion forum
It has been very good, perhaps the tutor could push the participants more, it is easy to put the studies aside
Perhaps one could stimulate some more discussion, but I am uncertain how it could be done
One could focus more on discussion in the forums, the activity has been quite low this year, perhaps one could introduce assignments that made it necessary to contribute to the forum discussions
It should be more discussion in the ‘classroom’, possibilities for discussion should be increased, and one could make it easier to ask questions, perhaps also introduce direct lectures from time to time (the student probably means face-to-face)
To some extent I miss the atmosphere of a classroom, a fixed timetable could have helped, but my work situation makes fixed timetables impossible
This form of study requires self discipline, but for those who manage it is clearly more efficient for learning
I would have liked some more personal follow-up
More immediate feedback from the tutor would increase quality

Table 12. "Technical support – Explanations are easy to understand and gives the necessary help".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	2	18	6	33	8	28
Agree	4	36	9	50	13	45
Uncertain	3	27	3	17	6	21
Disagree	2	18			2	7
Fully disagree
N	11	99	18	100	29	101

The overall impression is that the students in both groups are reasonably satisfied with information/explanations on technical matters. It might be a little surprising that some of the Multimedia students are negative, as one should believe that these students at this stage should not encounter technical difficulties.

Table 13. "Technical support – Problems are quickly solved/questions are quickly answered".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree			4	27	4	17
Agree	1	13	4	27	5	22
Uncertain*	5	63	5	33	10	44
Disagree	1	13	2	13	3	13
Fully disagree	1	13			1	4
N	8	102	15	100	23	100

* Some students have not answered this question. The reason probably being that they have not met technical difficulties. Also the answer "uncertain" may actually mean that they do not know, as they have not had technical problems.

Some few students indicate that they have met technical problems, which could have been solved more quickly.

Administrative support mainly concerns the students’ contacts and communication with the Study Manager in the NKI administration.

Table 14. "Administrative support – The Administration/Study Manager gives helpful feedback when I need help".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	9	82	8	57	17	68
Agree			4	29	4	16
Uncertain*	1	9	1	7	2	8
Disagree			1	7	1	4
Fully disagree	1	9			1	4
N	11	100	14	100	25	100

* Some students have not answered this question. The reason probably being that they have not yet had specific problems to be solved by the course administration. Also the answer "uncertain" may actually mean that they do not know, as they have not contacted the administration for support.

Generally, we may conclude that, with some few exceptions, the students are satisfied with support from the Administration/Study Manager.

Table 15. "Administrative support – Questions to The Administration/Study Manager are quickly answered".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	8	67	7	54	15	60
Agree	3	25	2	15	5	20
Uncertain*	1	8	4	31	5	20
Disagree
Fully disagree
N	12	100	13	100	25	100

We may clearly conclude that the students in this evaluation study are satisfied with response time for inquiries to the Administration/Study Manager.

In some previous evaluations studies we have examined the students’ general attitudes towards learning on the WWW/Internet (Rekkedal 1998, 1999, Keegan & Rekkedal 2000). Attitudes were measured by different 5-graded ‘Likert scale’ questions similar to many of the questions presented above.

Table 16. "From my participation in this course I can state that it is possible to achieve excellence in courses that are taught on the WWW".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	6	50	5	26	11	35
Agree	2	17	12	63	14	45
Uncertain	1	7	2	11	3	10
Disagree	3	25			3	10
Fully disagree
N	12	99	19	100	31	100

We can clearly conclude that the majority of the students in both groups "agree" to the statement that excellence can be achieved in courses taught on the WWW. However, some of the Multimedia students do "disagree".

Table 17. "My learning from the course was damaged because it was on the WWW".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree
Agree	3	35	1	5	4	13
Uncertain	1	8	5	26	6	19
Disagree	3	25	4	21	7	23
Fully disagree	5	42	9	47	14	45
N	12	100	19	99	31	100

This statement was put in a negative form. We may note that the same 3 Multimedia students and 1 Management student "agree" to this negative statement. Thus, it seems that online learning does not suit some few students trying this form of study. But, generally, as shown in our previous evaluation studies, the majority of online students hold the opinion that study on the Internet/WWW leads to satisfying learning results.

Table 18. "My learning from the course was damaged because it was on the Internet/WWW".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree
Agree	3	35	1	5	4	13
Uncertain	1	8	5	26	6	19
Disagree	3	25	4	21	7	23
Fully disagree	5	42	9	47	14	45
N	12	100	19	99	31	100

Table 19. "My enrolment in this course was facilitated because it was on the Internet/WWW".

	Multimedia		Management		Total
	n	%	n	%	n	%
Fully agree	10	83	13	65	23	72
Agree	2	17	6	30	8	25
Uncertain
Disagree
Fully disagree			1	5	1	3
N	12	100	20	100	32	100

With one single exception, the students "fully agree" or "agree" that the fact the course or study programme is offered on the Internet facilitates enrolment.

Positive and negative attitudes towards an activity can be measured in many ways. We have decided that whether the student would participate in the activity again or recommend the activity to a colleague or friend would constitute reasonably valid indicators of how the activity is valued. (See e.g. Boshier et al. (1997) who used the variable "Recommend this course to friends interested in the subject" as an indicator for the researchers’ subjective view of WWW courses.)

Table 17. "I would enrol again in a WWW based course (if there were courses suited to my learning needs)".

	Multimedia		Management		Total
	N	%	n	%	n	%
Fully agree	8	67	14	70	22	69
Agree	2	17	5	25	7	22
Uncertain	1	8	1	5	2	6
Disagree	1	8			1	3
Fully disagree
N	12	100	20	100	32	100

Over 90 percent of the students "fully agree" or "agree" that experiences are positive and opens for possible future enrolment in Internet courses. Only one Multimedia student "disagrees" to this statement.

Table 17. "I would encourage a colleague or friend to enrol in a WWW based course".

	Multimedia		Management		Total
	N	%	n	%	n	%
Fully agree	9	75	14	70	23	72
Agree	2	17	5	25	7	23
Uncertain			1	5	1	3
Disagree	1	8			1	3
Fully disagree
N	12	100	20	100	32	101

A large majority of the students in both groups have experiences from learning on the Internet, which make them positive towards recommending this form of learning to friends and colleagues.

This study was carried out mainly to evaluate the use of multimedia content in one specific course and one study programme offered on the Internet by NKI Distance Education. The experimentation with multimedia and the evaluation study were part of a EU Leonardo project lead by Ericsson Systems Expertise in Dublin, Ireland.

The project was quite small and did not finance development of very expensive and sophisticated multimedia. However, through the project the NKI project group had the opportunity to gain experience both in developing multimedia in some courses and evaluate these results by feedback from students. Regrettably, the project period was too limited for any final conclusions, and further collection of data should be undertaken at a later stage with more students.

The study also included some general questions on the students’ attitudes towards learning on the Internet.

Generally, the evaluation results show that both student groups in the study are positive towards their study experiences and learning on the Internet. The students taking the course ‘Multimedianett’ (Multimedia Networks) do put forward some critical comments on this specific course and the work of their tutor, while the students taking ‘Arbeidslederskolen’ (Management Programme) are generally positive.

Concerning multimedia content, the students’ attitudes differ. Most of the students look at the multimedia content as an increase in quality. In fact, the majority "agrees" to that multimedia in courses on the Internet "is important". On the other hand, they are not willing to pay more for courses containing multimedia. It also seems that they look at "variation" in ways of presenting course materials as just as important as possible increases in learning effects.

The students have encountered few technical problems in connection with downloading plug-ins, multimedia elements or using the multimedia content in their studies. The students clearly see some problems in multimedia in courses, mainly related to speed of downloading, but also that multimedia really must have a clear didactic objective to be helpful for learning.

Amundsen, A. (1997): FOSS – File Oriented Statistical Package. Veiledning for brukere. Oslo.

Bates, A. (1991): Interactivity as a criterion for media selection in distance education. Never Too Far, 16, 5-9.

Boshier, R., Mohapi, M., Moulton, G., Qayyum, A., Sadownik, L. & Wilson, M. (1997): Best and worst dressed web courses: Strutting into the 21^st century in comfort and style. Dist. Ed. 1997, 2, pp. 327-349. Toowoomba, University of Southern Queensland.

Clark, R.E. (1983): Reconsidering Research on Learning from Media. Review of Educational Research, volume 53, number 4, p. 445.

Clark, R. E. (1994): Media will never influence learning. Educational Technology Research and Development, 42, 21-29.

Dubin, R. & Taveggia, T. C. (1968): The Teaching-Learning Paradox. Eugene, Oregon: Center for the Advanced Study of Education Administration. ERIC Documant ED 026 966 (1968)

Hofstad, K., Løland, S. & Scott, P (1996): Norsk dataordbok. Oslo: Universitetsforlaget.

Keegan, D. & Rekkedal, T. (2000): Achieving excellence in courses on the WWW. Report from parallel evaluation studies at three institutions carried out as part of the EU Leonardo Project "Multi Media World Wide Kernel for Distance Education". http://www.nettskolen.com/alle/forskning/50/Jointarticle3.html

Mason, R. (1998): Globalising Education. Trends and Applications. London: Routledge.

Paulsen, M. F. & Rekkedal, T. (2001): The NKI Internet College: A Review Of 15 Years Delivery Of 10,000 Online Courses. In: International Review of Research in Open and Distance Learning (January 2001). (http://www.irrodl.org/) ISSN: 1492-3831

Rekkedal, T. (1998): Courses on the WWW - Student Experiences and Attitudes Towards WWW Courses. An Evaluation Report Written for the Leonardo Online Training Project.
http://www.nki.no/eeileo

Rekkedal, T. (1999): Courses on the WWW - Student Experiences and Attitudes Towards WWW Courses - II (1999). Evaluation Report Written for the Leonardo Online Training Project, MMWWW. http://www.nki.no/eeileo/

Schramm, W. 1979. Big Media, Little Media: Tools and Technologies for Instruction. Beverly Hills, Calif.: Sage Publications.

Mer interaktive multimedieinnslag
Flash innslag
Burde være tillegg til eventuelt skreven info – ikke repetisjon. Viktigst for meg vil det være at oppgaven som skulle leveres skulle bruke multimedieredskap, det ville øke mine praktiske ferdigheter som ikke er blitt nevneverdig bedre etter å ha fullført kurset.
Viser bedre at kurset benytter multimedieinnslag. En del av materiellet som det ble vist til hadde høyere kvalitet enn NKIs egen materiell. NKI har litt igjen for å nå det beste nivå på multimedia. Kunne også ha kjøpt inn materiell med større nedslagsfelt. Kurset ga et mye bedre inntrykk etter hvert som multimedia ble brukt også i NKIs eget kurs.
Hele kurset som multimedia.

Dersom du bare blir passiv lytter/tilskuer er nytteverdien begrenset
Kurs bør absolutt inneholde multimedia – uten overdrivelse av effekter.
Har du dårlig forbindelse tar det lang tid
Problemer dersom man har liten båndbredde
Må ikke bli for mye – det må ha en hensikt, ikke bare for multimedias skyld
Ingen ulemper egentlig. Det hadde vært fint med kombinasjon med samlinger og mer dialogmuligheter med medstudenter og lærere.
Ingen ulemper
Hvis det ikke er noe poeng med multimedieinnslag kan en like godt la være.
Krever god maskinkapasitet hos brukerne

Noen av lærerne fungerte dårlig og burde vært skiftet ut
Som student vil jeg gjerne vite hvor lang tid det tar før lærer gir tilbakemelding. Burde finnes i kurskatalogen – tar det en uke – to uker eller? Tilgang til tidligere eksamensoppgaver og eventuelt sensorveiledning. Siden fagene jeg har tatt er i en rasende utvikling, kunne jeg ønsket at lærer hadde en nettside med oppdatert info, nye lenker osv. Bøker er foreldet før de utgis.
Første runde med nytt kurs. Kritisk til at det nye kurset var bedre enn det gamle. Skeptisk til gamle lærebøker. Lite feedback fra lærer. Læreren må ha tid til å svare på spørsmål. Læreren har for mange studenter
Bedre tilbakemeldinger fra lærer, bedre kommentarer.
Svar på øvingsoppgavene
Lærere bør være mer aktive for å få i gang kommunikasjon med studentene.
Læreboka var tunglest – bytte ut læreboka. Går mye i dybden i forhold til eksamen.
Finn lærebøker som er mer egnet til selvstudier. Nåværende bok er egnet som støttebok til forelesninger. Studieveiledningen veileder lite, gir kun anvisninger på hva som skal leses.

Studieleder Gjorde en veldig god jobb.
Bøkene var gamle. Bortsett fra dette kurset, var kursene i studiet bra, har ingen kritiske kommentarer til lærere eller andre forhold på de øvrige kursene.
Som student bosatt i Thailand, vil jeg spesielt bemerke den gode hjelp og serviceinnstilling studieleder har møtt oss med hele tiden. Svært viktig med en fast person på NKI som har besvart henvendelser for og seriøst – takk for det.
Det jeg lærte var bra, men måten gjennomføringen var ikke bra. Dette gjelder spesielt Multimedianett og ikke studiet generelt.
Har benyttet teknologien med både bærbar og stasjonær PC, på jobb, hjemme og gjennom mobiltelefon på reiser. Har vært morsomt å se at til og med GSM virker. Etter hvert kommer disse mulighetene til å bety mye.
Er veldig fornøyd med studiene ved NKI – det fungerer utmerket for meg.
Svært misfornøyd med læreren. Læreren gidder ikke å lese fordi det er for langt. Detaljnivået i læreboka går mye lengre enn til eksamen. Ellers fornøyd med læringen, men det er et resultat av egen innsats mer enn læreren og kursopplegget.
På grunn av forhold nevnt i 48 blir kurs på nettet tunge å ta. Ikke så enkelt å stille spørsmål eller be læreren om hjelp for det er vanskelig å vite hva man skal spørre om uten å ha lest seg i hjel først. Amerikanske bøker er generelt tunge å lese da de bruker mange ord på å forklare ting. Dermed tar det lang tid å finne det som er viktig, og det kan være vanskelig å finne hva som er kjernen. Den beste studieveiledningen jeg har brukt er den på www-kurset. Den veiledet godt, den fortalte ikke bare hva du skulle lese.

Hastighet på nettet kan være et problem og irritasjonsmoment
Man kan bli litt for opptatt med selve innslagene
Bør være valgfritt om man bør se på. Tar ofte lang tid i dag. Kan skape mer ulempe nn nytte. Ikke bra om en må gjennom innslag for å komme videre.
Kan kanskje bli litt rotete, må ikke bli for mye, ikke oppstykking, fint med bilder ved siden av tekst
Fint med noe variasjon, men det må ikke bli for mye.
Noen ganger tidkrevende.
For mye "fancy" ting, slikt som forstyrrer konsentrasjonen vekk fra det man opprinnelig skal lære..

Få forumet til å virke bedre
Det har vært helt bra. Mulig for læreren å pushe på deltakerne. Lett å drøye litt for lenge.
Kjappere tilbakemelding fra læreren.
Det kunne legges opp til mer diskusjon, men usikker på hvordan – ingen formening.
Man kunne fokusere mer på forumene. I år har det vært liten aktivitet. Kanskje lage en oppgave som akkurat går på det området, det å bruke Internett som kanal.
Burde vært bedre diskusjon/debatt i "klasserom" og/eller mulighet for en forelesning en gang i blant for å øke kompetansenivået og selve læringen. Muligheten for diskusjon og debatt, samt lettere å stille spørsmål.
Mer personlig oppfølging.
Legge ut forslag til løsning på tidligere eksamensoppgavene. Økt aktiviteter på forumene på web.

Stort sett positivt
Meget fornøyd. Det har vært en utmerket måte å studere på spesielt for meg som har lang arbeidsuke.
Nyttig i den forstand at alt man lærer kna man benytte etter kort tid.
Er egentlig veldig fornøyd med det som er gjennomført til nå. Var ikke klar over hvor god kontakten på nettet kunne være.
Kurset ser bra ut, alt virker enkelt, virker veldig ordentlig. Kjente NKI som en god fjernundervisningsinstitusjon. Har lenge tenkt på å ta studier som fjernundervisning ved NKI eller NKS som jeg kjenner som de to anerkjente fjernundervisningsinstitusjonene.
Veldig fint tiltak – enkelt for alle – kan være hvor som helst – samtaler som hjelper å løse problemene
Fornøyd. Mer kompendier og på papir for de som ikke er helt fortrolige med å finne fram på nettet.
Generelt fornøyd – studiet fungerer slik jeg hadde forventet.
Jeg har ikke funnet frem til multimedieinnslagene på nettet, har bare sett den lille biten demo om Arbeidslederskolen. Det var svært lite. Ellers jeg ønsker raskere tilbakemelding fra lærer.
Jeg savner litt atmosfæren i et klasserom. Oppfølging og fastbestemt timeplan.
Men arbeidssituasjonen min gjør det til tider vanskelig å følge en fast timeplan..
Ellers er studiet meget interessant og nyttig !!!
Arbeidslederskolen: veldig jobbrelatert, positivt. Når jeg løste innsendingsoppgaver måtte jeg hele tiden tenke på jobb, jobb, og jobb. Når jeg skulle lese til eksamen fikk jeg en ny opplevelse, da så jeg mer helheten og bredden i det.
Dette krever klart mer selvdisiplin enn å følge klasseromsundervisning, men gir absolutt mer til de som har selvdisiplin til slike typer studier.

NKI Fjernundervisningen har de siste årene vært med i flere EU-prosjekter.
Ett av disse heter "MediaWEB".

Prosjektet har som mål å undersøke pedagogiske muligheter og eventuelle
problemer knyttet til bruk av multimedia på Internett. Som de fleste vet er
det både kostnadsproblemer, tekniske problemer og pedagogiske problemer med
bruk av multimedia. Det betyr at de fleste nettkurs i dag i stor grad er
tekstbasert.

Som del av prosjektet har NKI Fjernundervisningen eksperimentert litt med
multimediaelementer i 2 kurs/studier.

Vi har utviklet noen få multimediaeksempler i kurset Multimedianett som er
en del av studiet Informasjonsnett. Disse eksemplene består av ulike
løsningsforslag til øvingsoppgaver i Studieenhet 1, pluss innledninger i
ulik form til alle studieenhetene.

Videre har vi laget noen multimediaeksempler til studiet Arbeidslederskolen.

Hvis du ikke har vært inne og sett på de multimediapresentasjonene som er
laget til Multimedianett, hadde det vært fint om du tok deg tid til å se på
dem - gjerne også om du ikke har begynt å lese faget ennå.

Jeg ønsker å gjennomføre telefonintervju med deltakere på kurset. Det hadde det vært fint om du tok deg tid til å besvare dette skjemaet ved å krysse av og skrive i selve dokumentet og returnere det til meg. Skjemaet er et rent rtf-dokument. Jeg regner med at du eventuelt bare kan sette * foran q på det alternativ du velger, slik:

Jeg er spesielt takknemlig for utfyllende svar også på de åpne spørsmålene. Alle spørsmålene der det står kurset/studiet gjelder Arbeidslederskolen.

Dersom det er noe du føler at du ikke kan svare på, kan du la spørsmålet stå ubesvart.

Svarene på dette spørreskjemaet gjelder kurset/kursene Multimedianett/Arbeidslederskolen

5. Grunn til å begynne på kurset (sett bare ett kryss for den viktigste grunnen).

6. Hvorfor valgte du et nettbasert kurs (sett bare ett kryss for den viktigste grunnen).

11. Hvem var hovedansvarlig for at du begynte på kurset/studiet? (sett bare ett kryss).

15. Synspunkter på studiestøtte – Forklaringer, kommentarer og veiledning fra lærer er til god hjelp i læringen.

16. Synspunkter på studiestøtte – Tilbakemeldinger fra lærer kommer raskt.

17. Synspunkter på teknisk støtte – Forklaringer er greie og gir nødvendig hjelp.

18. Synspunkter på teknisk støtte – Problemer løses raskt/tilbakemelding på spørsmål kommer raskt.

19. Synspunkter på administrativ støtte – Administrasjonen/studieleder gir gode tilbakemeldinger når det er behov for hjelp.

20. Synspunkter på administrative støttefunksjoner – Svar på henvendelser kommer raskt fra administrasjon/studieleder

22. Jeg hadde tilstrekkelige tekniske kunnskaper til å begynne på nettbasert undervisning.

26. Min tilknytning til nettet har hatt tilstrekkelig kapasitet og hurtighet.

28. Jeg hadde tilstrekkelige tekniske kunnskaper til å nedlaste den programvare/plug-ins jeg har hatt bruk for.

29. Jeg har hatt tilstrekkelige ferdigheter når det gjelder bruk av nødvendig programvare/plug-ins for å benytte multimediaelementene.

33. Multimediaelementene representerte økning av kvaliteten når det gjelder læringseffekt

34. Multimediaelementene representerte økning av det totale inntrykk av kursets kvalitet

36. Multimediaelementene gjorde det mer motiverende/inspirerende å jobbe med kurset

39. Multimediainnslag i kurs på Internett kan ha ulike formål. Hva mener du eventuelt er viktigst.

42. På bakgrunn av min deltakelse i kurset/kursene mener jeg at det er mulig å få fullt læringsutbytte gjennom kurs som undervises på WWW.

43. Det at kurset ble tilbudt over Internett/WWW gjorde det enklere for meg å begynne på kurset.

44. Min læring i kurset ble dårligere fordi kurset ble undervist over Internet/WWW.

45. Erfaringene mine tilsier at jeg gjerne kunne ta et WWW-kurs ved en senere anledning.

46. Jeg anbefaler gjerne en kollega eller venn å ta kurs basert på undervisning over WWW.

48. Har du andre synspunkter på forhold som kunne øke kvaliteten på det kurset/ studiet du har tatt ved NKI Nettskolen?

Torstein Rekkedal:

Students’ Views on Multimedia in Distance Learning Courses on the WWW An Evaluation Study Among NKI Students

Introduction

NKI and the NKI Internet College

The Courses Evaluated

Multimedia in WWW Courses

Method and Design of the Evaluation Study

Results