has not kept pace with the development of increasingly rich IP-based delivery
platforms because the e-learning experience is far too often puerile,
boring, and of unknown or doubtful effectiveness.
don’t seem to be aware of how people learn, for they continue
to use mostly flawed models.
are more interested in throughput and low unit cost, so solid measures
of effectiveness are infrequently developed or applied.
available platform drives the instructional strategy, which may
not be appropriate to the learning style of trainees or to the learning
cost of development is high, so bad (cheap) programs drive out the
good ones in the absence of any commitment to measure effectiveness.
e-learning experiences for complex competencies are rarely scalable.
does the situation persist, when so many knowledgeable people have sat
through a course they know to be bad? Habit, and perhaps low expectations
by trainees—we don’t expect to find the courses stimulating
or engaging, so we don’t complain too much when they are pretty
much like the boring lectures we used to sit through.
flawed model of cost-effectiveness
a moment when higher education has become increasingly convinced that
the standard classroom lecture is not a particularly effective way of
teaching, how ironic that many of those responsible for e-learning say
the ultimate goal is to mimic the classroom experience as much as possible.
Perhaps that’s one indication that e-learning is no longer an unproven
cutting-edge experiment, but has moved into the mainstream. A few years
ago, only a minuscule percentage of corporate training was technology-based,
but in the year 2000, that figure was up to 24 percent and compares impressively
to the 57 percent delivered through traditional classroom methods. There
are other signs that higher education is looked to for systems of learning
management and measurement. The “Carnegie units” model of
counting noses (one person in one course for one term) is a standard component
of ROI calculations and, while no school system or college would ever
mention ROI publicly, they do employ all kinds of ratios to determine
is difficult not to conclude, however, that there is relatively less emphasis
on outcomes measurement in corporate training, certainly in comparison
with higher education, where it is intense; my experience over more than
30 years in the corporate world suggests that most businesses give more
weight to anecdotal accounts than to efforts to measure outcomes rigorously.
Where there is an effort, it seems to be directed toward measuring the
cost side of the dyad, especially where training staff can claim substantial
cost savings. The trade press is replete with articles quoting training
managers boasting of how many hundreds of thousands of dollars (or more)
they expect to save with e-learning, generally through less travel, fewer
hours lost to training, and lower staff costs. Years ago, ROI never came
up in discussions of corporate training budgets, primarily because the
knowledge/skill level of the workforce was regarded as an intangible asset
that did not show up on the balance sheet. That may still be the case,
but the telecommunications and systems infra-structure necessary to deliver
e-learning does appear on the balance sheet, so ROI has become
a tool of the trade in training departments. “In tough economic
times, you have to demonstrate the ROI of an e-learning project back to
the business sponsors,” said an HR director at a major firm quoted
in Online Learning.
development of standards is a distraction
addition to the emphasis on cost savings, there is another dimension that
has received considerable attention in e-learning circles—the development
of standards such as SCORM (Shareable Courseware Object Reference Model)
and IMS (Instructional Management System). These are not standards that
treat learning outcomes, but instead deal with tagging, coding, and indexing
Learning Objects to facilitate reuse of digitized training materials.
Some have likened that effort to “rearranging the deckchairs on
the Titanic,” but that is perhaps harsher than necessary. The emphasis
on adoption of standards is clear: “Implementation of SCORM specifications
can help learning technology to become reusable, interoperable, stable,
and accessible.” Who would be opposed to standards, except there
is nothing in any of those standards that focuses attention on the effectiveness
of the Learning Objects. Indeed, the term Learning Objects itself ought
to cause some unease. An LO (Learning Object) is defined as a “discreet
small chunk that can be used alone or dynamically assembled to provide
just enough and just-in-time learning. Learning Objects can also enable
learners to select the training that is most relevant for them, perhaps
even in a media format that matches their preferred learning style (auditory,
visual, etc.).” A Learning Object is, thus, a thing that has physical
dimensions (type, number of megabytes) that can be measured; it can be
tagged and indexed for future use. No one knows, however, whether that
LO has ever resulted in anyone learning anything or subsequently demonstrating
know that learning doesn’t happen in discrete chunks. An acquaintance
at the University of Colorado once said, “We have to cross the line
between ignorance and insight many times before we truly understand.”
We get it, then lose it, then kind of get it again, then find out we don’t
quite have it right, and ultimately, after struggling to master the concept,
we have it. Learning often appears a little ragged, and does not generally
come in nicely packaged objects, no matter how systematically tagged.
Efforts to measure outcomes are difficult enough, but to substitute for
those efforts a set of standards which tag and index inputs seems to me
to be mistaken.
lack of emphasis on outcomes
the e-learning industry attempts to quantify content elements, the concern
is misplaced; it diverts attention from the more important issue of measuring
effectiveness, e.g., under what conditions does e-learning work?
The drive for standards, originating in the mutual efforts of the aviation
industry and the Department of Defense, appears to be part of an attempt
to make e-learning programs more acceptable to IT departments, who are
reluctant to consider anything that involves audio, video, and other features
with bandwidth or security issues. The next step, presumably, will be
to measure the mean number of megabytes in a Learning Object, so IT can
estimate how much additional capacity they will need to add in order to
teach the sales force how to sell the company’s new gizmo.
appears that this push for standards really has little to do with measurable
learning outcomes. The move toward standards arose, we are told, because
of “complaints about previous generations of e-learning products
[which] range from integration issues and interoperability concerns to
bandwidth and scalability problems.” Those complaints did not, it
appears, come from trainees, who often found the training dull, rigid,
and not related to their work, but rarely complained about interoperability
and integration issues.
the absence of a sustained emphasis on measurable outcomes, there is little
incentive to value anything but “throughput” and low unit
cost. But dropout rates (defined as failure to complete a course) for
e-learning are much higher (70 percent) than for standard instruction
in four-year colleges (about 15 percent). Although three-fourths of corporations
use course completion as a measure of effectiveness, some vendors and
training executives seek to downplay completion rates as a significant
measure of success. Community colleges, on the other hand, pay close attention
to course completion rates and consider them a most significant indicator
(though not the only one) of their success.
concerned voices within the industry have been raised: Eliot Masie, in
response to an InformationWeek question about the scariest question
[regarding e-learning]: “Does it work? If I invite 50 people into
a session, is there learning? If it’s well-structured, there’s
the right content, we’ve taken care of who we invite, and there’s
a payoff at the end, they’ll probably learn as well as [they would]
in the classroom—which isn’t very well.” Still, Masie
is among those leading the push for adoption and dissemination of standards,
so he apparently sees no inherent contradiction between the centrality
of learning effectiveness to the long-range success of e-learning and
the drive for interoperability. Indeed, he specifically notes that “all
the work on standards and specifications will play a similarly critical
role in causing the ‘take-off’ of the learning industry, [but]
they do not, in and of themselves, look after ensuring the quality or
effectiveness of learning.”
fact is, e-learning has become well established and will only grow, whether
there are standards or not. The cost savings are too great to ignore,
regardless of the lack of measurable outcomes, and e-learning has made
available to people in remote locations a variety of courses they would
not otherwise have had access to; if their motivation was high and their
perseverance strong, I have little doubt that many of them learned. So
we are not talking about the survival of e-learning. But we may be talking
about a degrading of quality if we are content to measure only the cost
savings, the compliance with standards and the number of Learning Objects
dispensed. Clearly, we should be under no illusions about effectiveness
if the failure-to-complete rate remains at 70 percent.
can be done?
begin with the learning experience. If that is not engaging, only the
most highly motivated (or those under duress) are likely to complete the
course. How would the typical trainee describe the typical e-learning
experience? Boring is the first word that comes to mind, whether the instructional
strategy is reading text, watching a streaming video of the average instructor,
or following an audio-over-PowerPoint presentation. The developer’s
attitude seems to be similar to my high school biology teacher’s,
who often reminded us, “If you’re smiling, you’re not
may call it a masochistic tendency, but I have an irresistible urge to
examine e-learning courses whenever I get a chance. Not to complete the
course, but to sample it and see how the designer engages my interest,
allows me to move through the material, tests my understanding, reinforces
appropriate responses and my ability to apply the learning, and corrects
my mistakes. I like to see if that designer has made any attempt to adapt
to people with different learning styles or perhaps with a different purpose.
So I examine the free demonstration courses offered online whenever possible,
expecting that purveyors would put their best foot forward and show content
that was interesting and well designed. But it is not so. I hope those
demos are their throwaway material—dogs they couldn’t get
anyone to register for—because if they are representative of the
rest of their curriculum, a lot of customers are being taken.
the heart of the problem lie a couple of factors beyond the unwillingness
to insist on measurable outcomes: 1) available technology is driving the
instructional strategy, 2) developers don’t know anything about
how people learn, and 3) a desire to produce courses at the lowest unit
cost leads to cutting corners and/or to repurposing of material that wasn’t
very good to begin with. Absent the chance to network with peers, students
find e-learning technologies to be very unforgiving. Let’s examine
the first of those factors.
is not an e-Learning strategy
need to calculate the ROI for a training initiative should lead to an
insistence on definition of an e-learning strategy, which is a very good
thing. But the strategic statements I’ve seen are driven by technology,
not by corporate objectives. The infrastructure (largely network bandwidth
and telecommunications capability) is the strategy in some of those statements.
To me, that’s backward. Begin with the organization’s objectives,
extract the competencies required to attain those objectives, examine
the constraints (time, distance, trainee’s experience, corporate
culture, etc.), and then you can begin to outline the kind of learning
experiences that will be necessary to develop those competencies. Only
at that point (or when describing the constraints) do you consider the
technology and whether its capabilities and limitations are congruent
with the learning experiences necessary to achieve the outcome.
there is not an established track record for the effectiveness dimension
of e-learning, we might examine the extent to which the available programs
and the enabling technologies rely on established models of how adults
learn. There are two dominant learning models that, consciously or not,
are employed in IP-based learning systems: Presentation and Programmed.
models range from streaming audio and video to PowerPoint programs
that have been repurposed and sent over platforms such as PlaceWare. This
is the traditional learning model, used for centuries. Sometimes called
the “information transmission” model or, more skeptically,
“the-sage-on-a-stage,” it assumes that most people can learn
the content through aural and visual means. At its worst, it is simply
a talking head, or a voice over a slide show. Frank Zvi, President of
the webcasting vendor Interwise, makes it seem very simple: “If
you’re an enterprise, human resources can use [streaming audio,
video, and data] to have the CEO talk to everyone in [broadcast] mode,
and at the same time also talk to specific groups in [seminar] mode.”
best, the speaker may be excellent and the graphics, video clips, and
other visual aids add materially to the listener’s understanding.
Presentation models have been one-way until recently, when live, interactive
videoconferencing has become available, if somewhat unreliable. Still,
there are doubts.
teachers—perhaps all human beings—are in the grip of an astonishing
delusion. We think that we can take a picture, a structure, a working
model of something constructed in our minds out of long experience and
familiarity, and by turning that model into a string of words, transplant
it whole into the mind of someone else. Perhaps once in a thousand times,
when the explanation is extraordinarily good, and the listener extraordinarily
experienced and skillful at turning wordstrings into nonverbal reality
. . . the process may work, and some real meaning may be communicated.
Most of the time, explaining does not increase understanding, and may
even lessen it.
other dominant model, programmed instruction/tutorials
is particularly popular for asynchronous learning. Now frequently referred
to as “traditional (!) CBT,” most of the courses available
on the Internet are based on this model. The developer essentially chops
the content into manageable chunks of text (perhaps augmented by audio/video
clips and graphics), and lets the trainee work through the screens at
her own pace. There are frequent questions interspersed with the instruction,
and immediate feedback. Some programs offer remediation for wrong answers,
but most simply ask the trainee to try the question again. Tutorials can
be individualized (by means of a pretest or self-inventory) but few offer
contingent tracks based on the trainee’s profile. Many of the capabilities
are entirely consistent with basic learning theory, but the content is
mostly text and is frequently criticized as boring and puerile. The IP-based
platforms did not (until very recently) build in opportunities to interact
with other learners or to ask questions of the instructor. One feature
of this model is that the instruction was often built around quantifiable
learning objectives, which were usually measured in some kind of post-test.
That doesn’t meet Kirkpatrick’s Level Four criterion, but
it’s more than most presentation models build in.
are other instructional models that have occasionally been used with IP
technologies, including what might be called the apprenticeship/coaching
model. Combined with case studies, projects, or simulations,
there is exceptional potential for learning complex competencies. Unfortunately,
they are rarely employed, presumably because of the development cost and
the fact that case studies and projects are not particularly scalable.
An excellent example of the use of the project model is Unext.com’s
Cardean University course on Promotion and Principles of Marketing. Each
unit is structured around a project, which the trainee has to complete
(e.g., preparation of a brand marketing plan), and offers readings,
data, competitive information, etc.; it encourages interaction by means
of e-mail with other students and includes video/audio clips and rapid
feedback from the course’s instructor. [A demonstration course is
available at www.cardean.edu.]
my own experience, the case study/simulation model can offer an exceptionally
rich learning experience. Working with a network security firm whose objective
was to teach network field engineers how to configure a security system,
my company designed a series of increasingly complex networks, represented
by detailed network architecture diagrams, which trainees would have to
protect from a variety of viruses, Trojan horses, denial of service (DoS)
attacks, and other hacks by means of firewalls, VPNs, Intrusion Detection
Systems, honeypots, and DMZs, appropriately placed and configured. Trainees
have access to explanations of what the various security devices are,
how they work and how to configure them for several levels of protection
appropriate to different kinds of clients. Trainees were asked to design
a security system for a client by inserting symbols into the network architecture
diagram and identifying key configuration items. If they get it right,
the DoS is foiled, viruses are kept out, and no data is compromised. If
they get it wrong, they can “watch” hackers get in and destroy
data or use the site to launch DoS attacks on other sites. Trainees were
provided with a text-based briefing of the vulnerability/nature of the
hack, IDS records of the sequence, and operations involved in the hack,
and given another chance to reconfigure the security system.
are also hybrid models in use in higher education and
corporate training that combine e-learning with classroom or lab sessions;
my experience suggests these can be particularly productive, assuming
the learning model for each part has been carefully thought through. There
is no advantage to a hybrid delivery system, however, if both e-learning
and classroom use a lecture/presentation strategy. Community colleges
have employed IP technologies to make the lecture and lab sessions more
intense and better focused by assuring that students are well prepared
for them, then using e-mail and chat to respond to questions and reinforce
the experience. Toshiba’s Telecommunications division was using
this model six years ago to cut the lab time on digital key telephone
systems by half while increasing every measure of competency, including
helpdesk calls and time to install.
we're missing the real potential of IP technologies
there can be no such thing as a generic e-learning model because the range
of potential instructional strategies and learning models is significantly,
but not entirely, dependent on the capabilities of the delivery platform.
Vendors with a repository of content that has been repurposed for the
Internet favor the Programmed instruction model; vendors with a rich media/streaming
video platform favor the presentation (sage-on-a-stage) model. Those with
collaborative tools haven’t done much yet, so no convention has
emerged (in what little I’ve seen). Most common are the electronic
page-turners that are often PowerPoint programs or texts reformatted into
HTML; they don’t give any evidence that the developer has thought
much about how people learn. How else could one create 400 courses overnight,
as some firms claim? The delivery media drives the learning model, not
the other way around.
the rich videoconferencing-plus-collaboration platforms that are emerging
(Polycom, Tandberg), we still have a chance to show how the Internet can
enhance the learning experience and not merely extend traditional models
to wider audiences. There is the potential now to develop models that
are highly suitable for a wide variety of learners and objectives, so
let us examine what is known about how adults learn.
technology to adult learning styles
consider the broad conclusions about adult learning that have emerged
in recent years. Earlier generalizations that informed much of the best
practices of CBT remain largely valid (self-paced, individualized tracks,
frequent practice, immediate reinforcement, emphasis on outcomes), but
Howard Gardner’s work, Multiple Intelligences, stimulated
a lot of rethinking and research into learning styles. Among the most
suggestive conclusions to emerge from that work are these:
have different learning styles. Only 30 percent of adults say they
learn best by listening; another 30 percent report they’d
prefer to learn by reading and reflection.
subjective difficulty of the material (i.e., for that trainee) affects
the learning style, as does gender (sometimes) and perhaps (ethnic)
culture in certain areas.
complex topics/judgment issues, people need to get comfortable,
to mess around with the topic before they can understand it; understanding
does not necessarily flow in a linear manner from breaking the task/object
into simpler component parts.
is often a gradual process that happens through a series of shaping
activities, which are not always instructor initiated. This is sometimes
called tacit learning. The coaching process recognizes this, and
so do many lab courses where we expect student skills will develop
over the semester without explicit focus on those skills.
communities work; there is a social as well as cognitive dimension
to learning. Students transform the information they get from instructors
and texts into meaningful knowledge through conversations, arguments,
lunches, discussion groups, and other real-world activities. “Bull
sessions actually do have a lot of value.”
of IP-based platforms
let’s consider the capabilities of the current and developing IP-based
& collaboration, messaging & chat systems, such as Groove
and eRoom, hold exceptional promise for individual/group tutoring, as
well as for building learning communities. They have low bandwidth and
processing requirements, but high potential for many learning tasks, both
synchronous and asynchronous. This capability enables tutorial and presentation
models, of course, but may be particularly suited to those built around
case studies and projects.
systems: Streaming audio & video (live and canned), including
WebEx and HorizonLive, bring multimedia to multiple points at low cost.
There are relatively modest bandwidth/processing requirements compared
to conferencing systems, but the communications are essentially one-way,
so, in the absence of other capabilities, this technology is locked into
a presentation model. It is widely used for both synchronous and asynchronous
systems: Live, real-time audio & video conferencing, like
Polycom and Tandberg, offers an enriched classroom experience, plus the
power of collaborative tools. High bandwidth/processing requirements and
other issues related to security means this technology is not for the
casual user. The systems are not yet robust enough to move into the mainstream,
but close. There is a significant cost savings over ISDN-based systems,
as well as considerable improvement compared to the uneven quality of
that older technology. Conferencing systems offer potential for using
a variety of learning models, but they are largely intended for synchronous
learning. An e-learning strategy with access to this capability might
choose to offer a significant amount of instruction by means of other
IP-based technologies, then periodically use multipoint video-conferencing
to, say, review a case or project, asking the team to defend it in the
face of questions from other trainees or the instructor. But I fear that
once a robust IP-based conferencing system is in place, the tendency will
be to emphasize the sage-on-a-stage learning model because it will be
cheaper and faster to develop.
those capabilities, developers have the ability to create more effective
learning experiences by creating communities of online learners who can
share experiences, questions, and tentative solutions and generally noodle
with a task until they’ve solved it. They can question the instructor,
instead of just listen to him. Technology can offer alternative and complementary
ways of approaching a topic: read, listen, observe, discuss, reflect,
construct. Simulations may be inexpensively done, supplemented by Instant
Messaging and e-mail among the trainees.
we need to use all the capabilities of e-learning technology for every
training task in the curriculum? No. Some cognitive skills can be learned
with a minimal Internet platform, although pace, practice, feedback, and
remediation are probably necessary if you are to reach an 80-80 standard
(80 percent of trainees score 80 percent or better on the post-test).
The effectiveness of the course is less dependent upon the enabling technology
than on the skill with which the developer uses the available technology
to construct learning experiences appropriate to the trainee and to the
topic. But many firms are likely to be reluctant to embrace one platform
for one set of tasks and a different one for other instruction, so the
availability of a delivery platform is likely to continue to drive the
learning model unless management is unusually sophisticated.
rich delivery platforms are available, at a fraction of the cost of just
a few years ago, but a trainee’s e-learning experiences are mired
in a technology that’s not much advanced from the teaching machines
of the early 1960s. Developers don’t seem to be aware of how people
learn, or if they are, they nevertheless continue to use mostly flawed
models of adult learning. For those vendors, that business model may be
cost-effective in the short term. Corporations are giddy about the savings
the P&L statement is showing, but the hangover will come when they
realize that costs have been saved at the expense of competencies.
technology platform is driving the instructional strategy, warping our
focus, which should be on creating an engaging learning experience that
reliably contributes to the organization’s objectives. We are going
to have to accept the fact that the cost of development of good e-learning
courses is high (should that really come as a surprise to anyone?), and
that the effectiveness of those courses has to be measured as carefully
as one measures cost savings. Only then can e-learning realize its potential.
is the outlook?
For many learning tasks that are not too complex (and especially if motivation
is high), they will be accomplished via e-learning for many trainees at
least as well, cheaper, and with more people getting more training in
a convenient manner than before. For that, we should be grateful.
more complex skills, such as designing and configuring a network security
system, we’ll have the illusion of learning because we have our
headcounts, class hours, and certificates awarded, but competencies on
the job will be marginal until experience will gradually bring up the
more highly motivated people to a level that could have been achieved
with the application of better learning models. Dropout rates for e-learning
will continue to be considerably higher than those for traditional instruction.
Educational technology has long been seen as promising, but has rarely
lived up to the promises. Not because it wasn’t effective, but because
it was cumbersome, boring, and did not adapt to the way people wanted
to learn. The e-learning industry is in danger of repeating that cycle.
L. Greenagel is Managing Director of Guided Learning Strategies; he can
be reached at firstname.lastname@example.org.