Defining and Teaching Learning Outcomes

Sunday, June 24, 2001

Facilitator: Cynthia Wilson

Recorder: Bill Castellano

Resources: Robert Barr, Robert McCabe, and Ned Sifferlen

Learning Outcomes are statements of the knowledge, skills, and abilities the individual student possesses and can demonstrate upon completion of a learning experience or sequence of learning experiences (e.g., course, program, degree).

Processes for Defining Learning Outcomes

Collegewide Learning Outcomes

1. Core Competencies

• define core skills
• institutionalize them by incorporating into official course outline
• provide for evolution of core skills to allow for change

2. Research 

• review literature (broad environmental scan) to see what others have done
• make it your own; personalize/modify from your college  

3. Structure/Design

• course to introduce students to college outcomes 

4. Process 

• determine expectations of key external constituencies, including

§ industry

§ legislative

§ external accrediting bodies

§ national standards

§ K-12 and four-year institutions 

• determine views/expectations of key internal groups/partners

§ board

§ mission/values

§ K-12 and four-year institutions

§ faculty teams (x discipline) 

• implementation strategies

§ standardized syllabi

§ course for students to teach better consumerism  

5. Resources

• begin with broad base of stakeholders
• functional teams (internal) to facilitate, coordinate/formulate statements for review
• support teams with research

§ money; release time

§ clearinghouse/bank of ideas

§ institutional research for processing, cataloging, and dissemination

§ consultants with expertise to help others do this

Program Level Learning Outcomes

1. Definition of a program (i.e., something with specific outcomes) 

• learning experiences leading to a specific degree/certificate

• in a discipline (i.e., the math program)
• basic skills sequences
• course sequences within disciplines
• student services 

2. Process: external definitions 

• consulting with other colleges
• use advisory boards and needs assessments
• research on best practices; what other colleges are doing
• articulate with K-12, other colleges, and business and academic associations  

3. Process: internally defined 

• faculty teams are formed
• use of interdisciplinary teams
• use advisory boards and needs assessments
• creating a capstone course; (electronic) portfolios
• produce lists and definitions of competencies
• use student input to help define outcomes
• integrates with external standards
• adapts to individual circumstances of the college (i.e., a union/administration agreement about developing outcomes; resources provided to teams)
• use and move through the approval process (i.e., curriculum committees)

4. Guidelines

• Guidelines for defining task: Outcomes relate to college outcomes and to individual course outcomes
• Use common method for defining outcomes at the three levels (course, program, college)  

5. Outcome

• These processes create faculty and staff learning about their work, about how to go about doing this, and about the nature of outcomes, leading to continuous improvement of definitions and identifications, relationship to assessment means and methods.

Course Level Learning Outcomes

There are a number of processes across the institution that contribute to the identification and defining of course-level learning outcomes; for example: 

• individual faculty

• common course outline (how does syllabus reflect?)

• department defined
• core skills
• college committee
• external advisory committee  

Roles of Learning Facilitators in Student Achievement of Learning Outcomes

Observations

• Faculty has traditionally done it all: define learning outcomes, design learning experiences for student achievement of learning outcomes, and assess student achievement of learning outcomes.
• Experiential learning, web-based instruction, student development, advising, programs, and activities are examples of ways learning outcomes are generated outside the traditional classroom.

Needs

• Support for faculty in this work 

§ training in design and implementation

§ design resources (design specialists, time, incentives)

§ implementation resources (implementation specialists, time, incentives)  

• Redefinition of roles and workload

§ of faculty

§ of others

•  Consistency/evenness of support (data collection)
•  Learning outcomes embedded in institutional effectiveness

Assessing and Documenting Learning Outcomes

Monday, June 25, 2001

Facilitator: Cynthia Wilson

Recorder: Dianne Cyr

Resource: Ned Sifferlen

Agenda

· Revisiting Yesterday

· Assessment

• Identifying Challenges and Issues
• Finding Solutions
• Sharing the Learning

· Documentation

• Current Practices
• Meeting Needs
• Sharing the Learning

Revisiting Yesterday

Outline of yesterday’s discussion for review by participants for additions/amendments—stand as written for now; invitation to make changes throughout Session III prior to posting on website.

Assessment

· Why do we want or need evidence of student achievement of learning outcomes?

• to improve student learning
• to know where to invest our resources
• to plan for improvement
• to know what we need to change
• to provide accountability to external constituents
• to empower students to manage their learning
• to improve institutional effectiveness
• to become intentional about what we are doing

· How are we going to get this evidence?

• Internally (with external validation)
• Proxies like completion, retention, etc. (like CCSSE)
• Watch students
• Ask students
• Consistent measures of competency in a variety of ways
• Homegrown and standardized measures
• Intentional as opposed to haphazard
• Early and often
• Through application; authentic assessment measures
• Plan for data collection

· What are we going to do with it?

• Apply it for continuing improvement; collect data only when and if we know how we’re going to use it
• Change practice or continue successful practices
• Tell students how they’re doing
• Show stakeholders how we’re doing
• Use data in both aggregated and disaggregated formats
• Transcript in portable ways
• Continuing improvement of measures
• Replicate and share best practices
• Convergence of standards of what is important in similar programs
• Help students design their own portfolios; self knowledge and empowerment
• Relate it to other Institutional Effectiveness measures

· What do we fear will be done with it, and why does that scare us?

• Merit pay
• Tied to evaluation
• Additional workload
• Institutional worth at state level
• Standardized test
• Loss of autonomy
• That it won’t matter—so what?
• A disincentive for those working with the hardest to serve
• Learning can’t be measured
• Change—forced out of comfort zone
• Just another fad
• Tied to state funding mechanisms
• Wrong conclusions might be drawn from data
• Competition with other colleges that don’t require outcomes testing
• Assembly line education
• Lack of resources to implement
• Lack of variation in measures
• Might close the door—reverse the open door policy

· What can we do to overcome these fears?

• Guarantee risk-free assessment
• Information for faculty
• Get involved in design and development
• Provide training and professional development
• Clarify/determine use before implementation
• Front-end designing, etc.
• Provide incentives for implement
• Separate student assessment and faculty evaluation
• Build trust, an assessment culture rather than an evaluation culture
• Respect the privacy of individual
• Do a risk/benefit analysis—ask ourselves questions about the risk that we’re taking against the potential benefit to be gained
• Get all the stakeholders involved
• Leadership must provide a fair, consistent path to evaluation from the institutional level—take an honest look at ourselves
• Reality of media focus on institutional weaknesses—we must engage the media as partners—educate externally
• Empower faculty for their own improvement
• Address the technical skills needed
• Identify benefits through pilot programs—identify Vanguard faculty members
• Identify partnerships among institution, faculty, information resources, assessment/placement policies—holistic approach to assessment of student outcomes
• Balance challenge and support to facilitate change in assessment practices
• Acknowledge change in workload
• Plan, design and engage all stakeholders in the design of assessment (emulate learning-college principles in our own work toward outcomes measurement)
• Recognize assessment as learning
• Recognize the difficult of creating partnerships that move efficiently toward goal

Assessment Challenges

· Maintaining partnerships

· Lack of knowledge of methodologies and tools

· Limited resources (financial, time, personnel)

· Difficulty of getting buy-in

· Effective communication at all levels

· Assessing the assessment

· How does assessment compete with other institutional and workload priorities?

· Viewing assessment as integral to our mission as educators at every level and not separate

· Overcoming our fears

· Creating incentives and recognition for assessment efforts

Kay McClenney’s observations about why the work of learning outcomes is so difficult:*

· lack of collaboration among disciplines and other groups within the institution

· lack of knowledge about assessment processes and tools

· lack of awareness of the need for outcomes-based education

· lack of appropriate, effective assessment tools and models

· a perception that some important learning outcomes are not measurable

· traditional insulation from accountability for individual student learning at the classroom level

· traditional resistance to self-assessment in higher education

· traditional external requirements for accountability, funding, and policy that are rarely tied to individual student learning, leading to a lack of incentive for outcomes-based efforts

· increasing demands and constricting resources, which leave little time or incentive for educational reform efforts of this magnitude

*from an address to participants in the League’s 21^st Century Learning Outcomes Project, March 4, 2001, Atlanta, GA

Group Process: Identify strategies you know will work and strategies you think or believe will work.

Partnerships

· Interdisciplinary Outcomes Teams

• Palomar Assessment Learning Project; several faculty teams created an assessment pilot and implemented an assessment process
• Put the project on their Website for faculty review and possible implementation
• Disseminated results at Palomar by presenting in department meetings, at faculty orientations, in professional development workshops, to administrators and deans
• Provided handouts on benchmarks and asked for feedback
• Stipends funded initial project; some informal adaptations occurred without stipends

· Connect and communicate with the movers and shakers and keep them informed of progress and results

· Communicate well—get all stakeholders involved (faculty, board members, students, media). Involve as many people as possible on campus and in feeder schools

• Cascadia’s learning outcomes teams focus on integrating and assessing core values and involve all areas of the college—initiatives are reported through all-college network

· Emphasis on communication, implementation and shared ownership

Resources

· The League serves as a clearinghouse for information, creating a resource database

• Ask the right questions
• Collect relevant information
• Organize outcomes into useful reports
• Use information for reporting and evaluation
• Focus on faculty and others as users of information to make decisions about teaching and learning

· The League provides a Database/Evaluation Consultant to work with each of the Vanguard Learning Project colleges to identify common themes and data collection and to work with a college point person for customized data to create a hybrid instrument that can serve multiple purposes

· League colleges make a commitment to share information on assessment projects

• Made public and electronic
• Emphasis on assessment included in new faculty orientation
• Follow-up professional development emphasizes assessment

Needs

· support from across institution

· Palomar three-year project stealth approach not as successful. Needs to be a celebration of risking assessment.

· People need to be able to vocalize fears as part of moving through; goes along with fear of change, going out of comfort zone.

· Facilitates departmental discussions (Moraine) Needs to be across functionality.

· Lane has contract that calls out the parameters for undertaking assessment.

· Madison is trying to design email to college community celebrating success.

· Kirkwood has efforts to communicate best practices.

· Departmental review (academic and non-academic) at Moraine Valley

· Using student/faculty satisfaction measures to improve—Kirkwood multilevel assessment of current practice (levels and stages). Can target areas that need assistance and provide needed resources.

· Need to be able to find $ resources and identify untapped internal resources to assist in this effort.

· Kirkwood—train-the-trainer; brought in national experts to work on this effort (Barb Wolvaard, Notre Dame).

· Valencia—funds Title III and Title V; about 75 involved. Has gone through models of assessment; now working from outcomes backward. A resource library on assessment has been developed, 7 assessment tools have been created to address competencies. Destination 2001 is the name of the project.

· Hiring practices need to reflect the learning college culture. Valencia is moving forward on this—faculty serve on every board.

· Learning by Design is a two-semester program that measures inclusive learning and critical thinking. Hypotheses are developed about what students will need to do to demonstrate higher learning outcomes, strategies are applied and outcomes are measured again resulting in higher grades and retention rates. Used internal faculty to train others.

· Buddy system is used to call students who are not attending; also work as study groups.

· Sinclair—NSF architecture of courses requires teamwork, demonstrated transference of information, assessment and external evaluation.

· Involve adjunct faculty in assessment projects; it is like a grow-your-own for developing new full-time faculty.

· Use Classroom Assessment Techniques, Cross & Angelo, for potential assessment tools.

· Kirkwood—Work with Wolvaard allowed for common rubrics among disciplines and allowed for broader/fuller collection of data.

· Category of Awards and Recognition—Award for most improved programs (most improved in terms of learning outcomes; not measuring learning outcomes).

· Kings College model (Don Farmer-author); course-embedded assessments for core competencies—not overburdensome to faculty at discipline level. Faculty read other students’ exams (even if multiple choice final) and note items that demonstrate a core competency.

· Moraine Valley—knows good tool for measuring critical thinking—developing rubrics for beginning, developing and accomplished. Teach students how to self assess/validate their lifelong learning (how do students know they know rather than thinking they know?).

· Have students put examples of their work in portfolios—will work but need to tighten up benchmarks (instructor assessment, self-assessment and demonstrating improvement over time).

· Johnson County CC—interdisciplinary work to come up with rubrics for core competencies, collect artifacts from randomly selected classes—Institutional Research collects assignments and organizes them for reading and evaluation in cross-sectional groups working together.

· Budding partnerships—Interdisciplinary Learning Outcomes Team—Faculty/Student Services/etc. All full-time employees are expected to be part of one team (Learning Outcomes, Critical Thinking, Communicating, Active Learning, Interacting in Diverse Environments).

· Competencies/Values—Core Values need to be the same to develop viable strategies.

Documentation

In documenting individual student learning, what do users need and want to know?

· Depends upon the audience

· Digital/electronic portfolio

· Industry certifications that show competencies

· Log of experiences in internship or clinical experiences

· Checklist of competencies within a discipline to accompany the transcript and provide a record of demonstration of individual competencies with level of accomplishment

· Partnership opportunities with colleges and universities

· Cocurricular transcripts

· Documentation of service learning experiences and essential skills as measured by Work Keys

· Alternative assessments such as portfolio, special projects, student self-assessment, education plan, and faculty critiques