This goal of this paper is to establish an Assessment Framework for the Community College and to contextualize that framework within the concept of the Learning College. A major premise of the paper is that the assessment of student learning can generate data to support continuous improvement efforts necessary for documenting institutional effectiveness.

It must be noted that institutional effectiveness may be measured in a number of areas, such as graduation, retention, job placement, number and effectiveness of student services, management and administrative structure, physical infrastructure; while measurement in these areas is critical, it is intentionally not the focus of this paper. Rather, this paper will focus on the measurement of student learning and development throughout learning processes that take place in both the physical and virtual classroom.

Students attend community colleges for a variety of reasons—to earn a degree, earn a certificate, obtain transfer credits, develop specific skills through one or two courses—yet at the heart of each of these reasons is the desire to improve skills, increase knowledge, or change attitudes. Skills, knowledge, and attitudes may be affected through instructional programs inside physical and virtual classrooms or through student service activities outside of the classroom setting. Regardless of where the learning is taking place, measuring learning will help an institution gauge whether or not students are achieving their educational goals. If learning can be documented at the student level, then it is possible to aggregate assessment data in ways to provide information at the course, program, division, and institutional levels. In this way, assessment data gathered at the student level may be used to promote continuous improvement efforts, which over time will yield documentation for institutional effectiveness.

In some college settings, where the student population is constantly changing, assessment data is not as useful at the student level as it is at the aggregate level. In these settings, aggregate data can be used to benefit the curriculum and faculty development opportunities important to continuous improvement initiatives.

While assessment data provides beneficial information to instructors and administrators, students also benefit directly and indirectly from assessment results. Through the appropriate use of assessments, students receive direct feedback on their educational progress, helping them take ownership of and understand critical next steps in their educational plan. Students are also the beneficiaries of college, program, and course-level continuous improvement efforts supported and sustained through the systematic collection and analysis of assessment data.

In support of the measurement of student learning, the assessment framework will provide an assessment vocabulary, implementation processes, and methods for data generation and reporting. The framework does not intend to prescribe one assessment methodology for measuring student learning; instead, the framework should be viewed as a set of building blocks and best practices that can be assembled, modified, or shaped to a particular institution’s integrated approach to learning, teaching, and assessment. Similarly, the vocabulary is intended to provide a baseline of terms. Colleges should use the definitions to establish a common understanding amongst stakeholders so that, if necessary, college-specific definitions may be developed. The goal of the Assessment Framework is to provide a foundation of terms, processes, and procedures so that all stakeholders involved with the development or consumption of assessment results may operate from a common understanding.

Why an Assessment Framework for the Community College?

Community colleges are increasingly required to document institutional effectiveness to a broad group of stakeholders through assessment data. College accreditation agencies refer to a variety of assessment requirements, but the following statement excerpted from the Southern Association of Colleges and Schools’ Principles of Accreditation: Foundations for Quality Enhancement (section 3.3) highlights the importance of documenting the effectiveness of educational programs through assessment data:

“The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidences of improvement based on analysis of those results.”

While the need for assessment is clear, an implementation gap often exists between the desired end result and how one gets there. This gap prevents colleges from developing an effective assessment plan that will yield meaningful data at the student, course, program, and college levels. The breadth and width of the gap varies from institution to institution. However, the Assessment Framework intends to begin filling the assessment information gap by defining an assessment vocabulary, outlining a practical assessment implementation process, and establishing a methodology for how to use assessment data in an integrated fashion across a college campus.

But before it makes sense to talk about assessment, much less an assessment framework for the community college, the context in which assessment plays a role needs to be defined. Assessment may be thought of as a tool kit, a varied and marvelous set of devices and instruments, each honed and calibrated to tackle a very specific problem. This set of tools on its own may be impressive, but without an understanding of how the tools can be applied, it cannot be used to its potential. In addition, the tool kit without context or purpose doesn’t offer much help. It must be contextualized within the broader learning outcomes of a college before it can be applied successfully.

At first glance, the answer to the context question seems simple. The context is education, and assessments at the college level are applied for a variety of purposes: to ensure students are ready for college or their next course; to certify students for a profession or trade; to ensure courses, programs and departments meet their stated objectives; and to ensure that the institution provides a first class learning environment. But while these uses of assessment are valid, the educational context in which they are applied is changing, and the entry-exit-certification and accreditation models of assessment need to evolve, as well.

Terry O’Banion in a monograph entitled Launching a Learning-Centered College speaks about the “learning revolution” that took place in the last decade of the twentieth century, where colleges refocused mission and value statements on the learning process and transformed their institutional structures into learning-centered enterprises. Learning is the focus of the educational process. It focuses on the student and what the student needs to achieve and has achieved. It puts the student in the center and acts to realign all other college support systems— teaching, research, and support services—around the goal of helping students achieve their learning outcomes.

O’Banion named this revolutionized college the “Learning College” and has developed for it six guiding principles
(p. 5):

(Several people, including O’Banion, have mentioned, written about, or developed a seventh principle. To avoid confusion, this paper deals with only the original six guiding principles.)

When considering assessment within the context of the Learning College, the uses for assessment grow quickly beyond the entry-exit-certification model. The six principles are key, and different kinds of assessments may be used to provide evidence of progress towards and adherence to them. For example, it’s possible to define an array of student learning assessment opportunities that can support and sustain each of the six principles for the Learning College, as follows:

Continuous improvement efforts take place at the individual student level, where students use assessment data to understand academic and developmental areas needing improvement, and extend out to the much larger-scale institutional level, where baseline and trend data can be used to identify issues, root causes of issues, and pave the way for establishing benchmarks and measuring progress toward benchmarks. Regardless of the size of the continuous improvement effort, assessment data provides concrete evidence about a current state and helps students, faculty, and program administrators tailor an appropriate improvement plan. The purpose of defining an Assessment Framework for the Community College, therefore, is to provide a framework of use—instructions for when and how to use the specific tools in the kit for a given purpose. The instructions will include a vocabulary of terms, a process for implementing an assessment plan, and guidance on how to assemble assessment data into documentation of learners’ competencies and institutional effectiveness. In other words, instructions on how to use the assessment toolkit to prove to the learner (and other stakeholders) that learning is expanded and improved.

Views of the Stakeholders

An Assessment Framework is important for the range of stakeholders with interests in the performance of a college. The diagram below illustrates the variety of stakeholders associated with a community college.

This image illustrates the number of stakeholders with an interest in the performance of a college. All stakeholders have a need and right to receive and understand effectiveness indicators. Given the range of stakeholders, the institutional performance data need to be packaged and presented in a clear, concise, and precise fashion.

One of the primary goals for the Assessment Framework is to establish a vocabulary of assessment terms so that all stakeholders may easily understand the discussion and presentation of learning effectiveness data. In addition, the Assessment Framework aims to outline a process for measuring student learning that all stakeholders will be able to understand. Clearness of language and transparency of process are critical to the success of any organization and perhaps even more important in a college setting where stakeholder interests are so varied and diverse.

Focus on Student Learning as Evidence of Institutional Effectiveness

In the endeavor to document institutional effectiveness for accreditation boards and all institutional and community stakeholders, a number of effectiveness indicators come into play. Traditional measures of institutional effectiveness include rates of graduation, retention, and job placement. College accreditation boards have expanded the view of institutional effectiveness to include indicators of the quality, value, and use of student services, strength of the administrative structure, and robustness of an institution’s physical infrastructure. Many would argue that these latter kinds of indicators are necessary foundational components that contribute to higher rates of graduation, retention, and job placement.

When the idea of institutional effectiveness is contextualized within the college focused on learning as described by O’Banion’s principles of the Learning College, an even more fundamental component is added to the formula: student learning and student opinions (e.g., satisfaction). Students are the primary and central focus of the Learning College, and the strength of their opinions and quality of learning are essential for any college to be measurably successful. The diagram below illustrates the fundamental nature of the student within the equation of institutional effectiveness.

In this diagram, it’s clear that what happens at the student level is fundamental to the success of all other levels of a community college. The diagram illustrates an environment where college-level learning outcomes infuse all campus activities, both within and outside the physical and virtual classroom. The goal is to develop student services and curricula around the learning outcomes defined by the college. It is important to note that the learning outcomes will be related to academic goals as well as personal development goals such as leadership, social growth, and communication skills.

The diagram illustrates that student opinions affect the nature and composition of a college’s structure and student services offerings. If student opinions are negative, a ripple effect occurs: either the college manages change to provide better structures and services for the student or the student chooses to leave the institution or dissuade other students from attending the college.

Indices of student learning are even more powerful than student opinions. By measuring the amount and quality of student learning toward defined outcomes, it is possible to evaluate the quality of a curriculum. If an educational division is showing that its student learning is expanded and improved, then higher rates of retention, graduation, and job placement are likely to occur. The feedback process is critical. If aggregate measures indicate that student learning is not occurring, then appropriate actions need to be taken at the curriculum or departmental level. The nature of student learning and opinions will guide administrators and faculty in determining whether adjustments need to be made to learning outcomes, learning theories applied in the physical or virtual classroom, teaching styles, delivery methodologies, measurement practices, or the course content.

Another way to focus on student learning is to illustrate the fact that the learner is at the core of all curricular, assessment, and outcomes-definition activity. The diagram below submitted by the Worldwide Instructional Design System (WIDS) provides a graphical view of this concept: This diagram graphically depicts the learner as the target around which learning outcomes, assessment strategies, and learning strategies are developed. The point is to design all these components with the student in mind.

Similarly, the goal for the Assessment Framework is to focus on the student and provide best practice information on how to assess student achievement of college-level learning outcomes. By documenting progress toward and achievement of learning outcomes through assessment evidence, a large step will be taken towards documenting institutional effectiveness. In other words, colleges are designed to help students learn. If the learning process is sound, and student learning can be documented and proven, a strong argument can be made that the educational institution is effective.

Defining the Vocabulary

As evidenced by the terms already presented in this paper, a number of words specific to assessment exist and are used interchangeably by community college stakeholders. The problem though is that assessment terminology has various meanings, especially to the different groups of stakeholders. To parents, an exam may be the set of questions given at the end of a unit with the goal of informing the student and teacher about whether a review exercise is needed, while to an academic dean, an exam may be a high-stakes certification test. To prevent confusion and to promote a common understanding between and amongst stakeholders of the community college, it’s important to clearly define assessment terms.

Without a common understanding of assessment terms, it becomes very difficult to discuss assessment implementation at a college. Many colleges charter an assessment task force to map out assessment strategies at the college level. Consider how much time would be saved if all the members of the task force used the same vocabulary starting at the very first meeting. Using a common vocabulary will allow the task force to move more quickly and efficiently without spending a great deal of time defining terms or acting on unconfirmed or inaccurate assumptions.

The main point in providing the vocabulary is to provide a set of common terms. Colleges may opt to define their own assessment terminology, a language that fits more intuitively within the local structure of the college. However, the process of defining a college-specific set of terms will be greatly facilitated by starting with the common understanding of terms defined in this paper.

Assessment

In 1995 the Assessment Forum of the American Association of Higher Education led by Thomas A. Angelo went through an iterative process to develop a definition of assessment. The end-result of that definition process is as follows:

Assessment is an ongoing process aimed at understanding and improving student learning. It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards; and using the resulting information to document, explain, and improve performance. When it is embedded effectively within larger institutional systems, assessment can help us focus our collective attention, examine our assumptions, and create a shared academic culture dedicated to assuring and improving the quality of higher education (Thomas A. Angelo, AAHE Bulletin, November 1995, p.7).

This definition fits nicely within the context of O’Banion’s Learning College as well as the student-centered focus of the Assessment Framework. It also speaks to the idea of making assessment a transparent, clearly documented process so that the needs of disparate stakeholders may be met. The Questionmark white paper titled Assessments through the Learning Process also speaks to the embedded nature of assessments within learning environments and institutions.

Defining the Learning Assessment Process

In order to apply assessments effectively to determine whether student learning is expanded or improved, an assessment plan needs to be developed that incorporates assessment opportunities throughout the learning process. To be effective, assessment cannot be an afterthought or instructional add-on. It needs to be embedded, contextualized, and executed within the learning process as illustrated in the white paper Assessments through the Learning Process. This chapter will outline the process for developing and implementing an assessment plan for measuring student learning.

Following is a process to follow for the development and implementation of assessments:

1) Define measurable institutional learning outcomes. Establish outcomes at the institutional, program, major, course, and classroom level.

2) Design assessments to measure learning outcomes. Determine the outcomes to measure, determine the purpose for the assessment, determine the assessment method to employ, and determine the kind of assessment data you need to collect.

3) Design learning events based upon learning outcomes. Include assessment activities within the learning designs.

4) Deliver learning.

5) Assess learning and learning events.

6) Gather and format data generated from assessment activities.

7) Interpret the assessment data.

8) Use assessment data to make decisions at the student, classroom, course, major, program, or institutional level.

The following diagram from WIDS illustrates the assessment planning and implementation process.

Defining an Assessment Data Flow

This section will discuss the reports that can be generated from assessments of student learning and how student-level assessment data can be aggregated at the course, program, and department levels to provide evidence of institutional effectiveness.

For the assessment phases described in the previous sections, there are a number of ways in which assessment data gathered can be viewed and aggregated to support the process of documenting evidence of institutional effectiveness and to promote continuous improvement within the Learning College.

College Case Studies

Following are four case studies from community colleges that illustrate the process of developing a sound assessment plan focused on students and learning outcomes.

Overview

Blackhawk Technical College (BTC) located in Janesville, Wisconsin, was founded in the early 1900s and is one of sixteen technical colleges in the Wisconsin Technical College System (WTCS). The college serves the residents of Green and Rock Counties by providing up-to-date courses and programs in technical, basic skills, and continuing education.

The College offers 40 Associate Degree and Technical Diploma programs as well as apprenticeships, certificate programs, contracted business services, continuing education, basic skills, and personal enrichment classes. These programs provide a variety of career options and training opportunities to meet local business needs. BTC has defined core abilities at the college level and all programs have developed program-level learning outcomes. To assess student learning, BTC has developed an overall plan for assessing student learning outcomes. Key to the assessment plan is the use of an assessment management and tracking tool, and a program matrix. Recently, the College has added an electronic portfolio application to its toolbox to collect student artifacts that illustrate achievement of program learning outcomes or college core abilities.

Problem

BTC has a number of objectives that the assessment plan needs to accomplish. The college is committed to the use of portfolios for gathering and evaluating student work artifacts. Many program outcomes are written as performance-based outcomes, and gathering examples of student work is seen to be the best, most authentic approach to measuring student achievement of performance-based learning outcomes. The use of portfolios, while critical to sound measurement, can present challenges for instructors managing documents, and a successful portfolio system would need to reduce the organizational tasks associated with tracking and managing multiple work artifacts.

Second, BTC was looking for a way to provide more direct, authentic information than is currently provided in a transcript about student achievement of program-level learning outcomes to transfer institutions or businesses looking to hire BTC graduates. The portfolio system provides a vehicle for collecting work artifacts and helping instructors and students determine the best examples of work to share with potential transfer institutions and employers. However, the process for sharing work artifacts needs to be streamlined and easy to use for both BTC students and the recipients.

Third, BTC wanted the assessment plan to help instructors, students and management engage in continuous improvement activities. To facilitate this goal, any data collected within the portfolio system would need to be easily integrated with BTC’s electronic assessment data tracking system.

Finally, BTC would like to establish a reputation in the state of Wisconsin and within the technical college system as a leader in the use of innovative technologies, both for instruction and assessment. Developing this leading reputation will help draw students to the school as well as add value for the students obtaining degrees from BTC or transferring to 4-year institutions within the state.

Solution

To meet the needs defined for the assessment plan, BTC developed a specific assessment initiative for assessing student learning outcomes at the college and program levels. In late fall 2003, BTC implemented a portfolio system from Nuventive called iWebfolio, a web based tool that gives individuals an online portfolio to store and present student work products related to learning outcomes. The goal for implementing iWebfolio was to help faculty members and students review student work on learning outcomes and provide an easy way to store work artifacts electronically, which would facilitate the sharing of work with transfer institutions and potential employers.

The Teaching Learning Center (TLC) recruited three program instructors to participate in a pilot program for the use of iWebfolio. The participating programs were Early Childhood Education, Marketing, and the Legal Secretary program.

The first phase of the pilot entailed TLC staff training program faculty in the implementation and use of iWebfolio. Training included working with instructors to create a template in iWebfolio that included learning outcomes, courses associated with each outcome, and a macro-level scoring guide that could be applied to the scoring of each portfolio submission. This template created the structure in which students could submit portfolio items and allowed instructors to access the scoring rubric, evaluate portfolio items, and assign scores.

Training also included working with students on how to use the portfolio tool for submitting artifacts and determining who would have permission or rights to access the artifacts. iWebfolio could be used for submitting work artifacts related to course assignments or learning outcomes. After training, instructors in each program created the templates, and both students and instructors began using the portfolio system. Following is an overview of each of the three programs participating in the pilot portfolio project.

Early Childhood Education Program

This program had been using a paper portfolio for assessment of program outcomes and began using the iWebfolio system in late fall 2003. An electronic portfolio was appealing to the early childhood instructors for four reasons:

1) The electronic portfolio would be easier to maintain; instructors didn’t need to carry and track paper documents for grading and evaluation.

2) The electronic portfolio provided an opportunity to more easily share work artifacts with colleges that had articulation agreements in place with BTC.

3) It would be possible to integrate work artifacts and scores with the assessment tracking system.

4) It was showing the use of cutting-edge technology, an important objective for a technical college wanting to illustrate and demonstrate technical sophistication to schools with articulation agreements in place.

Marketing Program The iWebfolio pilot project was a good fit with the marketing program because of the program’s overall sophistication with software tools and because marketing courses often required students to generate media files as course assignments. The electronic portfolio system allowed marketing students to submit media files as work artifacts that instructors could evaluate. In addition, marketing students could more easily share portfolio artifacts with potential employers. The marketing program began their pilot program in spring 2004.

Legal Secretary Program

The pilot portfolio project was a good fit with the legal secretary program because faculty in the program are innovative, open to risk, and willing to take on additional work in the short-term to advance critical program objectives. One of the important goals for the legal secretary program is to provide a method for students to collect documents that can be shared with potential employers. Work examples such as findings of fact, last wills and testaments, living wills, and various legal motions are often required by employers as part of the hiring process. The legal secretary program felt the iWebfolio system could allow students to collect work artifacts and receive evaluations from instructors that would guide students in choosing the best work sample to submit to potential employers. The legal secretary program began their pilot in spring 2004.

Results

Because the pilot has only been operating for two semesters in the early childhood education program and one semester in the marketing and legal secretary programs, quantitative data has not yet been collected to guide the evaluation of the pilot. However, anecdotal information from students and faculty in all three programs indicate that the portfolio system is providing a valuable means of aligning work products with learning outcomes, collecting and evaluating work samples from students, easing the task of managing student portfolios, and providing a good method for selecting the most appropriate work samples to share with articulation institutions and potential employers.

Dallas County Community College District
Submitted by Dr. Allatia Harris

Overview

The Dallas County Community College District (DCCCD) is a multi-college district, with seven separately accredited colleges, a center for telecommunications, and a small business development institute. Dallas County is both urban and suburban, ethnically and economically diverse. The DCCCD offers a large transfer program as well as a strong workforce presence. Credit programs enroll more than 60,000 students each fall, with more than 80,000 students served through noncredit offerings annually. Even though the colleges are separately accredited, the district is funded as one entity by the state of Texas through the Texas Higher Education Coordinating Board. Students transfer freely and regularly between and among the seven colleges.

Problem

In 1997, the Texas legislature passed legislation that required colleges and universities to develop a core curriculum of not less than 42 nor more than 48 credit hours. The charge from the state was that the core curriculum should address eight broad perspectives and should integrate intellectual competencies (reading, writing, speaking, listening, critical thinking, and computer literacy) across the curriculum. The core curriculum should also address exemplary educational objectives which were determined by a state committee.

DCCCD faculty participated in a two-year course-selection process to identify courses which addressed the eight perspectives and which would prepare students adequately for transfer to a four-year institution. After open hearings on all seven campuses, revisions, further discussion, and final revision, a 48-hour core curriculum was approved in December, 1999. Then, discipline committees comprised of faculty from each of the seven colleges participated in another two-year process to determine which courses addressed which intellectual competencies and which educational objectives. By summer 2002, syllabi for all core courses reflected specific core intellectual competencies and exemplary educational objectives, as determined by district discipline committees.

A major element of the state-mandated core is the requirement to evaluate the effectiveness of the curriculum and to assess student achievement of learning outcomes. Initial plans must be filed with the state in 2004, and outcomes evidence is to be reported in 2009.

The Dallas faculty has long used a variety of instructional methods and assessment techniques in their individual classrooms. The challenge was to determine assessment methods that would be valuable in improving instruction and student achievement on a grand scale, across sections and across disciplines for the more than 45,000 students who enroll in core courses each semester.

Solution

Assessment data must be meaningful to support continuous improvement of educational programs. The decision was made to conduct college-based assessment (rather than district-wide assessment) to collect data for immediate use to improve delivery of instruction, enhance student support services, and foster faculty dialogue within and across disciplines. A district team comprised of representatives from each college was assembled to share results, best practices, and lessons learned.

Results

Student results are only now being collected for core curriculum assessment purposes, but the district team has observed positive results to accrue from the faculty-driven process of developing the assessment plans.

College plans are currently in the final stages of development. Each college has reported that systematic dialogue among faculty has been required to develop the college evaluation plan. At some locations, faculty dialogue began as discipline groups determined assessment methods for exemplary educational objectives within broad discipline areas. For example, science faculty met to discuss assessment in biology classes. The discussion led to a comparison of the basic objectives of introductory biology parallel with the objectives of introductory physics or chemistry.

Discussions are also occurring across disciplines regarding how and where in the curriculum the achievement of intellectual competencies can be assessed meaningfully. Where are the benchmarks to be established? Discussions on questions such as, "What does it mean to be computer literate in speech communication?" and "What does it mean to be computer literate in economics?" have led to new assignments across disciplines at one college. Another college is examining critical thinking, and hosting workshops to address questions such as, "How can we use a common language across disciplines when we teach critical thinking?” Interdisciplinary faculty groups are discussing the need for students to understand the connectedness of their studies, to understand that “this is critical thinking in mathematics, and this is how we approach critical thinking in the social sciences.”

College plans include maps to assess competencies across the curriculum, beginning with computer literacy-across-the-curriculum, writing-across-the-curriculum, and critical thinking- across-the-curriculum. Discipline-based assessment activities are scheduled to include a variety of assessment types including portfolio assessment, embedded questions, common assignments across sections and common scoring rubrics, departmental examinations, videotaped oral presentations, and group projects. The timeline for 2004 - 2009 reflects comprehensive assessment of competencies and exemplary educational objectives. Faculty members have aligned their efforts to address the question: “How best do we know what Maria knows and what she can do?”

Overview

Kirkwood Community College serves a seven county area in eastern Iowa. With an enrollment of more than 15,000 students, the college is the fourth largest higher education institution in the state and offers 62 liberal arts majors and more than 80 Applied Science programs. The Health Science department enrolls more than 2,000 students in 15 programs.

These Heath Science students must complete a core curriculum consisting of Heath Skills I and II, First Aid, Professionals in Health (including OSHA regulations) and OSHA Recertification before progressing to other program courses where they apply skills learned from the core content. These courses encompass 12 critical competencies, which are aligned with 53 specific learning outcomes.

Problem

Accreditation agencies want direct measures of student learning. NCA states, “Programs that have accountability as their only goal fail to provide faculty and students with information useful to the improvement of instruction and learning,” and Criterion Three wants to know how “the institution is accomplishing its education goals.” Likewise, other stakeholders such as certification boards, advisory committees, administrators and the general public seek direct evidence of student learning. Whereas grades, graduation and job placement rates are one indicator of student success, the critical question is, “How well have Health Science students mastering general health core competencies and learning outcomes?” Direct measurement of student performance and achievement is needed to answer this question, and data is also needed to help students identify areas needing more attention and to help instructors tailor learning activities to meet the needs of students. How can we get this data and what can this data tell us about improving instruction and learning? And how does this data contribute to assessing departmental and institutional effectiveness?

The Solution

An Assessment Plan. The first step in answering these critical questions is to develop an assessment plan that will identify measurable learning outcomes, provide outcome-level feedback to students, and yield performance and question data to guide instructors.

The Health core curriculum defines 12 measurable competencies: