Rationale for AIER
- The AIER merges departmental assessment, end-of-the-year reporting and planning into one document with a clear focus on mission and improvement of programs and services.
- The AIER is a web-based system providing relevant operational and survey data to report writers.
- The AIER process ties into university-wide planning and budgeting processes in Strategic Planning Council and Cabinet.
- The AIER provides a process that meets the SACS Core Requirement 2.5 for Institutional Effectiveness: “The institution engages in
ongoing, integrated, and institution-wide research-based planning and evaluation processes
- incorporate a systematic review of institutional mission, goals, and outcomes;
- in continuing improvement in institutional quality; and
- demonstrate the institution is effectively accomplishing its mission.”
The vice-presidents will review the departmental reports to create a divisional summary. In addition to looking for common themes and items that require divisional or university-wide support, the VPs will provide feedback to each department using this rubric.
You may want to review the Technology Introduction page before you start working with the AIER system.
Report Authors vs. Authorized Report Writers
Fields at the top of the start tab allow you to change who is authorized to log in and edit the report, but not all of those people may be active in writing the report in any given year. The Report Author(s) field is used to indicate who actually worked on the report this year.
Mission and Distinctives
The mission or purpose statement answers the question “What does my department do?” It should be broad enough to cover all of the important aspects of the department’s mission, yet focused enough to clearly be identified with the department.
The distinctives section of the AIER should clarify how the program connects and contributes to EMU’s vision and mission. It may help to ask, “How is this program different because it’s at EMU?” Several areas you may want to consider include:
- Faith: How does your program connect to EMU’s focus on faith development, the Mennonite Church or the Anabaptist heritage?
- Cross-cultural experience: How does your program support EMU’s emphasis on valuing and learning from differences in culture?
- Community: How does your program reflect EMU’s commitment to community and service?
- Justice, non-violence and peacebuilding: How do these values affect how we interact or what we do with students?
Attachments and Publishing
Functions at the bottom of the start tab allow you to attach a file and to publish the report when it is complete. The preferred format for the attachment is PDF. If you have multiple files, you can save each of them to PDF format and then merge the PDF files. Contact the HelpDesk or Institutional Research staff if you have questions about the process.
Highlights and Progress
Highlights and Other Significant Factors
This section provides departments an opportunity to summarize the significant achievements of the department, external factors affecting the department, changes in staffing or resource needs, significant changes in demand or focus, and their impact on the overall productivity of the department. You may want to come back to this section once you have completed the rest of the report.
Progress from Previous Year
The plan for improvement you submitted last year will appear in this section for you. Please comment on your efforts at improvement over the previous academic year. Be specific about what strategies you implemented, how you measured your success, and whether you met the targets you set for improvement.
Learning and Operational Outcomes
The two outcomes sections of the report have the same basic structure:
- A list of Outcome statements,
- Multiple Measures and Criteria or Benchmarks for each outcome, and
- A statement of Results and Analysis for each measure.
The outcome sections of the AIER include all intended outcomes important to the department mission. Learning outcomes refer to the specific knowledge, skills or developmental attributes that students develop through their college experience. Some administrative departments will not have learning outcomes. Operational outcomes define the desired impact of the department’s services on various internal and external audiences. All departments will have operational outcomes and we have a few suggestions for academic programs.
- The comprehensive list of outcomes should cover all aspects of the department’s work. This list of outcomes won’t change much over time — unless the focus of the program changes.
- Outcomes should focus on the results of processes or programs as opposed to the process itself. Student learning outcomes should be stated in terms of students’ knowledge, values and beliefs, and abilities or skills. Operational outcomes might be stated in terms of customer satisfaction, accuracy, efficiency, timeliness, etc.
- Outcome statements should be free of methods of measurement and broad enough that they can be measured several ways.
Customer satisfaction (students, alumni, employees, donors, etc.) provides an important perspective on program quality. In previous years, the satisfaction data collected through national and campus-wide surveys was included in the Quality section of the report. However, recognizing that departments should state and prioritize their own operational outcomes, the survey data section of the AIER has been removed. Please continue to report on the operational outcomes that are important to your department. Survey data that may serve as relevant measures for your outcomes are available on the IR website.
View the guide to survey data for help in interpreting the data. Contact the institutional research staff if you require further analysis.
A set of measures for each outcome should collectively cover all important aspects of the outcome from different points of view. Include both direct and indirect measures when possible. Measures may change more frequently than outcomes, but should not change each assessment cycle. The AIER process should include an evaluation of the quality and effectiveness of assessment instruments.
For each measure:
- Clearly indicate what data will be collected and the method used to collect it,
- include realistic criteria or benchmarks based on comparison and/or longitudinal data, and
- Specify the assessment cycle to be used for data collection and analysis (i.e. every 3 years, each semester, etc.).
Criteria or Benchmarks
Each measure should include a benchmark or other criteria used to interpret the results of that measure. A benchmark is:
- Used to describe either what is acceptable, expected, or desired; not some future unachievable ideal;
- Generally a specific, constant value or expectation, and not stated as a mandated increase each year,
- May involve a comparison to some other EMU or external group, but only if the other group is very similar in all important ways.
It may be helpful to think of benchmarking as similar to stating a hypothesis. A benchmark may be based on past performance, some agreed-upon standard of performance, or the performance level of some other group. Most importantly, the benchmark should not be set or changed arbitrarily.
Finding appropriate comparison groups as an aid in setting benchmarks is one of the most difficult challenges of assessment. The following resources may be of help:
- IPEDS Peer Analysis System
- Professional associations such as:
- American Mathematical Society
- Modern Language Association
- American Association of Colleges of Nursing
- American Association of Colleges for Teacher Education
- National Association of Student Personnel Administrators
- American Association of Collegiate Registrars and Admissions Officers
Note: These are just a few examples. Check out professional associations in your field!
Frequency of Assessment
How frequently should you collect and analyze data for a particular measure? It depends on the nature of the outcome, the type of measure, and many other factors. Here are a few points to consider:
- Data that are readily available (grades on assignments, evaluations of senior projects, number of work orders completed, etc.) can be collected every year.
- Data from surveys should generally not be collected every year unless each year targets a specific group (i.e. graduating seniors, new employees, etc.).
- Even when data are collected annually, analysis of the results can be performed less frequently.
- For small samples, a full analysis every three to five years may be more useful than a limited analysis every year.
- Do not choose a schedule which surveys the same group of people annually.
Results and Analysis
For each measure, you will collect data and evaluate that data to determine if you are satisfying the corresponding outcome.
- Clearly state the results of your data collection activity. For example:
- Include sample sizes and descriptive statistics (averages and/or frequencies);
- Include tables and graphs if they provide clarity;
- Explain missing data or any data collection anomalies;
- Provide reliability and validity information about assessment instruments whenever possible.
- Indicate to what extent your benchmarks or criteria are met.
- Analyze the results of your assessment. For example:
- What are the trends for this outcome?
- What factors contributed to your department’s success or failure to meet the criteria?
- What specific changes or improvements are needed?
- Are more data, or different kinds of data, necessary before you can identify what should be changed?
As noted above, it is perfectly reasonable to collect data on one schedule (e.g. yearly) and perform a complete analysis of the results less frequently (e.g. every 3 years).
The Resources section of the AIER gives you an opportunity to discuss the adequacy of the resources you have available to accomplish the work of your department. If the work of your department has changed or expanded significantly, discuss those changes and any adjustments you have made. If additional resources are required, explain what these resources might be, and how you would use them to better accomplish the mission of your department.
Specific sections within the Resources area include: faculty, staff, facilities, equipment, and library. There is also a space labeled “Other” in which to address resources not covered in the previous sections.
The Demand section asks you to examine the demand for your department’s services. The specific fields vary by type of program.
You’ll want to look at demand generated by both internal clients (students, faculty, and other staff, trustees), as well as by external clients (governing bodies, alumni, community groups and other clients). Discuss any changes in demand and your prediction for growth in demand for specific types of services.
Enrolled Students (academic programs)
The demand for academic programs is measured by the number of students enrolled in the program. You can copy in last year’s analysis of enrollment trends and update it based on new enrollment values and then note any new or changed factors affecting enrollment. You should also determine if the ideal program enrollment values are still set appropriately.
Note that undergraduate departments offer courses specifically for students majoring or minoring in their programs, but also offer courses in support of our general education curriculum and programs in other departments. Space is provided to comment on both types of enrollment.
In the cost section we are looking at the costs associated with the services provided by the department.
Faculty Workload: Student-to-Faculty Ratio (academic programs)
The principal driver of costs for academic departments is faculty salaries. The student-to-faculty ratio provides a method to quantify the amount of faculty work it takes to provide programs on a per student basis.
The “Summary data” link provides a table with the student FTE, faculty FTE, and the student-to-faculty ratios. The “Course Enrollment Detail” link includes data on enrollment in each course offered by your department over the last year, and “Faculty Load Detail” is a summary of the faculty loading for your department. See the footnotes for each table for an explanation of how faculty FTE and student FTE are calculated.
Program Cost per Student (Seminary, graduate, and ADCP programs)
The student FTE and the total program cost are used to calculate the cost per full-time equivalent student. The data for this cost measure are found on the Financial Summary-click the link to the Summary data beside the Program costs.
The Financial Summary is divided into four tables. The top table, Student and Faculty Data, includes the student credit hours and student FTE data. Please note that the data do include summer courses. The other three tables of the Financial Summary show cost, revenue and (income) loss data for each of the last three years. Please note that numbers shown in parentheses are positive values.
The rows of the tables are identified with line numbers shown in the right-hand column; if you scroll down below the tables, you will find footnotes explaining how the amounts for each line are calculated. The second table, Summary Data for AIER Reports, includes the cost per student FTE : Line 1 is the revenue per student. Line 2 is the cost per student, the second measure in the AIER cost section. Line 3 provides you the (income) or loss per student FTE.
Program Contribution to Overhead (Seminary, graduate, and ADCP programs)
The third cost measure subtracts the direct expenses of the program from the program income, and calculates the dollars available to cover overhead expenses. This remainder, expressed as a percent of total program income, is referred to as the percent contribution to overhead (line 4). The “Summary Data for AIER Reports” is based on data from the operating budget, and does not include any revenue or cost from your restricted accounts.
In this section, you are asked to discuss the factors that contributed to your program’s success or failure to meet your contribution to overhead target. The trend data should be useful in helping you to see what is happening to income and expenses over time.
The Operating Account Costs and Revenue Data table may help you understand the trends you see in the Summary Data for AIER Reports table. The Campus Web Financial Data (lines 5-8), summarizes the activity in your program’s operating accounts and should match up with the total revenue and cost for your accounts on Campus Web. You can look up the detail for 2005-06 for each of your accounts on the Campus Web.
The Overall Program Financial Data section is designed primarily to incorporate allocated tuition net revenue for the undergraduate departments. The section proceeds through a series of steps to calculate an overall program (income) loss. Note line 13 does give you a total for program revenue.
Data on restricted accounts is in the bottom table and includes contributions, endowment income, grant income and costs for some program activities. Because programs use restricted accounts in different ways, you will need to include a description of the activity in the restricted accounts and their impact on the finances of your program.
Other Departmental Costs (administrative programs)
In the cost effectiveness section we are looking at the amount of work produced by the department, and the costs associated with the resources required to accomplish that work. The principal driver of direct costs is salaries, so productivity is often expressed in units per staff FTE and will vary by department. Measures of costs are often expressed in costs per unit of service. Examples for both types of measures are given in the pink explanation box. If you have difficulty determining the staff FTE for your department, contact the Human Resources Office.
General questions about how to measure productivity or costs can be discussed with your supervisor. Professional associations can also be good sources of information on standard ways of measuring cost effectiveness and productivity. Longitudinal data as well as peer comparisons from survey data can be very useful in putting your department’s productivity and cost effectiveness numbers into perspective. Defining who your peers are can be a fairly complex process. The IR Office has done some work on defining our peers that might be helpful to you, so let us know if you have questions. Click on the Financial Summary Report link for assistance in evaluating the trend in departmental costs.
The Financial Summary includes two tables: one summarizing activity in accounts within the operating budget, and the other summarizing revenue and cost in the restricted accounts. Three years of data are included in the summary. Each row in the tables is identified with a line number shown in the right-hand column; if you scroll down below the tables, you will find footnotes explaining how the amounts for each line are calculated. The Campus Web Financial Data section summarizes the activity in your department’s operating accounts and should match up with the total revenue and cost for your accounts on Campus Web. You can look up the detail for 2005-06 for each of your accounts on the Campus Web. The department (income) loss figure is the bottom line, taking into account all revenues and costs. If your bottom line is a positive number, it will be in parentheses. In your report you will want to note changes over the three years in lines 5-8 and, where possible, identify the reasons for those changes.
The Overall Program Financial Data section proceeds through a series of steps to calculate an overall program (income) loss. The main purpose of this section is to bring the allocated tuition net revenue into the calculations for the undergraduate departments. It gives you a total department revenue line that adds any tuition or fees your department receives to the other department revenue, and program cost is added to give the overall program (income) loss (line 15).
Data on restricted accounts is in the bottom table and includes contributions, endowment and grant income and costs for some program activities. Because the programs use restricted accounts in different ways, you will need to include an analysis of the effect of activity in the restricted accounts and its impact on the finances of your department. More information on the revenue and cost included in each of your restricted accounts is available in the Financial Detail Report. The link to this report is in the pink explanation box on the Cost Effectiveness page in the AIER form.
Improvement Plan for Next Year
The focus of the improvement plan is on using the results of your assessment activities and analysis of other data to improve your programs and the services you provide.
Improvement Plan action items should:
- Be based on analysis in previous sections of the reports, particularly the learning or operational outcomes,
- Clearly state the desired impact to student learning, customer satisfaction, departmental operations, etc.,
- Identify specific activities that will be undertaken over the next year or years,
- Include specific, measurable targets to assess the impact of those activities, and
- Strongly support accomplishing the departmental purpose and mission.
Multi-year action items can result when there is a need for further information and analysis before deciding on a specific course of action. For any long-term activity, ensure that a series of targets are established so that you can be certain progress is being made each year.
You may have action items that are not the direct result of outcome measures. When this happens please provide a brief description of how you identified the need for the action item.
Next year you will be asked to review progress in implementing this plan in the “Progress from Previous Year” section of the AIER.
If you have additional questions or have problems with the AIER system, please contact Institutional Research.