You can count me among the skeptics of much of what is written about Generation Y, (or Millennials, or as I prefer to call them, the Digital Net-Gennials). I decided to take a look at our most recent PSOL data to see if we can see any evidence of any of the claims made.

One of the traits that is claimed by the “experts” about the Digital Net-Gens is that they are team-oriented. This is often promoted as meaning that they prefer to play in groups and work in groups. There is one statement on the PSOL that appears to get at this type of information.

#8: Student-to-student collaborations are valuable to me.

Of the 36 items on the PSOL that are ranked for both importance and satisfaction, this one always comes out as #36, at least in importance. It also has the distinction of usually getting a higher score for satisfaction than for importance. That is very odd for survey questions of this type. Roughly translated (IMO), they are saying some like this: “I really don’t care very much about this item, but you’re doing a pretty good job with it.”

The chart above shows something that I would not have predicted. The students 24 years old and younger (roughly making up the Digital Net Gens) did in fact score the importance of this item as significantly higher than did those students who are 25 & over. NOTE: 296 students in the PSOL survey indicated their age as 24 and under while 283 students indicated age ranges of 25 and over, so the total population was almost evenly split.

The difference of .34 in importance (5.34 for the younger and 5.00 for the older) is clearly a significant difference in importance. So, even though the importance of group work (student collaborations) rates low overall for our online students, the Digital Net Gens do seem to value group work more than the older students. Go figure. I guess every once in a while some of this junk has to work out this way.

Or does it? If I make my own counterpoint to the previous point, there could easily be other factors at play here that are related to age differences, but not necessarily generational trait differences, if you follow my drift. In other words, although the younger students expressed a different level of importance for this item, it might not be because they have different generational traits (Digital Net Gens versus the Gen X-ers and Boomers), but just because older people have different life constraints.

Here’s what I’m getting at. Some other demographic differences between these groups besides the age difference include:

• Primarily on-line student or primarily on-ground student?
  • 24 & under: 49% primarily on-line enrollments
  • 25 & older: 69% primarily on-line enrollments
• Full-time work status, or part-time, or not employed?
  • 24 & under: 27% work full-time
  • 25 & older: 52% work full-time
• Residence: Own home, renter, live with relatives, etc.?
  • 24 & under: 12% own home
  • 25 & older: 61% own home
• Marital status: Married, single, or other?
  • 24 & under: 9% married
  • 25 & older: 54% married
• Parental status?
  • 24 & under: 13% have kids
  • 25 & older: 60% have kids

So, let’s add that up. Twice as many of the older group work full-time, five times as many own a home (with all the responsibilities and time suckers that come with that), six times as many are married (with all the time suckers that are implied therein), and over four times as many have kids (you know the drill).

What a surprise that the older group isn’t as crazy about group work (defined currently as student-to-student collaborations) as the younger group. When exactly, do they have the time to fit this into their schedules? And how exactly does that make the generations different from one another?

For the 2009 PSOL survey I have been analyzing the results of our two largest groups who completed the survey; the students pursuing the Associate in Arts degree primarily online and those students pursuing the same degree primarily (but not completely) through on-ground courses.

Out of the 579 students who submitted the survey, 148 indicated that they are completing online A.A. degree while 131 indicated that they are completing the on-ground A.A. degree with an online course or two or three mixed into their schedules.

I’ve previously written about how I don’t find the summary questions to be completely useful, at least not the first one (what WERE their expectations?), but I do think that when you see a very large difference in the ratings that you do need to pay attention to the results. As shown in the slides above, the online A.A. students are clearly far more satisfied than the on-ground A.A. students.

As you can imagine, this is the type of data that almost no one on campus wants to hear about. The only way that this data could be generated is through suspect surveying methods or misleading questions. No way that students think online is better. NO WAY!!

Listed below are the 12 largest gaps out of the 36 items in the PSOL for Lake Superior College in spring 2009. The difference between importance and satisfaction as rated by the students is known as the gap, which represents the amount of improvement needed in your satisfaction score to bring it up to equal the importance score. Gap analysis is a way of concentrating your improvement efforts in the right areas to maximize value for students.

CC photo by Joe Shlabotnik

Many schools have gaps above 1.0, but we typically (in 5 years of surveying) have not had any gaps that large. Still, it is important for us to focus on a few areas each year for targeted improvement. In order to achieve an improvement in the survey scores over time, a couple of factors need to be present:

• the item needs to be something that you have some control over or can influence where that control is held
• the item needs to be controllable in the short-term (if you want to see reportable improvements in the next year or two)
• the item needs to be something where the cost of achieving the improvement is less than the benefits to be received, if possible

For example, item #9 is the statement that “Adequate financial aid is available.” Although we might be able to add another scholarship or two to those already offered, we really don’t have much ability to make a difference in the amount of financial aid available to students. So, although it’s nice to know what they think about the availability of financial aid, there’s little we can do about it. That is not a gap that we (as an institution) can do much about. However, the Feds and the State of Minnesota sure could make a difference there if they wanted to help educate the populace.

On the other hand, there are five items in this top 12 list which could be targeted for improvement with a major campaign to help faculty improve these items in their courses:

• #11 – Student assignments are clearly defined in the syllabus.
• #30 – Interactions I have with online instructors are useful to me in the learning process.
• #32 – Layout of courses, as designed by instructors, is easy to navigate and understand.
• #33 – Instructions to students about how to meet the course learning objectives are adequate and clearly written.
• #35 – Instructional materials have sufficient depth in content to learn the subject.

Our online faculty peer review process absolutely makes a difference in items 32, 33, and 35 shown above. In fact, I shudder to think how large the gap might be if we didn’t have that process in place and if we hadn’t made good progress over the past several years. Still, not all faculty participate in the voluntary process, and not all courses have been reviewed. Therefore, we have the opportunity to encourage all online faculty to consider spending some time to beef up these areas, and of course we can provide some professional development opportunities to help them do so. By doing so, we should also help make a difference in the largest gap on the list, item #20.

Although only a slight improvement this year, this continues the year-over-year improvement that can be seen by looking at the average satisfaction scores for the 26 basic items in the PSOL.

For the Noel-Levitz PSOL, there are spaces for 10 customized questions in addition to the 26 standard questions (statements, actually) where students indicate both their levels of importance and satisfaction. Since we give the survey in conjunction with many other MnSCU schools we all use five common statements that get at some of the data that they need to collect for the MnOnline system as a whole. That leaves five more survey slots for each college or university to ask whatever they want.

This year I made an effort to find five statements that the students would rank with high importance scores. I don’t fall into the camp that says that you need to keep asking the same questions each year for the sake of continuity. If a statement didn’t pan out with a high enough importance score in previous years, I want to try a new one in hopes of hitting those things that are most important to students.

I sent out a call to other MnSCU schools to try to find out which statements they had used in previous years that had the highest importance scores for students. I set up a wiki for people at other schools to share information about the added statements that they have used in previous years. Some good information was gathered through the wiki, but at the last second I had a bit of an epiphany – yet one that I wasn’t sure would bear fruit.

For five years now we’ve had an online course design peer review process, modeled after Quality Matters of MarylandOnline. It occurred to me that there would probably be some items in the course review rubric (sample completed rubric PDF) that would make worthy survey statements. BINGO! I picked five statements from different sections of the rubric and added them to the PSOL, statements 32-36. I was very pleased to see these five items come back with very high importance scores. In fact, the five LSC added items were all ranked in the top 10 (out of 36) for importance. Our five added statements are highlighted in red below.

In addition to the five highlighted in red above, statements #28 and #31 are part of the 5 added statements for MnOnline. Therefore, only 3 of the top 10 items come from the 26 standard statements on the basic PSOL.

This seems particularly reaffirming to me. One – it indicates that many of the items identified on the course design quality rubric are not just important to teachers, but they’re important to students as well (that’s not always the case). Two – because the satisfaction scores on those items are also pretty decent (all in the top 50% for satisfaction ranking), it also appears that our peer review process is making a difference that is recognized by students. Three – it’s always nice when you find out that what you thought should be important actually is important. Overall, I think all of us can be very proud of these survey results.

For the record, the five highlighted statements shown above correlate to the following standards on the LSC course design rubric:  I.2, II.2, III.2, IV.1, and V.3.

This year for the Noel-Levitz PSOL I tried a different approach with survey incentives. Every student who completed the student satisfaction survey received a 2GB flash drive for their trouble. Since this is the fifth time we have used the survey, I have some pretty good baselines to compare results with, and partly I was interested in what a difference the more liberal incentives policy would make. Last year we gave out 40 flash drives in a random drawing and saw an increase in submissions from prior years. This year we went another step down that road.

We received 121 more responses than the previous high number of returns from 2008. The percentage response rate only went up from 23% to 25%, which is a pretty modest increase. However, that increase of only 2 percentage points resulted in an increase of 24% more responses (121 more with a 2008 base of 458 responses) since our online student population was also a couple hundred students higher in 2009 than in 2008.

Basically, we spent about $3,000 in incentives in order to get significantly more information. You’ll see in the upcoming posts that the satisfaction data collected this year is very similar to the data in all the other years. One thing I was interested in seeing is whether the added incentives might change the results – would more students answer the survey without really putting out the effort to answer sincerely, for example? Apparently not (at least in my opinion).

This will be cross-posted at my e-Learning blog. I have been analyzing some of the data (again) from the 2008 PSOL survey. This is the fourth year that we have used this Noel-Levitz survey at Lake Superior College. The embedded slides explain a bit more about the survey, including the four sets of data that are compared for online student ratings of both importance and satisfaction.

[slideshare id=834231&doc=2008psoldatacharts-1228861953635614-9&w=425]

(NOTE: visit SlideShare to use the full screen option.)

There are 26 items that are included in all PSOL submissions. You can add other items but only the first 26 can be compared across other populations since these are the only items answered by all students. In order of descending importance, here are the top eleven items for LSC students on the 2008 PSOL (survey item number is indicated at beginning of each line).

1. (20) The quality of online instruction is excellent.
2. (25) Faculty are responsive to student needs.
3. (11) Student assignments are clearly defined in the syllabus.
4. (18) Registration for online courses is convenient.
5. (07) Program requirements are clear and reasonable.
6. (06) Tuition paid is a worthwhile investment.
7. (12) There are sufficient offerings within my program of study.
8. (23) Billing and payment procedures are convenient for me.
9. (04) Faculty provide timely feedback about student progress.
10. (03) Instructional materials are appropriate for program content.
11. (10) This institution responds quickly when I request information.

There are clearly other items that are very important to online learners but are not included in the 26 PSOL items. Please leave a comment if you have some ideas about what they might be. Thanks.

These slides show the comparisons between our 2008 PSOL results and the 2008 SSI (student Satisfaction Inventory) results. There are eleven questions that match up between the two surveys, including some that we added for that very purpose.

[slideshare id=591501&doc=psolssi2008charts-1221054827135496-8&w=425]

To clarify: here are the questions that were compared from the two groups.

The basic PSOL has used the same 26 questions during all four of the survey administrations that we have conducted here at LSC. I decided to make a few simple calculations that are not normally made by Noel-Levitz.

One thing that they do with the data is aggregate them into five separate categories, such as Enrollment Services, Academic Services, etc. It seemed to me that you could also get an overall feel for student satisfaction by calculating the average satisfaction score of all 26 items. This simple average seems somewhat relevant to me as a measure of overall student satisfaction, and is probably more relevant than a couple of the PSOL overall satisfaction survey questions – such as whether their expectations have been met, exceeded, or not.

The chart above (click to enlarge) indicates the average scores over the four survey periods for students of LSC Online. I think it’s instructive to see that the average satisfaction score has increased each time on a year-over-year comparison. I realize that there is still an issue about whether the rights things (most important ones) are increasing in satisfaction rating, but I do think that this is still something worth paying attention to.

This is cross-posted from my e-learning blog: Desire2Blog

The chart below shows the results over the past three years to the following statement:

The online course delivery platform (Desire2Learn or D2L) is reliable. (click photo to enlarge)

The PSOL is the main instrument that we use to gather information from students about the online programs and services that we provide. In two of the last three years, reliability of the VLE platform (we all use Desire2Learn) has been rated as the most important factor out of the 30 (31 this year) questions asked of all students. The satisfaction rating (6.01 in 2008), is also one of the highest scoring. This year it comes in with the 2nd highest satisfaction rating out of 31 statements, with first place going to “Registration for online courses is convenient.” (rating of 6.21)

I realize that the reliability factor does not capture all of the pertinent information about a VLE, but clearly it is an important one. Credit for the high student ratings goes both to Desire2Learn for the product development as well as to the MnSCU Office of the Chancellor staff who actually host and troubleshoot the service for our several hundred thousand user account holders.

Congratulations are in order for these high marks related to student satisfaction.

NOTE: the survey uses a 7 point scale where 6.0 is satisfied, 7.0 is very satisfied, 5.0 is somewhat satisfied, and 4.0 is neutral. The other 29 items rated below the D2L item ranged in satisfaction scores from 5.96 to 5.12.

The results from the PSOL were waiting for me in my inbox when I returned from vacation on Sunday. I haven’t been able to do a full analysis just yet, but the preliminary look is very encouraging. I will be making several posts to this blog over the next couple of months detailing what we’ve learned from our online students from this fourth administration of the Noel-Levitz survey.

In broad strokes, of the 26 items on the standard PSOL, our students were more satisfied than the national average on 21 items, and less satisfied on only 5. Even better, the mean differences for those five items were not statistically significant from the national average. Better still, ten of the positive differences were statistically significant when compared to the national average. Here are the ten items: (click to enlarge)

The asterisks system works as follows: 3 stars (***) indicates that the difference is significant at the .001 level. 2 stars (**) is a significance level of .01, and one star (*) is the .05 level.

We also have comparison data from the 2006 survey and the 2008 survey and I will be asking for a peer group report as soon as I get a chance. Once again the aggregate Minnesota Online data is not very pretty, but I’ll be posting some info soon about how our results compare with our consortium results.

Even though this is my first post on the 2008 results, I should probably repeat that I don’t actually put much stock in comparisons to the national group data since the demographics of that big group are not very comparable to our students. However, it is one of many comparisons that I make in order to see the whole picture.

We expect to receive our 2008 PSOL results any day now. This will be our fourth year of using the same instrument to gather data about importance and satisfaction for our online offerings and services. This year we had a 23% response rate with 458 students submitting the survey out of the pool of 2,012 students who were invited to do so. All students taking at least one online course at LSC are invited to submit the survey.

The last time we used the survey was in Spring 2006. We only had a 17% submission rate (325 out of 1,889). In an effort to significantly improve the submission rate, this year we offered 40 (2 GB) flash drives by random drawing to those who submitted the survey. There were no incentives on 2006.

For about $440 ($11 per USB drive) we were able to gather data from a more significant group of our online students. I’m thinking that was well worth it. Now we need to find out what they had to say to us.

Once again, Minnesota Online will be sponsoring the use of the PSOL for all interested schools in the Minnesota State Colleges and Universities. LSC will be participating again this year after taking last year off. The survey will be available to students in February and we will likely have the results in the latter part of March.

This will be our fourth time gathering data with this instrument. The previous three years have provided us with a good baseline on which to judge future changes in importance and satisfaction to the online students. We have five optional questions where we can ask any additional items that we see fit. We are planning on trying to match most or all of those five questions with similar questions on the N-L Student Satisfaction Inventory (SSI). The SSI is a similar survey that will be given this spring to our on-campus students. This will give us even additional data with which to compare the satisfaction of our online students with our on-ground students.

It’s one thing to get the data about importance to students and their related levels of satisfaction, it’s quite another to be able to do anything about those things. On the one hand you might have an opportunity to manage student expectations which could possibly affect either importance or satisfaction, or both. On the other hand, you might be able to improve services and affect satisfaction scores in a positive way.

On the third hand, there might not be much that you can do at all, at least not without a long time horizon, lots of patience, and maybe lots of money. As I look at our results of the PSOL, I believe that there are four items where we don’t have much of an opportunity to affect results. They are as follows:

01. This institution has a good reputation.
06. Tuition paid is a worthwhile investment.
08. Student-to-student collaborations are valuable to me.
09. Adequate financial aid is available.

College reputations are not built or destroyed overnight. It may be nice to know what your students think about your reputation, but there might not be too much that you can do about it. I suppose you could start an internal promotional campaign designed to convince your current students that you really are much better than they think, but that seems silly, self-serving, and somewhat pathetic.

Convincing students that their tuition dollars are being well spent is also a rather futile exercise in my opinion. Even though that may be true, many people think that there is always a better deal right around the corner. The grass is always greener somewhere else which means that tuition is always cheaper elsewhere or the quality is better or both. Good luck trying to convince students (or any consumers) that they have misjudged the value in what they’re paying for.

Student-to-student collaboration? I’ve already posted about that a couple of times. It’s a bit of a strange question for students to assess, especially on the satisfaction scale. My satisfaction with collaborations is more a function of who the collaborators are rather than something that the institution can control.

Finally, adequate financial aid? Are you kidding me? The only adequate financial aid for many people is something that covers all their expenses (that includes beer money) and doesn’t have to be paid back. Short of that they will rate financial aid adequacy as extremely important and satisfaction level as quite low. Again, this begs the question of what the institution can possibly do about the huge gap between the importance and satisfaction scores here. Start giving away money in the hallways? I doubt it.

All told, only four out of 26 survey items that you have very little control over isn’t too bad. The other items I believe that you either have a great deal or a moderate amount of control over, or ability to influence in the short term of 1-3 year. Those are the items where you may be able to see some increases in your survey results if you pick your strategies wisely.

(What does the Washington Monument have to do with this post? Nuthin’. I just like it and it’s mine, so why not?)

One of the important things to do with your PSOL data is to look at the gaps – the differences between the importance ratings and the satisfaction ratings expressed by the students. You’re always concerned about the larger gaps which indicate where you have significant room for improvement, if it is an item where you have much control over (or impact upon) the level of student satisfaction.

At the same time, I like to look at the smaller gaps. You also need to have some positive reinforcement for those things that are going very well – for those things where you are essentially meeting the student expectations. The seven items listed below come from the 26 standard items on the PSOL and indicate the lowest performance gaps for Lake Superior College in the FY06 satisfaction survey. The item number is shown first, followed by the full text of the item, and then the size of the gap (importance score minus satisfaction score).

08. Student-to-student collaborations are valuable to me = (.42)
This one is always interesting (see previous post about this item) because the students are basically telling you something like this: “I really don’t care much about this, but you’re doing a pretty good job with it!” This item has a decent (not great) satisfaction rating, but it has an extremely low importance score.

01. This institution has a good reputation = .02
This item is interesting to me because it tells me that we are serving the proper audience. Our students know who we are and what we can do for them. For the national numbers, the importance score is much higher than ours, which is why the national gap was .39 while ours was only .02. The satisfaction scores were almost identical, but reputation is more important to more of the students in the national survey. Keep in mind that many of those students are graduate students and have a very different demographic than our students.

18. Registration for online courses is convenient = .14
We have a low gap here because our registration system is completely online and has been for several years. It is managed by the state system (MnSCU) and seems to meet students’ needs quite effectively. This is always one of the most important survey items for students, but one with consistently high satisfaction scores.

21. Adequate online library resources are provided = .33
For several years now we have had significant online library resources available to all students. Our biggest issue is in getting them to use the resources, not whether the resources are online or not.

24. Tutoring services are readily available for online courses = .33
We are currently in our fifth year of offering online tutoring services through SMARTHINKING. We have always scored highly in this category, but you also have to keep in mind that only 10-15% of the students use tutoring services (either online or on-ground) and so the importance score is one of the lowest on the survey.

23. Billing and payment procedures are convenient for me. = .35
Much like our online registration, the billing and payment function is managed centrally for all 32 institutions. This is an item with a high importance score, but also a high satisfaction score.

19. Online career services are available = .35
This last one also is not very important to many of our online students, since so many of our online students are transfer students working on an A.A. degree of the Minnesota Transfer Curriculum. Our gap is low because this is not a hot topic for most of our students.

For me, the takeaways here are as follows:

1. Small performance gaps are often the result on items where the student ratings indicate a low importance score. That is the case for items 1, 8, 19, and 24 above. All four of those items have an importance score below 6.0 which generally indicates that most students rate them somewhere between “slightly important” and “important.”
2. Small performance gaps on the other three items (18, 21, and 23) indicate areas where you are really meeting the needs of the students. These items have high importance scores (above 6.0) and high satisfaction scores. These are things to be proud of.
3. Gap analysis is not an exact science. It is only a starting point in looking for places where you may be able to improve the services that are being offered to students.

(CC Flickr photo by Annie Mole)