PSIA - CSIA Compare and Contrast the Organizations (Training, Certification, and Teaching)

[Magi]

...
From what I've read, and considering your input, which you have expressed so clearly, confidently and authoritatively that I have no doubt about it's truth, it appears that there are indeed two separate problems: low pass rates on exams, and diminishing numbers of fully qualified of L3 and L4 instructors.
...

Neither of the above is perceived as a problem in PSIA. Why someone outside the organization with "no axe to grind" is so concerned about it baffles me.

Even so a pass rate of 30% seems very low for an exam to verify that those taking have learned what you have taught them. If the purpose of the exam is to see if they have learned what you were supposed to teach them, then either the teaching needs adjustment, or more folk need to be told up front they are not ready, and effectively discouraged from taking the exam.

The bolded portion of your quote is factually incorrect. The purpose of the certification exams is to signal that, on a day, in front of examiners trained to evaluate a standard s objectively as possible, you met that standard. Full stop.

Furthermore - no one that I've talked to that's a leader of, or member of the organizations in question - thinks that the pass rate is "too low" on an average basis. (WIDE open to hearing that this is perceived as a problem, but that's not my impression/opinion).


While no one is against having more people attempt and pass high level certifications - it turns out that hard things are hard - and there is no way to make them less hard without lowering the standard.

In PSIA - exams are scored 1-6, with 4 passing, 5 being occasionally superior, and 6 being always superior.

Most people take the exam when they're at a consistently (but not always) "4" place.

Let's try numbers!
If the candidate demonstrates the task X% of the time at a "4", assuming 10 tasks - they pass the entire exam day:
88% -> ~28% pass rate
89% -> ~31% pass rate
90% -> ~35% pass rate


The actual average pass rates of the L3 in PSIA-RM is about 29-35%. This indicates to me that people are (rightly) judging they can do it ~9/10 times and then deciding to take a shot at passing, knowing full well they probably won't get the whole thing in one shot.

Going from 88% to 89% success rate on one task (at the same level "4" of performance) is days to weeks of additional training. If you manage to get 2% better at all 10 tasks you gain approximately 6% increased chance of passing overall.

Taking the test twice (without getting any better between attempts) takes you from a 30% shot to 50%. Taking the test once, passing a section or two, then focusing your training attempts on specific sections would let you double or triple your effective training time in an interval. [And note - even if someone can demonstrate each task 99 out of 100 times, *they still fail the whole thing ~10% of the time!*]

That makes it eminently rational to get pretty good, take the exam, and then focus your training where you now have objective feedback that you need it.


One other note: being able to do a task in training is not the same as doing it on exam day. It is actually impossible for a trainer to give you more than "You are consistently showing me that you can perform the task and/or demonstrate the required movements". On exam day you have to do the task *cold*, *under pressure*, with conditions that may be "unfavorable" to put it mildly. You can't simulate the exam without... taking the exam.

Of course if it's merely a gate keeper to ensure that only the top 30 percent of the best prepared and best qualified instructors get the L3, then go ahead rate on a bell curve and call yourself the best of the best.

The Standard is the Standard - it isn't graded on a curve.
There is no limit on pins awarded. Enough pins are brought to each exam so that EVERY registered candidate to bring one home.

 

[LiquidFeet]

Those numbers are interesting. I'm working to get my head wrapped around them, but hitting the wall.
Can you explain how you get from 88% of the time performing at level 4 to 28% pass rate?
Congratulations on getting your LIII!

....In PSIA - exams are scored 1-6, with 4 passing, 5 being occasionally superior, and 6 being always superior.
Most people take the exam when they're at a consistently (but not always) "4" place.....
If the candidate demonstrates the task X% of the time at a "4", assuming 10 tasks - they pass the entire exam day:
88% -> ~28% pass rate
89% -> ~31% pass rate
90% -> ~35% pass rate
The actual average pass rates of the L3 in PSIA-RM is about 29-35%. This indicates to me that people are (rightly) judging they can do it ~9/10 times and then deciding to take a shot at passing, knowing full well they probably won't get the whole thing in one shot.....

Taking the test twice (without getting any better between attempts) takes you from a 30% shot to 50%. Taking the test once, passing a section or two, then focusing your training attempts on specific sections would let you double or triple your effective training time in an interval. [And note - even if someone can demonstrate each task 99 out of 100 times, *they still fail the whole thing ~10% of the time!*]

That makes it eminently rational to get pretty good, take the exam, and then focus your training where you now have objective feedback that you need it.....

 

[Magi]

Those numbers are interesting. I'm working to get my head wrapped around them, but hitting the wall.
Can you explain how you get from 88% of the time performing at level 4 to 28% pass rate?
Congratulations on getting your LIII!

Thanks!

If you have an 80% (.8) chance of passing one task, you have a (.8)*(.8) chance of passing two (assuming these are strictly independent events).
Generalized, that turns into: (% chance of passing)^(# of tasks)


Please note that I'm not claiming the above analysis is a rigorous model that accurately reflects all of reality. All I'm hoping the reader takes away is you can be pretty damn good a set of things, and have a non-zero chance of failing "the test" if failing any one problem causes you to fail.


Another thing that may be worth pointing out to people who haven't taken the exam. A "4" is probably closer to getting a "95" on a normal test than a 70. You either did it right, or you fail it. There's no partial credit.

 

[wutangclan]

@LiquidFeet, @Magi is referencing the binomial probability formula that is applicable to compound event pass/fail rates. I like this explanation -- I'm gonna steal it for laying out the cold, hard truth to my apprentices!

 

[wutangclan]

Let's turn the "problem" of low L3/L4 pass rates on its head.

At all three snow schools (Cypress, Grouse and Seymour) in Greater Vancouver, the full time staff are 99% comprised of L1 and L2 instructors. I personally know a couple dozen instructors who've passed their L3 in the last decade or two, and except for a handful who've moved into snow school management or become nomadic snow-chasers, most of them have either left the ski industry altogether or become part-timers who make their real money elsewhere. Why? Because someone who is smart, talented and perseverant enough to excel at skiing is also likely to excel at something else that pays better.

So, setting aside how it could be done (by lowering standards?), what "problem" would be solved by raising L3/L4 exam pass rates, when the industry is incapable of retaining the majority of people who've already passed?

There are many other industries suffering from skilled-labour "shortages", wherein employers are clamouring for training/certification bodies to increase their output yet refusing or unable to raise wages to truly attract/retain labour. Airline piloting is probably the most egregious example, however pilots can be replaced with an autopilot, but ski instructors cannot.

These are wide-ranging socioeconomic issues not endemic to the CSIA/PSIA.

 

[Mike King]

Let's turn the "problem" of low L3/L4 pass rates on its head.

At all three snow schools (Cypress, Grouse and Seymour) in Greater Vancouver, the full time staff are 99% comprised of L1 and L2 instructors. I personally know a couple dozen instructors who've passed their L3 in the last decade or two, and except for a handful who've moved into snow school management or become nomadic snow-chasers, most of them have either left the ski industry altogether or become part-timers who make their real money elsewhere. Why? Because someone who is smart, talented and perseverant enough to excel at skiing is also likely to excel at something else that pays better.

So, setting aside how it could be done (by lowering standards?), what "problem" would be solved by raising L3/L4 exam pass rates, when the industry is incapable of retaining the majority of people who've already passed?

There are many other industries suffering from skilled-labour "shortages", wherein employers are clamouring for training/certification bodies to increase their output yet refusing or unable to raise wages to truly attract/retain labour. Airline piloting is probably the most egregious example, however pilots can be replaced with an autopilot, but ski instructors cannot.

These are wide-ranging socioeconomic issues not endemic to the CSIA/PSIA.

This is, I think, a major failing of PSIA/CSIA. That is, they see themselves only as educational support institutions, not professional institutions. If they truly were professional institutions, they would do the hard work to promote the interests of their members -- that is, market to the general public the advantages of taking a lesson from a certified member of the institution and to ski area management the advantages of hiring certified instructors. That is, do the work to achieve higher pay for their members.

Why doesn't this happen? Might it have something to do with the fact that ski area management make up the vast majority of leadership of PSIA/CSIA?

Mike

 

[François Pugh]

"The bolded portion of your quote is factually incorrect. The purpose of the certification exams is to signal that, on a day, in front of examiners trained to evaluate a standard s objectively as possible, you met that standard. Full stop"
Sorry, my bad, I thought that was what you were teaching them to be able to do.

Congratulations on having an organization whose members are happy with the way the exams and certification is run, that you have no problem obtaining and retaining highly skilled (Level 3 and above) instructors and that there is no problems with the low pass rate.

You are correct, I don't really care about PSIA certification and training. I do care about having sufficient numbers of high level CSIA instructors available in Canada to teach advanced lessons.

I also agree with not lowering the bar. I'm glad to hear that it is scored to an objective standard, and not a bell curve.

Congratulations on getting your PSIA Level III certification too. Getting certification in a physical endeavour that requires years of dedicated hard work is worthy of celebration.

 

[markojp]

"

Congratulations on having an organization whose members are happy with the way the exams and certification is run, that you have no problem obtaining and retaining highly skilled (Level 3 and above) instructors and that there is no problems with the low pass rate.

I do care about having sufficient numbers of high level CSIA instructors available in Canada to teach advanced lessons.

The former... still grinding the ax it sounds like.

The latter, just curious, but are you looking for coaching? You could have booked Tobin at Mt. Washington while you were there. I guess I'm still doubting the veracity of your concern as I can't recall you ever mentioning looking for coaching, but I could well be mistaken. I'd say we all can use some input. Affordability is a whole 'nother and very legitimate topic for discussion and is probably the single largest barrior to accomplished skiers signing up.

 

[François Pugh]

I did try for a private full day lesson with Ski Nerd from Epic when I was at Mt. Washington, but he was otherwise occupied doing some multi day camp for Section 8.

 

[Doby Man]

Interesting topic ... In no other educational or industrial training setting that I am aware of, are there so many people willing to repeat an exam so many times before they either pass or quit altogether. Leave it to Malcolm Gladwell to explain why through his theory of socio-economic epidemiology. I actually believe that, among other theories of human behavior, it is the exclusivity of the low L3 pass rate itself that makes the plight even more attractive than would a high pass rate. Considering all the attempts and extra clinics taken to continue stringing the process along, this could be seen as a very crafty revenue model. Most or all other education and training business revenues would dry up with such a low pass rate. With so many stats floating around, the one I’d like to see is the average amount of money spent on exams, repeat exams, clinics and fees per instructor. Because the PSIA has all this (private) data for historical analysis, it is easily conceivable that they are able to control their incoming training/exam revenue by adjusting the pass rate based on that specific result far more than any other. It is a business after all. I would not be surprised if the current pass rates produce the highest training/exam revenue where it would drop if the pass rate either increased or decreased from where it is currently being held. It doesn’t matter why. In business, the dry mechanics of numerical data makes the lion’s share of decisions. If a company or organization sucks at business, it doesn’t matter how good the product or service is because it won’t be around long enough to matter.

 

[markojp]

Most or all other education and training business revenues would dry up with such a low pass rate.

I think many of us have had our moments of dispair/utter frustration in PSIA. The stat that's missing is pass rate after x attempts. Lot's of folks fail initially, but eventually get there. In a similar context, not many rise above a certain level in any organized sport, but lot's will continue to try and improve, get coaching, etc....

 

[wutangclan]

Interesting topic ... In no other educational or industrial training setting that I am aware of, are there so many people willing to repeat an exam so many times before they either pass or quit altogether.

Nonsense. CFA (Finance) and SOA (Actuarial) candidates often fail many times. And peeps often retake LSAT, MCAT and GMAT exams multiple times until their score is high enough to gain entry to the grad school of their choice. Those are just a few I can list off the top of my head with my limited knowledge of other industries. There must be many more.

 

[wutangclan]

This is, I think, a major failing of PSIA/CSIA. That is, they see themselves only as educational support institutions, not professional institutions. If they truly were professional institutions, they would do the hard work to promote the interests of their members -- that is, market to the general public the advantages of taking a lesson from a certified member of the institution and to ski area management the advantages of hiring certified instructors. That is, do the work to achieve higher pay for their members.

Oh boy. The topic of instructor pay is a whole other can o' worms that has been discussed endlessly before. I'll just respond with a couple of points and let others with more time and energy debate the topic thoroughly.

1) With few exceptions, wages are frequently low in a) "fun" industries and b) receding industries. The ski industry suffers from the double whammy of being both, therefore there'll always be a glut of labour that makes it hard to fight for higher wages. It's a bit simplistic to lay all the blame at the feet of the PSIA/CSIA.

2) It sounds like we do things differently up here in Soviet Canuckistan, as all Canadian resorts require instructors on their payroll to be CSIA-certified, and independent instructors are prohibited from teaching on resort property. Furthermore, there is a strong culture in most Canadian ski schools (certainly all the ones in BC) of instructor advancement through in-house training and CSIA certification. Heck, at Whistler, they don't let you out of "Base pod" unless you have your L3.

Why doesn't this happen? Might it have something to do with the fact that ski area management make up the vast majority of leadership of PSIA/CSIA?

Who else do you think should run our governing organizations? Would you suggest inexperienced outsiders? Paging Dr. Ben Carson ...

 

[Magi]

Nonsense. CFA (Finance) and ASA (Actuarial) candidates often fail many times. And peeps often retake LSAT, MCAT and GMAT exams multiple times until their score is high enough to gain entry to the grad school of their choice. Those are just the ones I can list off the top of my head with my limited knowledge of other industries. There must be many more.

I can vouch that Network Engineering certifications also work similarly.

 

[Bruno Schull]

@ markojp and others. I've been watching this thread with genuine curiosity. I don't really know anything about PSIA or CSIA so I have nothing to add there.

However, it seems as if Francois Pugh raised some points, others reacted, you guys went back and forth a little bit, and then FP posted a sort of "OK, I'm out, all the best," sort of a response. Yes, he made his point in the first sentences, i.e. he suggested that there still might be legitimate issues concerning his original post. There was an element of him taking the "moral high road" which can be annoying in an argument. I get that. But he also went out of his way--more than is customary on internet forums--to step back from controversy, and to congratulate successful instructors personally.

I think that, in your subsequent post, when you insisted that he had an axe to grind, and then asked if he was looking for coaching, that was unnecessary and somewhat patronizing. Is not FP one of the principal moderators (owners?) of this forums? I think that makes him deserving of some respect. Also, like anybody, he certainly entitled to his opinions, without having others insist that he fully backtrack on whatever they don't like about his posts. Last, who knows how well anybody skis? They might be an instructor or not, but they might be an amazing skier. I know a world famous top level IMGFA mountain guide and climber (the IMGFA ski tests are notoriously difficult to pass) who told me that he tries to take one ski lesson per season and has done for decades. I think there is little doubt that he skis as well or better than most instructors. Two points here. First, just because somebody is not instructor does not mean that they don't ski very well and somehow need instruction. Second, even the best can continue to improve. I have no doubt that you share a view of continual potential for growth for skiers of all levels, including the best instructors, but somehow this recognition and humility appear lacking in your response above.

 

[markojp]

Bruno, point taken. There's a bit of a history here that well preceeds this thread and even Pugski. I'm sure Phil is monitoring, and we're all free to hit the 'report' button if needed. I asked F.P. if he was looking for coaching knowing he had spent some time at Mt. Washington, BC this winter and that he had access to an excellent CSIA L4 instructor who can be contacted via PM here to ask about his availability or to recommend someone who might be if he wasn't. FWIW, I don't assume someone who isn't a fully certified instructor doesn't or can't ski well. Very far from it. I think F.P. would have much to contribute to a ski school if he chose to become involve and commensurately have access to great coaching and more accurate information about the umbrella organization(s). Encouraging him to get involved was not rhetorical poke.

The ski instruction world is a small subset of the ski experience, and I don't think it's unreasonable to ask questions about an organization when one isn't an active participant either as an instructor or customer. In this conversation though, there clearly there was a pretty deep misunderstanding of what the organization(s) do, how they operate, and why. I do believe that we all can benefit from having a good, educated, and experienced eye on or skiing if at all possible no matter how accomplished we are. On that note, we're at page 4 which is the usual point where what is useful in a thread has been said. I'll follow that adage and check out of this one.

F.P., apologies if I was a bit harsh questioning your motives and comments. In the end, it's just skiing, and skiing should be fun.

 

[KingGrump]

Is not FP one of the principal moderators (owners?) of this forums?

No.

 

[Doby Man]

I think many of us have had our moments of dispair/utter frustration in PSIA. The stat that's missing is pass rate after x attempts. Lot's of folks fail initially, but eventually get there. In a similar context, not many rise above a certain level in any organized sport, but lot's will continue to try and improve, get coaching, etc....

I'm not trying to discount the psia and, instead, speculating as to its business savvy. I am sure that the examiners are empowered with a certain amount of discretion, I am as sure that it is the PSIA administration that sets the bar that this discretion is predicated on. I would also be willing to bet that if the pass rate declined from 30% to 20%, revenue would decrease along with the associated decrease in ongoing training/exam revenue and, if the pass rate were to increase to 40%, revenue would also decrease along with the associated decrease in attraction with the lessening exclusivity of L3 acquisition. The more exclusive something is, the more attractive it becomes. The PSIA needs to raise enough revenue to run an operation the size and cost of which must meet some balance of scale with the number of members from which they acquire all their revenue. While I may be only speculating the model, not considering the connection between pass rate and revenue stream would be operating blind.

Nonsense. CFA (Finance) and SOA (Actuarial) candidates often fail many times. And peeps often retake LSAT, MCAT and GMAT exams multiple times until their score is high enough to gain entry to the grad school of their choice. Those are just a few I can list off the top of my head with my limited knowledge of other industries. There must be many more.

You may be correct, though, we don't have the pass rates for any of those other industries and just the anecdotal info you give. The comparison was speculation in support of my main point. PSIA is a very unique model compared to those situations where I believe the pass rate is far more influential on revenue than those more regulated industries that operate in a corporate, competitive, non-monopolistic environment. Ultimately, what I am saying is that from a business sense, a low pass rate for top certification may a smart thing in terms of the bigger picture.

 

[François Pugh]

I'm just a skier. Oh, and a volunteer ski patroller, but mostly a skier.

No matter how good you ski, you can always improve. I tried hard to get a private lesson with Tobin. I phoned the mountain numerous times, but was told he could not spare the time from his multi-day skiing camp. Of course I could have gone into debt for a few grand and taken that course, but that would have meant taking up too much of my vacation time, and not being able to spend any time skiing with my children. I also prefer a private lesson.

I have no association with this web site, other than spending far too much time on it.

 

[DanoT]

I'm not trying to discount the psia and, instead, speculating as to its business savvy. I am sure that the examiners are empowered with a certain amount of discretion, I am as sure that it is the PSIA administration that sets the bar that this discretion is predicated on.

I am not an instructor nor have I any detailed knowledge of PSIA or CSIA but I think it is fair to assume that it is primarily L1 and L2 courses and exams that form the bulk of revenue for PSIA/CSIA. So @Doby Man, imo your assumption that L3 pass/fail rates are based on revenue need is flawed. There may be a "maintain exclusivity" component to pass/fail rate but it has close to nothing to do with revenue because it doesn't need to.