L7: ROI for learning braindump

The learning 2007 conference has started yesterday evening and is now at full speed. During the opening session there was an interview with Doug Lynch from Wharton university. He claims ROI is a false number and the learning profession should move towards other forms of evidence to prove its value. Actually he used a quite strong word to describe the significance of an ROI number but I'm not going to repeat it here, both because I didn't get it well and because it's a dirty word that would bring down the level of this blog :-).

I went to his session this morning. Here is my dump:

• We live in a knowledge economy and more people learn in companies than in schools. However there are no peer reviewed studies of ROI in companies. That is worrying.
• It is not that ROI cannot be done, but it is very complex to do and not necessarily meaningful. A quick poll of the audience says 71% say their companies do not have an accurate ROI measurement in place.
  
• Learning is accounted for under GAAP as an expense, even if we talk about it as an investment in people
• Some reasons why ROI is not what we necessarily need: ROI is outcome based but in a learning organization for example it is the process of continuous improvement that matters. For ROI you need to define, measure and monetize all variables that make up the benefits and the costs of learning, while controlling for all variables except for learning. What goes into ROI is also very specific to the company and context.
  
• All things being equal, what is the impact of learning? Well guess what, all other things that affect performance are not equal, performance is affected by lots of things we cannot control for in the calculation.
  
• What we are actually are talking about is the IMPACT of learning, and evidences of that instead of the financial number related to outcome benefits and associated costs ratio.
• So we should all be like researchers and find evidence and validate that in a scientific way. Work with the evidence you can gather or with what is easily quantifiable.
  
• Americans are typically asking a lot of questions and interrupt the flow of the presentation and go into side discussions much more than you will see happening on European conferences. That is not necessarily bad, but at this point I lost it so I don't really know what we could do instead ROI. We did not get to the end or half of the slides, but they will be on the learningwiki.com site later on so maybe I'll get the point of the session later.
  

At a previous conference I remember in a similar session some people and vendors boosting on how they got a number and cracked the holy grail on proving learning value. Then the last speaker came and declared that mankind has been teaching each other stuff since the dawn of time, and we still do not know how to measure its value, although we all feel it does matter. So who started the whole ROI saga in learning anyway? Was it the business that noticed learning and wanted to manage it like any other service or investment? Or was it the learning function that wanted a place at the table and decided it should come up with a ROI number to achieve that?
At the end of the session I'm still as confused about the ROI debate as before. I have more reasons to believe that ROI is not the unifying answer. But I still have no idea as to what other evidences to replace it with. So hopefully I'll get it later and in the mean time stick to Kirkpatrick.