I’ve been doing a lot of program impact evaluation for the last few months. That comes with pluses and minuses. One of the minuses is that I needed to suspend writing this blog for a while. Pluses include problem-solving, working with staff throughout our company, and forensics engineering.
Impact Evaluation
But first, what is impact evaluation? I have a broad audience, and I get great feedback, some of which is, “I don’t know what you’re talking about sometimes.”
Entities delivering efficiency programs (implementers) are responsible for delivering savings or impacts. For custom efficiency portfolios, which entail a wide variety of programs, the impacts are determined, project by project. Implementers have goals for hitting energy savings targets, and to varying extents, they are incentivized for surpassing goals or achieving other measurable metrics.
The impact evaluator verifies, but more often, independently determines the actual impacts or savings that were achieved considering any number of 36 billion factors. Once in a while, implementers and the all-knowing impact evaluators disagree on the savings/impacts, chortle, and laugh break here.
Ok.
Forensics Engineering
What is forensics engineering? Something like estimating what caused a parking garage to collapse or a train to crash, and in the case of impact evaluation, doing so on a small budget, possibly with access to witnesses who have zero interest, personal, financial, or otherwise, to help. With luck, there might be scraps of invoices, a pile of emails, calculations, and drawings that may or may not represent construction. Utility-measured energy consumption for the facility at hand or an entire campus of facilities are typically available. Half the time, there are either more than one building on a meter or more than one meter for a building. The fun never ends.
But the glass is half full, and optimism prevails. The less and more fragmented the pile of information we have to work with, the more interesting creativity we deploy.
I’ve never met an engineer who could get enough information to solve a problem – beyond a textbook problem – the so-called “real world.” We only have five or six hours to come to a solution, and as described above, somebody at the customer site, any of hundreds or thousands of people, has no interest in what we need to do. Fuhgedaboutit! This is beautiful. Let the forensics begin!
Engineering Review v Bull Riding
Contrary to popular opinion, not all evaluators enjoy inflicting pain and suffering on program administrators and implementers. The implementer’s dump of files resembles the contents of a couple’s attic, in a house in which they raised six kids and lived for 50 years. Most of it isn’t relevant or useful. As I pick through this stuff and find the best nuggets, I root for it to be reasonably correct.
As I was telling a colleague, it’s very much like bull riding. Who doesn’t like to see an athlete stick to the back of a bull for eight seconds? Once I find the most informative document or calculation, as I review it, my mind’s voice would sound like this: “Oh, Yes! Go!
Go! Hang in there! Ooooooh! That had had to hurt.” In this way, it’s just like bull riding where you can watch twenty “cowboy” athletes, and only one can hang on for eight seconds. That’s about the batting average we see in project file reviews.
This fella (the bull), named Bushwacker had a 40 ride winning streak against cowboys, at the time in 2013.
By the way, bull riders primarily use core strengthening exercises and yoga. Finesse is the name of the game. “I won’t be able to outmuscle a 2,000-pound animal,” says Luke Snyder, bull rider. I admire the, athleticism, positive attitudes, and fearlessness of these guys. It’s inspiring. The bulls? They’ve got it made. Like racehorses, they are worth hundreds of thousands of dollars, are insured against injury, and they are loved and coddled – from a distance, of course.
Sympathy for the Devil
Folks complain about engineers, which is usually deserved, but one thing that is often not deserved goes like: “if six engineers calculate energy savings for a project, you will get six different answers.”
Consider the engineering analysis to be reviewed in an impact-evaluation scenario is like baking cookies. You might think all the ingredients are finely measured and displayed, ready to whip together for baking – that we can see, yep, all the ingredients are there. This looks A-O-K. Approved.
Never.
Ingredients provided by the implementer might include bacon, eggs, dill pickles, yeast, honey, oatmeal, and flaxseed. You have your fat, some sweet and salty, leavening, some grains – the functional requirements are there, but the end product is anyone’s guess. Six answers.
You might think this is a little ridiculous and absurd. It isn’t. It’s quite analogous to what I just described. I like it because we get to deploy creativity and make do with what we are provided. We also have recourse for our crappy cookies because we are stuck with the ingredients provided by the implementer, while the budget disallows the acquisition of better ingredients. It’s an exciting ride. No bull.
