Impact Evaluation: Or Your Money Back
Without fail, it seems that every custom efficiency or self-directed impact evaluation we do has a controversial, giant project accounting for 25% (or more) of the program’s savings. It turns out it shouldn’t even have been allowed into the program because it doesn’t qualify – and in many cases, aside from that, the savings calculation is demonstrably and by the laws of thermodynamics and utility meter readings, wrong.
An analogy to the message today might be a home energy assessment. Our house was built to our liking thirteen years ago. I laid out the floor plan in about fifteen minutes on an 8.5×11 sheet of paper (literally). The architect pretty well stuck to that but added a couple features (for the better) to mask the fact that it was designed by an engineer with no style points. I would do a few things differently today with dimensions, but primarily for better envelope characteristics – continuous insulation for sheathing, thermal breaks and insulation for the basement concrete walls, and larger soffits for better shading. When it comes to tightness, it’s probably mediocre.
If I hired a decent home assessor, I would expect a bunch of recommendations, especially with plugging air leaks. I’m sure a worthwhile home assessor has acquired learned and honed skills for plugging substantial air leaks.
Suppose I hire up an assessment, and I wait around for a couple hours and the guy says, “This house is quite new. [duh] There is nothing that can be done to save energy.” So what is my response? “That’s wonderful. What do I owe you for providing no value whatsoever? Get out of here.” Then I’d call the program administrator to complain, and after listening to 20 minutes of Al Jarreau, Air Supply, and Lionel Richie music, I’d be told a grunt would call me back.
What is the purpose of The Energy Rant (big picture)? As described in the recent post Program Evaluation – Nellie You are Toast, a major purpose of the rant is to improve the industry from within – like the home assessor is supposed to do.
This all leads up to a recent multi-million dollar portfolio evaluation report that was called to my attention. The portfolio and resultant report featured a typical menu of programs, but the results were shocking. Eleven out of twelve natural gas measure categories had realization rates[1] of 100%. Twelve out of seventeen electric measure categories had realization rates of precisely 94%. It was as though savings were decided by flipping a coin – one side featuring 100% and the other 94%. Perhaps next time at least use a Magic 8 Ball.
These results are physically and statistically impossible. So I read the description – desk reviews, site visits, data logging, the whole eight yards. Findings from a decent sample of projects should take the form of a normal distribution – the bell curve – but with the exception that there is typically a pile of outliers with zero or near zero realization rates, and/or some with double/triple savings (200% realization rates and up) on each end of the spectrum. They should be evenly distributed about 100%, meaning 100% should be the most likely in an ideal world. Even in an ideal, perfectly representative sample with random results, the probability of hitting 100% for any category of measures (say commercial air conditioning) is probably less than 10%.
What should the results look like in these cases? Some randomness and disorder, perhaps? I would expect that given this number of categories, at the low end there would be something around 60% and at the high end, possibly as high as 130%.
In this case, as the buyer of this report, I’d seriously but sardonically ask for my money back. This does no one any good – not even the implementers because only a chump would take stock in the results.
This is my fear for energy efficiency programs and evaluation, specifically – that the industry is heading down the cheap and ratty path to a metaphorical dive motel in the red light district.
These results are not only less likely than running the table in college football, not only less likely than running the table in college basketball (Indiana, 1976), and not only less likely than running the table in the NFL (Dolphins, 1972). It would be something like running the table on the PGA tour or 162 straight wins in MLB, plus sweeping every playoff series for 43 straight wins, ending the day after Thanksgiving with a World Series championship.
If I were buying evaluation reports, I’d look at examples – not just the ones provided. I’d find other ones and I’d talk to references (people) for those. The appearance of rubber stamps, or superficial hand waving with a smattering of shiny objects, as distractions should be obvious.
[1] Realization Rate is the adjusted gross savings (verified savings) divided by the originally estimated savings.
Impact Evaluation: Catch Lucky
For many years of my adult life I thought, “courts and judges: who cares?” I’ve learned that I do, big time. Why? Because courts can bend and twist laws and essentially rewrite them to the point that we might as well send legislators home, which in and of itself sounds like a good idea. The danger is the courts are the backstop to protect us from unconstitutional law, and if they are creating law by fiat, there is no backstop. There is no accountability. They have the final say. Energy efficiency impact evaluation is much the same, as evaluators declare what programs have actually accomplished. There are parallels, especially with the final judgment.
One of the jobs of the evaluator is to understand the rules of program for eligibility, baselines, and a raft of other issues. We have experienced evaluators injecting their own opinion into the evaluation process. For example, Wal-Mart builds all their box stores with certain features. “No savings for you” declared one evaluator – because this stuff would have been implemented anyway! This is like the first amendment. Vile ogres have the same right to petition government ogres as does everyone else. We can’t pick and choose. Evaluators can’t make their own laws either.
As I have blasted in this blog many times, I insist that programs be cost effective and serve as a net benefit to customers and a cheaper alternative to adding supply, transmission, and distribution systems. Who’s going to do it? You, Weinberg? I have more responsibility than you can possibly fathom.[1] To answer the question (I’ll answer the question, truthfully even), it’s program evaluators.
Evaluators are the supremes[2] of energy efficiency programs and program evaluation is the court case. Like the supremes, evaluators can interpret and report on reality (the law) or they can generate their own reality, in which case everyone loses, except “special interests”, which may include the utility or the program implementers.
The Dumb Down Low Ball
My fear in this isn’t so much intentional misrepresentation but lack of depth of investigation and digging and technical competence/expertise on impacts. Furthermore, what really concerns me is the Wal-Mart-ization of this critical element of energy efficiency programs. Selecting a program evaluation team is like house shopping from the curb. Evaluation service buyers can be juked into “buying” a fake town like the bad guys on Blazing Saddles. The pretty façade can resemble a proposal and eventually the report itself. Prop up a nice façade in front of a 1972 leaky trailer home and call it a Frank Lloyd Wright modern contemporary masterpiece.
As Wal-Mart learned, somebody will always undercut you. Refer to Dollar General. However, buying program evaluation is not the same as buying a box of Lucky Charms, which has been proven in blind taste tests to taste the same from Dollar General as it tastes out of Safeway.
Decent impact evaluation, particularly for commercial and industrial programs, requires substantial expense. Does the evaluation buyer really want to know what’s going on, or do they want a report to show the regulators / legislature, er I mean the court, that everything is hunky dory? I promise you, hunky dory is cheaper than even Dollar General.
Getting it Right Costs Money
Dollar General isn’t going to cut it. It costs money, in some cases quite a bit, to do a decent job on impact evaluation for C&I programs. The energy calculations performed by the program can be lousy and disheveled or may hardly exist. A new building automation system that affects a half-million square feet of office space may receive a $100,000 incentive. Evaluating this project and estimating the savings in the first place is a bit more complicated than doing the same for the Dollar General light bulb. If the savings aren’t there, why are they not there? This is a difficult and costly question to answer but if it isn’t answered correctly, send the legislature home and skip the evaluation altogether, because what is the point? The point is (1) get the right answer and (2) improve the program and that may mean switch to a consultant that knows how to do custom calculations.
Getting Wrong to be Right Costs More Money
And another thing: programs with lousy impact estimates cost more to evaluate. It takes longer to figure things out. It takes more digging on site, more data collection, more interviews, and more analysis. Then finally when it’s all pulled together, it may take several rounds with the utility/implementers to hash out the adjustments. The hunky dory report is met with no resistance.
Suggestion: Selecting an evaluation team should be based on qualifications, and the fee should be negotiated after selection based on needs and priorities. Typical fees are 2-5% of total energy efficiency portfolio spending – a higher percentage for smaller portfolios and lower percentage for huge ones.
Look, someone will always sell cheaper plastic cutlery. If buyers desire to pay $59 for Motel 6 and delude themselves to believe sitting on a stained chair at a Formica table with Canada Dry in a plastic cup with ice is the same as lounging with a Bombay martini at a Hilton lounge, you go man.
[1] That is correct – a Colonel Jessup rant.
[2] Supreme Court justices.
Saints Are Marching In
One of the most rewarding aspects of our business is ensuring people get it right and customers get the legitimate energy and monetary savings they should be receiving. My answer to “why should we pick you?” is: we are passionate about what we do; we get things right; and we want to make a difference – improve things. The problem occurs when others don’t give a rip about these things.
Michaels’ primary thrust in the evaluation arena is impact evaluation, which is – what difference does the program make? The questions are (1) what is happening versus what would have happened and (2) would it have occurred without the program? The first question is difficult and the second question is really difficult. The result of the first question is adjusted gross savings and the result of the second question is net savings.
Adjusted gross savings is simply the actual savings verified at the site, with customer interviews, visual inspection, and data recorded over time to verify savings via the 80/20 rule, which means don’t sweat the little stuff. Net savings starts with adjusted gross savings and applies psychoanalysis via a battery of questions and hypnosis mixed with bat wings, chicken bones, and Ouija board viewed through a crystal ball.
Efficiency measures are generally broken into two categories: prescriptive and custom. Prescriptive measures, as the term indicates, prescribe fixed savings and fixed incentive for a specific measure, regardless of how it is applied. For instance, a compact fluorescent may have an incentive of $2 and deemed savings of 10 kWh per year even though it may save 5,000 kWh (that’s a hell of a CFL) in one instance and 3 kWh in another. When averaged, the population of CFL measures should save 10 kWh apiece. Custom measures have savings and incentives calculated for the specific application via customer, vendor, or program personnel.
Everyone agrees that calculating savings for June Cleaver’s new energy efficient clothes iron is stupid because everyone knows Ward, Wally, and the Beaver wear seven neatly pressed outfits per week. (I don’t know about you, but my mom used to iron EVERYTHING, except maybe grundies, circa 1960s-1970s). This is well documented and the savings aren’t that great. But consider a variable frequency drive on a 100 horsepower pump for a manufacturer. There might be a half million kWh savings or their might be zero. Engineers have a major conniption over such uncertainty.
There is a balance between accuracy, program cost, and equity (as in fairness to customers).
- Accuracy includes that for the customer so they have confidence in their investment, and that for the program as a whole such that the net answer for the program is reasonably correct.
- Program cost includes costs to calculate and verify savings via program QA/QC and independent evaluation.
- Equity includes not giving customers a huge incentive for a technology application that results in a one month payback but also not giving a customer a huge incentive for an application that results in a 78 year payback. In the first case, it would be moronic to not do the project with or without an incentive. In the latter case, the customer is obviously doing the measure for non-energy reasons.
A segment of humanity chooses to avoid problems rather than addressing them, wallowing in the bliss of ignorance. They want everything jammed into the prescriptive bin and never look at it again. It’s one thing to cram ridiculous stuff like a 500 ton chiller into the prescriptive column where savings could vary immensely but another to say the savings are correct on average and you evaluator don’t worry about it and whistle past the bank robbery. In some programs, measures are sanctified into sainthood and once there, they are untouchable. 
The implementer is “made”, like a mob boss (a little extreme but you get the point). The result of a less than glowing evaluation sometimes is sanctify the measure(s) and insert head in sand rather than improve the program for customers. This is depressing.
We don’t need to evaluate CFLs year after year although prices and technology adoption change so an update is warranted once in a while. We don’t need to evaluate programmable thermostats (which incidentally have been elevated to sainthood in some programs) because their net savings is zero. (see ”Oh Behave” for why) It is so ridiculous that in some cases, if “it” appears to be installed, it gets full credit for savings that don’t exist. It may never be used. It may not be connected to anything. This is the stuff 20/20, 60 minutes, Glen Beck and John Stossel live for. Bogus results and collusion. Thankfully, in most precincts this is not the case.
Green is Dead
This week in Coon Valley, where I’ve lived for 15 years, we were hit with frost that ended the growing season for unprotected plants. I have said for years that the first killing frost is always around the first of October, depending on the timing of cold fronts, but this year was earlier than normal – earlier than its been for many years I think (I’m getting old so I’m becoming a walking almanac like your grandfather is/was).
To the point, I thought, the killing frost may be symbolic of the death of “green,” I don’t think the death of the concept just yet, but the death of the term. “Green” may join the trash heap of burned out terms that were symbolic of failure and/or some sort of scandal:
- “Jobs created” converted to “jobs created or saved”.
- “Global warming” converted to “climate change”.
- “Stimulus” converted to [I’m not sure what yet but Nancy Pelosi has declared the term “stimulus” to be off limits]
A few months ago when we were rebuilding our website and updating some content, I was pondering for a replacement for the word “sustainable”. The word is entirely overused and I don’t think the average Joe Public has a clue what it means – maybe something hippie-like, living in communes, drinking herbal tea, wearing hemp over a Bob Marley tie-dyed tee shirt and dreadlocks, and riding a crappy looking single speed bicycle.
Green is also wearing out its welcome if you ask me. Speeding it to its death is the raft of bad news coming from what I would call misallocated stimulus (ironically) funds. I mentioned the Solyndra bomb a couple weeks back, but then it was merely a half billion dollars of taxpayer money that might as well have been pelleted and mixed in with coal and burned to generate electricity. THE US GOVERNMENT BLOWS A HALF BILLION DOLLARS! READ ALL ABOUT IT!, as the boy would yell on the street corner selling newspapers in the old days. In 2011 however, a half billion dollars of totally wasted taxpayer money is barely worth mentioning. Something like this would be buried on page 21 of the Arts and Leisure page, next to the movie listings.
The Solyndra episode has ballooned into a scandal of sorts. When boiled down, it was actually a $500 million dollar television ad for the President to tout successful green jobs in a state of the art manufacturing facility, in California of all places – probably the worst place in the union to make anything. At first it was hyperventilating right wingers on Fox News parading the successful failure of green jobs and the Obama administration.
However, now everyone including numerous news outlets, congress, and even the FBI is piling on like a school of piranha attacking a case of bratwurst. ABC first broke the scandal part of this they claim. The CEO was a crony donor to the President. And the best part: venture capitalists (of the private sector stripe) are first in line to get their money out during bankruptcy “restructuring”. Taxpayers last – my anthem for all government activity. Taxpayers are always last in line for any sort of break. Solyndra will not emerge from bankruptcy. It will be liquidated. Anybody need some light fixtures, fancy cabinetry, office furniture or equipment? Robots? The liquidation is my prediction because any company that burns through cash this fast and manufactures a product that competes against a similar product at 1/6 the cost is unsalvageable.
In other galling cases, we have General Electric, with CEO Jeffrey total-failure Imelt on the President’s jobs task force maneuvering to pay zero income taxes thanks in large part to “green energy” tax credits. In ten years Imelt has guided the company with a starting share price of $55, now trading at $15. Imelt claims he’s advising the president on jobs, not tax policy. Right. His henchmen are lobbying the bajeebas out of Capitol Hill for these tax breaks and loopholes. Regarding the jobs advisory, he is doing a fine job as he moves GE Healthcare manufacturing from Wisconsin to China.
Look; companies are free to make stuff where they want. A major reason for moving offshore is high corporate taxes for the unfavorable companies (like ours) and red tape like, oh, SarbOx, Frank-Dodd, OSHA, EPA, NLRB, protesters, and an incomprehensible byzantine tax code. I do have a problem with lobbying and manipulating tax code to one’s benefit. I do have a problem with Imelt being in the White House all the time on a jobs and competitiveness advisory this or that with the President. Why doesn’t the President immediately boost his image by losing this guy and holding a rose garden ceremony to do so?
In another success story, Whirlpool, which purchased Maytag about five years ago and closed the Newton, Iowa clothes washer and dryer plant, also pays no income tax thanks to green tax credits. It also happens to be moving jobs south of the border. Are these green jobs that are being exported?
Combine once-great American companies that move most manufacturing overseas, while paying no tax on the money they do still earn onshore with throwing taxpayer money in the incinerator for political photo ops and you get “green jobs” becoming a vulgar taboo phrase.
In still more program-formerly-known-as-stimulus news, $38 million in weatherization funds do wonders in West Virginia. The money was filtered down from the state to local “anti-poverty agencies”. Half the projects failed inspection. Projects were doled out without bidding per state law. Employees (of the weatherizers) and their relatives served themselves first, one spending $10k with new windows and doors. I’m surprised they didn’t build a garage and call it a vestibule. A lawyer was paid $25,000 (that’s right, twenty-five thousand dollars) to write two sentences approving the awarding of funds to the agencies. The topper of them all, and there’s no way Jay Leno could come up with this, one person was paid $2,500 to inform Washington there was not enough money to track the money.
I get the feeling we won’t be hearing about the creation of green jobs much longer, if ever again. Actually, the word green may even become foul.
This is bad for our industry and we are going to suffer collateral damage and take some shrapnel from this I am concerned. As I’ve been beating on the past month, independent and in depth evaluation of program impacts and cost effectiveness are needed. The way to do it is to have government (regulators) police the private sector – utilities and other implementers, using other private sector evaluators.
Tidbits
I kid you not. After writing the above, I came across this article in the NYT. Yes, in addition to the banning of the word “stimulus”, “green” is now verboten per the White House. http://economix.blogs.nytimes.com/2011/09/12/the-green-jobs-numbers/
written by Jeffrey L. Ihnen, P.E., LEED AP
Tapeworm or Honeybee?
Earlier this year I was partaking in an interview for a large project and we were asked a formal list of specific questions, the last of which was, why should we hire your team? While one possible response would be to ramble on for a few minutes about how great we were as demonstrated by this, that and the other, I thought of a different direction.
We are passionate about energy efficiency. We are passionate about getting it right. We are passionate about making a difference and improving things.
If you do not want these things, we are not your firm; and I’m serious because I do not believe some clients really want these things.
Passion to get it right simply means savings are going to be as accurate as we can reasonably get them as the budget and client allow – not only are the calculations sound, feasible and the answer relatively accurate, but is the project eligible per the rules of the program? Is the baseline, or less expensive, less efficient alternative reasonable? Is the less efficient alternative less expensive?
Making a difference and improving programs covers both the project level and the program level. We’re not here to beat people up and go home. If savings are fake or not eligible, we want to help clients identify and develop other substantial projects that are actually having an impact compared to the status quo.
As I am quoted on our website, I will tell people what they need to know, not necessarily what they want to hear, and this is consistent with the entire company. On rare occasion we stray off the ranch by trying to engineer a less expensive half-baked solution to a problem that needs a larger more comprehensive overhaul to fix correctly and sufficiently – such as trying to use too much of the currently-installed crap rather than ripping it out, spending 50% more and doing it right. But this isn’t really what I’m talking about in this post.
Good clients want their consultants, vendors, and contractors to report what is happening and offer solutions to improve their programs or systems. In a recent post, Don’t Ask, Don’t Look, Don’t Tell, I explain how some program folks by all indications aren’t so much interested in what the real impacts of their programs are. They filter information, they limit the scope in their requests for proposals to neuter impact evaluation, and/or they provide too little budget to do a decent impact (program-attributable savings) evaluation.
In another recent post, Rogue Choir Boy, I explain that many people in our industry, for pretty much other bizarre reasoning, agree that impact evaluation is a waste of time because it can’t be accurately assessed anyway. If we as industry are skimming 1-2% off all energy bills that customers are paying and not using it cost effectively, as indicated by a number of cost effectiveness tests, especially ratepayer impact measure (RIM), where benefits should be greater than cost, I’m out of here. Is the program a parasite on society like a tapeworm, or is it a benefit to society like a honeybee, pollinating foodstuffs and making honey on the side – i.e., everyone wins. Without rigorous impact evaluation, prepare for the march of the tapeworm.
The American Council for an Energy Efficient Economy says energy efficiency is by far the least cost resource – i.e., doing things to not use energy and capacity (demand) is cheaper than building more capacity and producing more energy. In fact, ACEEE has an entire conference devoted to this later this month. Portfolio managers should have to prove programs are cost effective with third party evaluation. While not all programs may be cost effective, the portfolio has to be. Some non-cost-effective programs are needed for equity purposes. For example, small business pays a large percentage of EE funds but “giving it back” by boosting cost effective measures is very difficult to do because there is no economy of scale with small business projects – impacts for the needed marketing/promotion required to get these customers to act.
Portfolios should provide wins for all stakeholders – utilities, consultants, vendors, contractors, customer participants and non-participants. I get the strong sense that for some portfolios the utility/implementers and the participants get wins but the non-participants get the shaft because the participants may be getting too much incentive for the impacts they are accomplishing. Actually, the participants may be getting the shaft too because they may not be getting the savings they should be getting. If utility/implementers can “fake” their way through with no one looking under the hood, they likely aren’t saving what they should be, the regulators don’t know it, and the customers don’t know it.
I go back to my conversation with the guy at the ACEEE summer study for industry a couple years ago. He said all evaluations should be on behalf of the regulators. I agree and not because working for the utility would bias outcomes, but utilities don’t think they have problems with impacts in some programs and some of these programs are very large. We know from experience time and again that program verification and in some cases measurement and verification helps, but in many cases does not come close to ensuring reasonably accurate results. Regulators I would hope would want more than a wink and a nod for impact evaluation
There are consultants all too happy to tell clients what they want to hear. How smooth can that be? There are consultants on the implementation side who will challenge and defend indefensible evaluation findings – projects that are not eligible, bad applications of engineering principles in determining impacts, or facts like large volumes of logged data showing assumptions / stipulated values are clearly wrong. Then it’s a sampling problem, the time of year, the economy or the dog ate the hard drive.
On the flip side to all this of course are clients, utilities, program implementers that strive for accuracy and reality, and continual improvement. They want to learn how to do things better and avoid problems and improve poor realization rates (“actual” savings divided by initial estimated savings).
As consultants we also greatly appreciate partnerships with our clients, whether they are other consultants, utilities, or prime contractors. Partnerships include shared sacrifice and shared benefit. Shared sacrifice includes project or program development time and expense in return for substantial work, revenue, and profit. Tip offs include not wanting to pay for program development but asking, what would that look like? Good grief. For great long-term clients we do many things for free, including “what it would look like”, preliminary analysis and development, spending time and attending functions with their customers and not charging for it, rounding up some data. Bargaining tooth and nail for a blanket in Juarez may be novel, but it isn’t how we like to do business for real.
Finally, it’s always nice to get paid. The worst clients not only recoil when told replacing their early 20th-century HVAC system won’t save more energy than they currently use, and then they refuse to pay because they should have been told the answer ahead of time.
Tidbits
Imagine this, Johnson Controls Inc. wants congress to pass the President’s jobs bill, whatever it’s called this time. I’m sure they have a billion or two (billion) with their name on it in the bill.
Outrage of the Week
Meanwhile, a bunch of other companies are pressing for another energy bill that would “help companies make their supply chains more efficient; as provisions to update national model building codes to increase energy efficiency of new buildings”. Good grief! This better go down in flames. If anyone thinks the role of government is to fix company supply chains, their name would be Vladimir, Hugo, Raul, or Kim.
written by Jeffrey L. Ihnen, P.E., LEED AP
The Rogue Choir Boy
I spent last week at the International Energy Program Evaluation Conference, IEPEC, as in, I-E-P-E-C to hard core evaluators or I-Peck for the rest of us.
Ninety-five percent of the conference including content and networking was great. Of course with this being the Energy Rant, I will beat on the remaining 5%.
Recapping, there are generally two portions of program evaluation: impact and process. Impact evaluation, which is what we at Michaels do, involves the assessment of savings (impacts) programs achieve, including what the measure actually saves (gross savings) and what impact the program had on the savings (net savings). For example, my mother started buying LED Christmas lights and practically replaced her Las Vegas scale lighting system with LEDs in one year. She then showed me forms she could submit to get cash back from the utility. Mom loves the lights. The cash was just a handout. Gross savings may have been decent for a residential end user. Net savings were zero because she already bought them without knowledge of the incentives or program.
Per the California Evaluation Framework, 2004, process evaluation is a systematic assessment of an energy efficiency program for the purposes of (1) documenting program operations at the time of the examination, and (2) identifying and recommending improvements that can be made to the program to increase the program’s efficiency or effectiveness for acquiring energy resources while maintaining high levels of participant satisfaction. The term “program’s efficiency and effectiveness” refers to dollars spent on programs. Are they achieving real impacts or just handing out money?
The conference is dominated by process evaluators and their counterparts on the utility and government (state and federal) side of things. Many (10-30%?) of these people I think have orbited a little too far from earth and spun off to other galaxies. They live in a galaxy far away and they argue about things like Zeno’s Paradox, except more obtuse, nebulous, and alien than our pal, Zeno. From here on I’ll refer to life forms as ETs, as in extraterrestrials.
I’m sending out a call to program implementers – utilities and third party implementers – you need to get involved with these evaluation conferences because I think you might be surprised at what you hear. For example, the keynote speaker Naomi Oreskes, a professor of history and science, talked; well let’s just say the title of her book is “Merchants of Doubt” (about global warming). Her message involved the history of deniers going back to the end of the cold war, later risks of tobacco, and so on up to global warming. It was interesting but as you know, I have doubts about significant human derived global warming and the conference itself perpetuates the basis of my doubts. I believe the reason it isn’t getting traction is people do not perceive any climate changes or adverse effects. Change is normal. On the other hand in the tobacco fight, black rotten lungs and people dying a heinous cancerous death are clearly eye openers.
Like climate science, the conference is supposed to be based on engineering, science, social behavior, decision making processes and the like. But in a large sense, to some it has turned into an advocacy group for policies to mitigate climate change and the philosophy of evaluation is starting to look like the philosophy of climate change as presented by the ETs. Papers for the conference have the same “peer review” process that the climate scientists supposedly have. In fact, as moderator, I peer reviewed three papers for my session and those papers focused on subjects relative to program evaluation, including how do customer perceptions of their lighting hours match reality as determined by logged data. What are the market effects of high bay lighting in California relative to a control group of states with no programs? This was interesting stuff and the papers were informative. I learned a lot and even referenced some of the findings already in proposals. To be sure, other sessions presented much pertinent useful information.
However, some latter sessions were really whacko fringe stuff that had nothing whatsoever to do with program evaluation. I attended one session that I swear to my maker was a doomsday cult session. People were pounding away on their smart phones during the session, possibly to book flights to Guyana?? In fact, I was thinking of shooting off an email to my loved ones saying that it’s been great knowing you and thank you for everything, Mom. The presenter was talking about how megatons of methane locked up in silt from a Siberian river flowing into the arctic ocean was going to be released and temperatures would rise 20 degrees or so in 20 years and a billion people would die from drought and famine. No kidding! Sounds like the makings of a Superman IIX movie. And this guy was absolutely convinced this is about to happen and absolutely sure what the precise effects would be. He spoke factually and in reading his paper he speaks of certain things to come – i.e., he was not saying,” could”, “may”, or “worst case” – it was “will” for all claims.
The only “will” there is about the future is, I will die, and I will pay taxes.
At the same conference there were panels of ETs who would debate things for which I could not understand whatsoever. When a guy (me) who’s been in the business for 16 years doesn’t understand the message, the message is miserably and hopelessly off course. The ring leader would say, “so and so is a contrarian and the other three panelists are believers” [in something?]. I don’t know what. They are so far out in left field you have to remember the seven time zones difference to ask them a question so you wake them up in the middle of the night. Finally, one woman in the session – an implementer – stood up and commented that we need to start talking the language of ISOs (independent system operators that control the grid), power suppliers and utilities or we will lose relevance not only for EE and policy, but for evaluation as well. I was about to give her a standing-O right there for COMMON SENSE. The first words out of one of the panelists mouth was, “I disagree.” Boooooooooo! What is wrong with these ETs?
The day before this there was a panel of mostly ETs to discuss politics of evaluation, which turned into politics of global warming of course. Note that probably most old timers still inhabit our planet and they are clear thinking people with their eye on the ball, which is improving programs. They didn’t all arrive at Boston Harbor in the starship lollipop. Moving on, each of the panelists gave their sermons and then it was turned over to ETs in the audience voice their proposed chapters to the holy book. I actually had to leave before it was over. The typical rant would start with, “I just have a couple quick comments.” Five minutes later they were talking about chicken farming on a nickel a day in Kenya. They would babble incoherently like someone who had been dropped on their head and doped up on maximum doses of morphine prescribed by their doctor. Anyone home? Is there a question in there somewhere? Perhaps a comment regarding our industry? Have you ever been introduced to concise?
Many ETs argue that we cannot quantify the true impacts of EE programs so just fuggedaboutit. We cannot treat EE or demand response as a resource so fuggedaboutit. What the? Then why have these programs? They know for sure that they need plane tickets to Guyana but we can’t even come close on program impacts? See what I mean? Does this make sense? The juxtaposition with program evaluation and climate change seems to be, believe. We can’t really measure it. We know neither the day nor the hour. It is going to kill us and these programs are cost effective. Just keep the money coming.
As it turns out, I was not the only flat earther in attendance. Numerous other heretics including one who had been away from these conferences for 10 years said the same thing, unsolicited. “It has become a religion,” one elder statesperson said. Well, heeeyah!
Let’s get rid of the underlying stealth agenda and crap papers and refill with good stuff and save the ET rants for the lollipop ride back to planet Koozebane.
written by Jeffrey L. Ihnen, P.E., LEED AP
EE: LOOK and THINK!
An overarching theme of the Energy Rant is that much energy policy has a feel-good foundation of fluff. Last week I ranted about the feel-good dream of having plentiful, inexpensive renewable energy. This will take a miracle because conventional sources are still huge and growing. We have enough coal, natural gas, tar sands, oil shale, and offshore energy to last beyond our kids’ great grandchildren. Of course most readers of this are champions of energy efficiency, but energy efficiency also has too much feel-good fluff.
Consider compact fluorescent lights, which despite my rant about it’s mandate a few weeks ago has been a fantastically successful development from the private sector sped along with the aid of EE programs. That market has been pretty well transformed, especially in states with high rates and years of EE programs behind them. Here’s the “problem” – the program has been successful. The market is transformed. Programs can no longer take credit for it but they don’t want to let go of the “savings”. Well c’mon!
This guy’s letter from the National Resources Defense Council illustrates this. He is responding to a recent Wall Street Journal opinion piece describing the “ineffectiveness” of California CFL programs. An independent evaluation of the program demonstrated that savings were much less than claimed. Sounds familiar per our first hand evaluation of some similar programs. He says the op-ed is based on a “consultant report that makes arbitrary and unsubstantiated reductions to the benefits of the compact fluorescent lamp program”. Well if that isn’t the cat calling the kettle black. Talk about unsubstantiated. I’m sure there’s nothing in the report to back up its conclusions. The guy probably hasn’t even read the executive summary.
Per our experience, this hack’s comments are unfortunately not uncommon. Utilities, program administrators, and implementers do not want to be told their programs are saving less than they claim – as they almost always are. I’m not sure who did the above evaluation in California but I will bet my house that they did not underestimate savings because: (1) it jibes with results we see for similar programs and (2) evaluators do not hammer savings for fun because it can lead to confrontation. We tell it like it is; not how someone wishes it would be.
We’ve recently completed impact (savings) evaluations for programmable thermostats; let’s just say in a state with a temperate climate – a state that has been lampooned in this rant a couple times. A programmable thermostat is 98% a heating-energy-saving technology. In the referenced temperate climate, where you can heat the entire house with a toaster oven, or at most your basic kitchen oven, what do you expect? Even in states that need heating, the attributable impacts can be tiny. Reasons for poor attributable savings include customers not using their furnaces; they were the programmable thermostat, programmable thermostats replacing programmable thermostats, and programmable thermostats in permanent override.
Impact evaluation for residential end users is often done by billing regression, which is a sexy term for comparing the bills before implementation to the bills after implementation and making appropriate adjustments. Consider evaluation for programmable thermostats with the only gas-using device in the home being the furnace. Billing regression is the ONLY way to go. Any engineering analysis is going to have much lower precision and confidence. But noooo! The program people didn’t like the regression results. Can we “engineer” savings? NO!
The other thing I’m seeing is rules changes to capture more savings. Incentives are limited by total dollars per year per customer, minimum paybacks, and maximum percentage of measure cost. This of course protects against free riders. Then there is the incentive itself – how much incentive is there per kWh, kW, or therm saved? Some utilities are greatly increasing incentives, lowering payback limits, and increasing annual payout limits. Does this result in more attributable energy savings? Probably not much. Evaluations will probably show they are mainly making more projects eligible and thus claiming more savings. I estimate free ridership will go up a lot. Program evaluators walking into the evaluation of these “upgraded” programs should prepare for pushback and maybe a little firestorm in some cases.
Some utilities whine to regulators that they’ve already done a great job of saving energy and all the easy stuff is gone (hence the expanded pay out and slackening rules discussed above). I don’t buy it. First, their 20th century programs are running low on remaining opportunity. Could be, but there are alternatives if they AND the regulators would open up to program innovation. Second, opportunities are created every day by engineers, architects, contractors, building owners, tenants, the milkman, janitor, cooks… you name it.
I haven’t seen any studies yet but I would bet there is more opportunity for cost effective measures in NEW buildings – ones that are already built. You just need to be capable of seeing the hand in front of your face and know how to “read” – i.e., understand what you are looking at. Buildings are loaded with opportunities we find but rarely see coming out of programs. Why? Perhaps because in many cases there is no equipment to sell. Examples: grocery store has a main air handler maintaining 75F in the space and at the same time an adjacent one is struggling to maintain 70F. The little one is cooling like crazy in the summer and pumping cold outdoor air all winter to try to get to 70F while the main unit is burning gas like crazy to make up for it. Obviously, this is an incredible opportunity and a very simple concept. Somebody just has to LOOK. And THINK! This is far more common than a congressman would ever imagine.
Taking on Parmenides
We do a LOT of energy efficiency program evaluation and measurement and verification work all over the country; make that North America. Program evaluation consists primarily of process evaluation (process) and impact evaluation (impact). Our work is almost entirely in the impact side and I know just enough to talk dangerously about process.
Impact is the analysis of what energy savings are really attributable to the program. This includes verifying the physical installation and determining the actual savings using some sort of engineering analysis. This actual savings is known as gross savings in the business. It also includes determining whether the program actually influenced the project to happen. For example, some would do a project or buy an efficient piece of equipment regardless of the program and just take the money because they can – and hey, they are paying into the program so there is nothing wrong with this in my opinion. These program-influence factors are applied to the gross savings to determine net savings – savings the program can take credit for.
Largely, evaluation teams consist of economists (impact and process) and engineers (impact) although there are many people with liberal arts degrees in the business as well.
Many times in determining the gross savings we get into spats with program implementers and sometimes utilities regarding what the actual savings really are. Many times for large custom projects, the energy analysis we have to evaluate varies from pathetic to essentially non-existent. “We installed a control system. Savings = 15%.” That’s it. Analyze that! Other times we will have an actual analysis and just plainly an incorrect application of engineering and physics or the operating conditions are much different than originally assumed.
Last week we were preparing to do impact for a huge low income weatherization program. Past evaluations for that program have turned up results that are only a fraction of what the utilities think they ought to be.
Consider how to estimate heating savings in this case. A house is heated by natural gas, which is also consumed by other appliances including possibly a stove and a water heater. The analysis is easy. You can see on the monthly billing data (gas consumption) how much gas is used to heat the place. It’s everything above the June through July average. Savings in this case are more or less proportional to the consumption for heating. It is as plain as the nose on your face. But the utilities think otherwise. While I certainly don’t want to arm them with any arguments, they could use Parmenides, the 2500 year old and dead philosopher.
I took a four credit philosophy course as an undergrad. The discussions in class seemed bizarre but definitely thought provoking. If you haven’t studied or read philosophy, you would most likely think it bizarre. But I am far, far, far (way far) from an expert on the topic.
One thing I remember discussing at length was, what does it mean for a being to be? Is there really anything that exists other than your mind?
I had to do some “research” to find philosophical terms. I’m talking about idealism. Idealism is the argument that your mind is all that exists and that the world is mental itself or an illusion created by the mind. Sound bizarre? Not so much if you think about it.
You’ve probably seen the HDTV ads that have stuff jumping out of the screen – like the picture is so real viewers purportedly see footballs flying out of the TV, right at them. Consider a person comes into my office and I ask him what he sees out the window. After a looking around to make sure he’s not on candid camera, the answer is: Coney Island hot dog joint and Deaf Ear Records. “Really?”, I respond. How do you know? I can see it. How do you know it’s not just an illusion? How do you know it’s not the world’s most expensive and lifelike television? Good God! I can go downstairs, cross the street and touch it. What more do you want? I can prove motion is an illusion and that you won’t really go anywhere, much less get out of this room, but that’s beside the point right now. So go ahead and touch it. What does that tell you? Why do you call it Coney Island? It says so. Really? How do you know? I can read it. Read what? By touching it? Why don’t you ask that guy who just got off the plane from Moscow what it says? You can’t prove anything. It’s all an illusion formulated in your mind.
The sky is blue. OK. But what if blue in your figment-of-imagination world would be green in my world? Who is ever going to know? We can both look at the same color and declare it to be the same thing – yeah sure, it’s blue. But a color is a color only because somebody told you so way back when and you correlated it to what you saw and it has been as such ever since – in your fantasy world. What is the definition of blue anyway? My dictionary defines it in part as the color of a clear unclouded sky. Great. That doesn’t explain anything. What color is a blue car under a clear unclouded sky… AT NIGHT? Why don’t you ask that color-blind 100 lb rodent that is eating the seedling I just planted what color his snack is.
This brings me back to the illusionary energy savings. Now that we know energy savings like everything else is all an illusion anyway, we can fool ourselves and put any number to it that we want.
Quite possibly, the program evaluation industry may be a gold mine for out-of-work philosophers and theologians! Utilities could have a team of philosophers to take on the evaluation team’s philosophers. Engineers and economists on the evaluation team would argue with their counterparts on the implementation team regarding the illusionary savings and the philosophers could duke it out over… something. See what I’m sayin? If so, it’s just a figment of your imagination. These people only exist in your mind.
Epilogue
For more on Parmenides, see this article, and in particular the Achilles and Tortoise paradox. Since learning that we still earn vacation while taking vacation (eons ago), you never need to return to work.
I earn roughly 3 hours of vacation every week. So if I take a week off I’ve used 40 hours but earned another three. I’ll take those three Monday morning, but I’ve earned 0.225 hour during those three hours. While I take that 13.5 minutes of vacation, I earn another minute. And it goes on forever, like eternity. Now do you think this philosophy stuff is stupid?
Contest
Above I said I can prove motion is an illusion. That was a lie, at the time. Since I’m telling you it was a lie, it isn’t, is it? On “The Big View” website, number 3 from Zeno attempts to prove motion is an illusion. For $10, explain why his hypothesis is wrong. The best answer wins, unless they are all horrible. Prize money will be split in case of a tie. If there are 10 or more correct answers, it wasn’t difficult enough so no prize. Contest ends September 30, 2010 AD. Send responses to kjk@michaelsengineering.com. There is a 50 word limit. Responses that are too long will be rejected.
written by Jeffrey L. Ihnen, P.E., LEED AP
Energy Efficiency Stimulus and Oversight
Most energy efficiency programs are required by regulators to be evaluated to ensure ratepayer money is being spent wisely and reported savings are being achieved. If only such oversight were to happen for the millions/billions/gazillions being shelled out to state and local governments in the name of energy efficiency.
State and local governments have Amazon-wide budget gaps to fill, and I can assure you that earmarks (dirty word) for energy efficiency will find their way to plug budget holes to keep buildings open, replace roofs, buy new lawn mowers and pickup trucks, and avoid staff reductions.
We in Wisconsin have already experienced this during the last recession. Starting in about 2000, most money collected by utilities for programs was turned over to Madison to be distributed from the ivory tower. The recession of 2001 resulted in a major budget gap (major at that time – it probably looks like a hairline fracture compared to what we have now). There, coming in from investor owned utilities, was a nice cash stream of $80 million per year. The state government swiped half of it. It pretty much eviscerated the energy efficiency programs and brought the industry to a slow crawl. Incentives were pathetic.
Thankfully, the Public Service Commission has taken control of cash flow now to help ensure ratepayer money is used to save energy, reduce demand, and delay/avoid construction of power plants and transmission systems as intended, rather than filling in a tiny portion of a humongous budget hole. Now energy efficiency incentives in the state are what I consider to be very attractive.
These federal funds should either be funneled through established credible program delivery channels such as utility programs or, in some cases, state governments (as long as it is out of reach of the legislative and executive branches), or there should be third party impact evaluation of projects emanating from block grants to local governments and other private sector grant writers.
If there is no oversight, vendors, consultants, engineers, architects, whoever can declare whatever savings they want. Or worse, as noted above, the funds will go toward new park benches and decorative street lights.
We welcome the oversight and technical review of our work because we are going to do things right regardless of whether others review our work. In a competitive market, the more technically astute and persnickety the reviewers are, the better for us. While LEED® takes its lumps for being too cumbersome, time consuming, and nit-picky, I think it would be a big mistake to slack off the review process. It will weaken a strong brand.
The bottom line is, if you have no rigorous third party review, you can expect pennies on the dollar of proclaimed savings.
written by Jeffrey L. Ihnen, P.E., LEED AP
