It seems like hardly a day goes by that doesn’t see some news story splashed across our feeds that has something to do with Elon Musk and one or another of his myriad companies. The news is often spectacular and the coverage deservedly laudatory, as when Space X nails another double landing of its boosters after a successful trip to space. But all too often, it’s Elon’s baby Tesla that makes headlines, and usually of the kind that gives media relations people ulcers.
The PR team on the automotive side of Tesla can take a bit of a breather now, though. This time it’s Elon’s solar power venture, Tesla Energy Operations, that’s taking the heat. Literally — they’ve been sued by Walmart for rooftop solar installations that have burst into flames atop several of the retail giant’s stores. While thankfully no lives have been lost and no major injuries were reported, Walmart is understandably miffed at the turn of events, leading to the litigation.
Walmart isn’t alone in their exposure to potential Tesla solar problems, so it’s worth a look to see what exactly happened with these installations, why they failed, and what we as hackers can learn from the situation. As we’ll see, it all boils down to taking electrical work very seriously and adhering to standards designed to keep everyone safe, even when they just seem like a nuisance.
A Solar Panel on Every Roof
SolarCity installers on a residential job, before the Tesla takeover. Source: Green Car Reports
Solar panels have always been a key part of Musk’s vision for Tesla, which seeks to eliminate the need for fossil fuel transportation. Tesla’s battery-powered cars were designed from the outset to be charged by the sun, with rooftop solar arrays installed everywhere.
And so the idea for a company called SolarCity was dreamed up by co-founders Peter and Lyndon Rive during a road trip with their cousin, none other than Elon Musk. Musk would serve as the new company’s chairman and would be actively involved in running the company, whose early business model involved door-to-door sales of rooftop solar arrays directly to residential customers. Early adoption in the USA was partially fueled by federal and state green energy tax credits which partially offset the cost of installing an array. Later, SolarCity began hawking installations with no up-front costs that would be paid over 20 years with a cut of the power produced. That ended up being attractive to enough customers that business — along with SolarCity’s debt — boomed.
At about this time, SolarCity got into the commercial market, with large installations at businesses looking to both reduce their energy costs and burnish their green reputation with customers. Intel and eBay were early customers, as was Walmart. The retailer would ultimately enter into agreements with SolarCity to install and maintain arrays on more than 240 of its stores.
Burning Down the Store
According to Walmart’s suit (downloadable PDF), the first rooftop fire was in 2012, atop a store in Long Beach, California, which caused $90,000 in damage. It’s not clear what the root cause of that fire was, but 2016 fires at stores in Milpitas, California, and Lakeside, Colorado were blamed on “faulty connectors”. A fire at a store in 2017 was pinned on “Tesla’s faulty installation of conduits”. Tesla was blamed specifically in this fire because SolarCity had been purchased in full by Tesla in 2016 for $2.6 billion.
Then, between March and May of 2018, a series of fires occurred at three Walmart in rapid succession, first in Beavercreek, Ohio, then Denton, Maryland, and finally Indio, California. A firefighter was injured, customers were evacuated, sales were lost, and repair bills mounted. Having had enough, Walmart demanded that Tesla de-energize every surviving solar array on every Walmart. Even that was not enough to stop the carnage, as a de-energized array in Yuba City, California, somehow started arcing and burning. How that fire started is a mystery.
Remains of the fire at the Indio, California Walmart. Source: Supreme Court of the State of New York, New York County, Case No. 654765/2019
Walmart hired consultants to comb through the debris of these fires and to inspect the workmanship of intact arrays and found quite a few issues. Bearing in mind that lawsuits are a laundry list of claims that the plaintiff uses to put the maximum blame on the defendant, and that the consultants were hired by Walmart and are therefore likely to be biased toward their client’s position, the investigation revealed some pretty shoddy workmanship, which Walmart alleges was due in part to SolarCity having overextended itself during the huge increase in its business during the time these installations were performed.
Getting Torqued?
Among the problems that Walmart’s consultants discovered by investigating the surviving solar arrays were improper grounding, poor wire management leading to insulation abrasions and wear, and lack of as-built drawings and proper documentation. But the most glaring errors alleged by the inspectors were the presence of hotspots in the arrays, and improper installation of the connectors used to string together the solar panels.
This one got a bit toasty. Evidence of a hotspot on one array. Source: Supreme Court of the State of New York, New York County, Case No. 654765/2019
Hotspots in photovoltaic arrays occur when one or more cells in a series-connected string of cells are underperforming for some reason — say, by being shaded by leaves or dirt. The shaded cell or cells can then become the current limiting element in the series circuit, which can lead to reverse-biasing of the bad cells. This essentially dumps all the power from the good cells into the bad cells, heating them up to possibly the point of failure due to melted solder joints, cracked silicon, and, as appears to be the case with the Walmart fires, ignition of the materials used to encapsulate the cells. Walmart’s investigators discovered multiple examples of hotspots in intact arrays, which they allege Tesla either missed or didn’t bother to look for.
Walmart’s consultants also found fault with the installation of connectors. Like most solar arrays, the Walmart panels were strung together with MC4 connectors. These are industry-standard connectors manufactured in the billions, and are designed to provide fast, weatherproof connections in the field. They consist of a crimped metal male or female terminal, a locking weatherproof shell, and a strain-relief gland for the conductor. Walmart’s investigators provide a photo of one MC4 connector that shows the threads of the strain-relief partially exposed. The lawsuit contends that the installers failed to use “a special tool known as an MC4 torque tool” to tighten the glands. Stäubli, one of the leading manufacturer of MC4 connectors, recommends the use of a calibrated torque wrench to achieve a final torque of 3.5 to 4.5 N-m in its MC4 product guide, a tool which apparently was not in the Tesla installer’s toolkit.
Improperly torqued MC4? Hard to say. Source: Supreme Court of the State of New York, New York County, Case No. 654765/2019
It’s hard to say whether the exposed threads in the photo really mean that the glands weren’t properly tightened, though. Weatherproof glands like these work by squeezing a flexible sleeve against the outer jacket of the conductor, and since MC4 connectors can be used with wire between 14 AWG and 8 AWG (1.5 mm2 to 10 mm2), the amount the gland will settle on the body threads will vary. Since the point of the gland is to provide strain relief and protection from water and dust intrusion, Walmart’s consultants may have made a better case by showing compromised connections inside of an allegedly incorrectly torqued connector. Sadly, they did not.
In the final analysis, it’s hard to avoid the impression that SolarCity and Tesla did a poor job of installing and maintaining these arrays. What comes from this is up to the courts to decide, but for now, anyone who does solar installations would be well advised to pay attention to wire management, connector installation, and array cleaning to make sure a similar fate doesn’t befall. And anyone who has a SolarCity array on the roof of their house might want to have it inspected very carefully in the very near future.