DWP Explosion and Fire Blasts KFWB Surge Protector, Takes Station Off Air
Wednesday, April 23, 2003

KFWB has tried to protect itself in a multitude of ways from being off the air. We have generators,
Uninterruptable Power Supplies, multiple transmitters, Surge Suppressors, you name it. Lost air time
in Los Angeles incurs an incredibly high cost in a short amount of time. But the nature of trouble
always seeks out a way to to slip though even your best laid plans...

Wednesday was a weird day. Or, at least, weirder than usual. When I pulled into the KFWB parking lot at 8:30AM the generator started up. It is not on an exerciser because the power goes out in our area of Hollywood every 30-60 days for 8-18 hours at a time, so it is load tested very regularly. I knew something was up.

Newsroom staff reported that the lights had dimmed a number of times in the last few minutes and that the lights had gone out momentarily about the time I pulled into the lot. Some of the computers had rebooted, but came up all right. I didn't think too much of it, as our 200kW generator has plenty of capacity to run the entire building for days at a time and some of the UPSes respond too slowly to keep that from happening. We had to reset a couple of microprocessor controlled things that got confused, but that was about it.

Robert Arak, KFWB Staff Engineer, and I began setting up to do the "Ask the Mayor" show in our new interview studio that is videotaped by the Los Angeles City's cable channel. They were doing their lighting setup and we determined that nothing they did caused the power problems. After a few minutes we got a comment about a smell. We figured it was the TV lights heating up their diffusing gels, but the smell got stronger as we went beyond the studio. We followed it down the stairs into the Server Room where the smoke was pretty strong.

We started gingerly touching panels looking for a hot one. We found smoke coming out of a screw hole in the LEA Surge Suppressor panel. I opened its front door and a plume of smoke belched out. Then some little thing in there that was white hot as we opened the door burst into flames, because we gave it oxygen with the door open. Robert gave it a couple of blasts from the extinguisher but the fire kept going. So it was time to close the door and choke off its air. Then we looked for the disconnect. At 9:12AM, we found it, flipped it and took the station off the air, of course. We waited a minute and rechecked the suppressor box and the fire was out. We closed the door again to be safe. Next, engineer Kevin Graham called 911 and we all went outside to cough out lots of smoke.

We went upstairs to explain what was going on and started figuring out a quick way to get back on the air. The lights were still on because the lighting circuits don't run through the surge suppressor, as we found out. We quickly determined that the surge suppressor only fed the Tech Areas like the Control Rooms, the Studios and Newsroom and most of the wall outlets. So some of the building infrastructure, including the air conditioning, was still operational. The phone system stayed on because it runs on batteries as its primary power source. The studio lighting WAS on the Surge Suppressor so the studios and control rooms were very dark.

During this time, the LA Fire Department arrived, agreed that the fire was out and helped us ventilate the smoke from the basement.

So we decided to stay. Then Mayor Hahn arrived with his entourage.

We brought the Remote Kit, which includes a Shure M367 mixer, upstairs and ran extension cords to our new Dodgers Control Room, which has a 5.7kVA UPS with a typical 1.6kVA load on it and 168 minutes of power left in it. The Shure mixer uses very little power, so it could run for a long time on UPS power. I grabbed a portable radio from my office to act as an Air Monitor.

We plugged a couple of mikes into the mixer, ran the mixer output to the patch bay spigot in Master Control for the 15k line to the transmitter and I announced "KFWB, Los Angeles", like the old engineer I am. I handed the mike to one of the Anchors and we were back on. Anchors Cindy Dole, Dave Williams, Tracie Savage and Suzanne Whatley did a great job reading stories printed out just before the newsroom went down and ad libbed admirably. Richard Saxton even gave his financial reports.

The lead story was about the big explosion and fire at the DWP power substation down the street that knocked out power for much of Hollywood. Our reporter, who was at the DWP substation, came back to the studio to do his report. Traffic Reporter Jeff Baugh landed and drove in to the station to do his reports.

Chief Engineers Ron Russ from KTWV and Lynn Duke from KRTH/KROQ/KCBS-FM arrived to provide additional technical support. We got out our two 5kVA portable generators used for Dodgers Promotions inflatables and ran extension cords into the studio to supply power in case the UPS ran out before we had regular power restored. At that point we still had not found any wall outlets that had power available, fed from the main generator which, was still running. And we wanted power that was independent from the electrical panels in the basement. The fuel was low on both generators, so Silas Marshall went out and got a couple of cans of fuel for them. While he was gone, the firemen filled up the tanks with some of their fuel.

We, then, did the "Ask the Mayor" show with host Jack Popejoy and Mayor Hahn standing in the newsroom for an hour without commercials and it went surprisingly well. We couldn't do much else, as various UPS batteries were going dead and computers, including the Enco workstations, were dropping every few seconds. The UPSes are sized to provide power to their loads for only a short time, between the loss of city service and when the main generator comes on line. There was never an expectation for them to operate for hours at a time solo.

While the Mayor show was going on, news crews from KTLA, KTTV and KCBS-TV came and shot video of our temporary setup with our news people and the Mayor. The Fire Department finished up and an electrician from Key Electric arrived. He bypassed the Surge Suppressor Panel and restored "normal" power to the Studios and Newsroom at about 10:50AM. It was decided that we would finish out the hour with the temporary setup and pull the patch and go back to the regular studios at the top of the hour. This gave us a few minutes to get the computers and Enco workstations up and running.

Everything went back to normal fairly quickly. By 2:00PM, we connected to the Dodgers feed from Cincinnati and aired the game on time and without incident. We were unable to find a single piece of equipment damaged by the power surge. The Surge Suppressor did its job, sacrificing itself. The servers were all a little finicky starting up, but they all are fully operational.

Since the City of Los Angeles is our electric utility, the Mayor asked us on his way out, "You're going to sue us, aren't you?" We all laughed and then said, "yes."

The electrician is looking for replacement parts for the Surge Suppressor and we had a "disaster recovery" company clean up the extinguisher mess. The DWP substation fire was finally put out about 1 PM, when the DWP workers began dealing with their electrical problems. We ran on the main generator until about 4:30PM when City power returned and we were satisfied it was clean. We had about 18 days of fuel left for the main generator.

In the end we had about 10 minutes of dead air and an hour and fifty minutes without commercials, less than some periods of our recent war coverage. The News and Engineering staffs, and even Mayor Hahn, pitched right in and did what it took to keep the station going even in the unusual conditions.

Special thanks to my engineering crew of Robert Arak, Kevin Graham and Silas Marshall; to fellow Infinity Chiefs Ron Russ and Lynn Duke; to General Manager Roger Nadel for his help and acting as a single point of contact between me and station staff and for taking the still pictures; to Asst. News Director Andy Ludlum for staying cool and being able to jumpstart news programming in the chaotic situation; to Bill Nesbitt for becoming the Shure mixer "board op"; and to Stan Chambers from Channel 5, who gave me a "Good job, son" as I passed him. It has been a long time since I have been refered to as "son".

A video of the story KTLA ran Wednesday night on the Hollywood power outage that included coverage of the emergency operation at KFWB is available at: http://www.labaseballnetwork.com/video.