IceStorm 98 Study

Q. When or how did your department realise that there was an emergency?
Wood: I don’t think there was an actual departmental decision made, I tend to log in the morning to see what is going on. I guess it was Thursday morning when I got up and realised that there was a lot of ice on the ground. I connected in or dialled in to the system at Queen’s. I was in the middle of logging in to the monitor and my power went off and I realised that there was a problem. Luckily, my phone system was still working and I called in to work and found that no one was answering the phones. I tried again on the Friday and again, no one answered the phones but I finally managed to get a hold of Andy Hooper who is my supervisor. After I spoke to Andy I called one of our employees, Bob, who lives in the city and we asked Bob if he would come down and check a few things. The power was back on when Bob got in, and found Dupius Hall had power and that most of our systems were up and running. We have a data switch - the PACX -, which needed a fair amount of work to get running again but the network itself, was running. There were a number of buildings around campus that had no power, mostly the residences and this building (Mac/Corry) and anything that is fed out of this building was off. As the power came back on in the various parts of the campus most of our equipment came back on. We had one piece of equipment here in M/C that controls Ellis and Jeffery Halls and some of M/C that did not come back up properly and I think on Saturday, we changed that piece of equipment.

Q. Was that a server?
Wood: No, it’s a LAN emulation hub. It’s a device that controls where the signals go on the fibre. Once we replaced that, we had about 90% of our network up and running by Saturday. It was still the residences that were off line. I think by Tuesday their power came back up and they were up and running. Another employee, Geed, had to go and reset a few things. Everyone was back to work by the Thursday and everything was back on line by that day. As far as we were concerned, it was not an actual emergency except for some power being off for a short time.

Q. Were you ever asked by the City of Kingston to provide some outside communication hook-up?
Wood: No, we were not asked.

Q. Did they utilise any of the Queen’s network?
Wood: No, we were not asked that either.

Q. Were you aware of the problems that McGill was having?
Wood: No, only what you had mentioned to me earlier. I know Montreal was in a terrible state and much worse than we were but I didn’t have any specifics as to what was going on.

Q. There was a rumour going around that because the Queen’s network was still up and running and able to get through the ONET that someone set up an online communication with parents. Is there any truth to that?
Wood. Not that I know of. We didn’t set it up.

Q. Was there ever a concern on the issue of money? Damage to the equipment, how far do you go? Or was it a wait and see situation?
Wood: It was always our concern when something is not working, our network is quite critical on this campus and I think our attitude is what ever it takes to get it up and keep it running is worth it and what ever it costs to get it back, we spend. We were told that overtime was not a problem, whether it was working 24 hours a day, or whatever it took to get the system back and running.

Q. Who gave that directive?
Wood: It came from Stan Yagi, our Director of ITS and his authority came from the VP he works for.

Q. Have you had a chance to look at the cost of overtime?
Wood: Yes, it was 4 hours for two employees and mostly on the Sunday.

Q. As far as communication went, did you have any difficulties getting in touch and keeping in touch with your employees?
Wood: No, my phone was out for about a week but I had a cell phone and didn't have trouble getting through to Bob. This is a bit of an anecdote, but on the Sunday, I came into the city after moving my family in because it was getting desperate where we were (about 1 or 2 in the afternoon). I was quite bored and decided to come into the office and there was Bob and the Chief sitting there as bored as I was. So the three of us were there and we spent a couple of hours just checking the system to see if everything was ok.

Q. Did you have an emergency plan in place? Does ITS have a disaster recovery plan?
Wood: There is a disaster recovery plan in place, it's probably about five or six years old but we didn't have to go to it.

Q. Could you briefly explain what the disaster recovery plan is?
Woods: I think part of it is getting the main administration up, such as Richardson Hall. They have to be back on line.

Q. Did Richardson Hall ever go down, or was it simply the lack of power that prevented their operation?
Wood: It was simply the lack of power.

Q. What do you think was the priority was as far as the network was concerned? Were there security issues with respect to the public sites?
Wood: We don't need power to secure the areas, they all have locked doors, which remained locked in the buildings that were without power. The machinery in the sites is physically locked down but I do recall that some sites are secured with alarm systems and of course they wouldn't work. So it could have been a case of a smash and grab with some bolt cutters.

Q. What about the internal security of the machines? In particular, the potential damage to the electronic equipment from sitting in cold buildings?
Wood: These are things that I don't do. I suppose that it is Doug Crowe's responsibility as he supervises the sites. Our equipment will operate in temperatures of minus 20C. The only concern with things being cold and then getting warm is condensation.

Q. Have you seen any damage reports?
Wood: No, none.

Q. You mentioned that most of Queen's lines are 99.9% underground. Do you think that contributed to the network staying up?
Wood: Oh, for sure. We had no overhead wire to get damaged. Most of it is fibre.

Q. Could you take a few moments to explain how each residence is connected to the backbone and how it gets to the ONET and out?
Wood: Queen's is an Ethernet system at this time. It is distributed from a central hub in Fleming Hall, on fibre-optic cable through the duct work that's underground in our steam ducts. Queen's is centrally heated through the central heating plant, which distributes the heat throughout Queen's. Someone about 20 years ago had the foresight to install communication wiring/conduits along side the heating ducts. So when we moved from a telephone communication system to a network running fibre-optic we were able to run the fibre all over the campus. All of the buildings, including residences, are linked eventually back to Fleming Hall with the fibre optic. We have another fibre -optic system that runs from Fleming Hall over to Dupius Hall and then it runs to the ONET. That's also an underground link provided by Bell Canada. So we get to Toronto from any building on campus. The Ice Storm had no real impact on the network except for the buildings that were without power and the power being removed from the various devices. As soon as the power was restored, the system was back up and running. The network is described as this; we get into the building with the Ethernet signal on this fibre backbone. In each building we have a device called a router which looks at the traffic on that backbone and decides whether that traffic is destined for that building. Based on that decision the traffic is routed to the appropriate receiver/building. It's like a huge broadcast system. The analogy is radio broadcasting except they use airwaves. The router is to the radio receiver. If you're tuned to that station you receive the signal.

Q. Would you have put any other plans in place?
Wood: We have some devices called UPSs, (Un-interruptable Power Supplies) on the important nodes. These will keep our network running for up to two hours. It would be nice to have more of these devices installed because you can keep your network running. On that note though, if you don't have power to operate the systems after an hour they are kind of useless. However, we could have used them on a few critical pieces.

Q. So at least, having a few more UPSs would have allowed ITS staff time to get the equipment and safely shut them down?
Wood: Yes, it would give us an extra hour but most UPSs have a cycle down system that would shut the servers down. It would have preserved the integrity of the equipment. One thing that would have also helped would have been a generator in Dupius Hall. We have one in Fleming Hall where most of our distribution services are but our central services, main servers, and links to the outside world are housed in Dupius Hall. Dupius does not have a back-up generator. It just has a few UPSs.

Q. Is there any long-term plan to put one of those in Dupius?
Wood: It's already happening. We’re also putting together a new disaster recovery plan, we’re going to install alternate paths of fibre optic around campus so that we can get to any building from Dupius. There are two major streets on campus that we cross and we only cross them in one place and they are Union Street and University Ave. It is conceivable that an earthquake or road construction could sever that link. Splicing fibre together could be a multiple day job so we're putting in place an alternate path. We're looking into at least a generator or a contract with a power supplier to provide backup power to Dupius.

Q. When do you think that might be implemented?
Wood: We've hired a consultant to work this out and actually he was hired before the Ice Storm but now is incorporating this into his proposal. The Ice Storm spurred him on.

Q. Do you think the Administration is going to agree to all this?
Wood: Yes, there is a mandate to proceed.

Q. Do you have any emergency training?
Wood: That's what we do for a living, disaster recovery. It happens everyday where there are little emergencies happening and we respond to them. This was just a larger scale operation. I don't want to give you the impression that this was a major emergency for us. It wasn't. We were not impacted to the same degree as the City or other parts of the province.

Q. Did you receive any help from the province or federal government?
Wood: There was no need. We didn't have any requirements.

Q. What about contact with the media?
Wood: Not through me. They may have contacted Andy or Stan but then my phone was out.

Q. What about the levels of stress?
Wood: In my job, most of the stress comes from the telephone and since my phones were out, and there was no one on campus to call me, I was pretty low-key. We were at work for about 4 days to make sure everything was working and since the phones weren't ringing, we were able to get a lot done. It gave us the room to verify everything was functioning.

Q. Do you recall anything that was particularly funny or intriguing? Was the union co-operative?
Wood: There were no union problems or questions. Everyone just did what he or she had to do. The guys were told that they would be paid overtime and there were concerns.

Q. There was no concern about members’ safety or working conditions?
Wood: No, none at all.

Q. How did you decide who of your key people would come in? Was it the case of who could get in or was it based on specialisation?
Wood: The only criteria that we used were after I talked to my people, and said, "if you can get in, it would be nice" but it was based more on proximity. Who was closest.

Q. Would you have any documents to share with us?
Wood: Everything that we did was verbal. I can get you a copy of the draft of the emergency plan that the consultant is working on. John Travers is in possession of that document. Reg Locke wrote the original one (and he has since retired).

Q. Do you now have a sense of the overall costs?
Wood: The only costs were the overtime, which was a couple of hundred dollars.

Q. Is there anything else you'd like to add?
Wood: Well, it's something that I would never like to see happen again. I'm an optimist and I think that it won't happen again. Everyone is going to go around and spend a whole bunch of money to be ready for the next time and it's rather moot. I'm fifty years old and I've never seen this happen before and I doubt I'll see it again.