Prepared for Power Failure?

Although the concept of a checklist has been regarded as a foundation of standardization and operational safety in government and industry services for some time, such techniques have generally escaped the scrutiny of the human factor industry. Most laboratory operators budgeting for and installing new equipment tend to overlook key elements of their projects. This oversight results in missing the full scope of the required tasks, which usually lands the laboratory or project manager on the short side of funding, time, materials, and human resources when contemplating and/ or installing a capital improvement project. Additionally, in day-to-day operations, the improper use, or nonuse, of a standard operating procedures checklist or approved prescriptive-testing protocols by laboratory operators is often cited as a contributing factor to unnecessary rework/retesting, release of quality control errors, delays, increased cost, and, worst of all, accidents. The origin of the modern checklist and operational guidelines we use today had its roots in a loss-of-life accident and the complete destruction of a key frontline military strategic weapon system prototype on October 30, 1935. Checklists are not new; however, complying with them is an ongoing battle in laboratory management.

There are many types of checklists. The one for this discussion is associated with the acquisition and installation of new laboratory equipment, namely the site preparation or preinstallation guide/checklist available from all major instrument OEMs. Our interest is to examine this seemingly mundane, yet critical management instrument, including its functions, usage, and human limitations when preparing the lab to accept new equipment.

Whether the laboratory adopts an instrument OEM’s checklist or develops its own project checklist for new equipment, certain sociotechnical factors, such as the laboratory’s culture, internal operations/resource management, experimental design or production pressures, and other time constraints, influence the design and usage of the checklist. While the focus of this article is on the laboratory industry, most checklist concepts are universal and apply equally well to other high-risk industries such as maritime, air-carrier and rail transportation, power production, mining, refinery operations, weapons systems, space flight, continuous-process manufacturing, human factor, and veterinary health care/science.

What is a preinstallation checklist?

The major function of the site preparation or preinstallation guide/checklist is to ensure that the various laboratory staff, facilities management, external contractors, and executive/financial management understand the scope of the program and properly configure the laboratory to accept the new instrumentation system. The preinstallation guide forms the basis of installation and operation standardization in the laboratory before, during, and after the start-up of the instrumentation system. Only when the instrument is properly installed and deemed functional and staff training is conducted can benchmarking begin. Once benchmarking criteria have been satisfied, full production operations can initiate. There are many types of checklists that the laboratory manager should make available to her or his team in order to successfully and efficiently run the laboratory. This article’s focus is on the installation and start-up of the new instrumentation system, with particular emphasis on electrical facility improvements required for the lab.

An industry consultant and probably a seasoned laboratorian with extensive experience in new equipment acquisition and laboratory redesign will immediately recognize there is a void in most lab managers’ skills tool bags when it comes to knowing all there is to know to get a new system specified, budgeted, acquired, installed, benchmarked, and finally declared approved for production. Most lab managers do not have the experience required to smoothly manage a major revision to a lab’s facility infrastructure, often required to support a new capital instrumentation system. This is where developing the skills to manage the experts in each respective area of the project becomes very important. Most laboratory managers are generally not interested in learning the fine details of how infrastructure improvements are made. This initial decision to abdicate understanding of infrastructure is profound at the foundational level. It affects all other decisionmaking steps that are required in effecting a major capital improvement, such as a state-of-the-art instrumentation system.

Electrical infrastructure improvements are an area that consistently tax laboratory managers for three reasons:

The language of the electrical industry is regulatory and industry specific—arcane at best and only understood by few in an almost mysterious or secret way.
Understandably, most laboratory managers focus on their science and lab operations. They have not taken the time to understand what electricity is, how it is delivered, distribution impacts, that it is a common and unreliable utility, and whether the new instrumentation system should have its electrical power mitigated or even be connected to an emergency generator system.
Electrical infrastructure improvements require designbuild engineering, a bidding process, long lead-time scheduling, facilities management, licensing, permits, multi-trade contractor interfaces, municipal inspections, and, for a large project, an authorization-tooccupy, aka an occupancy permit indicating that the premises (laboratory) is “suitable to occupy” from health and safety points of view.

Reviewing the details: The electrical section of the preinstallation guide

Unless your laboratory staff has extensive and frequent experience with electrical installations for complex instrumentation applications, consider consulting with laboratory construction specialists, laboratory design/build contractors, and suppliers of certified instrument power protection systems. As you review the electrical requirements sections of the site installation/preparation guides below, you will note specific requirements for the instrumentation types.

Select site preparation/installation guide excerpts (Figures 1 - 5)

Figures 1 - 5 illustrate excerpts from various capital intensive and high-performance instrumentation systems. The key electrical terms and elements that are not common or not specifically defined in these reference documents, but must be understood, are listed below.

To truly understand the intricacies of instrument electrical power, the key electrical areas to focus on include:

Voltage Amplitude Specification
Maximum allowable percent sag (5%)
Maximum allowable percent swell (5%)
Phase (single or three/triple Ø) (1-Ø or 3-Ø)
Supply voltage is not guaranteed by the utility
Voltage waveform type (pure sine wave)
Supply Voltage for various instrumentation elements
Matched voltages for main system elements, accessories and peripherals
Safety Standards, i.e. national and local
North American examples (cUL, UL)
International examples (CE Marked, EN, IEC)
Check local codes and country specific requirements
Frequency Specification
Operating frequency (50 / 60 Hz)
Allowable frequency variance (± 1 Hz)
Waveform Specification
Maximum supply voltage total distortion (3%)
Maximum supply voltage distortion by a single harmonic (≤ 3%)
Current and Power specification
Typical peak current (3 x nominal)
Crest Factor (current) (3:1 minimum)
Minimum power (apparent power) rating in Volt – Amperes (xxxx VA)
Circuit breaker protection (MCB) types (slow-blow, et al)
Power Quality Description and Connection Limitations
Appropriate branch circuits and power delivery
Smooth, clean and free of transients – not exceeding x% distortion
Circuit type limitations
Not connected to other devices with large, frequent loads or inductive loads
Dedicated vs. isolated branch circuits
Galvanic isolation
Bonded neutral circuits (National Electrical Code / Local Codes)
Common ground requirements for instrumentation elements and computers
Patient/personnel contact approved, or not approved (usually not)
Personnel safety
Electrical connection devices (receptacles and plugs) types and ratings
Emergency Power Off (EPO) (local code)
Direct disconnect capability (remote controlled if possible)
Software / remote or local controlled capability
Peripheral device connectivity limitations
cUL, UL, EN, IEC or CE Mark approved and safety labeled
No extension cords
No power strips – especially surge protected devices for use with a UPS or IPPS
Step down (transformer) types and consistent facility supply voltages
“Delta” vs. “Y” (aka “Star”) transformer designs and supply voltage to instrument sections
208/120 Vac, 60 Hz, single-phase (1-Ø) is the standard in the USA
Note limitation for your specific instrumentation type
Consider higher voltages and appropriate combinations for your specific application Autonomy Time Requirements
Required backup / running time required for completion of testing protocols?
Provisions for Emergency Generator (Gen-Set) connections to an Automatic Transfer Switch (ATS)
Provisions for mitigating Gen-Set power distortions
Transient Voltage Surge Suppression (TVSS)
Alternate planning and Risk Management if Gen-Set capability is not available
Critical Power Quality and Mitigation Devices
Which type of UPS or instrumentation power protection system (IPPS) should be specified to protect this investment, production time, productivity, results integrity and—most of— all the reputation of the laboratory?
Commercial-Off-The-Shelf (COTS)?
Computer (IT) grade?
Lowest cost possible?
Certified and purpose built for the specific instrumentation application?
Instrument power protection system (IPPS)
What is the lab’s economic risk for downtime and loss of reputation?
The electrical industry power delivery standard (goal) is 99.99% uptime delivery
Electrical power is regulated to frequency delivery standards
Not voltage delivery
Not waveform delivery (free of harmonic distortion)

The electric utility’s (power generating company’s and/or independent system operator’s) goal is to provide reliable power 99.99% of the time. While 99.99% sounds impressive, it results in a minimum power delivery void of 53 or more minutes of productivity loss, which can occur at any instant day or night. This is a risk management checklist item. This statistical delivery goal can repeat itself thousands of times per year, resulting in significant downtime and severe instrumentation damage, loss of data, and, worst of all, loss of customers due to on-time deliverables failures.

As you develop your new equipment acquisition and installation budget, avoid “management blind sidedness” by thoroughly reviewing the OEM’s checklist and your own unique situation’s checklist in order to install and properly operate your intended instrumentation system. Spend time consulting with industry experts and colleagues who have installed identical instrumentation for similar applications and discuss their operational experiences. Evaluate whether your facility is sufficiently equivalent to your reference standard to accept a “same as” approach for a plugn- play installation. Investigate with your facilities engineering department the other types of electrical loads that may be on the intended circuit of your new equipment. Conduct a thorough analysis of the contingencies that are reasonable for your specific laboratory operation. A helpful starting point for a lab’s “Preparing for a Power Failure in the Lab” checklist was developed by Precision Power International, based on its over 35 years of instrumentation power protection experience. The checklist is available at http://precisionpowerinternational.com/wpcontent/ uploads/2012/12/PPI_Checklist.pdf.

Checklists help labs learn from the past and look toward the future

To be truly successful, laboratory managers need to develop an understanding and mastery of contingency management planning, daily operational efficiency goals, human resource management, and, even, electricity, which completely dominates our modern society.

Superstorm Sandy’s impact on the eastern seaboard of the U.S. in October 2012 will have lingering infrastructure effects well into 2013. During Sandy—the largest physical Atlantic hurricane on record with a wind field of nearly 1,000 miles in diameter—some laboratories were well prepared with appropriate electrical infrastructures, while others suffered devastating losses. One well-prepared facility in the midst of the devastation on New York State’s Long Island was Cold Spring Harbor Laboratory (CSHL), a major research center. As reported by various journals, CSHL’s success was attributed to advance planning, appropriate infrastructure development, emergency generator capacity, “smart monitoring and reporting” technology, and sufficient food supplies for up to five days. This laboratory is not only an exception in the proper use of checklists and advanced management planning, but also a hallmark of experience in laboratory operations learned from major storm events of the past.

While CSHL followed its institutional checklists, many laboratories did not and therefore suffered costly consequences, including laboratory instrumentation damage, sample loss, and significant lost production time. Looking beyond Sandy, following checklists and contingency plans enables a laboratory manager to keep the operation running in stride with very limited downtime and few, if any, unplanned expenses for repairs due to the occurrence of a planned exceptional event. This is a clear case for checklists allowing management to focus on exceptions, since the operation is now under the control of experienced staff using approved protocols.

Raymond L. Hecker, consultant, power protection solutions for the laboratory and life sciences industry, can be reached by email at rlhecker@precisionpowerinternational.com or by phone at 949-951-6784.

Product News