The Importance of Post-Installation Equipment Monitoring and Preventive Maintenance

In a March 2013 Lab Manager article, “Prepared for Power Failure?,” the concept of using a pre-installation checklist to plan for your successful, on-budget, on-schedule installation of your new laboratory equipment was presented and emphasized.

With that and many other such lab systems now installed, you, the lab manager, and your staff completed benchmarking protocols; the lab’s operating procedures were also well documented. Your lab and the equipment have been operating successfully for many years before you experience a catastrophic power failure. Your instrumentation and/or key supporting equipment go down and you find yourself in a bind to complete your testing and deliver promised reportable results. How or why did this happen? You had everything under control and the equipment was performing flawlessly, including the supporting instrument power protection system (IPPS) you had installed to specifically avoid this type of inconvenience. You have preventive maintenance (PM) contracts in place for the key instrumentation/equipment, and you had periodic software and/or hardware updates installed to meet the latest specifications. You had everything completely covered—or did you?

Related Article: Product Focus: Laboratory Power

As you consult with colleagues, other laboratory managers, and/or operators in your academic/business network, you find that your circumstance is not unique, but it was 100 percent avoidable.

You learn from your experienced colleagues that there was a small, but critical, oversight in the lab’s operational plan, which did not include providing PM checks for every piece of equipment or instrumentation that draws electrical power. With newly gained operational insight, your lab team develops and performs a secondary review that now includes computers, servers, chillers, gas generators, hoods, refrigerators, freezers, lab automation (robotics), and more, not to mention that generally unnoticed, tucked-away IPPS you had installed to prevent such a loss of productivity. It now becomes obvious to all in the lab that anything that has a fan and a computer controlling it should be inspected and cleaned annually or, at a minimum, at least once in a 24-month operational period.

Figure 1. Battery state of charge.Courtesy of The Stack.Cleaning the cooling fans is a routine task in hospital labs and a definite post-installation checklist item in labs that have rigorous operating procedures for critical system inspection and maintenance. Equipment, such as the IPPS, performs its function because it has a power reservoir (battery set) to allow it to store energy to operate key equipment to which it is connected during a power failure. When the electrical mains (utilities) fail in the laboratory, the IPPS uses its battery set’s reserve power to keep the electrical equipment running flawlessly until the emergency generator (genset) comes online. The reason for the periodic PM inspection is to ensure that the fans are providing sufficient cooling to keep control circuit boards and microprocessors from overheating and to ensure that backup battery sets are also sufficiently cooled. Heat is the nemesis of any electronically controlled device. Additionally, due to vibrations in the lab’s floor, the electrical connections to any device that has battery backup needs to be checked for electrical connection integrity/security. These PM inspections include reviews of signs of arcing due to loose connections. When electrical power connections are bolted, their tightness should be checked to specified torque values. Most lab power equipment incorporates “maintenance free” batteries, but that doesn’t mean you can ignore them in terms of low/ high (under/over) line supply voltage, duty cycle, time and temperature in service, loose connections, frayed wiring, weak battery “jar/block” performance, and a host of other items that are verified during a thorough and rigorous PM inspection.

Related Article: Lab Refrigerators and Freezers in a Power Outage

Figure 1 is a simplified depiction of how a secondary (rechargeable) battery performs in power mitigation service, such as we find in a laboratory’s backup power system. Batteries are generally happy with a residual power capacity above 25 percent and become totally abused when deep discharged to 10 percent or less. If a nominal 12-volt (Vdc) battery is discharged to below 10 Vdcs, it has been compromised and will have reduced service life. If a battery is overcharged (>14 Vdcs), it will vent built-up internal gases and will also be compromised. Over-discharging and overcharging will destroy a maintenance-free battery and will make it a prime candidate for failure at a critical time—when you need it the most!

Figure 2. Example of a swollen and failed (broken seal integrity) battery case due to overcharging/heating.Courtesy of PanasonicCertified IPPSs incorporate “superior battery management technology,” including sophisticated charging and monitoring systems that yield up to 60 months of continuous service. The typical battery set used in power mitigation products is a sealed lead acid design of the highest quality, which features high rate output/duty cycle capability. Battery reliability and performance are critical in a laboratory application. These types of standby power batteries are also known as valve-regulated absorbed glass mats, gel-cells (silica-bound electrolytes), or simply maintenance-free (no water addition) types that can be mounted in any orientation. Again, the maintenance- free labeling doesn’t mean set in place and forget about them.

Over time, any world-class high-performance and quality battery ages due to constant chemical reactions cycling with progressive anode/cathode sulfating between charged and discharged states. Under such cyclic conditions, the battery loses its capacity and becomes worn (depleted) beyond service limits. A certified IPPS compensates for aging by specifying a reliable and conservative autonomous backup time in its original specifications. All products have a design service life, and the IPPS battery is designed for approximately 60 months in controlled laboratory service and duty cycle. Typically, batteries are specified for up to 500 discharge cycles before they are depleted and require replacement. Batteries are generally warranted for 12 months by their original manufacturers. Certified IPPSs and their batteries are warranted for up to 36 months, with extended warranties for up to 60 months available. In a typical year of laboratory service, the IPPS is continually running for over 8,760 hours. Five years of continual service is approximately 44,000 hours. Operating an IPPS or any battery over 45,000 hours is an invitation for future catastrophic battery failure, loss of productivity in the laboratory, and an unplanned and generally unbudgeted expensive repair.

Case study:

A well-known and very highly respected Northern California research university recently experienced a failure of its instrumentation system after nearly 48,700 hours (~5.6 years) of continuous service. Its IPPS, original equipment at the time of the instrumentation system’s commissioning, was unable to provide electrical reserve power during a mandatory gens-Set cycle test. The battery set was depleted and damaged similar to those illustrated in Figure 2. The expected and specified in-service replacement time for the batteries was between 48 and 60 months of operation (35,000-44,000 hours). The equipment the IPPS supports is valued at $750,000. The instrumentation system sustained a hard crash (unintentional immediate shutdown) because the soft-shutdown protocol between the IPPS and the instrumentation could not be activated, due to the battery failure coincident with a mains failure. A standing IPPS PM service contract was not in place for this laboratory. The rapid response time to bring this system back online was delayed more than a week due to the approval process and release of a purchase order.

Summary

The lab case study above is typical when advanced planning is not in play. It is an illustration of an avoidable shutdown and loss of laboratory operational and personnel time. The on-site PM inspection and remedial service performed at the university was less than four hours in duration, with the main battery set and both main cooling fans replaced. The IPPS unit was completely inspected per standard field service inspection and PM protocols. All batteries were replaced and tested in a matched set (see figures 3a and 3b). The IPPS was brought back online, with the next PM service interval set for 24 months. The extended life of the IPPS is anticipated to be another 35,000+ hours, or approximately ten years of total time in continuous service.

All lab managers strive to maximize their equipment utilization and minimize downtime associated with an outage. Preplanned PM cycles allow the lab to run efficiently and economically. Unplanned outages that take the lab off-line are costly and avoidable. Consider coordinating periodic and planned equipment PMs with IPPS PMs. Both equipment elements are off-line at the same time and generally are not considered an a-priori event, which are dependent upon one another.

The bottom line is that your lab can now produce consistent results with the lowest cost per reportable result, in a timely manner, within budget. After all, that is what business performance and continuity are all about— consistent performance and results with high customer satisfaction. Your colleagues will be impressed by how your lab operates at near perfection with minimal interruptions and how you made the right choice in picking the best instrument power protection system and service provider.

↑Figure 3a. Matched battery set—20-element battery conductance test: Mho vs. battery jar/block.
→Figure 3b. Matched battery set—20-element battery conductance test: Mhos and Vdcs vs. battery jar/block. Courtesy of Precision Power International, Inc