Within a research environment, it is common for materials to be unique and quite literally priceless. Moreover, many of these need to be maintained in temperature-controlled environments.

It is an unfortunate fact that any controlled environment will fail at some point in time. The question is how organizations should prepare for this and manage the risk of loss. In this article, I will be referring primarily to ultra-low temperature (ULT) freezers as an example of critical equipment with valuable contents, and how we developed an equipment monitoring system to assist in our management of this risk of loss.

The development of an equipment monitoring system should be treated like any other risk assessment. It is only sensible that organizations monitor how they are insuring against loss through failure of critical equipment and how their policies reflect the replacement value of the contents versus the relatively insignificant cost of the piece of equipment itself.

When a freezer fails, it may be holding one (or a combination) of the following:

• A laboratory’s stock of commercially available, but often high-value reagents. In this case, the financial exposure and inconvenience of loss needs to be understood and managed.
• 15+ years of samples that cannot be replaced. 

How do you adequately insure against a loss of such crucial samples? This latter scenario should alarm those within an organization who manage exposure — whether it be the “business loss” insurer or the researcher whose research (and livelihood) depends on the integrity of those samples. Both the insurer and the researcher will be looking for physical and operational systems that effectively eliminate the risk of exposure to such loss.

At our institute, the vast majority of ULT freezers contain unique biological specimens. Given that a freezer will fail at some point and that the consequence to our research will be major, reference to a risk matrix (Figure 1) indicated we were exposed to an extreme risk.

Technology

The simplest form of an equipment monitoring system should produce an alarm when the parameter being monitored moves outside a predetermined range. Our previous equipment monitoring system was based on industrial Programmable Logic Controllers (PLCs) with Supervisory Control and Data Acquisition (SCADA) monitoring. Although robust and reliable when installed, the system was now obsolete (ten years old).

Additional features that we felt were desirable in any system we would implement were based on the limitations we had found with previous PLCs and SCADA systems. Key issues to be addressed included:

• Technology obsolescence (exposure if system failed) of a ten-year old system
• Poor technical support
• Non-intuitive user interface
• Un-informative alarm messages
• Reliance on proprietary PC-based software interfaces
• The desirability of ‘Open Platform’ technologies that allow equipment and services from a variety of suppliers to be connected cost effectively
• Remote access capabilities (preferably via a web interface).
• Alarm path flexibility
• Alarm messages with useful information
• Development of in-house expertise to expand and maintain the system

Figure 2: System Architecture

Where do alarms go?

The alarm function was of vital importance. When an alarm is triggered, we want the owner of the freezer to know and respond. If the owner is not interested or wishes to defer responsibility, then we must question the merits of the ULT freezers inclusion on the system. Our ambition was to have alarms that could be sent in both e-mail and sms forms and the flexibility to rapidly alter the pathway as required to ensure that the right people received the alarms depending on the time of day, week, or year. A severe limitation of our previous system was that the alarms were sent to a pager and the responsibility of acknowledging the alarms rested with that individual.

Review

Armed with our wish list, we went to the marketplace and reviewed what was available. During this process, we reviewed off-the-shelf monitoring systems and investigated the potential of building our own customized system.

After a comprehensive review, we were able to discount ready-built systems as we felt they did not offer enough of what we wanted. We did, however, have favorable discussions with companies that could build, to our specifications, a system using equipment adapted to meet our needs.

A clear message to put across in this article is that it is far too restrictive to limit any technology review to only what is commercially available at the time. Our experience strongly advocates that a fundamental question to ask is the following: can we build something new and better?

Budgets

Project costs are built from the job complexity, available technology, and the expertise of the provider. A cost penalty is associated with tailoring a product versus off-the-shelf with a large component being the time required to develop graphics and the interface for the user. The hourly rate of a skilled engineer can be significant.

The project budget should be the result of a risk assessment, a review of key issues to be resolved, and what solutions are available. A case can then be made to fund the solution that is correct for that set of circumstances.

Development

We opted to work with our Building Management System (BMS) maintenance contractor, Integration Control and Engineering (ICE), who proposed adapting BMS equipment to our needs.

There are many similarities with how a BMS works and what we wanted in our solution. A BMS is a system designed to take numerous inputs, manage that information, and provide a useful end product. We wanted to take numerous freezers, monitor them in real time, and when required, send out an alarm.

The ICE solution we went with is based on a Web — embedded, multi-protocol, controller connected to open architecture devices. The Web-embedded device meant that the system no longer relied on a PC loaded with proprietary software and operating systems for its functionality. In this case, the controller could be connected to the local intranet and viewed by a standard Web browser.

The installation of a multi-communication protocol device also ensured the long-term viability of the project. As the system grows, the ability to incorporate other equipment and gather information at a higher level will now be a possibility.

As with all IT-based products, we checked that it was technically suitable for our needs, could be supported long term, and could be networked. The development process was consultative (including interdepartmental communications) and met our desired outcomes.

Achieving clear communications between the client (us) and the ICE provider was challenging as each party was from a different background and skill set. We are not controls engineers and being able to document clearly our exact needs took some work. Equally, we were careful to ensure we had access to updates and progress reports so the contractor did not inadvertently go down the wrong path and waste time and effort.

Figure 2 displays the architecture of our system as it was designed and installed. To allow a Mac (common computer platform in our institute) user to remotely dial in to the system, we are using third-party software.

Timeline

Any organization thinking of going down the same path must allow for a significant investment in time if the end product is to be truly tailor-made. Our review process took two months to complete and the design/development/install process took an additional three months.

The tailor-made solution

We developed and collaboratively produced a system that gave us:

• New technology with an emphasis on future needs
• The capability to expand the number of monitored items as the institute grows
• Increased accountability across departments to respond to alarms
• A simplified interface between system and operator
• A system with limited points of failure
• A system we could control and manage in-house 

It was earlier noted that a goal of the project was to have increased coverage in where alarms were sent. In brief, we developed a system where alarms were sent to the “owners” of the devices during business hours via e-mail and to security after hours via sms.

Future changes

This article has focused on ULT freezers as the monitored device. However, the system was developed with a view to monitoring any controlled environment that could fail — these include:

• ULT freezers
• Freezers/Refrigerators
• CO2 incubators
• Liquid nitrogen dewars 

The system we developed allowed us to bring “in house” many of the skills required to manage and maintain our system where we were previously dependent on outside expertise.

Conclusion

Careful planning of the roll-out was critical to the success of the new system in that a single loss at this time might have seriously jeopardized its acceptance. To this end, we placed great emphasis on:

• Availability of skilled staff to quickly address system glitches
• Guiding staff through the changes in alarm notification procedures 

As it was at the selection phase and still is, there are numerous other systems that can be adapted for similar needs. I am also sure that there will be continual development of off-the-shelf systems for this niche. For the reasons outlined above, I hope it is clear why we chose the path we did; we hope other laboratories can also use this experience to go outside of the commercial market place and think about building a custom solution.

Thanks to Dr. Jeff Freeman and Mr. Robert Lane for critical review and comments on this article.

David Keenan is Operations Manager for the Garvan Institute. Garvan Institute, 384 Victoria Street, Darlinghurst, Sydney 2010 NSW Australia; d.keenan@garvan.org.au; www.garvan.org.au.