Newport News Waterworks Looks at SCADA to Meet New Challenges

Due to a number of current and future regulations, many of today's water and wastewater utility managers are forced to push the technology envelope. These technological challenges have a direct impact on an organization's operational effectiveness, reliability and bottom line performance. Virginia's City of Newport News Waterworks (NNWW) is a prime example of this principle in action. The NNWW, working with a team of information technology (IT) and engineering professionals from Rancho Cordavo, California-based Westin Engineering, has implemented an integrated supervisory control and data acquisition (SCADA)/process control system that provides utility-wide monitoring and control of both in-plant processes and remote distribution system facilities.

Newport News Waterworks is one of the 100 largest water utilities in the United States, and among the three largest in Virginia. It operates two surface water treatment plants (Lee Hill and Harwoods Mill) and provides drinking water to approximately 420,000 consumers. The service area covers approximately 250 square miles and includes the cities of Newport News, Hampton and Poquoson, as well as parts of York and James counties. In addition, NNWW provides raw water to the City of Williamsburg and major military facilities in the area.

Changing Needs Require Changing Practices
In the past, operation of one of the two water treatment plants (Harwoods Mill) and 17 remote sites (including pumping stations with ground storage tanks, elevated storage tanks, pressure monitoring sites, and raw water pumping facilities) in the service area were monitored and controlled by an aging SCADA system that proved increasingly difficult to maintain. The system's hardware and the software were out of production, and the system could not be expanded to support the planned development and expansions of the water treatment facilities over the coming decade. Further, the existing SCADA system did not have the expandability or performance capability to control and monitor a new reverse osmosis (RO) facility, a peak-shaving generator facility, and a residuals handling facility for land application that would be required to meet growing performance demands and expectations.

An additional driver in going ahead with this project was NNWW's participation in the Partnership for Safe Drinking Water Program. This program, sponsored by the American Water Works Association (AWWA), requires the utility to monitor water quality parameters by adding monitoring devices with serial communication ports, such as particle counters and other on-line devices, none of which could be connected to the existing SCADA/process control system. Through the utility's advanced public relations program and intensive planning, NNWW changed its disinfection practice from chlorine to chlorimes with less then a hand-full of customer complaints and inquiries.

To implement this change, ammonia feed processes and their associated in-line monitoring equipment were installed, tested and brought online in parallel with the new system. Also of note, the rebuild/upgrade of the Lee Hall Water Treatment Plant necessitated the implementation of a system that could readily integrate the operation of the two plants, which are eight miles apart, into a single unified SCADA/process control system.

For these technical and business reasons, the NNWW/Westin team moved forward with the project. HSQ Technology was selected as the system vendor. HSQ replaced vintage remote terminals units (RTUs) with modern intelligent RTUs at all of the remote sites. Additionally, two Allen Bradley programmable logic controllers (PLCs) were installed inside the Harwood's Mill Water Treatment Plant. Each PLC has redundant CPUs with bumpless hot failover.

Throughout the implementation of this system, there were numerous challenges and successes, including:

• Support integration of monitoring and control of planned development and expansions of the water treatment facilities over the coming decade.

• Hot changeover while the facilities remained in production from the existing control system to the new SCADA/process control system with no degradation in system performance, production, or water quality. This required a carefully planned and rehearsed sequential cutover of monitoring and control modules, process-by-process.

• Completely redundant process control servers, one server situated at each of the two surface water treatment plants. Each plant, as well as the remote facilities, can be fully monitored and controlled at either plant. By locating each server at a separate facility, the utility is better protected against both natural and man-made disasters. When the Lee Hall Water Treatment Plant comes on-line, it will be fully integrated into the SCADA system. The Lee Hall plant will be un-manned the majority of the time and controlled from the Harwood's Mill facility. The network between the facilities was designed to support both monitoring through the SCADA system and closed circuit television., its distributed processes capable of being monitored and controlled from each of the two plants.

• Integration of SCADA system with external Management Information Systems via the Department's existing Wide Area Network (WAN).

"Hello 21st Century, Meet My Infrastructure"
As is becoming the case with many of today's control system implementations, the design and implementation of the telecommunications infrastructure was critical to the success of the NNWW project. For this project, the Westin/NNWW team selected a combination of communications media to meet the project needs.

Between the two water treatment plants, a fiber optic cable was buried to support a wide band data link. In the unlikely event of a break in fiber optic communication, the system has automated failover to frame relay communication. Additionally, all I/O points in each plant communicate via campus-wide fiber optic Ethernet LAN between the control system and PLCs.

At the remote sites, the RTUs communicate with the primary SCADA server via frame relay circuits. An internal dial-up modem provides backup communication capability in the event of failure of the frame relay link. At three remote sites, where data is not needed on a continuous basis but only on operator demand, frame relay service is not provided, and communication is established solely by dial up modem.

The system comprises two virtual WANs, simultaneously sharing the telecommunications infrastructure. Every workstation at each of the two plants can log onto either the process control (PC) or Process Information (PI) virtual WAN. NNWW staff responsible for maintaining these systems use a network management software package that discovers new devices as they are added to the network and also reports failures of equipment as they occur. Having the ability to monitor and be notified (via alarms) of equipment failures, pinpoints the location of the failures and expedites the trouble-shooting and recovery process.

The degree of redundancy achieved should not be understated. At all times, one of the SCADA servers is assigned as the primary server, the other being in "hot" standby mode. The primary SCADA server continuously transmits all data and status changes to the standby server. In the unlikely event of a physical break in connection between the two plants, each server assumes all functions of a primary server for the water treatment plant in which it resides until interplant communications is restored.

NNWW Goes Online and Gets Results
Today, NNWW has the integrated SCADA/process control system up and running. For the first time in its history, NNWW staff have a fully redundant system that provides true "24/7" control and monitoring of their entire storage, treatment, and distribution system. This provides NNWW's customers with more reliable and higher quality service and product.

The staffs at NNWW and Westin have now taken this project one step further by developing on-line operations and maintenance (O&M) manuals for the integrated SCADA/process control system. O&M tasks now can be performed faster and more efficiently as operators and maintenance personnel will have a wealth of O&M information at their fingertips via the on-line system. Using laptops, both maintenance and operations personnel can connect to the network at remote facilities or via dial-up lines for direct access to the O&M manuals.

One of the greatest challenges faced by utilities abroad is how to keep documentation current as facilities and processes are added or modified. The O&M manuals are viewed using standard Internet browsers and are kept current by NNWW staff using standard off-the-shelf tools and in some cases free development software available through the Internet. The figure below is an example of one of the O&M screens.

Using laptops, both maintenance and operations personnel can connect to the network at remote facilities or via dial-up lines for direct access to the O&M manuals as seen here.

Starting with a vision of how the utility could function more reliably and efficiently, the NNWW/Westin team designed a solution that would meet NNWW's needs, and then delivered on that vision. Today, Newport News has one of the most modern process controls and SCADA systems in the country. Since these systems were designed using "open system" standards, different components of the system can be upgraded or replaced as necessary. Therefore, Newport News Waterworks will continue to have modern, flexible and expandable systems to meet their demands for many years to come.

About the Author:
Mike Smith is an independent consultant serving the utility IT and automation markets. He can be reached at phone 916-984-7430, or E-mail mikesmith@rcip.com.