Video security travels on existing control system
The task was 'simple.' Put cameras at 32 remote sites and bring back video clips over the utility's radio network
By Al Larson and Steve Rubin
On 11 September 2001, terrorists attacked America and the U.S. quickly acknowledged vulnerabilities at our airports, borders, food supply, and water supply systems.
Soon after, the government required vulnerability assessments (VAs) for all municipalities—with large cities required to go first. In 2002, Madison Water Utility (MWU) in Madison, Wisc., underwent its VA and saw a need for video cameras at many locations, including 32 remote sites.
Two obstacles stood in the way of Madison meeting this need:
Technology, as in whose cameras, network, and communication system to use? How can video work with our SCADA system?
Money, as in who will pay to protect Madison’s water supply?
While this is a high-level synopsis of the challenges Madison faced, many municipalities still face these same obstacles.
Our experience shows video security surveillance is possible over an existing SCADA network at a reasonable cost, and that funding is available to cover most of the expense.
Funding a video system
In January 2007, MWU learned the U.S. Department of Homeland Security had grant money available for security projects and Madison might qualify to receive some financial assistance.
MWU replaced its dialup telephone system with MDS iNet 900 radios in 2002 and 2003. They installed the system with the intent to use it for the SCADA system, a door access system, and for video surveillance. The radio network had a potential bandwidth range of anywhere from 100K to 400K bits per second depending on configuration. Radio testing indicated critical links were in the 100K range.
Any video surveillance system had to operate without affecting the SCADA system or the door access system. Madison was familiar with the Longwatch Video Surveillance system, and a demonstration proved video could transmit over our radio network.
Through the spring of 2007, Dane County, Wisc., developed a grant request in cooperation with MWU. Dane County wanted this grant to go toward protecting the water for the people of Dane County.
“The safety of our drinking water is vitally important,” said Mayor Dave Cieslewicz. “This new system will provide us with real-time surveillance of critical Water Utility facilities, providing an additional level of security for the community.”
Toward that objective, MWU developed a three-project grant request that included:
Access System estimated at $90,000
Video Surveillance Systems estimated at $360,000
SCADA System estimated at $1,500,000
This screen shows up to 24 still images from remote sites at one time. Each image updates every 20 minutes. If an alarm occurs, the operator receives a 15-second video clip of the incident and can switch to live video at any time.
By May 2007, Madison requested conceptual solutions from three video surveillance vendors. The task was simple; put cameras at 32 remote sites and bring back video clips over the utility’s radio network. We asked vendors to develop a project concept and then prove the concept will work. The utility’s requirements were:
Event-based video for the operators
Local recording per American Water Works Association security guidelines
Do not impede communications for the SCADA system or the Access System
If an event happens at 2:00 a.m., we want to know now
An estimated budget of $350,000 (estimating $10,000-12,000 per site over 32 sites)
Any installed system would provide a performance guarantee
Every camera every 20 minutes
Madison contacted three local video surveillance companies to propose a solution. Two of the three companies responded to the request and started developing concepts.
The challenge was not capturing video; the challenge was transmitting it back to the main office via existing radios. One of the two companies was successful in transmitting video from a single camera, and one company withdrew in frustration. This remaining company was able to transmit video from all 32 sites using the existing radio system.
To allow a conventional video surveillance system to function, one option was for MWU to use a faster, more reliable communication system. MWU looked into replacing the existing radios, installing a backhaul radio network, connecting to commercial fiber, and connecting to commercially available high speed Internet providers.
They rejected these options due to either high capital or high operations costs. MWU decided to stay with the existing radio system and to use the new installation.
In the summer of 2007, Madison Water received word that a federal grant for $388,000 requiring a Madison match of $97,000 would come through Dane County for a video system and an upgraded access system.
MWU and the vendor worked out an agreement whereby Madison supplies 50kb/sec bandwidth on the radios, and then the guard tour mode would produce an update from each camera at least every 20 minutes.
The system would also produce 25 video event clips per hour (five seconds in length and three frames per second) from around the network. Finally, with an optimized radio network, MWU decided to proceed.
Bringing up the system
After surveying the system and planning each site’s coverage and vulnerability, installation work commenced in November 2007.
Sites typically took one to two days to install the cameras and equipment. Afterwards we aimed, focused, and calibrated the cameras. Wonderware SCADA software was in use at the central office to manage and organize monitoring of the remote sites.
The video surveillance system uses a Wonderware HMI to allow the operator to monitor alarms, and view and manage video clips. The remote sites communicate with the main server via Ethernet radio connection to the HMI computer.
MWU is in the process of converting its aging outdated SCADA system to a PLC-driven platform. This conversion will probably take 18 months to two years to complete. Once the system change over is complete, the video surveillance system will interface directly with the new SCADA system.
The planned HMI SCADA integration will provide operators with information about system operations, records, logs real-time data, and will allow operators to view video and monitor system status. Monitoring and control of the system is our highest priority, so it is very important the access control and video do not interfere with process data on the radio network.
Following review and approval of the screen graphics for the system, sites went on line one by one.
The system uses 64 AXIS cameras at 32 remote locations connected to local hardware and software. A typical site video hardware setup includes one to four video cameras at each site connected to a “Video Engine.”
The Video Engine records high-resolution video 24 hours a day, seven days a week; and stores it up to 30 days. It simultaneously sends live video and event clips to the video control center (VCC) at the central office control center.
The Video Engine also monitors door switches, motion sensors, and other detection devices that indicate the presence of an intruder to create an alarm condition.
We configure the Video Engine so if any alarm goes off, it will retrieve previous footage and transmit the archived video and live video as a “video clip” to the VCC, using the existing radio system.
A video clip is a 10 to 15 second video snippet showing footage from before and after the alarm triggered, allowing the user to see the big picture.
The operator can also switch over to a live viewing mode and watch events occurring at the remote site in real time. Continuous live video from both of the remote site cameras stores on the Video Engine for up to 30 days, so an operator can recall as much high definition, archived video as needed to analyze a situation.
Video from 32 remote sites transmits via wireless to the central control room at the Madison Water Utility in Madison, Wisc. The video uses the same wireless link as the SCADA and access control system, and the video images appear on an HMI system.
Additional cameras in
The high-resolution video can write to a thumb drive or a portable DVR and then serve as evidence against intruders or to obtain increased detail of a scene.
Because of the limits of the radio system, it is impossible to transmit live video from every site, at all times. The bandwidth required would consume the network. Instead, the system stores all video locally and only transmits video over the network when an incident occurs or when operators request it.
In the absence of alarms, a system tour happens every 20 minutes delivering one frame of video to the operators giving them an updated view of the remote site.
When video transmits, it breaks down into pieces and weaves into the existing network traffic, essentially disguising the video data as normal process data. This allows the video to move over the existing communications network without interfering with the SCADA system or access control system.
Upon arrival at the central-office control room computer, the Longwatch software reassembles the video information from the individual blocks of data and presents the video clip directly onto the HMI/SCADA workstation screen for the operator.
The full system was completed and online by the middle of March 2008.
MWU continues to fine tune and adjust the system to minimize false alarms. Issues with car lights and lightning may require adjusting or modifying triggers at some locations.
Multiple frequent false alarms tend to desensitize pump operators and lessen the impact of the system. At one point, the system recorded incidences of graffiti and other mischief at our remote locations.
Because the system stores up to 30 days of high-resolution video at each remote site, we visited each site, downloaded the images onto a thumb drive, and gave the images to the police to help in prosecution.
Probably the most interesting incident so far was early on during testing. The cameras spotted a platoon of soldiers, armed with M16 machine guns and dressed in camouflage uniforms, deploying themselves across our property. Further investigation revealed that a ROTC unit from the University of Wisconsin was staging an exercise without telling anyone.
Today the system is operational and is monitoring the system 24/7. MWU can monitor its remote sites and know when someone has entered a site.
The Station Summary Screen shows the status of all doors, alarms, cameras, and wireless systems at 36 remote pump stations, wells, tanks, and other facilities.
Through the tour and live modes, we can check in on the assets protected by the camera system at any time of the day or night.
Additional cameras are under consideration for the utility’s operations center and vehicle storage garages. MWU will monitor camera operation and coverage and add cameras and event triggers to improve the overall system performance.
ABOUT THE AUTHORS
Al Larson P.E., (ALarson@madisonwater.org) is principal engineer at the Madison Water Utility in Madison, Wisc. Steve Rubin
(firstname.lastname@example.org) is president of Longwatch, a surveillance systems integrator, in Norwood, Mass.