Broadcast Writer

By Vicki W. Kipp
April 1, 2004
Society of Broadcast Engineers (SBE) Chapter 24 NewsletterApril 1, 2004

How do you monitor more than 17,000 different transmitter sites across the country from one monitoring facility with a staff of 21 people?   When I heard this claim made of Dielectric Corporation’s Flash Technology division in April 2003, I was a little doubtful. After all, 17,000 tower sites is an awful lot of potential light failures, security violations, and power outages to keep an eye on. I traveled to the Nashville suburb of Franklin, Tennessee to see for myself how monitoring on this grand of scale could be accomplished.

Dielectric Corporation acquired Flash Technology in 2003. Dielectric’s Flash Technology division includes a tower site monitoring service, tower light manufacturing plant, a post-manufacture tower light testing lab (Figure 1), a tower light research and development laboratory (Figure 2), and a tower light classroom known as “Flash University”. 

ARC

My first concern was trying to understand how any monitoring service could handle the sheer call volume of 17,000 wireless telephony, broadcast, and public safety tower sites. A great thing happens when you provide a service for more than 14 years: you get very, very efficient at it.

The tower monitoring service head end is called the Alarm Response Center (ARC). Staffed 24 x 7 x 365, the ARC always has at least three Monitoring Technicians on duty and an experienced supervisor on call.

The ARC contains three sections: the Call Center staffed by Monitoring Technicians who investigate alarms; Customer Support staffed by Technicians who diagnose problems, manage trouble tickets, and interact with clients; and Technical Support staffed by Service Coordinators who do troubleshooting, equipment support, and field support.

Flash Technology has analyzed the records of the millions of phone calls they have received over the years to create algorithms which help them predict which times of the day and night when they will be busiest. Each of the 17,000 sites monitored by Flash Technology will be called by the ARC once every 24 hours to log its parameters. Additionally, each site will call in anytime it experiences an alarm condition.

Each site monitored by the ARC has a sign posted at the site entrance listing a toll-free phone number for the ARC and a tower reference number. This contact information is useful to members of the general public, tower neighbors, tower technicians, and public safety personnel who need to access the site. The ARC can provide the access touch pad security code to admit people authorized to enter the site. The ARC monitors and records the names of all individuals who access a site. Having a toll-free contact number posted at the site has paid off. The ARC sometimes receives calls from tower neighbors complaining that a strobe light is running in day mode after dark, making their home resemble a bright, flashing disco dance hall. The ARC can manually place the tower lights in night mode, and then notify the tower’s owner of the problem in the morning. Addressing the neighbor’s lighting concerns promptly helps preserve good relations with tower neighbors. Timely acknowledgement of a problem may prevent a frustrated neighbor from taking matters into their own hands– perhaps even shooting an offending tower light.

ALARMING

Whenever a site calls in an alarm, Flash Technology’s Eagle software automatically creates a trouble ticket. An ARC technician analyzes the trouble ticket to see what action needs to be taken.

Monitoring Technicians in the Call Center answer 500 – 700 calls per day, responding to alarms, notifying authorities, and dispatching technicians. VoicePrint software records and logs incoming and outgoing calls. Site data is archived by the ARC for a minimum of three years and a maximum of five years after it is recorded. When the ARC receives an alarm about a tower light outage or notes an outage when the ARC calls a site for its daily logging cycle, the ARC Lighting Specialist will call a Notice to Airmen (NOTAM) in to the appropriate FAA Flight Service Station. The ARC calls in about forty NOTAMs a day to the FAA. The ARC will notify the site’s owner of the outage. For clients who contract with the ARC for tower maintenance, the ARC will dispatch a tower technician to the site to repair the light. In compliance with the FAA regulations, the ARC strives to get all light outages resolved within the FAA’s 15 day outage window.

When the ARC receives an alarm from a site, they handle the event according to the site owner’s preferences. If there is a light outage, the ARC will call the FAA Flight Service Stations (FSS) to report the failure within 30 minutes, as specified by FAA Circular AC No: 70/7460-1H and enforced by FCC Title 47 Code of Federal Regulations (CFR). But for less critical site events such as a door open or site temperature alarm, the ARC will respond according to the client’s customized escalation procedure. The ARC will call and/or email the people specified by the site owner. If the alarm occurs at a site that the ARC is contracted to maintain, the ARC will notify the owner and then dispatch service technicians. If an urgent situation arises which requires that the site owner be called in the middle of the night, the ARC manager will make that phone call.

Clients can pick a custom solution from the Dielectric-Flash Technology suite of services: FTM monitoring hardware for their site, leasing Dielectric’s Eagle software, or hiring Dielectric’s ARC to monitor their tower site. Before the ARC begins monitoring a site, Dielectric’s tower division must inspect the site. The tower division performs site erection, structural maintenance, and lighting maintenance and repairs. The inspection identifies any equipment issues. Since the ARC is responsible for filing NOTAMS, existing obstruction light deficiencies must be remedied before the ARC assumes responsibility for monitoring the site. 

SMART MONITORING

A large plasma television at the front of the ARC (Figure 3) displays the Weather Channel. ARC technicians constantly monitor the national weather map for storms, tornados, and hurricanes. They anticipate severe weather that could cause a power failure, lightning strike, or other damage to the sites that they monitor. When severe weather covers a wide geographic expanse, the ARC manager brings the on-call ARC technicians in to support the increased call volume. To the left of the weather monitor are two FTM 5000 units and multiple strobe lights. These units allow the lighting specialists to practice wiring and configuring the monitoring telemetry when there are lulls in incoming calls.

HARDWARE

Dielectric-Flash Technology manufactures the FTM 5000 remote telemetry module (Figure 4) for monitoring and control of various equipment and operating conditions. Each FTM 5000 can hold up to nine control/monitoring cards. Each card is capable of monitoring six to ten parameters. A single card can be shared among multiple devices such as monitoring lighting and the site door. The number of site parameters that a FTM 5000 can monitor in unlimited since any number of FTM-5000 modules can be bussed together. The FTM 5000 has an Ethernet port and a modem.

The FTM 5000 works in conjunction with Eagle monitoring software. Clients with more than 125 sites need to use Eagle Eye software, in addition to Eagle software, to manage the polling of all of their sites such that each site stays in compliance by calling the ARC once every 12 or 24 hours.

Dielectric- Flash Technology’s FTM monitoring hardware is highly customizable. You can use the FTM 5000 to monitor transmitter power levels and to control transmitter on/off. With the purchase of Dielectric’s VSWR Vision software and some additional hardware, the FTM can measure antenna VSWR and transmission line pressure, in conjunction with Eagle software.

Site owners can custom populate their FTM 5000 with the appropriate cards based on their monitoring and control desires. Monitoring and control card choices include:

• A dry contact input to monitor equipment functions, door closures, and alarm relays. A dry contact card indicates whether a device is off or on.

• An analog input to monitor generator fuel levels, building temperature, hydrogen fuel cells, power metering, battery levels, voltage, oil pressure, and other measurable levels. In conjunction with a relay output card, you can set customized thresholds so the card will turn a particular device on or off if a threshold is crossed. • A relay output card to monitor and control the generator, hydrogen fuel cell, climate control, and remote operation of equipment. A relay card is necessary for device control. When used in conjunction with a site generator, an analog card enables remote monitoring of the generator’s fuel level and status. You can remotely cycle the generator to run for a while as part of its maintenance. The relay card allows you to remotely cut AC power to ensure that the site properly switches to the generator’s DC power.

• A light monitoring card allows for tower monitoring of all types and brands of tower lights. Each site monitored by the ARC must have a connection to the ARC so that it can communicate alarms. The vast majority of sites monitored by the ARC use a combination of a modem and a landline (70%) or wireless telephone (30%) to call in alarms and status updates. Most new tower sites connect to the ARC through a fiber optic line or a low-bandwidth Very Small Aperture Terminal (VSAT) satellite system provided by Flash Technology. Although a fiber optic connection can be pricey, new towers typically have fiber run to them during the construction process.  

Dielectric’s Flash Technology site monitoring panels can monitor all FAA approved lighting systems. Eagle software, which communicates with the monitoring panel, can provide alarm information for any lighting type.

Sites who use Flash Technology’s FTM monitoring hardware and have Flash Technology’s lights installed on their tower receive especially elaborate light monitoring from the ARC. See how many triggers the flash tube has attempted and how many it has missed. If there is an alarm on Flash Technology brand tower lights, the ARC can diagnose what the cause of the problem is by analyzing the voltage, current, and emitted light for the failed lamp. If Dielectric- Flash Technology services that site, they guarantee that they will fix the lighting problem in one trip to the site. For sites not serviced by Dielectric- Flash Technology, the elaborate diagnosis of why the light failed helps the service know which lighting component needs to be serviced before they go to the site, cutting down on repeat trips.

COMPUTER APPLICATIONS

Since reliability is a top priority at the ARC, they have redundant computer operations in a separate location. The ARC runs diagnostics to check the health of their communications network every fifteen minutes.

The trouble ticketing program used by the ARC is a custom application developed from Remedy Action Request System. Dielectric-Flash Technology employs two Remedy developers who continuously make improvements to the software and personalize the software based on client needs. Remedy Trouble Ticketing tracks how much time elapses from when an alarm is received to when an ARC technician completes diagnostics on the problem. Incoming alarms are listed in a queue. An ARC technician selects an alarm to respond to as soon as they are available. The ARC has a system to ensure that each call is dealt with promptly. Once a staff member opens an alarm, their name is listed on the file. Each Monitoring Technician on duty is assigned a Monitoring Technician “buddy” during their shift. If a call isn’t acted on within a certain time period, the trouble ticketing software sends a reminder to the Monitoring Technician’s “buddy” and a supervisor.

VoicePrint digital call logging software is used to record and archive every phone call made to or from the ARC. Calls are saved with the records for the applicable tower site. Calls are recorded not only as a method of quality assurance, but to provide records of NOTAMs filed. VoicePrint ensures follow-through and accountability by recording the time each call was answered and the staff member who took the call.

The ARC uses NEC’s CallCenterWorX product with automatic call distribution and management information system modules to design their call-handling process and route calls to designated staff. While routing incoming calls, CallCenterWorX collects data which is converted into useful statistics and reports.

Flash Technology’s Eagle Monitoring System interfaces with all lighting systems, including their competitors’ products. Eagle allows the user to dial up an individual lamp on a tower to query it. Each site monitored has a graphic which shows the location and status with each light on the tower, along with the FAA circular which is applicable to each of the lamps. If a light fails, an X is placed over it on the tower lighting graphic. The software also lists each FAA circular so the ARC staff can reference the rules if there are any questions. All site alarms are automatically recorded by the Eagle Monitoring System. Eagle keeps records of each instance of the site building’s door opening and closing, and other monitored conditions.

Dielectric- Flash Technology creates real time web-enabled portals for clients to securely access and monitor their site status from any Internet connection. After logging in, clients can view tower operation status, alarms, and trouble-tickets. Clients can order equipment, track shipments, schedule maintenance, and track field technicians scheduled to perform repairs through the portal.

The Eagle Monitoring System allows in-depth monitoring and diagnosis of conditions for Flash Technology products installed on a tower. Eagle automatically monitors, diagnoses, anticipates, stores, and communicates high intensity strobe light events. I was impressed that Eagle Monitoring could remotely drill down to measure voltages and currents at various points in the lighting system. From the ARC, a technician could identify not only the specific tower light that is out or impaired, but also which lighting component was the likely cause of failure.

FEEDBACK

On the day that I visited the ARC, their largest client, American Tower Corporation (ATC), was also visiting. Michael Mitchell, VP of National Operations, expressed abounding satisfaction with the quality of monitoring that ATC has received from the ARC. 

RESPONSE

My trip to Tennessee yielded an answer to the question, “How do you monitor more than 17,000 different transmitter sites across the country from one monitoring facility with a staff of 21 people?”

If you happen to be the Alarm Response Center (ARC) of Dielectric’s Flash Technology division, the answer is that you monitor more than 17,000 tower sites with exceptional reliability and accountability. 

Acknowledgements: Sally Rich, Dielectric Corporation and Bryan Adams, Dielectric Corporation- Flash Technology.

Figure 1. All lamps made by Flash Technology get tested for 24 hours before shipping out.

Figure 2. The candelas of light output from this prototype lamp are being analyzed in a light tunnel (black square recess behind light).

Figure 3. Job tools help ARC technicians keep up with weather situations and lighting configurations.

Figure 4. Highly customizable Flash FTM-5000 remote telemetry module.