Four months after a county hard drive crash crippled operations, forcing some staff to drastically modify day-to-day procedures — if not close offices altogether — Information Technology director Chris Rayner said the situation “cannot happen again.”
Rayner gave a “state of the department” report to the Sampson Board of Commissioners Tuesday, talking about the learning experience and the two-month process it took to get departments fully-functional again. He also addressed steps that should be evaluated to ensure the county takes the learning experience and looks ahead.
The board is holding its annual planning session over two days this week, Thursday being the other.
“We all saw there was a major disaster in IT this year,” Rayner said. “It was probably one of the worst years physically that we’ve had. It took us down. Specifically, it took the Sheriff’s Department and the Library down. They lost the most efficiency and time in getting back up. In normal circumstances, systems are built where we don’t have these issues. This was not a normal circumstance.”
The county’s system has 15 drives, and should be able to withstand five of them going inoperable. The county only lost three drives, but data was corrupted within the aging system. Replacement drives were ordered, but the incorrect ones were sent — twice.
“As of now, both agencies are back fully-functional,” Rayner said. “They are finding some glitches here and there, but I think they’re satisfied. It took us two months to get to a point where it was fully functional.”
The county’s hard drive experienced the crash in mid-October, leaving the Sampson County Library system unable to complete circulation transactions or offer access to public computers and Wi-Fi. Library staff had to rebuild the database from an older backup database, re-cataloging and verifying inventory of all items in all branches and re‐inputting records not in the older database — the reason for the closures.
The Sheriff’s Office was similarly rendered unable to complete and log electronic reports.
While the departments had access to the internet, the server was inoperable rendering their software moot. That lack of accessibility closed libraries for nearly a month and forced officers to turn back to old methods of record-keeping.
“IT undergirds all departments. We often sit in the background,” said Rayner. “We’re not the most glamorous department. We try to make as little waves as possible, but I really found out at that time, if I didn’t know, what happens when we go down. We affect all services in some capacity. IT is good when it works. When it is broke, everyone is broke.”
While the issues were an inconvenience, county officials said upon functions being fully restored in December, they did not stop the daily operations of the county and all its departments. Rayner echoed that Tuesday.
“They are so used to having that capability, the ease in what they do,” said Rayner. “We have to make sure we’re looking at IT in a different light than what we’ve been in the past. We cannot afford to have this issue surface ever again.”
In the wake of the issues, a virtual network was established with Star Communications to maintain the “virtual environment,” which included nightly replicated backups of major servers within the county. Data was actually backed up in three locations.
“That backup system helps us tremendously,” said Rayner, who pointed to other measures that should be assessed going forward.
He floated a possible realignment of the department as well as the addition of a technology specialist for the Sheriff’s Office.
“The sheriff has around 200 devices located in his office,” the IT director explained. “Everybody has issues we need to get through and there is a lot of stuff to worry about. We are not abandoning the Sheriff’s Office but if they have a person that can handle some of the day-to-day issues and work between us and them … that would help out in many areas.”
Commissioner Clark Wooten said anyone who is brought on whether in lieu of existing staff or in addition to it, needed to have the acumen it takes to bring Sampson into the future, which is constantly evolving on the tech side.
“Technology is not going to get less complicated,” Wooten remarked, “it’s just going to get more complicated.”
Rayner said he was not advocating simply throwing money at the problem, but said resources have to be evaluated. Some cost-savings by the IT Department in recent years may have come as a detriment that now needs to be looked at further. Having other staff that is able to handle day-to-day would free Rayner up to look at the long-term.
“I have to have some time to look at some other things. I spend a lot of working and not planning. I have to start looking at where we’re going so we can build the correct infrastructure that we need.”
To that end, he spoke about having a centralized Data Center in a secured facility to install data, network and phone equipment.
“We have a Data Center that is housed inside Public Works, but we need to secure it closer to our location. These are ideas, not things that are needed immediately, but that we need to look at. We’re going to bring it back in-house.”
Developing a technology plan for the county and identifying “best practices” for the IT infrastructure were also a necessary step. That would allow key stakeholders to get input toward building an IT roadmap that includes targets and rotating replacement of aging equipment.
“If I didn’t understand it before, I clearly understand that our software and technology is so important that we need to do whatever is necessary to make sure those systems are in place to prevent those broad catastrophes from happening,” County manager Ed Causey stated. “I’m not saying those are anyone’s fault, but it does tell us we need to come back and take a good look at that foundation.”
“IT is the lifeblood of the county,” Commissioner Harry Parker concurred. “We need to maintain that and look down the road.”
Rayner said that new understanding goes both ways, with his eyes being opened up about just how many people use the libraries’ services and Mobile Patrol, an app that allows the public to get information logged by Sheriff’s Office.
“I never realized how many services the public saw in the library until it was down,” he said. “There’s also a product the Sheriff’s Office uses called Mobile Patrol, and I never knew how many people look at Mobile Patrol. I got more complaints about Mobile Patrol being down…”
Sheriff’s reports took a little longer but Rayner said the complaints he received weren’t about reports, Rayner said.
“It was all about Mobile Patrol. ‘When are you going to get Mobile Patrol back up,’” he recalled. “You see what the citizens like. I’m glad that everything is back up. It took a little bit of time, but it has laid the groundwork for figuring how to get this back in house and how to get this figured out.”
Reach Managing Editor Chris Berendt at 910-249-4616. Follow the paper on twitter @SampsonInd and like us on Facebook.