Cities across the world are grappling with an almost apocalyptic-sounding array of challenges. Many are dealing with dramatic upticks in natural disasters, choking traffic, air pollution, measles outbreaks, the threat of active shooters and, in some U.S. cities, a quick rise in homelessness.

And then there is city security as it relates both to networking and software. “Cybersecurity is probably my number one concern,” said Michael Lee Sherwood, the director of information technology for the city of Las Vegas, which has launched one of the most ambitious smart city programs in the country. “Guess how many attacks you think the city of Las Vegas gets in a year,” Sherwood said at the CIO Visions Leadership Summit conference in Las Vegas. “Half a billion. Right now we’re being attacked. And when we have a big event in Las Vegas, they spike up.”

The risk is growing daily as are the number of prominent victims in the past five years.

In 2014, the city of Ferguson, Mo. was hit with a cyberattack in apparent retaliation for the police shooting of the unarmed teen Michael Brown. The police department in surrounding St. Louis County was hit with an attack that brought down its website and email network.

Baltimore was recently a victim in early May to a ransomware attack known as “Robbinhood.” Some 10,000 local government computers were inaccessible for weeks. Mayor Bernard C. Young warned some systems could be adversely affected for months. The financial cost of the cyberattack could ultimately clock in around $18 million. Last March, Baltimore’s 911 network was hit with an attack as well.

This year, in particular, has been particularly bad when it comes to cyberattacks. In April, cybercriminals took nearly $500,00 from the employee payroll in Tallahassee Fla. while Augusta, Maine was forced to close its city hall because of a cyberattack. Last month, Laredo, Texas was hit with a ransomware attack. And last year, Atlanta fell prey to an attack that could ultimately cost taxpayers $17 million.

As many cities across the United States and the world pursue smart city projects extending their reliance on technology, they increase the potential risk of cyberattacks further.

Looking to Fort Lauderdale

It’s something of a truism that local governments don’t tend to excel at proactively confronting problems. That fact applies to everything from infrastructure investment to traffic management to cybersecurity. After Michael Maier, chief technology officer of the city of Fort Lauderdale assumed his role in 2012, he noticed the city had close to 90 employees with credit cards but little in the way of formalized security. “I asked: ‘Well, what do we do for security?’” Maier said at the CIO Visions event. “The answer I got from two elected officials was: ‘We got virus protection, Mike. You don’t need more security.’”

After Maier assumed the role, the city commissioned a network vulnerability study along with penetration testing and was able to address many of vulnerabilities, but his team struggled to convince city officials that plugging additional security holes was important has filling potholes.

A wake-up call arrived on “Cyber Monday,” December 1, 2014 in the form of a massive DDoS attack from the Anonymous hacking collective targeting the city’s website and that of its police.

The Anonymous group released a video of a speaker donning a Guy Fawkes mask, which had become emblematic for the hacking collective, insisting Fort Lauderdale abandon three controversial ordinances within 24 hours related to homelessness and panhandling. One of the ordinances, for instance, required organizations feeding the homeless to provide restroom facilities including wastewater disposal while providing certified food managers and receiving written consent to use public space.

Apparently miffed that local police arrested an elderly man who ignored those ordinances while feeding the homeless, Anonymous sent their video with demands asking the mayor to modify local ordinances not to any local government, but to local TV stations. “I was not aware of it, the staff was not aware of it, or the mayor or other elected officials,” Maier said.

After verifying their network was down, the city’s IT group called its ISP provider to troubleshoot.

Maier then gets a call from the city’s public information officer. “[Every TV station in the city of Fort Lauderdale is outside city hall. They want to talk to you,’” Maier recalled the PIO saying.

Shortly thereafter, Maier sees the Anonymous video for the first time. “I don’t know what’s going on until I see this. I unplug the internet. I don’t know if they’re inside my network,” he said.

How Fort Lauderdale Overhauled Its Cyberdefense

While the Floridian city’s tough cybersecurity luck caused chaos for city employees and residents for about a month, the incident can serve as a wake-up call for other municipalities. The lessons that follow are derived from Fort Lauderdale’s experience:

Have a Proactive Security Plan. Given an order to resolve the problem quickly, Fort Lauderdale’s networking team worked with its ISP provider to block the upstream attack, but Anonymous continued to change tactics. The attacks came in waves. “They’ll be silent for three or four hours and give you hope that they are going to go away and stop,” Maier said. “And all of the sudden everything pops back up.”

Maier stressed the importance of having a game plan for what to do in the event of an attack. It can be helpful to use so-called tabletop exercises, simulated emergencies to refine that plan.  

After the attack, Maier gathered a list of cyber vendors before he secured funding to use them while also reaching out to the local police department. The police connected with the FBI, while the FBI contacted the Department of Homeland Security. Meanwhile, he charged his internal staff to look at the inside network.

After the attack, the city’s IT team reset all its routers and changed all of the passwords while making sure all the firmware was up to date. It also invested in mobile device management and security information and event management technology.

To fight off the Fort Lauderdale attack, city officials created a war room while giving FBI staff members an office to use. It ultimately took 30 days to fully recover from the incident.  

The penetration testing and network vulnerability assessment the city commissioned prior to the attack, however, likely played a role in keeping attackers from getting inside their network. “I filled all of the low-hanging fruit,” Maier said.  

Have Clear Cyber Policies in Place

Because the city only had limited support for cybersecurity initiatives initially, Maier had to oversee the creation of cybersecurity policies and frameworks while patching servers, all of which he would have preferred to create earlier.

For instance, the city overhauled how it worked with external vendors and analyzed how they interfaced with its network. “Originally, they weren’t confined to the applications they support, etc. They could go anywhere,” Maier said. “So we bought software that limited their access only to things that we thought were appropriate.”  

It also updated its permissions globally. “We had administrator rights all over the place,” Maier said. The city also developed clear protocols for how to update its active directory of employees in the event that a worker quit or was terminated. A disgruntled terminated employee might be tempted to sabotage the city out of revenge. The city’s new policy would remove access — both to computer networks and city property — all within roughly 15 minutes, while also informing city officials of the city property the individual may still possess such as an iPad or smartphone.

Even though the city didn’t suffer a breach related to the Payment Card Industry data security standard, its officials changed all of the legal language in its vendor contracts with PCI, while ceasing business ties to vendors who didn’t support PCI.

The city eventually launched a cyber incident response plan. “We have a hurricane manual and in the back, it also has incident response,” Maier said.

Don’t Underestimate Phishing. Phishing attacks are so common it can be tempting to overlook them, but cyber adversaries tend to start with simple attacks before escalating them. For instance, the 2015 cyberattack in Ukraine that led to a partial shutdown of the nation’s electrical grid likely began as phishing campaign.

After the Anonymous DDoS attack on Fort Lauderdale, Maier learned that phishing scams were rampant inside of its network.

The city has since launched its own simulated phishing scams. “When I first started phishing, out of about 4,000 employees, they were at 10%,” referring to the percentage of employees who fell for those campaigns. Over time, that percentage has fallen to be close to less than 1%. “I phish every day with them,” Maier said. “It is to the point now that they are afraid to click on anything. I don’t mind because they’ll now send it to the security department and say: ‘Hey, is this OK?’”

The city has made cybersecurity awareness part of each employee’s performance review.

Dealing with Cyberattacks Retroactively May Be Expensive, but There’s a Silver Lining. After receiving an order for the mayor to “make [the cyberattack] go away, I immediately wrote $400,000 in emergency purchase orders for outside services, remediation, etc.,” Maier said.

But while the price tag was considerable for the city, the incident did lead to broad support from elected officials regarding the importance of cybersecurity and an overall cultural change. Maier said he went to the city commission meetings and mentioned the need to buy cybersecurity-related hardware or software or hardware, “all I had to do is put in the word ‘security,’” he recounted. “There was no question asked.”

A similar principle applies to rolling out security procedures that are so restrictive they temporarily interfere with everyday operations. While such actions can draw complaints from workers in the short-term, a recent cyberattack can help shore up support for them. “I stripped the firewall of everything as if we were a new organization,” Maier said. “I challenge you to go back and look at your firewall, and ask yourself: How many rules are really outdated?” After reconfiguring the firewall, the rules were strict that people practically couldn’t do their jobs. “I wanted it that way to start with because then I could taper back and find out whatever they’re trying to do, and I’ll open ports up,” Maier recalled.

The cyberattack also provided Maier with a justification to take away admin rights from users who didn’t need them.

Hiring Cyber Experts Can Be Hard for Local Governments. City officials decided to approve the creation of a cybersecurity team. Building that team, however, was a “major challenge,” Maier said. “Part of the reason is because I’m a government agency. Anybody in the private sector can outbid me.”

Maier attempted for several months to fill a chief information security officer role, and offered the position to several individuals. In each case, private sector companies offered to pay them an additional “$15,000 or $20,000.” “The cycle just kept on going,” Maier said.

So Maier decided to become a chief information security officer himself. He would get cybersecurity certification and take on a dual CISO/CIO role, while the city would hire security analysts to round out his team.

Cybersecurity Awareness Can Heighten Appreciation for Physical Security  

As Fort Lauderdale overhauled its cyber policies, the local police became inspired to help extend the city’s employees growing security awareness to the physical world. “They started doing active shooter drills inside the city in all the departments,” Maier said.

Police inform the employees in advance they plan on coming into a building with guns loaded with blanks on a given day. Employees have the choice to either participate in the drill or work remotely that day and read materials about how to prepare for an active shooter situation. “That’s what the police got from this Anonymous attack,” Maier said. “They saw the need to protect city employees.”