DHS vulnerability scanning program offline after Virginia office loses power

DHS vulnerability scanning program offline after Virginia office loses power:

Two cybersecurity programs the Department of Homeland Security offers both states and the private sector have been temporarily knocked offline due to a power outage, while other services have been shifted to backup locations, multiple sources tell CyberScoop.

The National Cybersecurity and Communications Integration Center (NCCIC), the 24/7 hub for monitoring cyberthreats across the government and critical infrastructure, has shifted operations to a backup location in Florida. The move was made after the Arlington, Virginia, building that houses NCCIC lost power last week due to heavy rains.

Additionally, two other programs under NCCIC’s National Cybersecurity Assessments and Technical Services (NCATS) — Cyber Hygiene vulnerability scans and Phishing Campaign Assessment — have been offline since July 26.

The Cyber Hygiene program remotely detects known vulnerabilities on internet-facing services. The Phishing Campaign Assessment program is part of a remote penetration testing service. Both programs are used by hundreds of customers across the country. Thirty-four states have received vulnerability scans through the Cyber Hygiene program, according to a DHS presentation given at the National Association of State Election Directors summer conference.

DHS Assistant Secretary for Cybersecurity and Communications Jeanette Manfra told CyberScoop that the disruption to Cyber Hygiene is temporary, and that election systems will be the first to resume service once the program comes back online. Officials expect scans to resume Aug. 6.

The building housing NCCIC suffered heavy damage on when portions of the façade ruptured due to the volume of rain that fell in the Washington, D.C., region. The roof of a restaurant on the building’s ground level failed during business hours on July 26.

… A number of DHS offices are in that building.

CyberScoop has learned that due to the water damage, the building completely lost power, which prevented server rooms used by DHS from staying cool. Once the room reached a certain temperature, a sprinkler system was activated. Those sprinklers damaged servers supporting the Cyber Hygiene and Phishing Campaign Assessment programs.

On Sunday, the NCATS office sent an email to its customers informing them that Cyber Hygiene and Phishing Campaign Assessment were offline and that contingency plans have been put in place.

“In order to minimize the operational impact, we immediately implemented our contingency plans and transferred functions to other sites, including NPPD’s facility in Pensacola, Fla.,” the email, obtained by CyberScoop, reads. “We are working to restore these services as quickly as possible. We will let you know when the service and reports will resume.”

NPPD is the National Protection and Programs Directorate, which oversees NCCIC.

The power outage has had a “minimal impact” on DHS’s cybersecurity operations, Krebs said. The incident has not, for example, affected the department’s ability to respond to cyber incidents or issue warnings to the private sector.

DHS has been at the center of the federal government’s efforts to fortify U.S. voting infrastructure following the 2016 presidential election, when Russian hackers probed systems in 21 states. Last week it was revealed that the same outfit of Russian hackers that meddled in the 2016 election appears to have targeted  Sen. Claire McCaskill‘s office.

(Via Cyberscoop)

With the DHS looking to create a central Risk Management program, seeing stories like this does not instill confidence that the U.S. Government, and the DHS in particular, are up to the challenge.

This slays me:

Chris Krebs, the undersecretary of NPPD, told CyberScoop that the department is “taking this opportunity to get some efficiencies into the system, but also to build resilience and redundancy.”

Those are the words uttered after every such event.

By the way for those not in the know, there is a well-known process call Disaster Recovery and Business Continuity Planning (DR/BCP) that has been around for decades to plan for just this sort of event.

Also on:

Your Olympics interests me and maybe earns my support?

In case you haven’t paid attention, I live in Tokyo. Tokyo hosts the 2020 Olympics.

Yesterday I briefly railed against the Olympics. I still think they are a waste of money, resources, and time. I specifically referenced the US.

And then there’s Tokyo.

Train stations across the city are being remodeled and improved. The stations not under construction have likely been already renovated. Hotels are shooing up. Transit plans started testing a year ago. Refinements to messaging to include English and other languages are in their late stages. More and more restaurants, shops, and other venues are taking credit cards.

If someone tries to tie specific economic benefits to hosting the Games, they will still be hard pressed.

But this city can and might just be, by 2020, the most globally accessible by language, culture, and disability. The country might be, too.

That will be a huge economic benefit.

According to what I’ve read, disaster planning includes Olympic scenarios.

You better believe that Olympic folk in Tokyo are taking notes about everything that happens in South Korea these next two weeks. They will break down, analyze, and game plan for everything seen.

I wonder how much raw data Japan gets from South Korea to prepare for their games?

I think the Tokyo and Japan governments are exemplifying something we talk about in Security circles – never let an emergency go to waste.

Ambivolence as Empowerment

Your lack of planning does not constitute my emergency

A former co-worker and former friend coined that phrase for me. He was fired for exercising that customer support philosophy far too stringently on a day-to-day basis (among many, many other sins). Both his mantra and eventual fate educated me.

Long after his departure, the poem “If” by Kipling stuck home:

“If you can keep your head when all about you / Are losing theirs and blaming it on you”

The opening quote to this post takes on a new meaning if the opening of “If” is included.

Emergency responders, be they Emergency Medical Technicians or police or fire fighters, do what they do to their training. For example, if you find yourself impaled on a piece of rebar your sense of urgency is to remove the rebar. The EMT knows removing the rebar is the wrong thing to do. Maintaining your blood pressure, treating for shock, and many other things are more important.

Case in point: About 10 years ago my team and I were pressed into service performing real life disaster recovery for a business unit that planned for no such disaster. We came in and assessed the situation. We told our customer truths about the current state of affairs they did not want to hear. We required them to make hard choices about which they wanted to dither and debate. We, as part of a larger response team, got them back up and running far faster than they deserved based on their lack of planning.

Ultimately our value was as much in our not being emotionally invested as our ability. They needed a rational actor in what they experienced as a highly irrational environment. My team and I offered independent advice, provided facts, and ultimately used our skills and creative thinking to help our customer get back on line from a catastrophic event.

We could have empathized. We could have offered platitudes. We could have told them everything would be okay, that being strong in the face of adversity would overcome.

Again, our emotional ambivalence offered more value. We didn’t coddle or make people feel good about themselves. We were there to fix things. We needed the business to make decisions based off of uncomfortable and ugly facts. These were hard business choices – short term, medium term, or long term; pick one. Shop floor production or back office support? Payroll or shipping?

Our stark A/B questions caused a number of recalculations. Back office staff could be easily relocated elsewhere, so they were moved off site. Some finishing and shipping could be shifted to another site, too.

Some of the stuff we tried failed. When that happened, we tried something else. Today that’s called agile development.

The catastrophe evolved to an event to a disruption to restored service in 6 days. I don’t presume my team’s approach was the primary catalyst for the speedy resolution of things, but without a doubt it made a positive impact.

This story highlights the value of an emotionally independent or ambivalent actor with their wits about them when yours are completely invested. This is true in an emergency, but also true when yours are too invested in the status quo.

Don’t Neglect Tampons

I visited an AT&T emergency response validation testing session several years ago. After Hurricane Katrina they were able to start restoring service as soon as the area was considered safe enough for their people to enter.

What makes this possible for such teams? Training. Equipment. Food. Water. Most people will flag those.

What about toilet paper? And washing machines for what the teams wear under the protection gear? And sunscreen for when they finally get out from under the protective gear? And tampons? And several hundred other taken-for-granted details that become huge and potentially life threatening in their absence while standing in a toxic soup of stagnant storm water who-knows-what infused trying to restore basic communications for emergency responders.

When working on disaster recovery or an emergency response plan, don’t draft it in isolation. Benefit from other’s learning and iterations (lessons learned). It’s much better to prepare for something with a shopping list than a blank piece of paper. This is not an area where non disclosure is good for anyone.

This is true in the command center as well. How can efficient effective direction & information get communicated when key people don’t have access to their insulin or blood pressure medication? How do you manage your technical expert’s dairy problem while all the food you have access to is a vending machine full of chocolate bars?

Which assumes the machines will be full. What if it’s the day before restock? What if the restocking person broke up with their significant other or was high or was distracted? Are humans part of your calculation?

Granted, squirrelling away prescription medication isn’t easy (& maybe illegal where you are) but knowing the challenge exists before it’s a problem is the first step to solving it. Other things, like stockpiling daily-free & gluten-free food, tampons, tissues, toilet paper, and everything else identified from other’s work and your own tabletop exercises is relatively easy to manage.

And you? What are your thoughts?