ISC Diary | Silent Traitors – Embedded Devices in your Datacenter

There’s a great post by Rob VandenBrink over at the ISC Handler’s Diary about embedded devices that are hiding in plain sight in your data center.

I was recently in a client engagement where we had to rebuild / redeploy some ESXi 4.x servers as ESXi 5.1. This was a simple task, and quickly done (thanks VMware!), but before we were finished I realized that we had missed a critical part – the remote managent [sic] port on the servers. These were iLO ports in this case, as the servers are HP’s, but they could just as easily have been DRAC / iDRAC (Dell), IMM or AMM (IBM) or BMC (Cisco, anything with a Tyan motherboard or lots of other vendors). These “remote management ports are in fact all embedded systems – Linux servers on a card, booting from flash and usually running a web application. This means that once you update them (via a flash process) they are “frozen in time” as far as Linux versions and patches go. In this case, these iLO cards hadn’t been touched in 3 years.
So from a security point of view, all the OS version upgrades and security patches from the last 3 years had NOT been applied to these embedded systems.

This is a thorny issue as systems often need downtime to patch these systems. Check out the thread there for how others are handing or mitigating this.
Oh, and I’ll throw in Sun’s LOM (Lights Out Management) to the list.
via ISC Diary | Silent Traitors – Embedded Devices in your Datacenter.

Why Marissa Mayer’s ban on remote working at Yahoo could backfire badly — Tech News and Analysis

Mathew Ingram at GigaOm wrote an article on Yahoo’s new policy on remote workers:

Not long after her arrival at Yahoo, new CEO Marissa Mayer started handing out carrots to her new employees, including new smartphones, free food and other Google-style amenities. Now she has brought out the stick: namely, a directive that employees are no longer allowed to work from home, something that is expected to affect as many as 500 Yahoos. Mayer’s move has its supporters, who argue that she is trying to repair Yahoo’s culture — but in doing so, she could be sending exactly the wrong message for a company that is trying to spur innovation after a decade of spinning its wheels.

The moment I first heard Yahoo proclaimed this policy I became angry. It does not impact me directly, but as a highly skilled and experienced IT Security and Networking professional now on the market I can say that Yahoo is no longer on my list of companies I’d care to work for. Here’s why.
About 15 years ago while I worked for EDS as a Network Security Administrator my marriage fell apart. Up until then I rarely if ever worked from home. With divorce looming I had sole custody of my two young kids. I had to work from home when they were sick or were off of school. At the same time my role at EDS changed to include firewall administration, demanding more of my time to cover on-call and odd support hours.
I was fortunate to report to managers that understood my situation and worked to help me. I worked with a great group of professionals who didn’t complain about my flexible work schedule. In fact we all worked together so everyone could have the same flexibility I had. How did I handle things? I became infamous for keeping sleeping bags, pillows, snacks, and toys for my kids in my cube. I don’t know how many nights I carried the two of them into the data center in the middle of the night, each slumped over a shoulder while I badged through the security doors. They slept on the floor swaddled in their sleeping bags and little heads resting on Disney-themed pillows, lullabied by the white noise.
When I interviewed with Magna I was very upfront about what I needed to do to take care of my kids and what I would to do in return. They took me on without hesitation, and I always appreciated and respected the trust they placed in me. Similar to my days at EDS, the team at Magna embraced me and the flexibility I needed. I repaid my boss’ and team’s trust in many of the same ways I did for EDS, but there was one case that was  above and beyond.
For reasons that escape my memory the IT staff in Europe all quit on the same day. The organization I worked for was very lean. There were no extra people around to help fill in while they hired new staff. I stepped up, waking between 03:00 and 04:00 Eastern time to support Europe until I had to get my kids ready for school. I’d drop them off (no bus service) and return to cover the rest of the European day and my normal work. I was caretaker of servers and services in addition to the network and security. I did this for almost 6 months from my basement, buying the European IT director time to hire some great team members.
When I moved into management my team earned with me the same opportunities and respect that I earned. With instant messaging and email, IP telephony and video conferencing, and cheap Internet-based VPNs back to the company they could do everything they needed to do from home that they could do from work. Yes, you cannot replace face-to-face interaction. But by the same token how much hallway and water cooler talk is mere friendly trivia?
I’ll leave how companies chose to handle working from home to what makes sense for them and their business. But I want the conversation rephrased to talk about working from home as a tool and not a benefit. It can help both the employer and the employee, and that can’t be taken lightly.
I sincerely hope Marissa Mayer reconsiders her decision. She’s closing a door on quality hard-working talent that will go elsewhere just at the time when she needs them in Yahoo.
via Why Marissa Mayer’s ban on remote working at Yahoo could backfire badly — Tech News and Analysis.

[Storage] The Problem With Noisy Neighbors in the Cloud – Matthew Wallace – Voices – AllThingsD

Matthew Wallace at AllThingsD wrote up a great article about how organizations employ a myriad of tactics to avoid the risks of shared storage environments, often inefficiently and ultimately self defeating:

Massive overprovisioning of resources in clouds, dedicated storage platforms attached to shared compute platforms, dedicated shelves in shared storage platforms, or massive horizontal scaling are options used every day. They don’t solve the problem — they avoid the problem, often at great expense or through significant architectural shifts.

My take away from this article is to ask the right questions of your cloud storage provider or your storage infrastructure vendor to make sure you’re not impacted by “Noisy Neighbors”:

For instance, does your CSP work with a storage vendor that offers guaranteed QoS on a storage platform? … Cloud environments empower you with the business agility of service on demand and flexibility to respond to changing business needs rapidly. Adding resources for a time and then giving them up when they are no longer needed is a major benefit. While the advancement of cloud computing has made those accessible on the compute side, the storage side was left behind by the limitations of rotational disks and the inability to offer ironclad QoS guarantees.
The power of a such a solution … is not only in knowing that you can guarantee a certain number of IOPS on each volume, but to pair that with cloud environments to allow the business agility to burst as needed on the storage array the way that cloud environments offer that flexibility for compute.
The rapid and automated provisioning world of the cloud demands that storage companies build APIs rich enough to control every aspect of an array. Building the user interface as a layer on top of the API is a demonstration of API and design maturity that shows a solution is future-proofed against demanding cloud orchestration requirements. Designing the solution to be linearly scalable without artificial breakpoints or step functions in performance keeps the provisioning and growth simple and reliable, shutting out the noisy neighbors once and for all.

via The Problem With Noisy Neighbors in the Cloud – Matthew Wallace – Voices – AllThingsD.

Move Over, APTs — The RAM-Based Advanced Volatile Threat Is Spinning Up Fast – Dark Reading

Advanced Volatile Threat (AVT) is an attack on RAM, not on data or programs stored on the system. It’s fast, ephemeral, & hard to detect, according to the article:

And that concern, (John) Prisco (CEO of Triumfant) says, could drive more attackers to drop their APT strategies and turn to AVTs instead. “The AVT is going to be attractive to sophisticated attackers because it’s there, and it’s gone,” he says. AVTs take a bit more effort, Prisco observes, because they only work once, but attackers who are highly concerned about attribution will likely be willing to do the extra work.
Using an AVT is no guarantee against detection, DeMesy says. “Detection of advanced volatile attacks is extremely difficult, even when best practices are followed,” he says. “However, you may be able to detect what the attackers are trying to do. Internal honeypots are an excellent way to entice attackers to reveal their presence. Attackers employing advanced volatile attacks are looking to get in and out of a network quickly, bringing with them as much information as possible, so seemly vulnerable targets, such as a honeypot, are a prime target.”

via Move Over, APTs — The RAM-Based Advanced Volatile Threat Is Spinning Up Fast – Dark Reading.

Malware on oil rig computers raises security fears – Houston Chronicle

This article from the Houston Chronicle highlights the need for layered security including proper VLAN design to segregate & contain malware as part of security:

Malicious software unintentionally downloaded by offshore oil workers has incapacitated computer networks on some rigs and platforms, exposing gaps in security that could pose serious risks to people and the environment, cybersecurity professionals told the Houston Chronicle.
The worst-case scenario could be catastrophic: A malfunctioning rig and safety systems could cause a well blowout, explosion, oil spill and lost human lives, experts said.

The way the article reads it seems like these platforms have large flat LANs, where employees’ personal equipment is on the same network as the production equipment. I’m a fan of placing SCADA systems in their own VLAN with non-routable IP addressing – Internet and the rest of your local network. Place a physical firewall device between the SCADA LAN and the regular LAN, but lock that firewall down. Selectively open ports for maintenance and restrict when done. Monitor the heck out of the thing.
InfoSec professionals, how would you handle this type of situation?
via Malware on oil rig computers raises security fears – Houston Chronicle.

[Administrivia] AdSense Advertising Experiement

Part of my professional change this year includes looking for a new position with a great company. One of the jobs and employers I am considering is writing for myself.
I’m still thinking this through, then I’ll get advice from friends and family. This could go many different ways. I need more data, so I’ll perform a number of experiments in the coming weeks.
The first experiment starts this week: I’m enabling Google AdSense on this site and probably all of my others. I’m curious what the impact is both financially and from the user experience.
I am not a fan of advertising. I go out of my way to avoid it. This leaves me in a quandary, and opens me up to hypocrisy. I’ll give it 4 to 6 weeks, then re-evaluate.
What are your thoughts? Are you generally pro advertising or do you find it a distraction here and/or the rest of the web?