Systems Admins: We Need To Talk.

I’m frustrated. Frustrated because I keep seeing articles about businesses, specifically hospitals, being ransom-wared into submission. In the past month, I can recall three specific instances. Some paid the ransom. One is still in limbo. Each system claims that their data is being held hostage, and that the ransomer is demanding somewhere between $1600 and $3.7 Million dollars – all negotiable, of course. Hospital administrators cry foul, sysadmins look to expensive solutions, and patient care suffers.

None of this has to happen.

Sysadmins, we need to talk. I know the struggle – I’ve been a systems administrator for 15 years. You have too few resources, too small a budget, and no respect. I get it. I do. Your users click links they shouldn’t, download things without forethought, and go to websites that you would firebomb from afar if you had your way. I understand that ransomware is a fast-changing, ever evolving beast that is mitigating your defenses as quickly as you’re mitigating its attacks. Its impossible to stop every attack. I get that. However, I’d like to pose question to you, and I ask this with as little snark as I can muster: Is that really an excuse? Can we really throw up our hands because “its hard,” and not even attempt good, basic security measures?

Admins, lend me your ears. With good, basic, and built-in tools, you can defend against ransomware. With just a few hours of configuration (at most!), you can stop this madness. Let’s talk turkey.

Fix Your Email

  • Filtering extensions. Do you block incoming file attachments? Most companies don’t, and can’t – that’s fine. However, you can certainly block the dangerous ones. All modern email systems block executables (.exe) and batch (.bat/.cmd) files from the get go – most will also block VB scripting (.vbs), screen savers (.scr) and a few others. Lets get to whats not being blocked:
    • .doc / .xls files – Yep, MS Office. No, I am not suggesting you disallow your users from sharing office files – but modern Office extensions are .docx and .xlsx – so ditch the old versions. Inside these files are malicious macros that will grab the ransomware payload and pull it onto your machine. While you’re at it, block .rtf
    • .js files – Nobody emails you raw Javascript, with a glaring exception: Locky. Locky’s vector is commonly a .js file attached to an email (often in a zip)
    • .zip files and .rar files – Yes, some businesses use these to transfer files. Say it with me (and if you’re a sysadmin, you’ve been shouting this for years) – email is not a file transfer mechanism. Find an alternative. Utilize network shares or a third party system like OwnCloud. Ransomware often comes in a .zip, and sometimes even password protected (with the password in the email body). Why? Mail scanners can’t look inside zipped files. Block them outright if you can.
  • Filtering countries. Does your company do business with China, Romania, or the Ukraine? What’s the business impact of never receiving mail from Russia again? In a great majority of cases, this will not impact you at all – but will cut down exponentially on both spam and phishing. Many email servers will allow you to block based on region or country. Take heavy advantage of this. If not, you can look at netblocks by country and black/grey-list them manually.
  • Crank up your spam protection. A lot of ransomware coming through is going to be flagged as spam by the same criteria that “13UY V1@GAR4” ads get stopped with. It doesn’t have to be turned to max, but it does have to be turned on.
  • Consider blocking any of the generic gTLD domains out there. Domains such as “.xyz” and “.info” are cheap and used as throwaways by spammers. Stop them from entering your email environment and you’ll reduce the number of phishing attacks and spam emails your users receive.

Defend Your Servers

  • Software Restriction Policies. Via group policy, you can restrict any executables from running out of the %TMP% directory – which is how all ransomware I have encountered or read about starts. Pushing this down to your users should be a no-brainer. Now, I say that with a grain of salt – this will break things. In my experience, Quickbooks installers, MS Office installers, and Spotify all break with an SRP is in place. These, however, can be whitelisted. This takes testing and should be rolled out slowly, especially in complex environments. Here’s a very thorough tutorial with screenshots on how to implement a Software Restriction Policy.
  • File Server Resource Monitor. FSRM is a method for actively monitoring file shares. One of the first things ransomware does is drop a file explaining how to pay the ransom. With FSRM you can easily alert on those files and run a script. The script I wrote is extremely basic – it kills the file sharing service, sends the admin an email, and writes the event to the event log. Here’s a list of filenames I monitor for.
  • Follow good security practices. Does everyone have read, write, execute access on every share? They shouldn’t. Follow good security practices for accessing data – use the principal of least privilege and role-based access control. This is good practice aside from ransomware, but will help contain the damage should something slip through your other controls. Users in Groups, Groups assigned to Folder/File permissions. Add/Remove users from groups as their access or roles change. This makes management easy
  • Monitor Handles. Consider setting up a “canary” to alert you of processes generating a high handle count. There are a few that we should expect to do so – system, SqlServer, and lsass come to mind – but a process actively encrypting or modifying thousands of files at once will generate a high number of handles. I wrote this script when the first CryptoLocker hit, and run it as a scheduled task every 15 minutes; feel free to modify it as you wish. Be warned that it is fairly ugly, but it does what it says on the box.

Defend Your Endpoints

  • Antivirus. Some people will tell you that antivirus is dead. There are certainly arguments for that – an antivirus can act as a last line of defense if your other controls fail. Make sure your definitions are updated, and the antivirus is up to date. Microsoft Security Essentials is free, and will defend against known ransomware. Teach your users to report virus alerts, not ignore them.
  • Patching. Keep your endpoints patched. You can download and install Windows Server Update Service for free, and have it manage your updates and reboot cycles.
  • Phish your users. It can be done for free and teaches them not only to be suspicious of emails they aren’t expecting, but helps train them on indications that an email is not from who they think.
  • Remove local administrator rights from machines. Users may kick and scream that they can’t install Skype, but reducing the local machine rights drastically reduces the damage that can be done. Without admin rights, you can only install and run applications out of very limited folders (My Documents and %TMP%), so its easier to mitigate malicious software trying to do you harm.

Defend Your Network

  • DNS. If you’re using your ISP’s DNS server, I would encourage you to change it to the free OpenDNS service. OpenDNS is good about blackholing known-bad IP addresses and command & control channels. It will reduce malware from web-browsing significantly and costs you nothing.
  • Block Tor. Tor has many legitimate, and noble uses. However, many pieces of ransomware use it to establish a connection to a C&C channels to generate the key used to encrypt data. If this step fails, ransomware stops. Block Tor unless you are actively using it for business – which you likely are not.

Defend Your Data

  • Backups. If all else fails, you need the security of having recent, tested, GOOD backups. Windows Server Backup is not the most elegant solution, but it works – and costs you nothing. A large USB drive is all you need to back up your data. Find out what your company’s tolerance for data loss is, and take the drive off-site that often. If they can tolerate a week of lost data, take it off-site every Friday. If they can tolerate no more than a day, take it off-site every night. A note about ransomware: if the backup drive is plugged in, and the system infected? It will encrypt your backup drive. Its important that you eject the USB or physically remove the drive every time you complete a backup. If you can spare a few dollars and some bandwidth, a service like CrashPlan runs about $8 per month and backs up changes in real time, and maintains a version history. Not an ideal way to recover the data should you lose everything, but it’s a “set it and forget it” approach that requires little maintenance and no drive swapping.

 

Sysadmins: this is what the phrase Defense-in-Depth means. Multiple solutions to solve a problem that may mitigate one or more defenses you have in place. An antivirus and firewall are no longer enough. There is no excuse for a ransomware infection resulting in lost data and days/weeks/months offline. You can accomplish every step outlined above with a zero-dollar budget.

Any other tips, tricks, or $0 mitigations you’d like to share? Please comment below!

21 thoughts on “Systems Admins: We Need To Talk.

  1. I’m pretty sure that you can enable whitelisting for executable files in group policy. I haven’t tried this approach yet, but I think that would stop most of the malware.

    Like

    1. yes you can whitelist with Group policy. I have several doctor offices I manage and we do block.exes from running in appdata, we also restrict zip files in email and use an alternative for sharing files with user outside

      Like

  2. Not a bad advice but some of it is impractical or even incorrect.

    1) Blocking DOC/XLS/RTF files is unacceptable. We still have Office 2002 users.

    2) Blocking ZIP files is unacceptable, although it is probably peculiar to us. We often receive e-mail submissions via such files.

    3) Mail scanners CAN look inside archives. The good ones can even look inside ENCRYPTED archives. The encryption hides the contents of the file, but it doesn’t hide the file name, extension or CRC. Known malware can be detected by its CRC even if you cannot decrypt the archive containing it.

    4) Filtering e-mail from countries is unacceptable for an international business. At best, you could increase the spam weighting of the e-mail coming from them.

    5) Blocking execution from %TMP%, while tempting, is impractical. Way too much self-updating software (even AV software) insists on running its updaters from there – and these updaters often have random names, so explicitly allowing only them isn’t easy.

    6) Using canaries to monitor network shares is a good way to protect your servers, but the ransomware can still encrypt the workstations on which it runs. There are products that do such monitoring locally, but they are still in their infancy, and some ransomware doesn’t touch files at all (e.g., Petya).

    7) Removing local administrator rights is very good if you can get away with it, but Microsoft doesn’t make it easy. Way too many things require them and “right-click run-as” is by far not as easy and convenient as “sudo”.

    8) Blocking Tor is a fascist policy I would never impose on our users. People have the right to stay anonymous on-line.

    9) Backups is the most important advice – shouldn’t have been left at the end. Everything else is likely to fail sooner or later.

    Like

    1. “We still have Office 2002 users.” If true, you’re a damn fool and should be fired! There is no excuse for using antiquated MS Office software.

      Like

  3. Amazing post! Couple of pointers for future admins…

    1)Install Linux (on *all* the PCs)

    2)Keep a (hidden) admin account with a complex password (which you don’t write and keep somewhere)
    -> allows for SSH, therefore allowing remote access and management

    3) Run a background service to monitor network access (“iftop” is a nice lightweight and easy to use command/package) create custom scripts (I prefer python; fast, dynamic and sensible)

    4) keep users in loop about possible malware, make it interesting…

    This is coming from a Linux user (past 5-6 years). Trust me, I may not be a SysAdmin (yet – still doing undergrad) but this kinda stuff does make sense.

    Like

  4. Hospital IT is a nightmare story among sysadmins. noone knows why they are so much worse, but it’s what it is. Meet two people in a day coming from a Raid loss where primary data was on a raid5 and there was no backup? It would be a hospital.

    I can read from your article and the checks you put in place on your systems that you’re on top of things.
    But where’s the difference comping from then?

    Honestly, the sysadmins in these places should just tell their management to fuck off, this is management decisions having effects. Stop telling people to ruin their life’s for the greater good.
    If management doesn’t give end users a slight bit of training, if your budgets are cut to the point where you need to cheat on raid levels or backups, you’re also not staffed well enough to just even read this blog post, unless they exploit their own time to the max.

    Turning up email sec to a reasonable level is a good advice.
    But you fail to notice that there has to be a reason why they didn’t. they didn’t have the resources to do it. That could be skill. Or time. Anything. Something you did have and they didn’t.
    Otherwise they would have done it. Even if they spent half of their days banging with the medical staff out of boredom, if they had that much spare time they’d get around to email sec.
    They don’t, otherwise hospital IT wouldn’t be this famous mess.

    Or maybe, someone who saw that that person is better at dealing with end users and business goals than any of the complete psychos that have filled the mail admin newsgroups for decades.

    Yes, let’s try to raise the bar as much as possible, and your scripts should be a good help for many people that don’t know what to do / where to start.
    But I think as long as the windows world sustains hiring underskilled people and the skilled ones aren’t able to fill the other job goals we’ll not get out of this by any technical means.
    We can try to bring enterprise standards of infra management to the small guys, but it’ll only fill half the gap.
    The other half is the people problem of messed up budgets, understaffing, feature craze and management ignorance. As long as mgmt keeps thinking of IT staff as useful idiots who don’t powerplay we will not really improve anything for the long term.
    And this is what it is, otherwise IT could ensure people are trained and can’t just lie about their computer skills at hiring time.

    tl;dr:
    distribute best practices but don’t try owning up to management decisions.

    Like

  5. Good backup policy > Ransomware
    I had an incident where a user had gotten infected with Ransonware but a daily backup prevented the payment. A 2hr format-reinstall and back in business with $0 paid. Remember HDD storage is cheap!

    Like

  6. Nice post and I totally concur on many recommendations.

    I’d add two things: 1) Deploy Microsoft’s EMET to endpoints is the one thing I’d add to your list. It’s an anti-exploit tool written by Microsoft 2) Don’t Hate (UAC), Elevate

    Like

  7. Excellent post!
    I thought I’d take a minute and add my two cents.

    1. Backups! There is no excuse for not backing up critical data.

    2. Why are people still not using shadow copies? I’ve restored hundreds of gigs of encrypted files from shadow copies.

    3. FSRM is amazing. That can shut ransomware down cold when properly configured.

    4. Folder redirection / user training. Users should not be storing critical data exclusively on their PC. Either redirect their folders or train then to save files on the server.

    5. Backups and shadow copies. Even if ransomware gets in, a good backup will win every time.

    Like

Leave a comment