Any engineer or physisist will tell you that Entropy is like Gravity - theres no fighting it, its the law! However, they can both be used to advantage in lots of situations.
In the IT industry, a files entropy refers to a specific measure of randomness called Shannon Entropy, named for Claude Shannon. This value is essentially a measure of the predictability of any specific character in the file, based on preceding characters (full details and math here: http://rosettacode.org/wiki/Entropy). In other words, its a measure of the randomness of the data in a file - measured in a scale of 1 to 8 (8 bits in a byte), where typical text files will have a low value, and encrypted or compressed files will have a high measure.
How can we use this concept? On its own, its not all that useful, when you consider that many data filetypes (MS Office for one) are highly compressed, and so already have a high entropy value. So using just entropy, theres no telling a good MS Office file from one encrypted by ransomware.
However, most data files have a specific file header, which includes a set of identification bytes (called magic bytes) that identify what the data file is. For instance, those bytes for PKZIP files are PK, and PE32 executable files use MZ. You can identify these files by using the files program. While this command is native to Linux, there are nice ports of this for Windows.
If files cant identify a file type, we can then use the entropy value as a second check. If a file is encrypted, it will have a higher entropy value than one that isnt. In this case you are looking for a file where the character distribution is random, or at least much more random than a normal data file.
You need both of these checks to identify suspect files - as mentioned todays complex data files are becoming much more compressed - Office files are actually PKZIP compressed these days, so will identify as PKZIP when checked with files.
Using these two checks to identify suspect files depends on a couple of things:
- Ransomware encrypts the whole file, so it clobbers the magic bytes of data files when it encrypts them (at least all the variants that Ive encountered)
- Ransomware generally employs a decent encryption algorithm, so the entropy will be high. So far this has been true - for instance, were not seeing ramsomware using simple XOR techniques (which would by terrible encryption, but would preserve a files average entropy value).
Using these two checks, I started with a simple powershell script that copies a subdirectory from one location to another, but leaves all of the suspected infected files behind. A log file is created that lists all files copied and all files that are suspect and should be looked at. I used Sigcheck (from sysinternals) to compute the entropy value. If the entropy is above 6 (8 is the maximum) and the magic bytes are unknown, I flag the file as suspect.
And for a test ransomed file, Im using Didier Stevens sample file: ransomed.bin (see the bottom of this story for links).
I used Powershell because most Ransomware affected shops that Ive worked with have been infected in MS-Office files and other Windows data files, so Powershell seemed simpler than adding install python into a customers crash IR situation. You could certainly take this same logic and code an equivalent python script that would cover Windows, Linux and OSX.
After finishing my replication script and taking a step back, I realized a few things:
- I was trying to recreate rsync, which does the replication job much better than I could ever hope to.
- Using Sigcheck was adding some Powershell overhead to extract the entropy value from the output text, which was adding up to seconds and minutes for larger volumes.
So I re-wrote the script to be more of a single-minded. The final script simply lists suspect files rather than copies them. And I wrote a short C program to compute the entropy value (thanks to Rosettacode for the starting point on this!), which simply spits out the numeric value, rather than lots of other stats I dont need for this job.
The final output list of suspect files can be used in a few ways:
- It can be used as an exception list, to tell rsync which files to skip in a replication job. This starts to approach some commercial backup, storage and replication products, which have product features that allow you to create sanitized copy of a data volume (these are generally new new features, so your mileage may vary).
- It can also be used to catch infection incidents early. Once you are in the midst of IR, you can usually identify infected files by something more obvious - like the file extension is now locky for instance. But if you dont know that youre infected, you wont know which IOC to look for - this can help identify the problem early.
- For many of us, the prevalence of malware that uses encryption has changed the focus of our jobs. Were not just protecting the perimeter and the workstations, were now tasked with protecting the actual data (which philosophically should have been the case all along). What this means is that we should start getting more familiar with the data in our organization, what the file type and size distribution is, how quickly it changes and what data is actually in the files. Hopefully the approach Ive outlined can be helpful in this effort.
- Lastly, Ill be using this to extend the files command to include more file types. Ive added the ID for 7zip files already, but look for a story on this project sometime in the next few weeks. So far Ive identified that I need to add file definitions for wma, mobi, m4a, epub, pgp, gzip / tgz and mp3/mp4 files. VMware Workstation appicon files (and several other workstation files) also arent correctly identified. If you find more filetypes that are not correctly detected please let us know via the comment form. The more effective the files">">.SYNOPSIS
This is a simple Powershell script to recursively scan a subdirectory tree, and identify any files that:
a/ do not have magic bytes that identify it as a specific file type to the file">The default path is .">#">function checkfile($FNAME) {
# check file magic char for type - unknown file type has data as type
$cmd = file -m c:\utils\magic ` + $FNAME + `
$ftype = iex $cmd
$copy = 0
if ($ftype -notlike data*"># check on entropy. If less than 6, its not encrypted well and is not compressed well, so it"># this entropy computation method uses sysinternals sigcheck
#
#$cmd = sigcheck -a ` + $FNAME + `
#$a = (iex $cmd | select-string -pattern Entropy:) -split :"># this entropy computation method uses dedicated C code, slightly faster
#
$cmd = entropy ` + $FNAME + `
$a = iex $cmd ">if ($args[0]) { $src = $args[0] }
else { $src = . }
">$files = Get-ChildItem -recurse $src
$files | foreach {
# directories
if ($_.psiscontainer) {
# nothing to do here
"> else {
$go = checkfile($_.fullname)
if ($go -eq 0) {
write-host $_.fullname
}
}
}">
int makehist(FILE *fh,int *hist,int len)
{
/* define a reasonable buffer to read the file - 1 byte at a time is too slow */
i
"> {
buflen = fread(
"> {
if(wherechar[(unsigned int)c[j]]==-1)
{
}
}
}
}
">
">
}
>">{
if ((fh = fopen(argv[1],rb)) == NULL )
printf(Error opening file %s\n
else
{
fstat(fileno(fh),
}
//hist now has no order (known to the program) but that doesnt matter
printf(%lf\n
>From this script, you can see that cleaning ramsomware infected files isnt an insurmountable problem. A simple script like this feeding rsync can be used to create a clean copy of a datastore, and identify suspect files. Just be SURE to keep up with evaluating that suspect file list - as noted, depending on your data store there might be lots of clean files in that list at the moment (give me a few weeks to improve this). If you run the script as-is to blindly feed rsync, you wont have a complete copy of your datastore.
As always, this was a 1 evening coding effort, so Im sure that there is more elegant Powershell syntax for one thing or another. Also, youll see that my C code chooses readability over efficiency in a few spots. For either piece of code, if you find any errors, or if you identify better syntax to get the job done, please do use our comment form and let me know. Of more interest, if you find this code useful in your environment and you want to see a version 2 - let me know in the comments also!
As I work on this code, youll find the most up-to-date version at: https://github.com/robvandenbrink/Ransomware-Scan-and-Replicate
Didiers diaries on Ransomware and Entropy can be found here:
https://isc.sans.edu/forums/diary/Ransomware+Entropy/20271/
https://isc.sans.edu/diary/Ransomware+%26+Entropy%3A+Your+Turn/20321===============
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
Rob VandenBrink
Compugen