Find Log4j with SaltProject and Everything

Another day, another Log4j patch to apply. When I first learned about CVE-2021-44228 I was terrified. I have been working in IT for 15 years and one thing I’ve learned in my org is you can’t throw a rock in IT without hitting a derelict java app somewhere. They’re everywhere. And even if you think you’ve found them all unfortunately software inventory is only part of the problem when it comes to Log4j. Maybe the app uses this library, maybe it doesn’t, who knows?

So the first thing I did when I heard of this vuln was to set off and find a way to quickly search every file on all systems. Yes, every file on all systems (Windows anyway). Enter voidtools Everything. If you’ve never used this application before it is basically black magic. It can provide a searchable database of every file on a Windows system in a few seconds. It uses the NTFS Master File Table (MFT) and I wish Microsoft would too. And it works on systems with millions of files, it’s incredible.

Only problem is I’ve never used Everything at such a large scale. I have hundreds of systems to search. It’s easy enough to double click the executable and enter a search term but how can I leverage this amazing software at scale. The answer, SaltProject (formerly SaltStack). Salt is an infrastructure management tool that is so incredibly powerful, it can be scary at times – and a little intimidating. I have run a ~6,000 system SaltProject implementation for a few years and it has saved my hide more times than I can count.

For this Log4j problem, Salt provides a single console that we can use to execute any command we can imagine on all systems simultaneously. Client systems (minions) connect to the Salt server (master) and report the results of commands in about the time it takes for the command to run locally (+ a little overhead for Salt). By combining Salt with Everything we should be able to search all Windows systems in the organization in minutes (we can search macOS and Linux systems too but not with Everything). After we get it all setup first of course. 😉

To get started, we need a Salt master.

Step 1, provision a Linux server with TCP ports 4505 and 4506 accessible from clients (I’ll be using Ubuntu). I recommend starting with something modest and scale resources vertically if needed. For ~1,000 – 2,000 clients, I have found 8G of ram and 4 cores to be sufficient. You won’t need a lot of disk storage, 200G is plenty.

Step 2, install the Salt master.

# Download key
sudo curl -fsSL -o /usr/share/keyrings/salt-archive-keyring.gpg https://repo.saltproject.io/py3/ubuntu/20.04/amd64/latest/salt-archive-keyring.gpg
# Create apt sources list file
echo "deb [signed-by=/usr/share/keyrings/salt-archive-keyring.gpg arch=amd64] https://repo.saltproject.io/py3/ubuntu/20.04/amd64/latest focal main" | sudo tee /etc/apt/sources.list.d/salt.list
# Update apt
sudo apt-get update
# Install salt-master
sudo apt-get install salt-master

Step 3, deploy the Salt clients. Now there are about a million ways to do this so don’t feel wed to my method. To deploy software far and wide I like to use a GPO with an immediate task that runs a PowerShell script. Here’s an example of the script I use. If you would like to use this script you will need to host the installer on a web server, set the server name and master name in the script. You may also need to change the file name and hash of the MSI if there has been an update since this post.

Step 4, accept the salt minion keys. Once the minions are installed and communicating with the master there will be a key exchange. The clients will cache the master’s public key and the master caches the minion’s public key. The minions are ready to accept commands from the master as soon as the minion’s keys are accepted on the master. To accept all keys run salt-key --accept-all.

You can verify connectivity to the minions by running salt '*' test.ping.

Congratulations you now have an incredibly powerful shell with access to run any command as ‘nt authority\system’, give it a shot salt '*' cmd.run 'whoami'.

root@salt:~# salt '*' cmd.run 'whoami'
PRRPT01.example.com:
    nt authority\system
PRRPT02.example.com:
    nt authority\system
PRPRC01.example.com:
    nt authority\system
PRWEB10.example.com:
    nt authority\system

...

PRDB01.example.com:
    nt authority\system
TSTDB01.example.com:
    nt authority\system
root@salt:~#

Step 5, create a Salt “state” file (sls) to deploy Everything. By default, salt creates a shared file store that is accessible to all minions. The default path of the file store is /srv/salt. Create an /srv/salt folder, sudo mkdir /srv/salt. It’s a good idea to keep stuff in subdirectories, so create a directory for Everything too, sudo mkdir /srv/salt/everything. Download and extract Everything from the 64-bit portable zip for and the es.exe cli tool. When your done the directory should look like this.

The state file tells the minion where the files should go and where it can get them from. Since Salt is a state management system it makes sense to name your states in past tense such that when the minion returns it has either made the appropriate changes and validated that the system in the correct state or it’s just going to validate the system is in the correct state already. My state file is /srv/salt/everything/installed.sls and here are the contents.

Everything.exe:
  file.managed:
    - name: 'C:\ProgramData\Everything\Everything.exe'
    - source: salt://everything/Everything.exe
    - makedirs: True

Everything.lng:
  file.managed:
    - name: 'C:\ProgramData\Everything\Everything.lng'
    - source: salt://everything/Everything.lng

es.exe:
  file.managed:
    - name: 'C:\ProgramData\Everything\es.exe'
    - source: salt://everything/es.exe

This state file is formatted in YAML and described three states. Each state is merely checking for the existence of a file and if the file is not found in the named location, it will download the file from the source and put the contents in the target name. Notice the first state includes “makedirs: True”. This is a nice little shortcut to create the C:\ProgramData\Everything folder.

Now our directory looks like this.

Step 6, deploy Everything. Now that we have a state we can deploy it one system as a quick test to make sure we got everything right.

root@salt:~# salt 'TESTSERVER.example.com' state.sls everything.installed
TESTSERVER.example.com:
----------
          ID: Everything.exe
    Function: file.managed
        Name: C:\ProgramData\Everything\Everything.exe
      Result: True
     Comment: File C:\ProgramData\Everything\Everything.exe updated
     Started: 19:58:00.731509
    Duration: 5107.766 ms
     Changes:
              ----------
              diff:
                  New file
----------
          ID: Everything.lng
    Function: file.managed
        Name: C:\ProgramData\Everything\Everything.lng
      Result: True
     Comment: File C:\ProgramData\Everything\Everything.lng updated
     Started: 19:58:05.840275
    Duration: 1269.513 ms
     Changes:
              ----------
              diff:
                  New file
----------
          ID: es.exe
    Function: file.managed
        Name: C:\ProgramData\Everything\es.exe
      Result: True
     Comment: File C:\ProgramData\Everything\es.exe updated
     Started: 19:58:07.109788
    Duration: 438.849 ms
     Changes:
              ----------
              diff:
                  New file

Summary for TESTSERVER.example.com
------------
Succeeded: 3 (changed=3)
Failed:    0
------------
Total states run:     3
Total run time:   6.816 s

Notice in our command, “everything” corresponds to the folder /srv/salt/everything and “installed” corresponds to “installed.sls” in that same folder. And we can see three states succeeded with three states changed.

Deploy Everything to all systems, salt '*' state.sls everything.installed.

Step 7, Find Log4j. Finally the time has come! We can run some inline PowerShell to return a list of all files everywhere with “log4j” in the name. Here is the command.

root@salt:~# salt '*' cmd.run shell=powershell '& C:\ProgramData\Everything\Everything.exe -admin; Start-Sleep 3; & C:\ProgramData\Everything\es.exe log4j | Sort; & C:\ProgramData\Everything\Everything.exe -quit'
PRRPT01.example.com:
PRRPT02.example.com:
PRDB01.example.com:
TSTWEB01.example.com:
DEVDB01.example.com:
TSTDB01.example.com:
    C:\Program Files\Microsoft SQL Server 2017\MSSQL14.MSSQLSERVER2017\MSSQL\Binn\Hadoop\FOO\slf4j-log4j12-1.6.1.jar
PRPRC03.example.com:
PRPRC02.example.com:
PRDB02.example.com:
PRWEB13.example.com:
PRPRC04.example.com:
PRWEB05.example.com:
PRWEB11.example.com:
DEVPRC01.example.com:
PRWEB12.example.com:
PRWEB06.example.com:
PRWEB09.example.com:
PRWEB10.example.com:
PRWEB01.example.com:
PRWEB04.example.com:
PRWEB03.example.com:
    D:\Program Files\ArcGIS\Server\framework\lib\shared\apache-log4j-extras-1.2.17.jar
    D:\Program Files\ArcGIS\Server\framework\lib\shared\ignite-log4j-2.7.0.jar
    D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-1.2.12.jar
    D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-1.2-api-2.11.1.jar
    D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-api-2.11.1.jar
    D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-core-2.11.1.jar
    D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-jcl-2.11.1.jar
    D:\Program Files\ArcGIS\Server\framework\lib\shared\slf4j-log4j12-1.7.25.jar
    D:\Program Files\ArcGIS\Server\framework\runtime\spark\conf\log4j.properties.template
    D:\Program Files\ArcGIS\Server\framework\runtime\spark\jars\apache-log4j-extras-1.2.17.jar
    D:\Program Files\ArcGIS\Server\framework\runtime\spark\jars\log4j-1.2.17.jar
    D:\Program Files\ArcGIS\Server\framework\runtime\spark\jars\slf4j-log4j12-1.7.16.jar
    D:\Program Files\ArcGIS\Server\framework\runtime\tomcat\lib\log4j.jar
    D:\Program Files\ArcGIS\Server\framework\runtime\tomcat\lib\log4j.xml
    D:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\conf\log4j.properties
    D:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\lib\log4j-1.2.17.jar
    D:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\lib\log4j-1.2.17.LICENSE.txt
    D:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\lib\slf4j-log4j12-1.7.25.jar
    D:\Program Files\ArcGIS\Server\tools\configurebasedeployment\lib\log4j-api.jar
    D:\Program Files\ArcGIS\Server\tools\configurebasedeployment\lib\log4j-core.jar
    D:\Program Files\ArcGIS\Server\tools\configurebasedeployment\lib\log4j-jcl.jar
    D:\Program Files\ArcGIS\Server\tools\createsite\lib\log4j-api.jar
    D:\Program Files\ArcGIS\Server\tools\createsite\lib\log4j-core.jar
    D:\Program Files\ArcGIS\Server\tools\createsite\lib\log4j-jcl.jar
    D:\Program Files\ArcGIS\Server\tools\upgradebasedeployment\lib\log4j-api.jar
    D:\Program Files\ArcGIS\Server\tools\upgradebasedeployment\lib\log4j-core.jar
    D:\Program Files\ArcGIS\Server\tools\upgradebasedeployment\lib\log4j-jcl.jar
    D:\Program Files\ArcGIS\Server\tools\upgradeserver\lib\log4j-api.jar
    D:\Program Files\ArcGIS\Server\tools\upgradeserver\lib\log4j-core.jar
    D:\Program Files\ArcGIS\Server\tools\upgradeserver\lib\log4j-jcl.jarPRWEB07.example.com:
PRWEB08.example.com:
PRWEB02.example.com:
PRPRC01.example.com:
TSTPRC01.example.com:
PRPRC05.example.com:
DEVWEB01.example.com:
root@salt:~#

Enjoy!

And if you have Linux systems, Salt can of course help there. Install salt-minion, use a Salt grain to target Linux hosts and run find.

salt -G 'kernel:Linux' cmd.run 'find / -type f 2>/dev/null | grep -i log4j'

Have fun patching y’all! Happy Holidays and yippee kai yay!

1 thought on “Find Log4j with SaltProject and Everything

Leave a Reply

Your email address will not be published. Required fields are marked *