Another day, another Log4j patch to apply. When I first learned about CVE-2021-44228 I was terrified. I have been working in IT for 15 years and one thing I’ve learned in my org is you can’t throw a rock in IT without hitting a derelict java app somewhere. They’re everywhere. And even if you think you’ve found them all unfortunately software inventory is only part of the problem when it comes to Log4j. Maybe the app uses this library, maybe it doesn’t, who knows?
So the first thing I did when I heard of this vuln was to set off and find a way to quickly search every file on all systems. Yes, every file on all systems (Windows anyway). Enter voidtools Everything. If you’ve never used this application before it is basically black magic. It can provide a searchable database of every file on a Windows system in a few seconds. It uses the NTFS Master File Table (MFT) and I wish Microsoft would too. And it works on systems with millions of files, it’s incredible.
Only problem is I’ve never used Everything at such a large scale. I have hundreds of systems to search. It’s easy enough to double click the executable and enter a search term but how can I leverage this amazing software at scale. The answer, SaltProject (formerly SaltStack). Salt is an infrastructure management tool that is so incredibly powerful, it can be scary at times – and a little intimidating. I have run a ~6,000 system SaltProject implementation for a few years and it has saved my hide more times than I can count.
For this Log4j problem, Salt provides a single console that we can use to execute any command we can imagine on all systems simultaneously. Client systems (minions) connect to the Salt server (master) and report the results of commands in about the time it takes for the command to run locally (+ a little overhead for Salt). By combining Salt with Everything we should be able to search all Windows systems in the organization in minutes (we can search macOS and Linux systems too but not with Everything). After we get it all setup first of course. 😉
To get started, we need a Salt master.
Step 1, provision a Linux server with TCP ports 4505 and 4506 accessible from clients (I’ll be using Ubuntu). I recommend starting with something modest and scale resources vertically if needed. For ~1,000 – 2,000 clients, I have found 8G of ram and 4 cores to be sufficient. You won’t need a lot of disk storage, 200G is plenty.
Step 2, install the Salt master.
# Download key sudo curl -fsSL -o /usr/share/keyrings/salt-archive-keyring.gpg https://repo.saltproject.io/py3/ubuntu/20.04/amd64/latest/salt-archive-keyring.gpg # Create apt sources list file echo "deb [signed-by=/usr/share/keyrings/salt-archive-keyring.gpg arch=amd64] https://repo.saltproject.io/py3/ubuntu/20.04/amd64/latest focal main" | sudo tee /etc/apt/sources.list.d/salt.list # Update apt sudo apt-get update # Install salt-master sudo apt-get install salt-master
Step 3, deploy the Salt clients. Now there are about a million ways to do this so don’t feel wed to my method. To deploy software far and wide I like to use a GPO with an immediate task that runs a PowerShell script. Here’s an example of the script I use. If you would like to use this script you will need to host the installer on a web server, set the server name and master name in the script. You may also need to change the file name and hash of the MSI if there has been an update since this post.
Step 4, accept the salt minion keys. Once the minions are installed and communicating with the master there will be a key exchange. The clients will cache the master’s public key and the master caches the minion’s public key. The minions are ready to accept commands from the master as soon as the minion’s keys are accepted on the master. To accept all keys run
You can verify connectivity to the minions by running
salt '*' test.ping.
Congratulations you now have an incredibly powerful shell with access to run any command as ‘nt authority\system’, give it a shot
salt '*' cmd.run 'whoami'.
root@salt:~# salt '*' cmd.run 'whoami' PRRPT01.example.com: nt authority\system PRRPT02.example.com: nt authority\system PRPRC01.example.com: nt authority\system PRWEB10.example.com: nt authority\system ... PRDB01.example.com: nt authority\system TSTDB01.example.com: nt authority\system root@salt:~#
Step 5, create a Salt “state” file (sls) to deploy Everything. By default, salt creates a shared file store that is accessible to all minions. The default path of the file store is /srv/salt. Create an /srv/salt folder,
sudo mkdir /srv/salt. It’s a good idea to keep stuff in subdirectories, so create a directory for Everything too, sudo mkdir /srv/salt/everything. Download and extract Everything from the 64-bit portable zip for and the es.exe cli tool. When your done the directory should look like this.
The state file tells the minion where the files should go and where it can get them from. Since Salt is a state management system it makes sense to name your states in past tense such that when the minion returns it has either made the appropriate changes and validated that the system in the correct state or it’s just going to validate the system is in the correct state already. My state file is /srv/salt/everything/installed.sls and here are the contents.
Everything.exe: file.managed: - name: 'C:\ProgramData\Everything\Everything.exe' - source: salt://everything/Everything.exe - makedirs: True Everything.lng: file.managed: - name: 'C:\ProgramData\Everything\Everything.lng' - source: salt://everything/Everything.lng es.exe: file.managed: - name: 'C:\ProgramData\Everything\es.exe' - source: salt://everything/es.exe
This state file is formatted in YAML and described three states. Each state is merely checking for the existence of a file and if the file is not found in the named location, it will download the file from the source and put the contents in the target name. Notice the first state includes “makedirs: True”. This is a nice little shortcut to create the C:\ProgramData\Everything folder.
Now our directory looks like this.
Step 6, deploy Everything. Now that we have a state we can deploy it one system as a quick test to make sure we got everything right.
root@salt:~# salt 'TESTSERVER.example.com' state.sls everything.installed TESTSERVER.example.com: ---------- ID: Everything.exe Function: file.managed Name: C:\ProgramData\Everything\Everything.exe Result: True Comment: File C:\ProgramData\Everything\Everything.exe updated Started: 19:58:00.731509 Duration: 5107.766 ms Changes: ---------- diff: New file ---------- ID: Everything.lng Function: file.managed Name: C:\ProgramData\Everything\Everything.lng Result: True Comment: File C:\ProgramData\Everything\Everything.lng updated Started: 19:58:05.840275 Duration: 1269.513 ms Changes: ---------- diff: New file ---------- ID: es.exe Function: file.managed Name: C:\ProgramData\Everything\es.exe Result: True Comment: File C:\ProgramData\Everything\es.exe updated Started: 19:58:07.109788 Duration: 438.849 ms Changes: ---------- diff: New file Summary for TESTSERVER.example.com ------------ Succeeded: 3 (changed=3) Failed: 0 ------------ Total states run: 3 Total run time: 6.816 s
Notice in our command, “everything” corresponds to the folder /srv/salt/everything and “installed” corresponds to “installed.sls” in that same folder. And we can see three states succeeded with three states changed.
Deploy Everything to all systems,
salt '*' state.sls everything.installed.
Step 7, Find Log4j. Finally the time has come! We can run some inline PowerShell to return a list of all files everywhere with “log4j” in the name. Here is the command.
root@salt:~# salt '*' cmd.run shell=powershell '& C:\ProgramData\Everything\Everything.exe -admin; Start-Sleep 3; & C:\ProgramData\Everything\es.exe log4j | Sort; & C:\ProgramData\Everything\Everything.exe -quit' PRRPT01.example.com: PRRPT02.example.com: PRDB01.example.com: TSTWEB01.example.com: DEVDB01.example.com: TSTDB01.example.com: C:\Program Files\Microsoft SQL Server 2017\MSSQL14.MSSQLSERVER2017\MSSQL\Binn\Hadoop\FOO\slf4j-log4j12-1.6.1.jar PRPRC03.example.com: PRPRC02.example.com: PRDB02.example.com: PRWEB13.example.com: PRPRC04.example.com: PRWEB05.example.com: PRWEB11.example.com: DEVPRC01.example.com: PRWEB12.example.com: PRWEB06.example.com: PRWEB09.example.com: PRWEB10.example.com: PRWEB01.example.com: PRWEB04.example.com: PRWEB03.example.com: D:\Program Files\ArcGIS\Server\framework\lib\shared\apache-log4j-extras-1.2.17.jar D:\Program Files\ArcGIS\Server\framework\lib\shared\ignite-log4j-2.7.0.jar D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-1.2.12.jar D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-1.2-api-2.11.1.jar D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-api-2.11.1.jar D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-core-2.11.1.jar D:\Program Files\ArcGIS\Server\framework\lib\shared\log4j-jcl-2.11.1.jar D:\Program Files\ArcGIS\Server\framework\lib\shared\slf4j-log4j12-1.7.25.jar D:\Program Files\ArcGIS\Server\framework\runtime\spark\conf\log4j.properties.template D:\Program Files\ArcGIS\Server\framework\runtime\spark\jars\apache-log4j-extras-1.2.17.jar D:\Program Files\ArcGIS\Server\framework\runtime\spark\jars\log4j-1.2.17.jar D:\Program Files\ArcGIS\Server\framework\runtime\spark\jars\slf4j-log4j12-1.7.16.jar D:\Program Files\ArcGIS\Server\framework\runtime\tomcat\lib\log4j.jar D:\Program Files\ArcGIS\Server\framework\runtime\tomcat\lib\log4j.xml D:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\conf\log4j.properties D:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\lib\log4j-1.2.17.jar D:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\lib\log4j-1.2.17.LICENSE.txt D:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\lib\slf4j-log4j12-1.7.25.jar D:\Program Files\ArcGIS\Server\tools\configurebasedeployment\lib\log4j-api.jar D:\Program Files\ArcGIS\Server\tools\configurebasedeployment\lib\log4j-core.jar D:\Program Files\ArcGIS\Server\tools\configurebasedeployment\lib\log4j-jcl.jar D:\Program Files\ArcGIS\Server\tools\createsite\lib\log4j-api.jar D:\Program Files\ArcGIS\Server\tools\createsite\lib\log4j-core.jar D:\Program Files\ArcGIS\Server\tools\createsite\lib\log4j-jcl.jar D:\Program Files\ArcGIS\Server\tools\upgradebasedeployment\lib\log4j-api.jar D:\Program Files\ArcGIS\Server\tools\upgradebasedeployment\lib\log4j-core.jar D:\Program Files\ArcGIS\Server\tools\upgradebasedeployment\lib\log4j-jcl.jar D:\Program Files\ArcGIS\Server\tools\upgradeserver\lib\log4j-api.jar D:\Program Files\ArcGIS\Server\tools\upgradeserver\lib\log4j-core.jar D:\Program Files\ArcGIS\Server\tools\upgradeserver\lib\log4j-jcl.jarPRWEB07.example.com: PRWEB08.example.com: PRWEB02.example.com: PRPRC01.example.com: TSTPRC01.example.com: PRPRC05.example.com: DEVWEB01.example.com: root@salt:~#
And if you have Linux systems, Salt can of course help there. Install salt-minion, use a Salt grain to target Linux hosts and run find.
salt -G 'kernel:Linux' cmd.run 'find / -type f 2>/dev/null | grep -i log4j'
Have fun patching y’all! Happy Holidays and yippee kai yay!