using the osquery carver to pull files

The osquery carver is a feature of osquery that allows you to pull files back to your distributed endpoint from a client machine. This feature is available in all versions of osquery after 2.5.0.

configuration

The carver is an advanced feature of osquery and requires several flags to be set for it to work properly or at all. These are the flags I'll be working with in this explanation. Note that you must also have flags set up for a distributed endpoint if you intend to run the example commands.

--disable_carver=false
--carver_compression=true
--carver_block_size=300000
--carver_start_endpoint=/start_uploads
--carver_continue_endpoint=/upload_blocks
--carver_disable_function=false

To go through these in order:

  • Enable the carver as it's disabled by default
  • Turn on Zstandard (Zstd) compression for files sent back to the server
  • How many bytes should each block be that's sent to the server
  • Where is the endpoint to tell the server we're starting a carve
  • Where is the endpoint to give the server a block of data
  • Allow the use of the carver function

carver as a table

The simplest way to use the osquery carver is to directly query the carves table.

select * from carves where path like '/some/path/%' and carve=1;

This is the easiest way to issue a carve and will have osquery copy all files in that directory, tar them, compress the tar, and send it to the endpoint. You can also query the table without any where clauses to get the status, IDs, and hashes of all previous carves. Using the carver as a table is generally a safe operation because you more or less know exactly what you're going to get.

Carver as a function

A more advanced function of the carver is the ability to use its osquery aggregate function to collect a bunch of paths and carve them. An example of this might be:

select carve(path) from file where directory like '/Users/%/Downloads/' and mode='0755' and type == 'regular';

This would carve all files with permission 0755 from every user's download folder. The aggregate function makes it easy to carve a many paths based on a criteria but may result in large carves.

Carver compression

Using the carver compression flag, the resulting tar file will be compressed with Zstd before being sent to the server. We use Zstd because it's highly performant and provides very good compression (generally better than Zlib) depending on how compressible the data is. Zstd is a Facebook open source project and you can find it on github to build or make pull requests here. However if you feel integrating Zstd into your backend would be too cumbersome you can disable it and the file uploaded will be an uncompressed tar.