About

sift is a fast and powerful open source alternative to grep. It comes as a single executable with no external dependencies - just download it and you are ready to go.

Why sift?

Most of the existing tools for searching through big amounts of text are either fast but inflexible (the original grep) or slightly more flexible but slow or complicated to install.

sift is an alternative that aims for both speed and flexibility - i.e. adding features while trying to reach (or even surpass) the performance of the original grep.
The additional features include multiline matching, conditions (e.g. match A only when preceded by B within X lines) and full multicore support.

sift is not tailored towards a specific use case (like system administration or code development/auditing), but you can customize it to your needs through a global or local configuration file. Don't want to search through .git directories? Just execute sift --exclude-dirs '.git' --write-config and you are done.

Please see the features and samples to get a better impression of what you can do with sift.

Performance

The following benchmarks show the performance of sift. While speed is not everything, it is quite a difference whether you get your results after 2 seconds or 2 minutes, especially when you are grepping through many files over and over (e.g. when searching in large source code repositories or big log files).
These results were achieved even though sift introduces new features that other tools do not have, like conditions and multiline matching.

In this comparision, all tools were configured to search the complete test data. Of course sift too can be configured to only search in specific paths/files/etc., but the aim was being fast while searching through the complete test data.

All searches were performed with the complete data files cached by the operating system. Three runs per test were performed and the best result was taken.
The weblog searches were done on a big server, while the rest was done on a desktop system.


Benchmark grep ack ag pt sift
Web log files search
This search simulated searching for a specific pattern in web logs.
The search was performed over 35GB data, split over 32 files, using the pattern 'IntWebApp.*ParamName'. The logs were synthesized from real logs, and there was one valid match to find ('IntWebApp' was part of the logged URL path, while 'ParamName' was a query parameter).
23.630s
40.81x
226.154s
390.59x
4.665s
8.06x
222.487s
384.26x
0.579s
1x
Web log files search for 10 strings in parallel
In this search, the same data as above was used, but the search was done for 10 static strings listed in a file. Some tools do not support searching for multiple patterns in parallel, here the result for a single search x 10 was taken as result.
148.497s
27.30x
(~2261s)
415.70x
(~46s)
8.46x
(~2224s)
408.90x
5.439s
1x
Linux source code
Listing all exported crypto symbols with line numbers - searching for "EXPORT_SYMBOL_GPL.*crypto" in the Linux kernel version 3.18.2 (637 MB).
0.747s
1.75x
25.929s
60.87x
1.140s
2.67x
17.840s
41.88x
0.426s
1x
Wordlist search
Searching a large wordlist (1.8 GB, used in password cracking attempts) for all word variations containing 'qwertz' (returning 2722 results).
1.285
1.74x
155.801s
211.11x
6.484s
8.79x
24.651s
33.402x
0.738s
1x
Userlist search (ignore case)
Searching a list of usernames (125 MB) for all name variations containing 'grep' (ignore case, returning 121 results).
0.381s
1.36x
10.050s
35.89x
0.497s
1.78x
7.485s
26.73x
0.280s
1x
Log search
This search was done over DNS logs (228 files, 15 MB). Some of the log files were already gzip'ed and the newest files were not.
For grep, two calls had to be made: grep for the normal files and zgrep for the gzip'ed files.
1.465s
6.37x
- 0.485s
2.11x
- 0.230s
1x