signatures (@data-leakage-protection/signatures)
Identify confidential and sensitive info in source code repositories by data-loss "signatures".
@data-leakage-protection/signatures is a Node.js
module
for storing and accessing to data-leakage detection definitions.
We call the data structure that represents a data-leakage detection
defintion a "signature." We store a community-tested list of signatures in a
file called signatures.json
.
Table of Contents
- 1. Security
- 2. Install
- 3. Usage
- 4. API
- 5. Accessing signatures with other tools and programming languages
- 6. Maintainers
- 7. Contributions
- 8. License
- 9. References and Attributions
1. Security
Data leakage is the unauthorized transmission of data from within an organization to an external destination or recipient.1
One of the most common forms of data-loss (aka, "data leakage") happens when developers (inadvertently) commit and push passwords, access-tokens, and sensitive data to a source-control management system (like Git). Consequently, confidential information "leaks" into search results and commit history.
The signatures.json contains a growing list of definitions to help you detect secrets in your source code repositories.
Signature | Detected in | |
---|---|---|
1 |
.asc file extension Potential cryptographic key bundle |
extension |
2 |
.p12 file extension PKCS#12 (.p12): potential cryptographic key bundle |
extension |
3 |
.pem file extension Potential cryptographic private key |
extension |
4 |
.pfx file extension PKCS#12 (.pfx): Potential cryptographic key bundle |
extension |
5 |
.pkcs12 file extension PKCS#12 (.pkcs12): Potential cryptographic key bundle |
extension |
6 |
1Password password manager database file Feed it to Hashcat and see if you're lucky |
extension |
7 |
AWS API Key __ |
contents |
8 |
AWS CLI credentials file __ |
path |
9 |
Apache htpasswd file __ |
filename |
10 |
Apple Keychain database file __ |
extension |
11 |
Azure service configuration schema file __ |
extension |
12 |
Carrierwave configuration file Can contain credentials for cloud storage systems such as Amazon S3 and Google Storage |
filename |
13 |
Chef Knife configuration file Can contain references to Chef servers |
filename |
14 |
Chef private key Can be used to authenticate against Chef servers |
path |
15 |
Configuration file for auto-login process Can contain username and password |
filename |
16 |
Contains word: credential __ |
path |
17 |
Contains word: password __ |
path |
18 |
DBeaver SQL database manager configuration file __ |
filename |
19 |
Day One journal file Now it's getting creepy... |
extension |
20 |
DigitalOcean doctl command-line client configuration file Contains DigitalOcean API key and other information |
path |
21 |
Django configuration file Can contain database credentials, cloud storage system credentials, and other secrets |
filename |
22 |
Docker configuration file Can contain credentials for public or private Docker registries |
filename |
23 |
Environment configuration file __ |
filename |
24 |
Facebook Oauth __ |
contents |
25 |
FileZilla FTP configuration file Can contain credentials for FTP servers |
filename |
26 |
FileZilla FTP recent servers file Can contain credentials for FTP servers |
filename |
27 |
GNOME Keyring database file __ |
extension |
28 |
Generic API Key __ |
contents |
29 |
Generic Secret __ |
contents |
30 |
Git configuration file __ |
filename |
31 |
GitHub __ |
contents |
32 |
GitHub Hub command-line client configuration file Can contain GitHub API access token |
path |
33 |
GnuCash database file __ |
extension |
34 |
Google (GCP) Service-account __ |
contents |
35 |
Google Oauth __ |
contents |
36 |
Heroku API Key __ |
contents |
37 |
Hexchat/XChat IRC client server list configuration file __ |
path |
38 |
Irssi IRC client configuration file __ |
path |
39 |
Java keystore file __ |
extension |
40 |
Jenkins publish over SSH plugin file __ |
filename |
41 |
KDE Wallet Manager database file __ |
extension |
42 |
KeePass password manager database file Feed it to Hashcat and see if you're lucky |
extension |
43 |
Little Snitch firewall configuration file Contains traffic rules for applications |
filename |
44 |
Log file Log files can contain secret HTTP endpoints, session IDs, API keys and other goodies |
extension |
45 |
Microsoft BitLocker Trusted Platform Module password file __ |
extension |
46 |
Microsoft BitLocker recovery key file __ |
extension |
47 |
Microsoft SQL database file __ |
extension |
48 |
Microsoft SQL server compact database file __ |
extension |
49 |
Mutt e-mail client configuration file __ |
filename |
50 |
MySQL client command history file __ |
filename |
51 |
NPM configuration file Can contain credentials for NPM registries |
filename |
52 |
Network traffic capture file __ |
extension |
53 |
OmniAuth configuration file The OmniAuth configuration file can contain client application secrets |
filename |
54 |
OpenVPN client configuration file __ |
extension |
55 |
PGP private key block __ |
contents |
56 |
PHP configuration file __ |
filename |
57 |
Password Safe database file __ |
extension |
58 |
Password in URL __ |
contents |
59 |
Pidgin OTR private key __ |
filename |
60 |
Pidgin chat client account configuration file __ |
path |
61 |
PostgreSQL client command history file __ |
filename |
62 |
PostgreSQL password file __ |
filename |
63 |
Potential Jenkins credentials file __ |
filename |
64 |
Potential Linux passwd file Contains system user information |
path |
65 |
Potential Linux shadow file Contains hashed passwords for system users |
path |
66 |
Potential MediaWiki configuration file __ |
filename |
67 |
Potential Ruby On Rails database configuration file Can contain database credentials |
filename |
68 |
Potential cryptographic private key __ |
extension |
69 |
Potential jrnl journal file Now it's getting creepy... |
filename |
70 |
Private SSH key _rsa |
filename |
71 |
Private SSH key _dsa |
filename |
72 |
Private SSH key _ed25519 |
filename |
73 |
Private SSH key _ecdsa |
filename |
74 |
RSA private key __ |
contents |
75 |
Recon-ng web reconnaissance framework API key database __ |
path |
76 |
Remote Desktop connection file __ |
extension |
77 |
Robomongo MongoDB manager configuration file Can contain credentials for MongoDB databases |
filename |
78 |
Ruby IRB console history file __ |
filename |
79 |
Ruby On Rails secret token configuration file If the Rails secret token is known, it can allow for remote code execution (http://www.exploit-db.com/exploits/27527/) |
filename |
80 |
Rubygems credentials file Can contain API key for a rubygems.org account |
path |
81 |
S3cmd configuration file __ |
filename |
82 |
SFTP connection configuration file __ |
filename |
83 |
SQL dump file __ |
extension |
84 |
SQLite database file __ |
extension |
85 |
SSH (DSA) private key __ |
contents |
86 |
SSH (EC) private key __ |
contents |
87 |
SSH (OPENSSH) private key __ |
contents |
88 |
SSH configuration file __ |
path |
89 |
Sequel Pro MySQL database manager bookmark file __ |
filename |
90 |
Shell command alias configuration file Shell configuration files can contain passwords, API keys, hostnames and other goodies |
filename |
91 |
Shell command history file __ |
filename |
92 |
Shell configuration file (.exports): Shell configuration files can contain passwords, API keys, hostnames and other goodies |
filename |
93 |
Shell configuration file (.functions): Shell configuration files can contain passwords, API keys, hostnames and other goodies |
filename |
94 |
Shell configuration file (.extra): Shell configuration files can contain passwords, API keys, hostnames and other goodies |
filename |
95 |
Shell configuration file (bash, zsh, csh): Shell configuration files can contain passwords, API keys, hostnames and other goodies |
filename |
96 |
Shell profile configuration file (profile): Shell configuration files can contain passwords, API keys, hostnames and other goodies |
filename |
97 |
Slack Token __ |
contents |
98 |
Slack Webhook __ |
contents |
99 |
T command-line Twitter client configuration file __ |
filename |
100 |
Terraform variable config file Can contain credentials for terraform providers |
filename |
101 |
Tugboat DigitalOcean management tool configuration __ |
filename |
102 |
Tunnelblick VPN configuration file __ |
extension |
103 |
Twilio API Key __ |
contents |
104 |
Twitter Oauth __ |
contents |
105 |
Ventrilo server configuration file Can contain passwords |
filename |
106 |
Windows BitLocker full volume encrypted data file __ |
extension |
107 |
cPanel backup ProFTPd credentials file Contains usernames and password hashes for FTP accounts |
filename |
108 |
git-credential-store helper credentials file __ |
filename |
109 |
gitrob configuration file __ |
filename |
2. Install
Before you begin, you'll need to have these
Programming languages:
Skills:
You'll need to know how to access the command line (aka, "Terminal") on your machine.
Open a Terminal and enter the following command:
# As a dependency in your Node.js app
npm i @data-leakage-protection/signatures --save-prod
3. Usage
Use @data-leakage-protection/signatures.signatures
to find file extensions, names, and paths
that commonly leak secrets.
const { signatures } = require('@data-leakage-protection/signatures')
// ⚠️ Note: the 'recursive-readdir' module is not bundled with
// @data-leakage-protection/signatures. 'recursive-readdir' is referenced
// only as an example.
const recursiveReaddir = require('recursive-readdir')
const potentialLeaks = recursiveReaddir('/path/to/local/repo')
.then(files => files
.map(file => signatures
.map(signature => signature.match(file)))
)
.catch(err => err)
4. API
The @data-leakage-protection/signatures module provides a
Signatures
class, which validates @data-leakage-protection/signatures and
converts regular expression strings to RE2 (whenever possible).
The @data-leakage-protection/signatures module's public API provides:
-
factory
method: a convenience function that creates a signature object. -
nullSignature
: implements a default object literal with all signatures properties set tonull
. -
Signature
: a class that constructs a signature object. -
signatures
: an array ofSignature
instances. -
toArray(data: {String|Array.<Object>})
: generates anArray.<Signature>
from a JSON string or object literal array. -
validParts
: a constants enum of validSignature.prototype.part
values. -
validTypes
: a constants enum of validSignature.prototype.type
values.
@data-leakage-protection/signatures.Signature
4.1. A class that constructs Signature objects.
const { Signature, validParts, validTypes } = require('@data-leakage-protection/signatures')
const signature = new Signature({
caption: 'Potential cryptographic private key',
description: '',
part: validParts.EXTENSION,
pattern: '.pem',
type: validTypes.MATCH
})
@data-leakage-protection/signatures.Signature.prototype.match
4.2. Discover possible data leaks by match
ing a Signature pattern
against file extensions, names, and paths.
const rsaTokenSignature = new Signature({
'caption': 'Private SSH key',
'description': '',
'part': 'filename',
'pattern': '^.*_rsa$',
'type': 'regex'
})
const suspiciousFilePath = '/hmm/what/might/this/be/id_rsa'
rsaTokenSignature.match(suspiciousFilePath)
// => ['/hmm/what/might/this/be/id_rsa']
const fileThatIsJustBeingCoolBruh = 'file/that/is/just/being/cool/bruh'
rsaTokenSignature.match(suspiciousFilePath)
// => null
Review the source code for signature
.
5. Accessing signatures with other tools and programming languages
You can access signatures.json
without the @data-leakage-protection/signatures
Node module. Select a tool or programming language below to view examples.
cURL
You can access data-loss rules using HTTPS. You can GET all signatures directly from Gitlab with cURL.
curl -X GET \
'https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json'
Golang
package main
import (
"fmt"
"net/http"
"io/ioutil"
)
func main() {
url := "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json"
req, _ := http.NewRequest("GET", url, nil)
req.Header.Add("Private-Token", "<your-personal-token>")
req.Header.Add("cache-control", "no-cache")
res, _ := http.DefaultClient.Do(req)
defer res.Body.Close()
body, _ := ioutil.ReadAll(res.Body)
fmt.Println(res)
fmt.Println(string(body))
}
Java (OK HTTP)
OkHttpClient client = new OkHttpClient();
String signaturesJson = "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json";
Request request = new Request.Builder()
.url(signaturesJson)
.get()
.addHeader("Accept", "*/*")
.addHeader("Cache-Control", "no-cache")
.addHeader("Host", "gitlab.com")
.addHeader("accept-encoding", "gzip, deflate")
.addHeader("Connection", "keep-alive")
.addHeader("cache-control", "no-cache")
.build();
Response response = client.newCall(request).execute();
Node (native)
const http = require('https')
const options = {
method: 'GET',
hostname: ['gitlab', 'com'],
path: ['api', 'v4', 'projects'],
headers: {
'Private-Token': '<your-access-token>',
'cache-control': 'no-cache'
}
}
const req = http.request(options, res => {
const chunks = []
res.on('data', chunk => {
chunks.push(chunk)
})
res.on('end', () => {
var body = Buffer.concat(chunks)
console.log(body.toString())
})
})
req.end()
Python (versions 2 and 3)
Python3
import http.client
conn = http.client.HTTPConnection("gitlab,com")
payload = ""
headers = {
'Accept': "application/json",
'cache-control': "no-cache"
}
conn.request("GET", "data-leakage-protection/signatures,raw,master,signatures.json", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
Python2
import requests
url = "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json"
payload = ""
headers = {
'Accept': "application/json",
'cache-control': "no-cache"
}
response = requests.request("GET", url, data=payload, headers=headers)
print(response.text)
Ruby (NET::Http)
require 'uri'
require 'net/http'
url = URI("'https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json")
http = Net::HTTP.new(url.host, url.port)
request = Net::HTTP::Get.new(url)
request["Private-Token"] = '<your-personal-token>'
request["cache-control"] = 'no-cache'
response = http.request(request)
puts response.read_body
6. Maintainers
The Maintainer Guide has useful information for Maintainers and Trusted Committers.
7. Contributions
We gratefully accept Merge Requests! Here's what you need to know to get started.
Thanks goes to our awesome contributors (emoji key):
Semantic Release Bot |
gregswindle |
Christina Valdes |
sairam pooraj |
This project follows the all-contributors specification. Contributions of any kind welcome!
7.1. Adding a Signature
Before adding a new Signature, please review all current definitions: the Signature might already exist.
If the Signature does not exist, please be sure to add your Signature with the following properties:
-
caption
: A succinct summary for the Signature. Think of caption as a well-written email subject. -
description
: Provide more details about the Signature if necessary. description is especially useful for differentiating similar Signatures. -
hash
: A hexidecimal SHA256 representation of a Signature (with ordered properties). -
name
: The Signature'scaption
, converted to kebab-case. -
part
: An enumeration that defines what the Signature is evaluating. Valid values are:-
contents
: The string(s) within a file. -
extension
: A file extension (which defines the Content-Type or mime-type). -
filename
: The unique name of the file. -
path
: The directory path relative to the repo and without the filename.
-
-
pattern
: The string or regular expression to look for. -
type
: An enumeration that defines how to evaluate for secrets. Valid values are:-
match
: A strict string equivalency evaluation. -
regex
: A regular expression "search" or "test".
-
7.2. Editing a Signature
Edits are welcome! Just be sure to unit test.
7.3. Removing a Signature
Please provide a testable justification for any Signature removal.
8. License
© 2019 Greg Swindle.
9. References and Attributions
-
What is Data Leakage? Defined, Explained, and Explored | Forcepoint. (2019) Retrieved January 27, 2019, from https://www.forcepoint.com/cyber-edu/data-leakage
↩