data-loss-signatures

    1.0.5 • Public • Published

    data-loss-signatures

    All Contributors

    logo

    Apache License FOSSA Status standard-readme compliant conventional commits JavaScript Style Guide

    Product summary Identify confidential and sensitive info in source code repositories by data-loss "signatures".

    data-loss-signatures is a Node.js module offsite web page for storing and accessing to data-leakage detection definitions. We call the data structure that represents a data-leakage detection defintion a "signature." We store a community-tested list of signatures in a file called signatures.json.

    Table of Contents

    1. Security

    citation Data leakage is the unauthorized transmission of data from within an organization to an external destination or recipient.^1

    One of the most common forms of data-loss (aka, "data leakage") happens when developers (inadvertently) commit and push passwords, access-tokens, and sensitive data to a source-control management system (like Git). Consequently, confidential information "leaks" into search results and commit history.

    The signatures.json contains a growing list of definitions to help you detect secrets in your source code repositories.

    Secret Detected in
    1 .pem file extension
    Potential cryptographic private key
    extension
    2 Log file
    Log files can contain secret HTTP endpoints, session IDs, API keys and other goodies
    extension
    3 .pkcs12 file extension
    Potential cryptographic key bundle
    extension
    4 .p12 file extension
    Potential cryptographic key bundle
    extension
    5 .pfx file extension
    Potential cryptographic key bundle
    extension
    6 .asc file extension
    Potential cryptographic key bundle
    extension
    7 Pidgin OTR private key filename
    8 OpenVPN client configuration file extension
    9 Azure service configuration schema file extension
    10 Remote Desktop connection file extension
    11 Microsoft SQL database file extension
    12 Microsoft SQL server compact database file extension
    13 SQLite database file extension
    14 Microsoft BitLocker recovery key file extension
    15 Microsoft BitLocker Trusted Platform Module password file extension
    16 Windows BitLocker full volume encrypted data file extension
    17 Java keystore file extension
    18 Password Safe database file extension
    19 Ruby On Rails secret token configuration file
    If the Rails secret token is known, it can allow for remote code execution (http://www.exploit-db.com/exploits/27527/)
    filename
    20 Carrierwave configuration file
    Can contain credentials for cloud storage systems such as Amazon S3 and Google Storage
    filename
    21 Potential Ruby On Rails database configuration file
    Can contain database credentials
    filename
    22 OmniAuth configuration file
    The OmniAuth configuration file can contain client application secrets
    filename
    23 Django configuration file
    Can contain database credentials, cloud storage system credentials, and other secrets
    filename
    24 1Password password manager database file
    Feed it to Hashcat and see if you're lucky
    extension
    25 Apple Keychain database file extension
    26 Network traffic capture file extension
    27 GnuCash database file extension
    28 Jenkins publish over SSH plugin file filename
    29 Potential Jenkins credentials file filename
    30 KDE Wallet Manager database file extension
    31 Potential MediaWiki configuration file filename
    32 Tunnelblick VPN configuration file extension
    33 Sequel Pro MySQL database manager bookmark file filename
    34 Little Snitch firewall configuration file
    Contains traffic rules for applications
    filename
    35 Day One journal file
    Now it's getting creepy...
    extension
    36 Potential jrnl journal file
    Now it's getting creepy...
    filename
    37 Chef Knife configuration file
    Can contain references to Chef servers
    filename
    38 cPanel backup ProFTPd credentials file
    Contains usernames and password hashes for FTP accounts
    filename
    39 Robomongo MongoDB manager configuration file
    Can contain credentials for MongoDB databases
    filename
    40 FileZilla FTP configuration file
    Can contain credentials for FTP servers
    filename
    41 FileZilla FTP recent servers file
    Can contain credentials for FTP servers
    filename
    42 Ventrilo server configuration file
    Can contain passwords
    filename
    43 Terraform variable config file
    Can contain credentials for terraform providers
    filename
    44 Shell configuration file
    Shell configuration files can contain passwords, API keys, hostnames and other goodies
    filename
    45 Shell configuration file
    Shell configuration files can contain passwords, API keys, hostnames and other goodies
    filename
    46 Shell configuration file
    Shell configuration files can contain passwords, API keys, hostnames and other goodies
    filename
    47 Private SSH key filename
    48 Private SSH key filename
    49 Private SSH key filename
    50 Private SSH key filename
    51 SSH configuration file path
    52 Potential cryptographic private key extension
    53 Shell command history file filename
    54 MySQL client command history file filename
    55 PostgreSQL client command history file filename
    56 PostgreSQL password file filename
    57 Ruby IRB console history file filename
    58 Pidgin chat client account configuration file path
    59 Hexchat/XChat IRC client server list configuration file path
    60 Irssi IRC client configuration file path
    61 Recon-ng web reconnaissance framework API key database path
    62 DBeaver SQL database manager configuration file filename
    63 Mutt e-mail client configuration file filename
    64 S3cmd configuration file filename
    65 AWS CLI credentials file path
    66 SFTP connection configuration file filename
    67 T command-line Twitter client configuration file filename
    68 gitrob configuration file filename
    69 Shell configuration file
    Shell configuration files can contain passwords, API keys, hostnames and other goodies
    filename
    70 Shell profile configuration file
    Shell configuration files can contain passwords, API keys, hostnames and other goodies
    filename
    71 Shell command alias configuration file
    Shell configuration files can contain passwords, API keys, hostnames and other goodies
    filename
    72 PHP configuration file filename
    73 GNOME Keyring database file extension
    74 KeePass password manager database file
    Feed it to Hashcat and see if you're lucky
    extension
    75 SQL dump file extension
    76 Apache htpasswd file filename
    77 Configuration file for auto-login process
    Can contain username and password
    filename
    78 Rubygems credentials file
    Can contain API key for a rubygems.org account
    path
    79 Tugboat DigitalOcean management tool configuration filename
    80 DigitalOcean doctl command-line client configuration file
    Contains DigitalOcean API key and other information
    path
    81 git-credential-store helper credentials file filename
    82 GitHub Hub command-line client configuration file
    Can contain GitHub API access token
    path
    83 Git configuration file filename
    84 Chef private key
    Can be used to authenticate against Chef servers
    path
    85 Potential Linux shadow file
    Contains hashed passwords for system users
    path
    86 Potential Linux passwd file
    Contains system user information
    path
    87 Docker configuration file
    Can contain credentials for public or private Docker registries
    filename
    88 NPM configuration file
    Can contain credentials for NPM registries
    filename
    89 Environment configuration file filename
    90 Contains word: credential path
    91 Contains word: password path

    2. Install

    Before you begin, you'll need to have these

    Terminal Open a Terminal and enter the following command:

    # As a dependency in your Node.js app 
    npm i data-loss-signatures --save-prod

    3. Usage

    Use data-loss-signatures.signatures to find file extensions, names, and paths that commonly leak secrets.

    const { signatures } = require('data-loss-signatures')
    // ⚠️ Note: the 'recursive-readdir' module is not bundled with
    //    data-loss-signatures. 'recursive-readdir' is referenced
    //    only as an example.
    const recursiveReaddir = require('recursive-readdir')
     
    const potentialLeaks = recursiveReaddir('/path/to/local/repo')
      .then(files => files
        .map(file => signatures
        .map(signature => signature.match(file)))
      )
      .catch(err => err)

    4. API

    The data-loss-signatures module provides a Signatures class, which validates data-loss-signatures and converts regular expression strings to RE2 (whenever possible).

    The data-loss-signatures module's public API provides:

    1. factory method: a convenience function that creates a signature object.
    2. nullSignature: implements a default object literal with all signatures properties set to null.
    3. Signature: a class that constructs a signature object.
    4. signatures: an array of Signature instances.
    5. toArray(data: {String|Array.<Object>}): generates an Array.<Signature> from a JSON string or object literal array.
    6. validParts: a constants enum of valid Signature.prototype.part values.
    7. validTypes: a constants enum of valid Signature.prototype.type values.

    4.1. data-loss-signatures.Signature

    A class that constructs Signature objects.

    const { Signature, validParts, validTypes } = require('data-loss-signatures')
     
    const signature = new Signature({
      caption: 'Potential cryptographic private key',
      description: '',
      part: validParts.EXTENSION,
      pattern: '.pem',
      type: validTypes.MATCH
    })

    4.2. data-loss-signatures.Signature.prototype.match

    Discover possible data leaks by matching a Signature pattern against file extensions, names, and paths.

    const rsaTokenSignature = new Signature({
      'caption': 'Private SSH key',
      'description': '',
      'part': 'filename',
      'pattern': '^.*_rsa$',
      'type': 'regex'
    })
     
    const suspiciousFilePath = '/hmm/what/might/this/be/id_rsa'
    rsaTokenSignature.match(suspiciousFilePath)
    // => ['/hmm/what/might/this/be/id_rsa']
     
    const fileThatIsJustBeingCoolBruh = 'file/that/is/just/being/cool/bruh'
    rsaTokenSignature.match(suspiciousFilePath)
    // => null

    source code Review the source code for signature.

    5. Accessing signatures with other tools and programming languages

    You can access signatures.json without the data-loss-signatures Node module. Select a tool or programming language below to view examples.

    cURL

    You can access data-loss rules using HTTPS. You can GET all signatures directly from Gitlab with cURL.

    curl -X GET \
      'https://gitlab.com/gregswindle/data-loss-signatures/raw/master/signatures.json'
    Golang
    package main
     
    import (
        "fmt"
        "net/http"
        "io/ioutil"
    )
     
    func main() {
     
        url := "https://gitlab.com/gregswindle/data-loss-signatures/raw/master/signatures.json"
     
        req, _ := http.NewRequest("GET", url, nil)
     
        req.Header.Add("Private-Token", "<your-personal-token>")
        req.Header.Add("cache-control", "no-cache")
     
        res, _ := http.DefaultClient.Do(req)
     
        defer res.Body.Close()
        body, _ := ioutil.ReadAll(res.Body)
     
        fmt.Println(res)
        fmt.Println(string(body))
     
    }
    Node (native)
    const http = require('https')
     
    const options = {
      method: 'GET',
      hostname: ['gitlab', 'com'],
      path: ['api', 'v4', 'projects'],
      headers: {
        'Private-Token': '<your-access-token>',
        'cache-control': 'no-cache'
      }
    }
     
    const req = http.request(options, res => {
      const chunks = []
     
      res.on('data', chunk => {
        chunks.push(chunk)
      })
     
      res.on('end', () => {
        var body = Buffer.concat(chunks)
        console.log(body.toString())
      })
    })
     
    req.end()
    Python

    Python3

    import http.client
     
    conn = http.client.HTTPConnection("gitlab,com")
     
    payload = ""
     
    headers = { 
      'Accept': "application/json", 
      'cache-control': "no-cache"
    }
     
    conn.request("GET""gregswindle,data-loss-signatures,raw,master,signatures.json"payloadheaders)
     
    res = conn.getresponse()
    data = res.read()
     
    print(data.decode("utf-8"))

    Python2

    import requests
     
    url = "https://gitlab.com/gregswindle/data-loss-signatures/raw/master/signatures.json"
     
    payload = ""
    headers = { 
      'Accept': "application/json", 
      'cache-control': "no-cache"
    }
     
    response = requests.request("GET"urldata=payload, headers=headers)
     
    print(response.text)
    Ruby (NET::Http)
    require 'uri'
    require 'net/http'
     
    url = URI("'https://gitlab.com/gregswindle/data-loss-signatures/raw/master/signatures.json")
     
    http = Net::HTTP.new(url.host, url.port)
     
    request = Net::HTTP::Get.new(url)
    request["Private-Token"] = '<your-personal-token>'
    request["cache-control"] = 'no-cache'
     
    response = http.request(request)
    puts response.read_body

    6. Maintainers

    @gregswindle

    Information for Maintainers The Maintainer Guide has useful information for Maintainers and Trusted Committers.

    7. Contributions

    We gratefully accept Merge Requests! Here's what you need to know to get started.

    Before submitting a Merge Request, please read Before submitting a Merge Request, please read our:

    Thanks goes to our awesome contributors (emoji key):

    Semantic Release Bot
    Semantic Release Bot

    🚧
    gregswindle
    gregswindle

    💻 ⚠️ 📖 🐛 🚧
    Christina Valdes
    Christina Valdes

    👀
    sairam pooraj
    sairam pooraj

    👀

    This project follows the all-contributors specification. Contributions of any kind welcome!

    7.1. Adding a Signature

    Before adding a new Signature, please review all current definitions: the Signature might already exist.

    If the Signature does not exist, please be sure to add your Signature with the following properties:

    1. caption: A succinct summary for the Signature. Think of caption as a well-written email subject.

    2. description: Provide more details about the Signature if necessary. description is especially useful for differentiating similar Signatures.

    3. part: An enumeration that defines what the Signature is evaluating. Valid values are:

      • contents: The string(s) within a file.
      • extension: A file extension (which defines the Content-Type or mime-type).
      • filename: The unique name of the file.
      • path: The directory path relative to the repo and without the filename.
    4. pattern: The string or regular expression to look for.

    5. type: An enumeration that defines how to evaluate for secrets. Valid values are:

      • match: A strict string equivalency evaluation.
      • regex: A regular expression "search" or "test".

    7.2. Editing a Signature

    Edits are welcome! Just be sure to unit test.

    7.3. Removing a Signature

    Please provide a testable justification for any Signature removal.

    8. License

    Apache-2.0 © 2019 Greg Swindle

    FOSSA Status

    9. References and Attributions

    Retrieved January 27, 2019, from https://www.forcepoint.com/cyber-edu/data-leakage

    Install

    npm i data-loss-signatures

    DownloadsWeekly Downloads

    2

    Version

    1.0.5

    License

    Apache-2.0

    Unpacked Size

    850 kB

    Total Files

    6

    Last publish

    Collaborators

    • gregswindle