Nomadic People Migration


    0.6.1 • Public • Published


    dissect dead node service core dumps with mdb via a smart os vm



    There is a Unix-like operating system called SmartOS whose ancestry represents a strong investment in low-level introspection tools (such as dtrace for instance).

    Once such tool is mdb, a high quality modular debugger which ships with SmartOS - it can be used to inspect execution context from the kernel to application layers.

    Some rather clever people wrote a debugging module called mdb_v8 that allows introspection of node core dumps from a high level (e.g. inspecting closure scope) to a low level (e.g. memory addresses).

    It turns out that mdb can analyse Linux core files as well as SmartOS core files. We just have provide the core file and the node binary that was running when the core dump was generated.

    autopsy installs a a SmartOS VM and then acts as a stdio proxy to mdb.

    For using mdb see the mdb reference docs


    • VirtualBox
    • 2gb of RAM for VM
      • The VM runs entirely in RAM and mdb can be memory intensive also, 2gb is a safe bet


    Install autopsy from npm:

    npm install -g autopsy

    Once finished the following executables will be available

    • autopsy setup - runs setup
    • autopsy start - starts the vm
    • autopsy stop - stops the vm
    • autopsy status - gets vm status
    • autopsy rm - removes vm
    • autopsy ssh - use exactly as you would ssh, provides a tunnel from a server to locally installed vm
    • autopsy - provides the CLI proxy to mdb in the vm

    Next, set up the VM

    VM Setup

    autopsy setup

    This will install autopsy on the system, download smartos virtual machine assets and setup a smartos vm in virtual box.

    Assets for the VM are ~150mb and downloads from S3.

    If setup is interupted for any reason (including network failure during assets download), simply try again. Partial downloads will be resumed.

    Starting the VM

    Before we can do an autopsy the VM needs to be running.

    Simply run

    autopsy start

    Autopsy takes a snapshot of the initial VM state on first run to optimize subsequent boots, so the first autopsy-start will be the longest.

    The VM runs SmartOS completely in ram (there are no zones). This means VM state is immutable.

    Performing an autopsy

    The autopsy command takes the following args

    autopsy [node-binary] core-file

    On OS X the node binary is not optional, on Linux if not supplied the current installed node binary will be used.

    When this command is run the following occurs

    1. copy the node binary and core file into the vm (in a purpose built smartos zone)
    2. ssh into the vm, login to the relevant smartos zone
    3. run mdb with the copied core dump and node binary files
    4. inject ::load v8 to get the v8 related debugging commands
    5. pipe host machine process.stdin to the mdb interactive environment and pipe the mdb stdout to the host machine stdout

    For using mdb see the mdb reference docs

    Stopping the VM

    When we're done we may wish to free memory by stopping the VM with

    autopsy stop


    The example folder has a core and node file that we're generated by the die.js file

    You can try out autopsy with these two files (on OS X and Linux), from the same folder as this readme do

    autopsy example/node example/core 

    Once the mdb console appears you can try

    > ::jsstack

    For starters, and then if you want to get fancy

    > ::findjsobjects -p myproperty
    > 137289672551::jsprint

    Autopsy on EC2 or in a VM

    EC2 (and other VPS-type solutions) runs "machines" in virtualized containers, it's very tricky to make a virtual machine run on a virtual machines, and even where it is possible there is either an insufferable performance cost and/or certain low level features must be enabled which risk of introducing security issues. That aside copying node, a core file and using mdb all in a ram-only VM is memory intensive - not something we want to do (or maybe even can do) on a production server.

    But autopsy provides a way to do seamless postmortems on an EC2 server or any kind of linux VM - by setting up an SSH tunnel back to through the local machine and into the SmartOS vm running locally.

    This can be achieved in a few easy steps

    1. install autopsy on local system and run autopsy setup, ensure its working locally
    2. install autopsy within a VM or on an EC2 server (etc.)
    3. ssh into the server in the same way as always, except prefix the command with autopsy, like so:
    autopsy ssh -i myKey.pem

    Simply use whatever ssh flags you normally would, and autopsy will additionally set up the tunnelling (for the curious we inject the -R flag with the port of VM mapped through to the same port on the server.)

    How to generate a core file

    In production, if we run our node processes with --abort-on-uncaught-exception we will always get a core dump when a process crashes (that is, as long as our linux environment is set up correctly)

    You can also manually generate a core file using process.abort().

    Finally a core file can also be obtained by attaching gdb to a running processing and executing generate-core.

    Setting up Linux to generate core files

    If you're using an ubuntu server (and probably debian etc. etc.) you may have apport installed - this intercepts core files so we need to get rid of it

    sudo apt-get purge apport

    Next you need to make sure that linux is configured to allocate space for the core file, like so

    ulimit -c unlimited


    Port 2222

    The VM currently maps port 2222 to the port 22 (ssh), at the moment is non-configurable - so to use autopsy port 2222 needs to be free on the host system.

    Removing the VM

    Currently there's no command for removing the vm, follow these steps, in order

    1. open virtual box, right click the vm, click remove - then click "delete all files"
    2. rm the assets folder from the autopsy module folder rm $(npm get prefix)/lib/node_modules/autopsy/assets
    3. make sure there isn't a smartos folder left in the virtual box virtual machines folder (~/VirtualBox\ VMs)

    Singleton VM - Run Globally

    We recommend installing globally, since there can (currently) only be one smartos vm.

    Virtualbox Filesystem Coupling

    If the smartos.iso file or any parent folder is moved/renamed the vm will fail to start because virtualbox won't be able to locate the the iso. In this case you would need to manually update virtual box with the paths.

    Debug Logs

    For troubleshooting (or the curious), debugging can be turned on like so

    DEBUG=autopsy:* <cmd>

    At present the following commands have debug output

    • autopsy
    • autopsy setup
    • autopsy start




    npm i autopsy

    DownloadsWeekly Downloads






    Last publish


    • davidmarkclements
    • matteo.collina
    • pelger