Nostalgic Punchcard Missing

    md5-wasm

    2.0.0 • Public • Published

    MD5-WASM

    MD5-WASM is a fast asynchronous md5 calculator, optimized for large files.  It returns a Promise which resolves to the md5 string.  WebAssembly is seamlessly applied to calculate values for files above a certain size threshold.

    Highlights

    ● 30x faster than the most popular md5 utility 
    ● Server-side (NodeJS) or client-side (browser) 
    ● Non-blocking, uses Promise syntax 

    Raison d'être  

    Faster and non-blocking

    Our md5 hashing was initially performed using this simple and popular utility:  https://www.npmjs.com/package/md5  (called "MD5" herein)   However, MD5 is synchronous, blocking code execution, and slow — impractically slow for video files.  (On our low-powered server platform, we clock it at about 1 second per megabyte.) 

    30x faster?

    On larger files, yes.  Here are the benchmarks, comparing MD5 to MD5-WASM, run on our (slow) production server platform using NodeJS (v10.18.1) on Ubuntu. 

                       ELAPSED MILLISECONDS        MEGABYTES PER SECOND
                       MD5         MD5-WASM         MD5        MD5-WASM
    0.2 Mbytes          260            90          1.05            2.2           
    0.3 Mbytes          390           300          0.98            1.4            
    0.5 Mbytes          520           360          0.98            1.4            
      1 Mbytes        1,000           170          1.00            8.5          
      2 Mbytes        2,000           240          1.00            8.5          
      4 Mbytes        4,000           330          1.00           12
      8 Mbytes        7,600           400          1.05           20
     12 Mbytes       12,400           490          0.96           24
     24 Mbytes       23,600           700          1.02           34
     37 Mbytes       38,500           990          0.96           37
    

    On our benchmark system, MD5-WASM gives up 150 ms to complete WebAssembly instantiation.  After that, the relative performance gap between the two keeps growing, reaching 30x for a 37Mbyte file. 

    Why the huge improvement?

    It would not be surprising to see a 3x improvement up to 5x improvement from WebAssembly but 30x is definitely surprising.  For md5 calculation, WebAssembly holds one other big advantage.  Any JavaScript implementation does a lot of number format conversion during md5 calculation, while WebAssembly implementations need not. 

    JavaScript runs native with 64-bit floating point numbers but all bitwise operations are done with 32-bit integers.  Since calculating a checksum is just scads of bitwise operations, Javascript implementations spend more time converting between number formats than they do on the checksum itself. 

    Is there a downside?

    You need do nothing different to accomodate WebAssembly — MD5-WASM loads in a browser or Node environment just like a pure JavaScript utility would.  Unlike MD5, MD5-WASM does not take parameters in a string format — you must convert the string before injecting it into MD5-WASM.  There is no synchronous version; you must use a promise instead of a simple blocking function call. 

    Javascript Calls And Parameters

    Usage example

    let data  = contentsOfAFile();                        // Get the data any which way you can
    
    // 'data' must be a Buffer, ArrayBuffer or Uint8Array
    md5WASM(data)                                         // Our function
        .then( hash => console.log(hash) )
        .catch( err => console.log(err) )
    

    Loading MD5-WASM

    At less than 32K, the code file does not justify minification.  It is all-inclusive and has no external dependencies. 

    HTML tag

    <script type="text/javascript" src="path/md5-wasm.js"></script>
    

    You will find the function at window.md5WASM

    In NodeJS

    md5WASM      = require("md5-wasm");
    

    Problems, questions

    Please open an issue at the GitHub repo.

    Install

    npm i md5-wasm

    DownloadsWeekly Downloads

    79

    Version

    2.0.0

    License

    MIT

    Unpacked Size

    58.1 kB

    Total Files

    5

    Last publish

    Collaborators

    • brianbutton