Apache Arrow in JS
Arrow is a set of technologies that enable big data systems to process and transfer data quickly.
apache-arrow from NPM
npm install apache-arrow or
yarn add apache-arrow
(read about how we package apache-arrow below)
Powering Columnar In-Memory Analytics
Apache Arrow is a columnar memory layout specification for encoding vectors and table-like containers of flat and nested data. The Arrow spec aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors.
Apache Arrow is the emerging standard for large in-memory columnar data (Spark, Pandas, Drill, Graphistry, ...). By standardizing on a common binary interchange format, big data systems can reduce the costs and friction associated with cross-system communication.
Check out our API documentation to learn more about how to use Apache Arrow's JS implementation. You can also learn by example by checking out some of the following resources:
- Observable: Introduction to Apache Arrow
- Observable: Manipulating flat arrays arrow-style
- Observable: Rich columnar data tables - Dictionary-encoded strings, 64bit ints, and nested structs
- /js/test/unit - Unit tests for Table and Vector
Get a table from an Arrow file on disk (in IPC format)
;;const arrow = ;const table = Table;console;/*foo, bar, baz1, 1, aanull, null, null3, null, null4, 4, bbb5, 5, cccc*/
Create a Table when the Arrow file is split across buffers
;;const table = Table;console;/*origin_lat, origin_lon35.393089294433594, -97.600730895996135.393089294433594, -97.600730895996135.393089294433594, -97.600730895996129.533695220947266, -98.4697799682617229.533695220947266, -98.46977996826172*/
;const LENGTH = 2000;const rainAmounts = Float32Array;const rainDates = Array;const rainfall = Table;
Load data with
;const table = await Table;console;
Columns look like JS Arrays
;;const table = Table;const column = table;// Copy the data into a TypedArrayconst typed = column;;for let i = -1 n = columnlength; ++i < n;;
Usage with MapD Core
;;const port = 9091;const host = `localhost`;const db = `mapd`;const user = `mapd`;const password = `HyperInteractive`;MapD;/*Index, origin_city0, Oklahoma City1, Oklahoma City2, Oklahoma City3, San Antonio4, San Antonio*/
Even if you do not plan to contribute to Apache Arrow itself or Arrow integrations in other projects, we'd be happy to have you involved:
- Join the mailing list: send an email to email@example.com. Share your ideas and use cases for the project.
- Follow our activity on JIRA
- Learn the format
- Contribute code to one of the reference implementations
We prefer to receive contributions in the form of GitHub pull requests. Please send pull requests against the github.com/apache/arrow repository.
If you are looking for some ideas on what to contribute, check out the JIRA issues for the Apache Arrow project. Comment on the issue and/or contact firstname.lastname@example.org with your questions and ideas.
If you’d like to report a bug but don’t have time to fix it, you can still post it on JIRA, or email the mailing list email@example.com
apache-arrow is written in TypeScript, but the project is compiled to multiple JS versions and common module formats.
apache-arrow package includes all the compilation targets for convenience, but if you're conscientious about your
node_modules footprint, we got you.
The targets are also published under the
npm install apache-arrow # <-- combined es5/UMD, es2015/CommonJS/ESModules/UMD, and TypeScript packagenpm install @apache-arrow/ts # standalone TypeScript packagenpm install @apache-arrow/es5-cjs # standalone es5/CommonJS packagenpm install @apache-arrow/es5-esm # standalone es5/ESModules packagenpm install @apache-arrow/es5-umd # standalone es5/UMD packagenpm install @apache-arrow/es2015-cjs # standalone es2015/CommonJS packagenpm install @apache-arrow/es2015-esm # standalone es2015/ESModules packagenpm install @apache-arrow/es2015-umd # standalone es2015/UMD packagenpm install @apache-arrow/esnext-cjs # standalone esNext/CommonJS packagenpm install @apache-arrow/esnext-esm # standalone esNext/ESModules packagenpm install @apache-arrow/esnext-umd # standalone esNext/UMD package
Why we package like this
The JS community is a diverse group with a varied list of target environments and tool chains. Publishing multiple packages accommodates projects of all stripes.
If you think we missed a compilation target and it's a blocker for adoption, please open an issue.
Full list of broader Apache Arrow committers.
- Brian Hulette, committer
- Paul Taylor, Graphistry, Inc., committer
Powered By Apache Arrow in JS
Full list of broader Apache Arrow projects & organizations.
Open Source Projects
- Apache Arrow -- Parent project for Powering Columnar In-Memory Analytics, including affiliated open source projects
- rxjs-mapd -- A MapD Core node-driver that returns query results as Arrow columns
- Falcon is a visualization tool for linked interactions across multiple aggregate visualizations of millions or billions of records.
Companies & Organizations
- CCRi -- Commonwealth Computer Research Inc, or CCRi, is a Central Virginia based data science and software engineering company
- GOAI -- GPU Open Analytics Initiative standardizes on Arrow as part of creating common data frameworks that enable developers and statistical researchers to accelerate data science on GPUs
- Graphistry, Inc. - An end-to-end GPU accelerated visual investigation platform used by teams for security, anti-fraud, and related investigations. Graphistry uses Arrow in its NodeJS GPU backend and client libraries, and is an early contributing member to GOAI and Arrow[JS] working to bring these technologies to the enterprise.