TileDB Cloud library for NodeJS
Installation
Basic usage
Browser usage
Cap'n Proto serialization
Queries
Development
TileDB Cloud.
TileDB-Cloud-JS is an axios-based NodeJS API forNote: This is an experimental package and aims to be similar to TileDB-Cloud-Py.
Installation
npm install @tiledb-inc/tiledb-cloud
Basic usage:
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// First we must create the client.
const tiledbClient = new client(config);
tiledbClient.userProfile().then((res) => {
console.log(res);
}).catch((ex) => {
console.log(ex);
});
Browser usage:
TileDB-Cloud-JS can also be used directly in the browser. Please see the following demo for more information:
Download a file in TileDB format:
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// First we must create the client.
const tiledbClient = new client(config);
client.downloadFile('namespace', 'tiledb_file').then((res) => {
console.log('saved!');
});
Download a TileDB notebook:
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// First we must create the client.
const tiledbClient = new client(config);
client.downloadNotebookToFile("kostas", "test_plot_widget").then((res) => {
console.log('saved as test_plot_widget.ipynb');
});
Serverless SQL example:
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// Create a client instance
const tiledbClient = new client(config);
// SQL query
const sql = "select `rows`, AVG(a) as avg_a from `tiledb://TileDB-Inc/quickstart_dense` GROUP BY `rows`";
// Run SQL and print any returned data to console
tiledbClient.sql.exec("<tiledb-cloud-username>", sql).then((res) => {
console.log(res.data)
/*
Will print:
[
{ avg_a: '2.5000', rows: 1 },
{ avg_a: '6.5000', rows: 2 },
{ avg_a: '10.5000', rows: 3 },
{ avg_a: '14.5000', rows: 4 }
]
*/
}).catch((ex) => {
// handle error
});
Queries
TileDB-Cloud-JS supports TileDB queries, by serializing data to capnproto. bufferSize
dictates the server the number of bytes that should allocated to make this query. In case the bufferSize
is not enough, it will result to an incomplete query. For this reason ReadQuery
is an async generator so a user could get results in batches.
Basic read query
Dimensions should always be an array of 2 (start of the range and the end of the range).
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// Create a client instance
const tiledbClient = new client(config);
const query = {
layout: "row-major",
ranges: [dimension1, dimension2],
bufferSize: 1500000000,
};
const generator = tiledbClient.query.ReadQuery("namespace", "arrayName", query);
generator.next().then(({value}) => {
console.log(value)
});
Incomplete reads in batches
Dimensions should always be an array of 2 (start of the range and the end of the range).
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// Create a client instance
const tiledbClient = new client(config);
const dimension1 = [636800,637800];
const dimension2 = [851000,853000];
const query = {
layout: "row-major",
ranges: [dimension1, dimension2],
bufferSize: 1500,
};
// Iterate over all results
(async function() {
for await (let results of tiledbClient.query.ReadQuery("namespace", "arrayName", query)) {
console.log(results);
}
})()
// Or manually iterate over the results
const generator = tileDBQuery.ReadQuery("namespace", "arrayName", query);
(async function() {
const result = await generator.next();
console.log(result.value);
const result2 = await generator.next();
console.log(result2.value);
})()
Multi range read queries
A dimension could be an array of ranges as well
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// Create a client instance
const tiledbClient = new client(config);
const dimension1 = [636800,637800];
const dimension2 = [[1577836800, 1588878856], [1577836800, 1578878856]];
const query = {
layout: "row-major",
ranges: [dimension1, dimension2],
bufferSize: 15000000000000,
};
// Iterate over all results
(async function() {
for await (let results of tiledbClient.query.ReadQuery("my_namespace", "my_array", query)) {
// returns an object with keys the name of the attributes and values the result
console.log(results);
}
})()
Selecting whole dimension
By setting a dimension as an empty array, query will select the whole dimension.
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// Create a client instance
const tiledbClient = new client(config);
const dimension1 = [636800,637800];
// Setting empty array, query will select whole 2nd dimension
const dimension2 = [];
const query = {
layout: "row-major",
ranges: [dimension1, dimension2],
bufferSize: 15000000000000,
};
// Iterate over all results
(async function() {
for await (let results of tiledbClient.query.ReadQuery("my_namespace", "my_array", query)) {
// returns an object with keys the name of the attributes and values the result
console.log(results);
}
})()
Write queries
For write queries user should provide an object with the attribute values and the coordinates of the cells (rows and cols in the object below). In this case we are writing to cells [1, 1] up to [1, 3].
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// Create a client instance
const tiledbClient = new client(config);
const query = {
layout: "unordered",
values: {
rows: {
values: [1, 1, 1],
},
cols: {
values: [1, 2, 3],
},
attr1: {
values: [2, 3, 38],
},
attr2: {
values: [10, 20, 30],
},
},
};
tiledbClient.query.WriteQuery("my_namespace", "my_array", query)
.then((result) => {
// returns the query object
console.log(result);
})
Dense writes with subarray
For Dense arrays we can provide a subarray instead of the coordinates and set the order (e.g. layout set to row-major).
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// Create a client instance
const tiledbClient = new client(config);
const query = {
layout: "row-major",
subarray: [[1, 1], [1, 3]],
values: {
attr1: {
values: [2, 3, 38],
},
attr2: {
values: [10, 20, 30],
},
},
};
tiledbClient.query.WriteQuery("my_namespace", "my_array", query)
.then((result) => {
// returns the query object
console.log(result);
})
Write queries with nullables and var-length attributes
For nullables and var-length attributes user should provide validity
attribute and/or the offsets
.
import client from "@tiledb-inc/tiledb-cloud";
// API tokens are the recommend way to access the cloud apis
const config = {
apiKey: "<insert token from setting page here>"
};
// Create a client instance
const tiledbClient = new client(config);
const query = {
layout: "unordered",
values: {
a1: {
values: [100, 200],
validity: [1, 0],
},
a2: {
values: [10, 10, 20],
offsets: [0, 8, 12],
validity: [1, 0],
},
a3: {
values: ['abcdewxyz'],
offsets: [0, 3, 5],
validity: [1, 0],
},
rows: {
values: [1, 1]
},
cols: {
values: [1, 2]
}
}
};
tiledbClient.query.WriteQuery("my_namespace", "my_array", query)
.then((result) => {
// returns the query object
console.log(result);
})
Development
For capnproto encoding you need to install capnproto
Debian / Ubuntu: apt-get install capnproto
Arch Linux: sudo pacman -S capnproto
Homebrew (OSX): brew install capnp
and the schema compiler
npm install -g capnpc-ts
Then run:
npm run capnp
Useful links
TileDB Cloud documentation
TileDB Cloud console
TileDB website