datamodel-to-oas
Data model to Open API Specification (FKA - Swagger) generator is an NPM module that generates OAS from a data model in JSON format.
Why should I automate the design of APIs?
Before getting into the how to use the framework, let's provide some context about it.
datamodel-to-oas framework intends to solve DRY (don't repeat yourself) principle that many teams face when designing and building *data-oriented APIs. In my experience, every time API teams decide to expose a new resource, and it needs to debate over and over again the same design process. Through hundreds or even thousands of resources. You have a major bottleneck here, if you want to scale you API team(s), you need to find a better way automate this.
What if you could instead automate this process into something that could produce fast, more reliable and consistent results? Enter datamodel-to-oas framework.
so, why the urgency of Data-Driven APIs?
The intent of Data-driven APIs is to address the following concerns:
- Consistency: APIs automatically generated by machines are more consistent than those produced by human beings by hand.
- Pluggability: As more data becomes available from underlying data sources, APIs expose these resources with minimal effort and in a controlled manner.
- Maintainability: your APIs evolve and require changes over time. Modifying hundreds of resources should not take more time than changing a single one.
- Affordance: By leveraging a data model as the foundation of your API, you provide access to your APIs from a data standpoint, similar to how we are used to accessing databases with SQL statements. Accessing your API resources should be similar. For instance, being able to filter, include or exclude attributes and nested entities, establish relationships and include hypermedia/HATEOAS automatically.
- Abstraction: APIs are becoming more mature by specification and standards produced by the industry. There is no need to invent a new standard. datamodel-to-oas uses standard JSON as input to define a model along with its relationships. Then, it converts entities within the model into an OAS specification with paths, parameters, and annotation/vendor extensions with models. The OAS can be used as the input to integrate backend systems with abstraction layers. e.g. ORMs with Sequelize.js for relational databases, Mongoose for MongoDB for document oriented databases, SOAP stubs generated from WSDL files or other Web APIs via their SDKs.
so what is datamodel-to-oas?
- It consists of a JSON file (data model JSON file, sample-data-model.json) with the definition of the data model that represents an API. For instance:
- For an API management company, an API might consist of resources such as accounts, organizations, api proxies, etc.
- For a retail company, it might look like a collection of resources such as products, orders, order items, vendors, etc.
- For a banking company, products, services, customers, accounts, etc.
- what else contains a data model JSON file?
- Entities, attributes, and relationships between them.
- Also annotations or vendor extensions for:
- Generating OAS paths, query params, headers, etc.
- or used by other frameworks to automate building APIs even further
- A Node.js module and command-line tool that generates OAS from data model JSON file.
What does the output look like?
Check out test/swagger-sample-datamodel-to-oas-generated.json file
Getting Started
Installation
npm install datamodel-to-oas -g
Using the CLI
Using the CLI is easy:
$ git clone git@github.com:dzuluaga/datamodel-to-oas.git $ cd datamodel-to-oas $ cd test $ datamodel-to-oas generate sample-data-model.json
That's it. You should be able to pipe the output of the generated oas file to either a file or the clipboard with pbcopy.
Add a new resource
Let's add a new resource at the end of the JSON collection in sample-data-model.json and run again datamodel-to-oas generate sample-data-model.json
.
"model": "FoobarResource" "isPublic": true "resources": "collection": "description": "A collection of Apigee APIs. TODO API Proxy definition." "parameters": "$ref": "./refs/parameters.json#/common/collection" "responses": "$ref": "./refs/responses.json#/apis/collection" "entity": "description": "A single entity of an API Proxy." "parameters": "$ref": "./refs/parameters.json#/common/entity" "responses": "$ref": "./refs/responses.json#/apis/entity" "name": "Foobar Resource" "path": "/foobarresource" "includeAlias": "FOOBARS" "listAttributes": "id": "type": "string" "is_primary_key": true "alias": "foobar_resource_id" "org_name": "type": "string" "model": "Org" "description": "The org name." "foobar_name": "alias": "foobar_name" "type": "string" "is_secondary_key": true "description": "The foobar name." "created_date": {} "last_modified_date": {} "created_by": {} "last_modified_by": {} "associations": "rel": "org" "foreignKey": "org_name" "modelName": "Org" "type": "belongsTo" "schema": "schema_name" "table": "v_foobar"
If you want to include /foobar
resource as a subresource of /org/{org_id}
, under Org model, include the following:
"associations": "rel": "foobars" "foreignKey": "org_name" "modelName": "FoobarResource" "type": "hasMany"
Voila!
Your OAS Spec should now include a /foobar
resource at the end
Try visualizing your resource by paste the output in Swagger Editor.
Using the API
The following example can be found under test/app.js:
var datamodelToSwagger = ; datamodelToOas ;
The Node.js module returns a promise with an Open API Specification resolving sample-data-model.json.
How can I take this even further with an API and a database
As mentioned earlier, the OAS generated by this tool includes x-data-model
annotations for each path. Another layer leverages these annotations that the API uses to lookup models to support http verbs. This adapter layer can represent anything that it can interact with. So, not necessarily a database. For instance other services such as Web Services, other REST APIs, FTP servers, you name it.
The implementation of this adapter layer is up to you. However, I built Nucleus-Model-Factory, a lightweight framework on top of Sequelize.js for the purpose of supporting Postgres database.
The Database/Backend Adapter
- For an example of a Sequelize.js adapter check this example.
Node.js API Example
An example of a Node.js app leveraging the data model to generate the OAS spec and Postgres models (Sequelize.js ORM):
'use strict'; var app = path = all_config = utils = config: all_config config = utils modelFactory = dataModelPath = './api/models/edge-data-model.json' edgeModelSpecs = http = swaggerTools = dataModel2Oas = var serverPort = 3000; // swaggerRouter configurationvar options = controllers: './api/routers' useStubs: processenvNODE_ENV === 'development' ? true : false // Conditionally turn on stubs (mock mode); dataModel2Oas
The Router/Controller
The router/controller is referenced by each path generated within the OAS by datamodel-to-oas. Then based on x-data-model annotation, it will know how to execute query based on all parameters and whereAttributes annotation.
moduleexports { var datamodel = reqswaggeroperation 'x-data-model' ; ; ;}; { var utils = reqapp; ; var where = ; ; ; var model = reqappdatamodelmodel; if !model ; else if reqquerydescribe && reqquerydescribe === 'true' // returns table description res; /*options.model.describe() .then( function( describe ){ res.json( describe ); } )*/ else modeldatamodelcardinality where: where attributes: utils offset: reqqueryoffset || 0 limit: utils order: reqqueryorder || include: utils ; }; /* * Dynamically generates where object with attributes from the request object */{ var _where = ; var utils = reqapp; ; routeSpecwhereAttributes || ; ; var where = utilsdb_connectionssequelizeUtils_; ; //where = utils.db_connections.sequelize.Utils._.merge( where, applySecurity( routeSpec, req.security.account_list, where ) ); ; return where;} { ; ; var _where = {} if account_list && account_listlength > 0 && account_list0 !== '*' _where optionssecurityAttributeName || 'account_id' = $in: account_list ; else if !account_list throw "User Credentials require user account mapping."; return _where;}
Are there any examples of APIs using this paradigm?
Yes. This framework is the product of drinking-our-kool-aid in my team. Nucleus API uses it.
Summary
By following above steps, we've just achieved automation of building of our API based while adhering to the principles of data-driven APIs. Trying to do this by hand would be untenable.
References
-
The sample code introduces a few steps that aren't necessarily documented in this readme file. It is recommended to check out their documentation. For (Swagger-Tools)[https://github.com/apigee-127/swagger-tools] and (Nucleus-Model-Factory)[https://www.npmjs.com/package/nucleus-model-factory] take either the datamodel.json file or the output of datamodel-to-oas.
-
*This framework is based on (data-oriented API principles)[http://apigee.com/about/blog/developer/data-oriented-designs-common-api-problems] introduced by Martin Nally and Marsh Gardiner. Also check this (webcast)[http://apigee.com/about/resources/webcasts/pragmatic-rest-next-generation].
TODO
- datamodel-to-oas currently lacks the support of verbs that modify resource state. Although, it could be extended to meet those needs. Stay tuned for this functionality.
Feedback and pull requests
Feel free to open an issue to submit feedback or a pull request to incorporate features into the main branch.