Object Detection (coco-ssd)
Object detection model that aims to localize and identify multiple objects in a single image.
This model is a TensorFlow.js port of the COCO-SSD model. For more information about Tensorflow object detection API, check out this readme in tensorflow/object_detection.
This model detects objects defined in the COCO dataset, which is a large-scale object detection, segmentation, and captioning dataset. You can find more information here. The model is capable of detecting 90 classes of objects. (SSD stands for Single Shot MultiBox Detection).
This TensorFlow.js model does not require you to know about machine learning.
It can take input as any browser-based image elements (
elements, for example) and returns an array of bounding boxes with class name and confidence level.
via Script Tag
<!-- Load TensorFlow.js. This is required to use coco-ssd model. --><!-- Load the coco-ssd model. --><!-- Replace this with your image. Make sure CORS settings allow reading the image! --><!-- Place your code in the script tag below. You can also use an external .js file -->
// Note: you do not need to import @tensorflow/tfjs here.const cocoSsd = ;const img = document;// Load the model.const model = await cocoSsd;// Classify the image.const predictions = await model;console;console;
You can also take a look at the demo app.
Loading the model
coco-ssd is the module name, which is automatically included when you use the
<script src> method. When using ES6 imports,
coco-ssd is the module.
cocoSsd.loadconfig: ModelConfig = ;
Args: config Type of ModelConfig interface with following attributes:
base: Controls the base cnn model, can be 'mobilenet_v1', 'mobilenet_v2' or 'lite_mobilenet_v2'. Defaults to 'lite_mobilenet_v2'. lite_mobilenet_v2 is smallest in size, and fastest in inference speed. mobilenet_v2 has the highest classification accuracy.
modelUrl: An optional string that specifies custom url of the model. This is useful for area/countries that don't have access to the model hosted on GCP.
Detecting the objects
You can detect objects with the model without needing to create a Tensor.
model.detect takes an input image element and returns an array of bounding boxes with class name and confidence level.
This method exists on the model that is loaded from
model.detectimg: tf.Tensor3D | ImageData | HTMLImageElement |HTMLCanvasElement | HTMLVideoElement, maxDetectionSize: number
- img: A Tensor or an image element to make a detection on.
- maxNumBoxes: The maximum number of bounding boxes of detected objects. There can be multiple objects of the same class, but at different locations. Defaults to 20.
Returns an array of classes and probabilities that looks like:
bbox: x y width heightclass: "person"score: 08380282521247864bbox: x y width heightclass: "kite"score: 074644153267145157
Technical details for advanced users
This model is based on the TensorFlow object detection API. You can download the original models from here. We applied the following optimizations to improve the performance for browser execution:
- Removed the post process graph from the original model.
- Used single class NonMaxSuppression instead of original multiple classes NonMaxSuppression for faster speed with similar accuracy.
- Executes NonMaxSuppression operations on CPU backend instead of WebGL to avoid delays on the texture downloads.
Here is the converter command for removing the post process graph.
tensorflowjs_converter --input_format=tf_frozen_model \--output_format=tfjs_graph_model \--output_node_names='Postprocessor/ExpandDims_1,Postprocessor/Slice' \./frozen_inference_graph.pb \./web_model