homography

Homography.js

Homography.js is a lightweight High-Performance library for implementing homographies in Javascript or Node.js. It is designed to be easy-to-use (even for developers that are not familiar with Computer Vision), and able to run in real time applications (even in low-spec devices such as budget smartphones). It allows you to perform Affine, Projective or Piecewise Affine warpings over any Image or HTMLElement in your application by only setting a small set of reference points. Additionally, Image warpings can be made persistent (independent of any CSS property), so they can be easily drawn in a canvas, mixed or downloaded. Homography.js is built in a way that frees the user from all the pain-in-the-ass details of homography operations, such as thinking about output dimensions, input coordinate ranges, dealing with unexpected shifts, pads, crops or unfilled pixels in the output image or even knowing what a Transform Matrix is.

Features

• Apply different warpings to any Image or HTMLElement by just setting two sets of reference points.
• Perform Affine, Projective or Piecewise Affine transforms or just set Auto and let the library decide which transform to apply depending on the reference points you provide.
• Simplify how you deal with canvas drawings, or subsequent Computer Vision problems by making your Image transforms persistent and independent of any CSS property.
• Forget all the pain-in-the-ass details of homography operations, even if you only have fuzzy idea about what an homography is.
• Avoid warping delays in real-time applications due to its design focused on High-Performance.
• Support for running in the backend with Node.js.

Install

To use as a module in the browser (Recommended):

<script type="module">
  import { Homography } from "https://cdn.jsdelivr.net/gh/Eric-Canas/Homography.js@1.4/Homography.js";
</script>

If you don't need to perform Piecewise Affine Transforms, you can also use a very lightweight UMD build that will expose the homography global variable and will charge faster:

<script src="https://cdn.jsdelivr.net/gh/Eric-Canas/Homography.js@1.4/HomographyLightweight.min.js"></script>
...
// And then in your script
const myHomography = new homography.Homography();
// Remember to don't override the homography variable by naming your object "homography"

Via npm:

$ npm install homography
... 
import { Homography } from "homography";

Usage

In the Browser

Perform a basic Piecewise Affine Transform from four source points.

    // Select the image you want to warp
    const image = document.getElementById("myImage");
    
    // Define the reference points. In this case using normalized coordinates (from 0.0 to 1.0).
    const srcPoints = [[0, 0], [0, 1], [1, 0], [1, 1]];
    const dstPoints = [[1/5, 1/5], [0, 1/2], [1, 0], [6/8, 6/8]];
    
    // Create a Homography object for a "piecewiseaffine" transform (it could be reused later)
    const myHomography = new Homography("piecewiseaffine");
    // Set the reference points
    myHomography.setReferencePoints(srcPoints, dstPoints);
    // Warp your image
    const resultImage = myHomography.warp(image);
    ...

Perform a complex Piecewise Affine Transform from a large set of pointsInY * pointsInX reference points.

    ...
    // Define a set of reference points that match to a sinusoidal form. 
    // In this case in image axis (x : From 0 to width, y : From 0 to height) for convenience.
    let srcPoints = [], dstPoints = [];
    for (let y = 0; y <= h; y+=height/pointsInY){
        for (let x = 0; x <= w; x+=width/pointsInX){
            srcPoints.push([x, y]); // Add (x, y) as source points
            dstPoints.push([x, amplitude+y+Math.sin((x*n)/Math.PI)*amplitude]); // Apply sinus function on y
        }    
    }
    // Set the reference points (reuse the previous Homography object)
    myHomography.setReferencePoints(srcPoints, dstPoints);
    // Warp your image. As not image is given, it will reuse the one used for the previous example.
    const resultImage = myHomography.warp();
    ...
    

Perform a simple Affine Transform and apply it on a HTMLElement.

    ...
    // Set the reference points from which estimate the transform
    const srcPoints = [[0, 0], [0, 1], [1, 0]];
    const dstPoints = [[0, 0], [1/2, 1], [1, 1/8]];
    
    // Don't specify the type of transform to apply, so let the library decide it by itself. 
    const myHomography = new Homography(); // Default transform value is "auto".
    // Apply the transform over an HTMLElement from the DOM.
    myHomography.transformHTMLElement(document.getElementById("inputText"), squarePoints, rectanglePoints);
    ...

Calculate 250 different Projective Transforms, apply them over the same input Image and draw them on a canvas.

const ctx = document.getElementById("exampleCanvas").getContext("2d");

// Build the initial reference points (in this case, in image coordinates just for convenience)
const srcPoints = [[0, 0], [0, h], [w, 0], [w, h]];
let dstPoints = [[0, 0], [0, h], [w, 0], [w, h]];
// Create the homography object (it is not necessary to set transform as "projective" as it will be automatically detected)
const myHomography = new Homography(); 
// Set the static parameters of all the transforms sequence (it will improve the performance of subsequent warpings)
myHomography.setSourcePoints(srcPoints);
myHomography.setImage(inputImg);

// Set the parameters for building the future dstPoints at each frame (5 movements of 50 frames each one)
const framesPerMovement = 50;
const movements = [[[0, h/5], [0, -h/5], [0, 0], [0, 0]],
                   [[w, 0], [w, 0], [-w, 0], [-w, 0]],
                   [[0, -h/5], [0, h/5], [0, h/5], [0, -h/5]],
                   [[-w, 0], [-w, 0], [w, 0], [w, 0]],
                   [[0, 0], [0, 0], [0, -h/5], [0, h/5]]];

for(let movement = 0; movement<movements.length; movement++){
    for (let step = 0; step<framesPerMovement; step++){
        // Create the new dstPoints (in Computer Vision applications these points will usually come from webcam detections)
        for (let point = 0; point<srcPoints.length; point++){
            dstPoints[point][0] += movements[movement][point][0]/framesPerMovement;
            dstPoints[point][1] += movements[movement][point][1]/framesPerMovement;
        }
        
        // Update the destiny points and calculate the new warping. 
        myHomography.setDestinyPoints(dstPoints);
        const img = myHomography.warp(); //No parameters warp will reuse the previously setted image
        // Clear the canvas and draw the new image (using putImageData instead of drawImage for performance reasons)
        ctx.clearRect(0, 0, w, h);
        ctx.putImageData(img, Math.min(dstPoints[0][0], dstPoints[2][0]), Math.min(dstPoints[0][1], dstPoints[2][1]));
        await new Promise(resolve => setTimeout(resolve, 0.1)); // Just a trick for forcing canvas to refresh
    }
}

*Just take attention to the use of setSourcePoints(srcPoints), setImage(inputImg), setDestinyPoints(dstPoints) and warp(). The rest of code is just to generate coherent sequence of destiny points and drawing the results

API Reference

new Homography([transform = "auto", width, height])

Main class for performing geometrical transformations over images.
Homography is in charge of applying Affine, Projective or Piecewise Affine transformations over images, in a way that is as transparent and simple to the user as possible. It is specially intended for real-time applications. For this reason, this class keeps an internal state for avoiding redundant operations when reused, therefore, critical performance comes when multiple transformations are done over the same image.

• [transform = "auto"]: String representing the transformation to be done. One of "auto", "affine", "piecewiseaffine" or "projective":
  • "auto": Transformation will be automatically selected depending on the inputs given. Just take "auto" if you don't know which kind of transform do you need. This is the default value.
  • "affine" : A geometrical transformation that ensures that all parallel lines of the input image will be parallel in the output image. It will need exactly three source points to be set (and three destiny points). An Affine transformation can only be composed by rotations, scales, shearings and reflections.
  • "piecewiseaffine" : A composition of several Affine transforms that allows more complex constructions. This transforms generates a mesh of triangles with the source points and finds an independent Affine transformation for each one of them. This way, it allows more complex transformation as, for example, sinusoidal forms. It can take any amount (greater than three) of reference points. When "piecewiseaffine" mode is selected, only the parts of the input image within a triangle will appear on the output image. If you want to ensure that the whole image appears in the output, ensure that you set reference points on each corner of the image.
  • "projective": A transformation that shows how the an image change when the point of view of the observer is modified. It takes exactly four source points (and four destiny points). This is the transformation that should be used when looking for perspective modifications.
• [width]: Optional width of the input image. If given, it will resize the input image to that width. Lower widths will imply faster transformations at the cost of lower resolution in the output image, while larger widths will produce higher resolution images at the cost of processing time. If not defined (or null), it will use the original image width.
• [height]: Optional height of the input image. Same considerations than width.

Homography.setSourcePoints(points[, image, width, height, pointsAreNormalized])

Sets the source reference points ([[x1, y1], [x2, y2], ..., [xn, yn]]) of the transform and, optionally, the image that will be transformed.
Source reference points is a set of 2-D coordinates determined in the input image that will exactly go to the correspondent destiny points coordinates (setted through setDstPoints()) in the output image. The rest of coordinates of the image will be interpolated through the geometrical transform estimated from these ones.

• points : Source points of the transform, given as a ArrayBuffer or Array in the form [x1, y1, x2, y2, ..., xn, yn] or [[x1, y1], [x2, y2], ..., [xn, yn]]. For large set of source points, performance improvements come when using Float32Array. These source points can be declared in image coordinates, (x : [0, width], y : [0, height]) or in normalized coordinates (x : [0.0, 1.0], y : [0.0, 1.0]). In order to allow transforms with upscalings (from x0 to x8), normalized scale is automatically detected when the points Array does not contain any value larger than 8.0. Coordinates with larger numbers are considered to be in image scale (x : [0, width], y : [0, height]). This automatic behaviour can be avoided by using the pointsAreNormalized parameter. Please note that, if width and height parameters are setted and points are given in image coordinates, these image coordinates should be declared in terms of the given width and height, instead of the original image width/height).
• [image] : Optional source image, that will be warped later. Given as an HTMLImageElement or ImageData in the browser version or as the output of await loadImage('path-to-image') in the Node.js version. Setting this element here will help to advance some calculations, improving the later warping performance. Specially when it is planned to apply multiple transformations (same source points but different destiny points) to the same image. If width and/or height are given, the image will be internally rescaled before any transformation if it is given as HTMLImageElement (if image is given as ImageData these parameters will be ignored).
• [width]: Optional width to which rescale the input image. It is equivalent to the width parameter of the constructor.
• [height]: Optional height to which rescale the input image. It is equivalent to the height parameter of the constructor.
• [pointsAreNormalized]: Optional boolean determining if the parameter points is in normalized or in image coordinates. If not given it will be automatically inferred from the points array.

Homography.setDestinyPoints(points[, pointsAreNormalized])

Sets the destiny reference points ([[x1, y1], [x2, y2], ..., [xn, yn]]) of the transform.
Destiny reference points is a set of 2-D coordinates determined for the output image. They must match with the source points, as each source points of the input image will be transformed for going exactly to its correspondent destiny points in the output image. The rest of coordinates of the image will be interpolated through the geometrical transform estimated from these correspondences.

• points : Destiny points of the transform, given as a ArrayBuffer or Array in the form [x1, y1, x2, y2, ..., xn, yn] or [[x1, y1], [x2, y2], ..., [xn, yn]]. The amount of source points given must match with the amount of source points that should have been previously setted.
• [pointsAreNormalized]: Optional boolean determining if the parameter points is in normalized or in image coordinates. If not given it will be automatically inferred from the points Array.

Homography.setReferencePoints(srcPoints, dstPoints[, image, width, height, srcpointsAreNormalized, dstPointsAreNormalized])

This function just makes a call to Homography.setSourcePoints(srcPoints[, image, width, height, srcPointsAreNormalized) and then Homography.setDestinyPoints(dstPoints[, dstPointsAreNormalized). It can be used for convenience when setting reference points for first time, but should be substituted by Homography.setSourcePoints() or Homography.setDestinyPoints() when performing multiple transforms where one of srcPoints or dstPoints remains unchanged, as it would decrease the overall performance.

Homography.setImage(image [, width, height])

Sets the image that will be transformed when warping.
Setting the image before the destiny points (call to setDestinyPoints()) and the warping (call to warp()) will help to advance some calculations as well as to avoid future redundant operations when successive calls to setDestinyPoints()->warp() will occur in the future.

• image : Source image, that will be warped later. Given as an HTMLImageElement or ImageData in the browser version. If given as ImageData, width and height will not be used. In the Node.js it should be the output of await loadImage('path-to-image').
• [width]: Optional width to which rescale the given image. It is equivalent to the width parameters of the constructor or setSourcePoints().
• [height]: Optional height to which rescale the given image. It is equivalent to the height parameters of the constructor or setSourcePoints().

Homography.warp([image, asHTMLPromise = false])

Apply the setted transform to an image. Apply the homography to the given or the previously setted image and return it as ImageData or as a Promise. Output image will have enough width and height for enclosing the whole input image without any crop or pad once transformed. Any void section of the output image will be transparent. In case that an image is given, it will be internally setted, so any future call to warp() receiving no image parameter will apply the transformation over this image again. Remember that it will transform the whole input image for "affine" and "projective" transforms, while for "piecewiseaffine" transforms it will only transform the parts of the image that can be connected through the setted source points. It occurs because "piecewiseaffine" transforms define different Affine transforms for different sections of the input image, so it can not calculate transforms for undefined sections. If you want the whole output image in a Piecewise Affine transform you should set a source reference point in each corner of the input image ([[x1, y1], [x2, y2], ..., [0, 0], [0, height], [width, 0], [width, height]]).

• [image] : Image that will transformed, given as an HTMLImageElement. If image was already setted through setImage(img) or setSrcPoints(points, img), this parameter doesn't need to be given again. If an image is given, it will be internally setted, so any future call to warp() will reuse it. When possible, this reusage of the image will improve the overall performance.
• [asHTMLPromise = false] : If true, returns a Promise of an HTMLImageElement containing the output image, instead of an ImageData buffer. It could be convenient for some applications, but try to avoid it on critical performance applications as it would decrease its overall performance. If you need to draw this image on a canvas, consider to do it directly through context.putImageData(imgData, x, y).

This function will return the transformed image, without any pad or crop in format ImageData or as a Promise of a HTMLImageElement if asHTMLPromise was set to true.

Homography.transformHTMLElement(element[, srcPoints, dstPoints])

Apply the current Affine or Projective transform over an HTMLElement. Applying transform to any HTMLElement will be extremely fast.
If srcPoints and dstPoints are given, a new transform will be estimated from them. Take into account, that this function work by modifying the CSS trasform property, so it will not work for the "piecewiseaffine" option, as CSS does not support Piecewise Affine transforms.

• element : The HTMLElement to which apply the transform. It can be also an HTMLImageElement. In this case, the difference with warp() will be that the transformation will be not persistent, as it will be only applied over its current view (as a style) and not to its beneath image data. Usually, it is enough if the image does not need to be drawn in a canvas or to suffer subsequent transformations.
• [srcPoints] : Source points of the transform, given as a ArrayBuffer or Array in the form [x1, y1, x2, y2, ..., xn, yn] or [[x1, y1], [x2, y2], ..., [xn, yn]]. If not given, they should have been set before through setSourcePoints().
• [dstPoints] : Destiny points of the transform, also given as a ArrayBuffer or Array in the form [x1, y1, x2, y2, ..., xn, yn] or [[x1, y1], [x2, y2], ..., [xn, yn]]. If not given, they should have been set before through setDestinyPoints().

Homography.HTMLImageElementFromImageData(imgData[, asPromise = true])

Transforms an ImageData object in an HTMLImageElement. Remember that ImageData is the output format of warp().

• imgData : ImageData object to convert.
• [asPromise=true] : If true return a Promise of a HTMLImageElement, if false returns directly a HTMLImageElement. In this case, you will have to wait for the onload event to trigger before using it.

Performance

Benchmark results for every kind of transformation.

• Image Data Warping section indicates the time for calculating the transformation matrix between a pair of Source and Destiny reference points and appling this transform over an image of size NxN. It generates a persistent ImageData object that can be directly drawn in any Canvas at a negligible computational cost, through context.putImageData(imgData, x, y).
• 400x400 ↦ NxN, indicates the size of the input image and the size of the expected output image. The CSS Transform Calculation section does not include this information since these sizes does not affect to its performance.
• First frame column indicates the time for calculating a single image warping, while Rest of Frames column indicates the time for calculating each one of multiple different warpings on the same input image. Frame Rate (1/Rest of Frames) indicates the amount of frames that can be calculated per second.
• You can test the concrete performance of your objective device just by executing the benchmark.html. Take into account that this execution can take some minutes, since it executes 2,000 frames for each single warping experiment, and 200,000 for each CSS experiment.

Performance tests on an Average Desktop PC.

Intel Core i5-7500 Quad-Core. Chrome 92.0.4515.107. Windows 10.
Image Data Warping
	400x400 ↦ 200x200			400x400 ↦ 400x400			400x400 ↦ 800x800
Transform	First Frame	Rest of Frames	Frame Rate	First Frame	Rest of Frames	Frame Rate	First Frame	Rest of Frames	Frame Rate
Affine	5 ms	0.7 ms	1,439 fps	14 ms	2.7 ms	366.7 fps	13 ms	10.8 ms	92.6 fps
Projective	6 ms	1.9 ms	527.4 fps	21 ms	7.2 ms	139.7 fps	30 ms	27.5 ms	36.3 fps
Piecewise Aff. (2 Triangles)	7 ms	1.1 ms	892.9 fps	19 ms	4.4 ms	227.9 fps	40 ms	16.5 ms	60.6 fps
Piecewise Aff. (360 Tri.)	26 ms	2.1 ms	487 fps	21 ms	4.6 ms	216.1 fps	41 ms	22.4 ms	44.6 fps
Piecewise Aff. (~23,000 Tri.)	257 ms	24.3 ms	41.2 fps	228 ms	11.5 ms	87.1 fps	289 ms	62 ms	16.1 fps
CSS Transform Calculation
Transform	First Frame			Rest of Frames			Frame Rate
Affine	4 ms			0.00014 ms			1,696,136.44 fps
Projective	4 ms			0.016 ms			61,650.38 fps

Performance tests on a budget smartphone (a bit destroyed).

Xiaomi Redmi Note 5. Chrome 92.0.4515.115. Android 8.1.0
Image Data Warping
	400x400 ↦ 200x200			400x400 ↦ 400x400			400x400 ↦ 800x800
Transform	First Frame	Rest of Frames	Frame Rate	First Frame	Rest of Frames	Frame Rate	First Frame	Rest of Frames	Frame Rate
Affine	25 ms	4.5 ms	221.5 fps	84 ms	16.9 ms	59.11 fps	127 ms	64.7 ms	15.46 fps
Projective	38 ms	15.5 ms	64.4 fps	150 ms	56.8 ms	17.6 fps	232 ms	216 ms	4.62 fps
Piecewise Affine (2 Triangles)	35 ms	8.8 ms	113.9 fps	316 ms	31.7 ms	31.6 fps	138 ms	118 ms	8.5 fps
Piecewise Aff. (360 Tri.)	151 ms	14.3 ms	70 fps	138 ms	30.2 ms	33 fps	274 ms	149 ms	6.7 fps
Piecewise Aff. (~23,000 Tri.)	1.16 s	162 ms	6.15 fps	1.16 s	75 ms	13.3 fps	1.47 s	435 ms	2.3 fps
CSS Transform Calculation
Transform	First Frame			Rest of Frames			Frame Rate
Affine	21 ms			0.0104 ms			96,200.10 fps
Projective	22 ms			0.025 ms			40,536.71 fps