camera-capture
Portable Camera, audio, desktop capture Node.js library.
Contents
What / Why ?
After searching for an easy to use portable library to access the webcam directly from node.js I didn't found a library that works in windows, macOs and linux, without native dependencies that users need ot manually install (or even so, they won't work).
This library solves the problem with an easy approach. Use headless browser to capture the video, draw in canvas and pass the image data the Node.js context as fast as possible (age.exposeFunction()
) and with minimal processing. It uses HTMLCanvasElement getImageData when returning raw image data or HTMLCanvasElement.toBlob() when retuning encoded images such as png, jpg. In both cases using ArrayBuffer
Install
npm install camera-capture puppeteer
(puppeteer
is a peer dependency you must install it by yourself)
JavaScript API
Managed frame read
const c = c// pause / resume frame emission (without tunning off the camera)// shutdown everything, including, camera, browser, server:console;await cstart // promise will be resolved only when `stop`console;
Manual frame read
Instead of using start() and being notified on each frame, just call initialize()
and read frames programmatically:
const c = mime: 'image/png'await clet f = await c // PNG as configuredf = await c // take another shot this time as webp imagef = await c // jpegf = await c // raw image data (as default)
Recording camera video
The following uses DOM MediaRecorder API to record video. Notice that it all happens in the browser, on memory, so the result is a excellent quality video but it could consume lots of memory on long recordings. If that's an issue perhaps it's better to store frame by frame to hard drive and then use a video assembler like ffmpeg / imagemagick. (in the roadmap):
await c.initializeawait c.startRecordingawait sleep500writeFileSync'tmp6.webm', data
Command line
TODO - TBD
Summary
I didn't found any library that provides an interface to capture webcam video so I show the video and filter frame by frame in my Node.s desktop app (not based on electron - no canvas / HTML5 available - rendering on cairo/opengl surface that complies with
- Don't require users to install native complex dependencies (like opencv or native applications installed)
- Don't include any binary code that needs to be compiled.
- works on windows, macOs, and linux
- provides a stream-like API for video frames
- fast so it can be used for a "real-time" video filter demo
- usable without electron/canvas/html5 - imagine I'm rendering in a native surface like cairo, gtk, etc
- portable - no surprises - working in latest node.js versions
- Optionally the frames can be encoded as in jpg/png or even a video created .
- Also provides simple filtering API.
Design summary
- Use puppeteer (which is google chrome headless browser) to capture camera video. Expose frames as fast as possible.
- not focused on encoding more than the ones supported by the browser
- API based on raw image data - users responsible of compose an output video with ffmpeg, imagemagick, opencv, etc. Format encoding is not the objective of this project
Status
Observed behavior:
About, 30 frames per second (size 600x400, format: raw image data)
- JavaScript API (managing the capturing loop)
- javaScript API (manual capture)
- image encoded as jpeg, png, webp
- camera video recording using DOM MediaRecorder
Reference API
TODO / Road map
- investigate why/how to pass the buffer / array buffer view directly without transforming it to number[] / and array buffer views
- se TextEncoder/TextDecoder to serialize the data as a single char-per-byte string (using windows-1252 encoding) and deserialize it in Node on the other side which is fast (since passing strings is much faster).
- test if toDataUrl is faster than toBlob
- probably for frames a generator / or observable is more appropriate than even listeners.
- perhaps is faster to do the capture loop all together inside the DOM, instead calling evaluate() on each iteration?
- CLI
- pause/resume / start/stop should work for recording too.
- performance tests (fps raw image data and encoded images)
- do we really need to serialize constrains ?
- video recording formats other than webm?
- video recording constraints - size -
- audio recording only API
- record desktop ? possible ?
- desktop screenshot only API
- browser screenshot only API
- webcam screenshot only API
- geo location (get the coords) ? (need https?)
- change video size dynamically ?
- check c.addFrameListener() with encoded images
- real world example: native app
- encode in browser supported formats (png, jpg)
- c.readFrame() users read manually instead listener - loop controlled by users.
- listener API managed loop
- API docs
- add api docs descriptions to class, options and
- record capture using dom api (output is mp4/avi video)
low priority
- research how fast/slow is painting canvas pixel by pixel from image data than showImage in node-gui
- TODO: support fps control like in opencv
- [ ]