vision-camera-doc-scanner-plugin
Plugin for VisionCamera that adds a Document Scanner to your app.
Plugin is powered by: OpenCV and Deep Learning for the Document Detection (Image Segmentation).
TODO'S:
NOTE: This plugin is still under development
- [ ] add steps to reproduce iOS
- [ ] some improvements
- [ ] test expo compatability
- [ ] expo config plugin
- [ ] Retrain model with more data
Installation
yarn add vision-camera-doc-scanner-plugin
cd ios && pod install
Add the plugin to your babel.config.js:
module.exports = {
plugins: [
[
'react-native-reanimated/plugin',
{
globals: ['__detectDocument'],
},
],
// ...
],
};
Note: You have to restart metro-bundler for changes in the
babel.config.js
file to take effect.
Add dependencies to package.json:
"react-native-reanimated": "2.12.0",
"react-native-vision-camera": "2.15.2"
Add import on top of index.tsx
:
import 'react-native-reanimated';
Android
Register package in MainApplication.java
(example/android/app/src/main/java/com/example/visioncameradocscannerplugin/MainApplication.java):
import com.visioncameradocscannerplugin.VisionCameraDocScannerPluginPackage;
// ...
@Override
protected List<ReactPackage> getPackages() {
@SuppressWarnings("UnnecessaryLocalVariable")
List<ReactPackage> packages = new PackageList(this).getPackages();
// Packages that cannot be autolinked yet can be added manually here, for example:
// packages.add(new MyReactNativePackage());
packages.add(new VisionCameraDocScannerPluginPackage());
return packages;
}
Add Camera Permission to /android/app/src/main/AndroidManifest.xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.visioncameradocscannerplugin">
<uses-permission android:name="android.permission.INTERNET" />
// add following line
<uses-permission android:name="android.permission.CAMERA" />
//...
</manifest>
Apply fix for OpenCV in android/app/build.gradle
android section:
// fix for opencv4 duplicated files
packagingOptions {
pickFirst 'lib/arm64-v8a/libc++_shared.so'
pickFirst 'lib/x86/libc++_shared.so'
pickFirst 'lib/x86_64/libc++_shared.so'
pickFirst 'lib/armeabi-v7a/libc++_shared.so'
}
iOS
Add camera usage description in your projects info.plist
<key>NSCameraUsageDescription</key>
<string>Scan Documents</string>
Usage
See the example for how to use this in your app.
import { cropImage, detectDocument } from "vision-camera-doc-scanner-plugin";
// ...
const frameProcessor = useFrameProcessor((frame) => {
'worklet';
const coords = detectDocument(frame);
}, []);
const takeSnapshot = async () => {
const cachedCoords = {value: coords.value};
if (
(cachedCoords.value.x1 !== 0 && cachedCoords.value.y1 !== 0) ||
(cachedCoords.value.x2 !== 0 && cachedCoords.value.y2 !== 0)
) {
const data = await camera.current
?.takePhoto({qualityPrioritization: 'speed'})
.catch(e => console.log(e));
const dataPath = data?.path;
if (dataPath) {
const base64Images = await cropImage(
cachedCoords,
dataPath,
);
const imageStr = `data:image/jpeg;base64,${base64Images
.replace('\n', '')
.replace('\r', '')
.replace(' ', '')}`;
} else {
console.log('no data path');
}
} else {
console.log('There is nothing to crop !');
}
};
Data
detectDocument(frame)
returns Coords
with the following data shape.
type Coords = {
x1: number;
y1: number;
x2: number;
y2: number;
x3: number;
y3: number;
x4: number;
y4: number;
width: number;
height: number;
}
cropImage
takes the Coords
to crop and an path to the image.
It returns a Base64-String
where the cropped image is stored.
Additional:
The project contains a python folder which contains the python script to train the image segmentation model and further information.
Contributing
See the contributing guide to learn how to contribute to the repository and the development workflow.
License
MIT
Made with create-react-native-library