Smartmeter2mqtt

This application can listen to your (Dutch) Smartmeter with a P1 connector, and send the data to several outputs. Currently supporting the following methods:

JSON TCP socket
Raw TCP socket
Website with websockets (and ajax fallback) for client side refresh
Http json endpoint to get the latest reading
Webrequest to external service
MQTT (with home assistant integration)

Supporting other services like some website where you can monitor historic data is also possible. Building your own output is explained a bit lower on this page.

Getting started

Connect smartmeter
Choose the run method docker or bare
Start the application (for testing)
Run in background using docker container or PM2
Send out a tweet that you are using smartmeter2mqtt on twitter @svrooij

Running in docker

My setup is a Raspberry Pi 3 model b rev 1.2 (See cat /proc/cpuinfo) with a P1 cable running Hypriot as an OS (because of the pre-configured docker/docker-compose).

The docker image is currently automatically build from every new release. We support the following platforms (can be extended if we have someone willing to test it out):

AMD64 linux/amd64
ARM v6 (raspberry pi ?) linux/arm/v6
ARM v7 (raspberry pi 3/3b) linux/arm/v7
ARM64 (raspberry pi 4 running in 64 bit) linux/arm64

Find device ID

If you're reading from an USB to P1 cable, it's important that you connect the device to the container. The mapped location might change on reboot or if you connect other devices. That is why I recommend to connect the device by serial. You will need the real device location, type ls /dev/serial/by-id and note the device string that looks like usb-FTDI_FT232R_USB_UART_A13LN4ZS-if00-port0 for my cable.

Be sure to replace this device id in the docker compose file.

Docker compose

version: "3.7"

services:
  smartmeter:
    image: svrooij/smartmeter:latest
    devices: # Replace the device id with your found id, the device is mapped as /dev/ttyUSB0 inside the container.
      - /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A13LN4ZS-if00-port0:/dev/ttyUSB0
    restart: unless-stopped
    ports: # Depending on your situation you'll need to expose some ports
      - 3000:3000
      - 3010:3010
      - 3020:3020
    environment:
      - TZ=Europe/Amsterdam
      - SMARTMETER_PORT=/dev/ttyUSB0
      - SMARTMETER_web-server=3000
      - SMARTMETER_tcp-server=3010
      - SMARTMETER_raw-tcp-server=3020
      # - SMARTMETER_sunspec-modbus=192.168.x.x # if you want to also read your solar inverter.

You can set every setting with an environment variable prefixed with SMARTMETER_, choose the settings you need.

Running locally

Install npm i -g smartmeter2mqtt --production
Figure out what source you want to connect, see below.
Start application to see if it works
Configure to run in background see stackoverflow answer.

Usage

smartmeter2mqtt 0.0.0-development
Publish data from your Smartmeter with a P1 interface to your MQTT server.

Read from P1 to USB serial:
index.js --port /dev/ttyUSB0 [options]

Read from tcp socket:
index.js --socket host:port [options]

Options:
  --port                   The serial port to read, P1 to serial usb, eg.
                           '/dev/ttyUSB0'
  --socket                 The tcp socket to read, if reading from serial to
                           network device, as host:port, like '192.168.0.3:3000'
  --web-server             Expose webserver on this port                [number]
  --post-url               Post the results to this url
  --post-interval          Seconds between posts         [number] [default: 300]
  --post-json              Post the data as json instead of form parameters
                                                                       [boolean]
  --mqtt-url               Send the data to this mqtt server
  --mqtt-topic             Use this topic prefix for all messages
                                                         [default: "smartmeter"]
  --mqtt-distinct          Publish data distinct to mqtt               [boolean]
  --mqtt-discovery         Emit auto-discovery message                 [boolean]
  --mqtt-discovery-prefix  Autodiscovery prefix       [default: "homeassistant"]
  --tcp-server             Expose JSON TCP socket on this port          [number]
  --raw-tcp-server         Expose RAW TCP socket on this port           [number]
  --debug                  Enable debug output                         [boolean]
  --sunspec-modbus         IP of solar inverter with modbus TCP enabled
  --sunspec-modbus-port    modbus TCP port               [number] [default: 502]
  --version                Show version number                         [boolean]
  -h, --help               Show help                                   [boolean]

All options can also be specified as Environment valiables
Prefix them with 'SMARTMETER_' and make them all uppercase

Inputs

This application supports two inputs, you'll need either one.

P1 cable

For a direct connection you'll need a Smartmeter cable like this one at sossolutions, and connect it to a free usb port on the device reading the meter. I've been using this cable for years without any problems.

TCP socket

You can also connect to a TCP socket, this way you don't need the device running this program to be on a device near your meter.

Tested with:

Device	Meter Type	User	Date
Slimme lezer by Marcel Zuidwijk	DSMR 4.2	@svrooij	`2021-02-23`

I'm tempted to recommend the device above since it doesn't require your pi to be near your smartmeter. The latest version of the Slimme lezer in combination with a DSMR 5 meter, doesn't even require an external power source.

This app can also output a raw socket so you can use one instance running all the time and have a second instance connected to the socket for debugging.

Solar panel inverter (optional)

This library can also read current solar production from an inverter supporting SunSpec. We use @svrooij/sunspec for this, see other package for more info.

By reading data from the solar panel as well, the properties houseUsage and solarProduction will become available. The first one is computed as follows solarProduction + calculatedUsage if you're producing 1000 watt and the calculatedUsage is -400 (delivering 400 watt), the houseUsage must be 600 watt.

Outputs

This application supports multiple (concurrent) outputs. Enable at least one!

Output -> Webserver

You can enable the webserver. This will enable you to see a simple webpage with the latest data from your smartmeter (PR with styling appreciated!). It will also enable an endpoint that responds with json with all the available data. Start this output with --web-server [port], and the webpage will be available on http://[ip-of-server]:[port]/ and the json endpoint will be avaiable on http://[ip-of-server]:[port]/api/reading.

This webpage uses WebSockets for automatic server side data refresh. So the browser will show the latest data as it comes in. If your browser doesn't support websockets it should fallback on ajax loading.

Output -> JSON tcp socket

This output creates a tcp socket where you will receive a newline delimeted json stream, to be used in your other applications. Start it with the --tcp-server [portnumer] parameter. You can then see immediate result when you connect too it with for instance telnet telnet [ip-of-server] [specified-port]. Maximum 3 connections.

> telnet 192.168.1.20 3010
Trying 192.168.1.20...
Connected to black-pearl.localdomain.
Escape character is '^]'.
{"header":"KFM5KAIFA-METER","p1Version":"42","powerTs":"2020-04-10T10:20:23","powerSn":"45303032xxx","totalT1Use":3000.497,"totalT2Use":1000.243,"totalT1Delivered":1000.458,"totalT2Delivered":3000.394,"currentTarrif":2,"currentUsage":0.049,"currentL1":2,"currentUsageL1":0.049,"deviceType":"003","gasSn":"4730303xxx","gas":{"ts":"2020-04-10T10:00:00","totalUse":2000.671},"crc":true,"calculatedUsage":49}
{"header":"KFM5KAIFA-METER","p1Version":"42","powerTs":"2020-04-10T10:20:33","powerSn":"45303032xxx","totalT1Use":3000.497,"totalT2Use":1000.243,"totalT1Delivered":1000.458,"totalT2Delivered":3000.394,"currentTarrif":2,"currentUsage":0.048,"currentL1":2,"currentUsageL1":0.048,"deviceType":"003","gasSn":"4730303xxx","gas":{"ts":"2020-04-10T10:00:00","totalUse":2000.671},"crc":true,"calculatedUsage":48}

Output -> Raw tcp socket

This output creates a tcp socket where you'll receive the raw data as it comes in. This is usefull if you want to debug the data coming in and don't want to restart your smartmeter2mqtt application all the time. This can in turn be used as an TCP socket input. Start it with --raw-tcp-server [port]. Maximum 3 connections.

Conect to it with telnet [ip-of-server] [specified-port] and see the data coming in on your windows machine.

This socket can also be used in domoticz as P1-Wifi Gateway.

> telnet 192.168.1.20 3020
Trying 192.168.1.20...
Connected to black-pearl.localdomain.
Escape character is '^]'.
/KFM5KAIFA-METER

1-3:0.2.8(42)
0-0:1.0.0(200410102433S)
0-0:96.1.1(4530xxx)
1-0:1.8.1(003000.497*kWh)
1-0:1.8.2(001000.248*kWh)
1-0:2.8.1(001000.458*kWh)
1-0:2.8.2(003000.394*kWh)
0-0:96.14.0(0002)
1-0:1.7.0(00.105*kW)
1-0:2.7.0(00.000*kW)
0-0:96.7.21(00000)
0-0:96.7.9(00000)
1-0:99.97.0(1)(0-0:96.7.19)(000101000001W)(2147483647*s)
1-0:32.32.0(00000)
1-0:32.36.0(00000)
0-0:96.13.1()
0-0:96.13.0()
1-0:31.7.0(001*A)
1-0:21.7.0(00.105*kW)
1-0:22.7.0(00.000*kW)
0-2:24.1.0(003)
0-2:96.1.0(4730xxx)
0-2:24.2.1(200410100000S)(02000.671*m3)
!5305

Output -> Webrequest

This output will posts the new data to an URL, at an interval (to prevent overloading of remote). You can provide the url to post to with --post-url [url].

You can also configure the interval with --post-interval 300 (to set it to 300 seconds).

By default the data is posted as form variables, if you want you can have it post as json by specifing --post-json.

Output -> MQTT

This will output the data to the specified mqtt server. You'll need to submit the mqtt url with --mqtt-url mqtt://[host]:[port] like --mqtt-url mqtt://localhost:1883.

Mqtt - Topics

Once enabled mqtt, this application will send several messages to your mqtt server. All prefixed with smartmeter (configurable with --mqtt-topic).

Topic: smartmeter/status/energy Payload:

{
  "header" : "KFM5KAIFA-METER",
  "p1Version" : "42",
  "powerTs" : "2020-04-13T18:22:59",
  "powerSn" : "453030",
  "totalT1Use" : 3000,
  "totalT2Use" : 1000,
  "totalT1Delivered" : 1000,
  "totalT2Delivered" : 3000,
  "currentTarrif" : 1,
  "currentDelivery" : 1.772,
  "currentL1" : 7,
  "currentDeliveryL1" : 1.77,
  "deviceType" : "003",
  "gasSn" : "47303",
  "gas" : {
    "ts" : "2020-04-13T18:00:00",
    "totalUse" : 2000
  },
  "crc" : true,
  "calculatedUsage" : -1772,
  "houseUsage": 400,
  "solarProduction": 2172
}

Topic: smartmeter/status/usage Payload:

{
  "previousUsage" : -1746,
  "relative" : -8,
  "message" : "Usage decreased -8 to -1754",
  "val" : -1754,
  "tc" : 1586795117062
}

MQTT - Auto discovery homeassistant

If you're running home assistant, be sure to enable mqtt discovery --mqtt-discovery and --mqtt-discovery-prefix (defaults to homeassistant). This will make sure the following sensors will automatically show up in home assistant:

Current usage: Your current total usage (can be negative when delivering power)
Total used T1: Total power consumed from the grid in T1
Total used T2: Total power consumed from the grid in T2
Total delivered T1: Total power delivered to the grid in T1
Total delivered T2: Total power delivered to the grid in T2

Developer section

This section is for the curious ones.

Support for output X

This package comes with several outputs, they all extend Output. Every new output should implement the start(p1Reader: P1Reader): void and the close():Promise<void> methods. They all get the instance of the current P1Reader. So your new output should subscribe to one of the events. All events are defined in P1ReaderEvents and you should use the statics from the class, (even though they are just strings).

P1ReaderEvents.ParsedResult to get the parsed result (if crc check validates), probably the one you want.
P1ReaderEvents.UsageChange to get the changes in current usage. Already computed you dont have to.
P1ReaderEvents.Line to get every line when they come in.
P1ReaderEvents.Raw to get every raw message. Is transmitted as a complete string when the endline is received.

If you start some kind of server, be sure to stop it in the close() method.

Every output is wired to the input in the index.ts file in the startOutputs() method. You initialize the correct output and push it to the this.outputs array. The start and close methods are called automaticcaly.

Configuration is done in the config.ts file. Depending on your output you might want to create a single config entry to enable it or a nested configuration like for mqtt. This class is also responsible for parsing the input and converting it to the strong-typed config.

if (this.config.outputs.mqtt) {
  console.log('- Output: Mqtt to %s', this.config.outputs.mqtt.url);
  this.outputs.push(new MqttOutput(this.config.outputs.mqtt));
}

Your output will get an event every 10 seconds, if you only want daily results you will need to build some logic to skip events.

Run tests before PR

This library enforces these eslint rules (with some exceptions that will be removed soon):

eslint:recommended
plugin:@typescript-eslint/eslint-recommended
plugin:@typescript-eslint/recommended
airbnb-typescript/base

In the tests folder are several tests defined. So we don't break any existing code. Both the javascript styles and the tests can be run with npm run test in the main folder.

Build docker image

The Dockerfile is setup to support multi-architectures builds. You can build this image on you regular 64 bit computer for these platforms. arm / arm64 / amd64 by using the following command:

# For pushing to docker hub
docker buildx build -t svrooij/smartmeter:alpha --platform linux/amd64,linux/arm/v7,linux/arm64 --push .
# For loading it to your local machine
docker buildx build -t svrooij/smartmeter:alpha --platform linux/amd64,linux/arm/v7,linux/arm64 --load .
# Regular build (no buildx installed)
docker build -t svrooij/smartmeter:alpha .

DSMR - P1 Sample data

My Keifa meter outputs the following data as you connect to the serial connection. Other meters should be supported as well. Else please start with --debug and send one full output to us.

/KFM5KAIFA-METER            // Header, Manufacturer specific

1-3:0.2.8(42)               // Version information P1 output
0-0:1.0.0(190514213620S)    // Timestamp YYMMDDhhmmssX (X=S DST/W no DST)
0-0:96.1.1(453xxxxxxxxxx)   // Electricity Meter serial number
1-0:1.8.1(002000.123*kWh)   // Total used in T1
1-0:1.8.2(001000.456*kWh)   // Total used in T2
1-0:2.8.1(001000.456*kWh)   // Total delivered back in T1
1-0:2.8.2(002000.789*kWh)   // Total delivered back in T2
0-0:96.14.0(0001)           // Current Tarrif
1-0:1.7.0(00.329*kW)        // Current use in P+ (watt resolution)
1-0:2.7.0(00.000*kW)        // Current delivery in P- (watt resolution)
0-0:96.7.21(00000)          // Number of power failures in any phase
0-0:96.7.9(00000)           // Number of long power failures in any phase
1-0:99.97.0(1)(0-0:96.7.19)(000101000001W)(2147483647*s) // Long power failures log (can be multiple)
1-0:32.32.0(00000)          // Number of voltage sags in phase L1
1-0:32.36.0(00000)          // Number of voltage swells in phase L1
0-0:96.13.1()               // Text message max 1024 characters?? Undocumented
0-0:96.13.0()               // Text message max 1024 characters??
1-0:31.7.0(002*A)           // Instantaneous current L1 in A resolution.
1-0:21.7.0(00.329*kW)       // Instantaneous power L1 (P+) in watt resolution.
1-0:22.7.0(00.000*kW)       // Instantaneous active power L1 (P-) in watt resolution.
0-2:24.1.0(003)             // Device type
0-2:96.1.0(473xxx)          // Gas meter serial number
0-2:24.2.1(190514210000S)(01543.012*m3)  // Gas usages timestamp and gas usage
!90E4                       // CRC

Parsing messages explained

The p1-reader is responsible for connecting to one of the sources, it is an eventemitter that outputs the following events line, dsmr, raw, usageChange. It will send each line to the p1-parser for parsing and checking the message. To support extra data, you'll need to take a look at the p1-map file, it contains the id used in the DSMR standard, the name in the result object and a valueRetriever. The valueRetriever is passed an array of values that where between brackets in the current line.

Supporting other data fields is just a matter of changing the p1-map file.