Update 3.0.1 (#61)

* Script to merge device additions and changes into devices.conf (#49)
* Add republish on reconnect
* Filter additional invalid characters '+','#','/' from topic line (replace with '_')
* Minor bugfixes for parse values
* Include script for merging new devices into existing config
* Update dependency

Co-authored-by: tsightler <tsightler@gmail.com>
Co-authored-by: Doug Krahmer <doug.git@remhark.com>

.gitignore

@@ -1,6 +1,8 @@
 old/
 test/
 config.json
+.vscode/
+.bak
 
 # Logs
 logs

README.md

@@ -1,7 +1,7 @@
 # tuya-mqtt
 This project provides a method for locally controlling IOT devices manufactured by Tuya Inc., and sold under many different brands, via MQTT.
 
-Using this script requires obtaining the device ID and local keys for each of your devices after they are configured via the Tuya/Smart Life or other Tuya compatible app (there are many).  With this information it is possible to communicate locally with Tuya devices using Tuya protocol version 3.1 and 3.3 without using the Tuya Cloud service, however, getting the keys requires signing up for a Tuya IOT developer account or using one of several other alternative methods (such as dumping the memory of a Tuya based app running on Andriod).
+Using this script requires obtaining the device ID and local keys for each of your devices after they are configured via the Tuya/Smart Life or other Tuya compatible app (there are many).  With this information it is possible to communicate locally with Tuya devices using Tuya protocol version 3.1 and 3.3 without using the Tuya Cloud service, however, getting the keys requires signing up for a Tuya IOT developer account or using one of several other alternative methods (such as dumping the memory of a Tuya based app running on Android).
 
 To acquire keys for your device please see the instructions at the TuyAPI project (on which this script is based) available at the [TuyAPI GitHub site](https://github.com/codetheweb/tuyapi/blob/master/docs/SETUP.md).
 
@@ -29,7 +29,7 @@ npm install
 ## Configuration
 Tuya-mqtt has two different configuration files.  The first is config.json, a simple file which contains settings for connection to the MQTT broker.  The second is devices.conf, a JSON5 formatted file which defines the Tuya devices that the script should connect to and expose via MQTT.  This file uses the same basic format as the "tuya-cli wizard" outputs when used to acquire the device keys, so it can be used as the basis for your tuya-mqtt device configuration.
 
-### Seting up config.json:
+### Setting up config.json:
 ```
 cp config.json.sample config.json
 ```
@@ -58,7 +58,7 @@ Note that, because the format is JSON5, which is a superset of JSON, you can use
 
 By default tuya-mqtt will attempt to find the device and automatically detect the Tuya protocol version, however, this only works if the system running tuya-mqtt is on the same network/subnet as the devices being controlled.  If this is not the case, or if automatic detection fails for some other reason, it is possible to specify the IP address and protocol manually by adding the "ip:" property to the devices.conf file.  Note that if the IP address is specified manually it is required to also manually specify the protocol version using the "version:" parameter as either "3.1" or "3.3".  The easiest way to determine the protocol version is to try controlling the device with tuya-cli and try each version to see which one works.
 
-While the above syntax is enough to create a working tuya-mqtt install with generic devices, the full power and simplicity of tuya-mqtt 3.0 is only unlocked by configuring device types to get .  Please see the full [DEVICES](docs/DEVICES.md) documenation for details.
+While the above syntax may be enough to create a working tuya-mqtt install with raw DPS values accessible via DPS topics, the full functionality of tuya-mqtt 3.0 is only unlocked by configuring device types to get.  Please see the full [DEVICES](docs/DEVICES.md) documentation for details.
 
 ### Starting tuya-mqtt
 ```
@@ -69,13 +69,18 @@ To enable debugging output (required when opening an issue):
 DEBUG=tuya-mqtt:* tuya-mqtt.js
 ```
 
+### Updating devices.conf with new and/or changed devices:
+After adding or changing devices to your Tuya account the devices.conf file can be automatically updated with all new devices and name/key changes by using the merge-devices.js script.  Create a file named new-devices.conf with the new "tuya-cli wizard" output then run ```node merge-devices.js```.  A dated backup of the original devices.conf file will be created automatically before changes are made.  Devices are only added and updated, never removed.  The resulting devices.conf file will be neatly formatted and sorted alphabetically by device name.
+
+To prevent device entries from being updated by the merge script, add property "allowMerge: false" to the device definition in the devices.conf file.
+
 ### Usage Overview
-Tuya devices work by mapping device functions to various values stored in data points (refered to as DPS values) which are referenced via an index number, referred to as the DPS key.  For example, a simple on/off switch may have a single DPS value, stored in DPS kep 1 (DPS1).  This value is likely to have a setting of true/false representing the on/off state of the device.  The device state can be read via DPS1, and, for values that can be changed (some DPS values are read-only), sending true/false to DPS1 will turn the device on/off.  A simple dimmer might have the same DPS1 value, but an additional DPS2 value from 1-255 representing the state of the dimmer.  More complex devices use more DPS keys with various values representing the states and control functions of the device.
+Tuya devices work by mapping device functions to various values stored in data points (referred to as DPS values) which are referenced via an index number, referred to as the DPS key.  For example, a simple on/off switch may have a single DPS value, stored in DPS kep 1 (DPS1).  This value is likely to have a setting of true/false representing the on/off state of the device.  The device state can be read via DPS1, and, for values that can be changed (some DPS values are read-only), sending true/false to DPS1 will turn the device on/off.  A simple dimmer might have the same DPS1 value, but an additional DPS2 value from 1-255 representing the state of the dimmer.  More complex devices use more DPS keys with various values representing the states and control functions of the device.
 
 The tuya-mqtt script provides access to these DPS keys and their values via MQTT, allowing any tool that can use MQTT to monitor and control these devices via a local network connection.  In addition to providing access to the raw DPS data, there is also a template engine that allows those DPS values to be mapped to device specific topics, called "friendly topics", allowing for consistent mapping even between devices that use different DPS keys for the same functions.  These friendly topics also support various transforms and other functions that make it easier for other devices to communicate with Tuya devices without a detailed understanding of the data formats Tuya devices use.
 
 ### MQTT Topic Overview
-The top level topics are created using the device name or ID as the primary identifier.  If the device name is available, it will be converted to lowercase and any spaces replace with underscores('_') characters so, for example, if the device as the name "Kitche Table", the top level topic would be:
+The top level topics are created using the device name or ID as the primary identifier.  If the device name is available, it will be converted to lowercase and any spaces replace with underscores('_') characters so, for example, if the device as the name "Kitchen Table", the top level topic would be:
 ```
 tuya/kitchen_table/
 ```
@@ -85,11 +90,11 @@ tuya/86435357d8b123456789/
 ```
 All additional state/command topics are then built below this level. You can view the connectivity status of the device using the status topic, which reports online/offline based on whether tuya-mqtt has an active connection to the device or not.  The script monitors both the device socket connection for errors and also device heartbeats, to report proper status. 
 ```
-tuya/kitche_table/state --> online/offline
+tuya/kitchen_table/state --> online/offline
 ```
 You can also trigger the device to send an immediate update of all known device DPS topics by sending the message "get-states" to the command topic (this topic exist for all devices):
 ```
-tuya/kitche_table/command <-- get-states
+tuya/kitchen_table/command <-- get-states
 ```
 As noted above, tuya-mqtt supports two distinct topic types for interfacing with and controlling devices. For all devices, the DPS topics are always published and commands are accepted, however, friendly topics are the generally recommended approach but require you to use a pre-defined device template or create a customer template for your device when using the generic device.
 
@@ -98,17 +103,17 @@ If you do create a template for your device, please feel free to share it with t
 If you would like to use the raw DPS topics, please jump to the [DPS topics](#dps-topics) section of this document.
 
 ## Friendly Topics
-Friendly topics are only available when using a pre-defined device template or, for the generic device, when you have defined a custom template for your device.  Friendly topics use the tuyq-mqtt templating engine to map raw Tuya DPS key values to easy to consume topics and transform the data where needed.
+Friendly topics are only available when using a pre-defined device template or, for the generic device, when you have defined a custom template for your device.  Friendly topics use the tuya-mqtt templating engine to map raw Tuya DPS key values to easy to consume topics and transform the data where needed.
 
-Another advantage of friendly topics is that not all devices respond to schema requets (i.e. a request to report all DPS topics the device uses).  Because of this, it's not always possible for tuya-mqtt to know which DPS topics to acquire state information from during initial startup.  With a defined template the required DPS keys for each friendly topic are configured and tuya-mqtt will always query these DPS key values during initial connection to the device and report their state appropriately.
+Another advantage of friendly topics is that not all devices respond to schema requests (i.e. a request to report all DPS topics the device uses).  Because of this, it's not always possible for tuya-mqtt to know which DPS topics to acquire state information from during initial startup.  With a defined template the required DPS keys for each friendly topic are configured and tuya-mqtt will always query these DPS key values during initial connection to the device and report their state appropriately.
 
 For more details on using friendly topics, please read the [DEVICES](docs/DEVICES.md) documentation which discusses how to configure supported devices or define a custom template.
 
 ## DPS Topics
-Controlling devices directly via DPS topics requires enough knowledge of the device to know which topics accept what values.  Described below are two differnt methods for interfacing with DPS values, the JSON DPS topic, and the individual DPS key topics.
+Controlling devices directly via DPS topics requires enough knowledge of the device to know which topics accept what values.  Described below are two different methods for interfacing with DPS values, the JSON DPS topic, and the individual DPS key topics.
 
 ### DPS JSON topic
-The JSON DPS topic allows controlling Tuya devices by sending Tuya native style JSON messages to the command topic, and by monitoring for Tuya style JSON replies on the state topic.  You can get more details on this format by reading the [TuyAPI documentaiton](https://codetheweb.github.io/tuyapi/index.html), but, for example, to turn off a dimmer switch you could issue a MQTT message containing the JSON value ```{dps: 1, set: false}``` to the DPS/command topic for the device.  If you wanted to turn the dimmer on, and set brightness to 50%, you could issue separate messages ```{dps: 1, set: true}``` and then ```{dps: 2, set: 128}```, or, the Tuya JSON protocol also allows setting multiple values in a single set command using the format ```{'multiple': true, 'data': {'1': true, '2': 128}}```.  JSON state and commands should use the DPS/state and DPS/command topics respectively.  Below is an example of the topics:
+The JSON DPS topic allows controlling Tuya devices by sending Tuya native style JSON messages to the command topic, and by monitoring for Tuya style JSON replies on the state topic.  You can get more details on this format by reading the [TuyAPI documentation](https://codetheweb.github.io/tuyapi/index.html), but, for example, to turn off a dimmer switch you could issue a MQTT message containing the JSON value ```{dps: 1, set: false}``` to the DPS/command topic for the device.  If you wanted to turn the dimmer on, and set brightness to 50%, you could issue separate messages ```{dps: 1, set: true}``` and then ```{dps: 2, set: 128}```, or, the Tuya JSON protocol also allows setting multiple values in a single set command using the format ```{'multiple': true, 'data': {'1': true, '2': 128}}```.  JSON state and commands should use the DPS/state and DPS/command topics respectively.  Below is an example of the topics:
 ```
 tuya/dimmer_device/DPS/state
 tuya/dimmer_device/DPS/command
@@ -121,11 +126,11 @@ tuya/dimmer_device/DPS/2/command  <-- 1-255 for brightness state
 tuya/dimmer_device/DPS/1/state    --> accept true/false for turning device on/off
 tuya/dimmer_device/DPS/2/command  <-- accepts 1-255 for controlling brightness level
 ```
-**!!! Important Note !!!**\
+**!!! Important Note !!!**
-When sending commands directly to DPS values there are no limitation on what values are sent as tuya-mqtt has no way to know what are valid vs invalid for any given DPS key.  Sending values that are out-of-range or of different types than the DPS key expects can cause unpredicatable behavior of your device, from causing timeouts, to reboots, to hanging the device.  While I've never seen a device fail to recover after a restart, please keep this in mind when sending commands to your device.
+When sending commands directly to DPS values there are no limitation on what values are sent as tuya-mqtt has no way to know what are valid vs invalid for any given DPS key.  Sending values that are out-of-range or of different types than the DPS key expects can cause unpredictable behavior of your device, from causing timeouts, to reboots, to hanging the device.  While I've never seen a device fail to recover after a restart, please keep this in mind when sending commands to your device.
 
 ## Issues
-Not all Tuya protocols are supported.  For example, some devices use protocol 3.2 which currently remains unsupported by the TuyAPI project due to lack of enough information to reverse engineer the protcol.  If you are unable to control your devices with tuya-mqtt please verify that you can query and control them with tuya-cli first.  If tuya-cli works, then this script should also work, if it doesn't then this script will not work either.
+Not all Tuya protocols are supported.  For example, some devices use protocol 3.2 which currently remains unsupported by the TuyAPI project due to lack of enough information to reverse engineer the protocol.  If you are unable to control your devices with tuya-mqtt please verify that you can query and control them with tuya-cli first.  If tuya-cli works, then this script should also work, if it doesn't then this script will not work either.
 
 ## Integration with other Home Automation tools
 openHAB examples are [here](docs/openHAB.md).
@@ -136,7 +141,7 @@ openHAB examples are [here](docs/openHAB.md).
 - [Tycale](https://github.com/Tycale)
 - [crashdummymch](https://github.com/crashdummymch)
 - [GadgetAngel](https://github.com/GadgetAngel)
-
+- [dkrahmer](https://github.com/dkrahmer)
 
 ## Related Projects:
 - https://github.com/codetheweb/tuyapi

cleanup.js

@@ -1,29 +0,0 @@
-// Object to capture process exits and call app specific cleanup function
-var debug = require('debug')('Cleanup');
-
-function noOp() {};
-
-exports.Cleanup = function Cleanup(callback) {
-
-    // attach user callback to the process event emitter
-    // if no callback, it will still exit gracefully on Ctrl-C
-    callback = callback || noOp;
-    process.on('cleanup', callback);
-
-    // do app specific cleaning before exiting
-    process.on('exit', function () {
-        process.emit('cleanup');
-    });
-
-    // catch ctrl+c event and exit normally
-    process.on('SIGINT', function () {
-        debug('Ctrl-C...');
-        process.exit(2);
-    });
-
-    //catch uncaught exceptions, trace, then exit normally
-    process.on('uncaughtException', function (e) {
-        debug('Uncaught Exception...', e.stack);
-        process.exit(99);
-    });
-};

config.json.sample

@@ -3,6 +3,5 @@
     "port": 1883,
     "topic": "tuya/",
     "mqtt_user": "",
-    "mqtt_pass": "",
+    "mqtt_pass": ""
-    "qos": 2
 }

devices/tuya-device.js

@@ -17,7 +17,7 @@ class TuyaDevice {
             id: this.config.id,
             key: this.config.key
         }
-        if (this.config.name) { this.options.name = this.config.name.toLowerCase().replace(/ /g,'_') }
+        if (this.config.name) { this.options.name = this.config.name.toLowerCase().replace(/\s|\+|#|\//g,'_') }
         if (this.config.ip) { 
             this.options.ip = this.config.ip
             if (this.config.version) {
@@ -249,10 +249,10 @@ class TuyaDevice {
         // Perform any required math transforms before returing command value
         switch (deviceTopic.type) {
             case 'int':
-                value = (deviceTopic.stateMath) ? parseInt(Math.round(evaluate(value+deviceTopic.stateMath))) : value = parseInt(value)
+                value = (deviceTopic.stateMath) ? parseInt(Math.round(evaluate(value+deviceTopic.stateMath))) : parseInt(value)
                 break;
             case 'float':
-                value = (deviceTopic.stateMath) ? parseFloat(evaluate(value+deviceTopic.stateMath)) : value = parseFloat(value)
+                value = (deviceTopic.stateMath) ? parseFloat(evaluate(value+deviceTopic.stateMath)) : parseFloat(value)
                 break;
             }
 
@@ -602,6 +602,14 @@ class TuyaDevice {
         this.connectDevice()
     }
 
+    // Republish device discovery/state data (used for Home Assistant state topic)
+    async republish() {
+        const status = (this.device.isConnected()) ? 'online' : 'offline'
+        this.publishMqtt(this.baseTopic+'status', status)
+        await utils.sleep(1)
+        this.init()
+    }
+    
     // Simple function to monitor heartbeats to determine if 
     monitorHeartbeat() {
         setInterval(async () => {

merge-devices.js

@@ -0,0 +1,116 @@
+#!/usr/bin/env node
+const fs = require('fs');
+const json5 = require('json5');
+const utils = require('./lib/utils');
+
+// Setup Exit Handlers
+process.on('exit', processExit.bind(0));
+process.on('SIGINT', processExit.bind(0));
+process.on('SIGTERM', processExit.bind(0));
+process.on('uncaughtException', processExit.bind(1));
+
+async function processExit(exitCode) {
+    if (exitCode || exitCode === 0)
+        console.error('Exit code: ' + exitCode)
+    process.exit();
+}
+
+// Main code function
+const main = async() => {
+    const date = new Date();
+    const dateTimeStr = date.getFullYear() + '-' + ("0" + (date.getMonth() + 1)).slice(-2) + '-' + ("0" + date.getDate()).slice(-2) + '_' + ("0" + date.getHours()).slice(-2) + ("0" + date.getMinutes()).slice(-2) + ("0" + date.getSeconds()).slice(-2);
+    const configDevicesFilename = 'devices.conf';
+    const configDevicesBackupFilename = `${configDevicesFilename}_${dateTimeStr}.bak`;
+    const configNewDevicesFilename = `new-${configDevicesFilename}`;
+
+    let configDevices;
+    let configNewDevices;
+
+    try {
+        console.log(`Loading ${configNewDevicesFilename}...`);
+        configNewDevices = fs.readFileSync(`./${configNewDevicesFilename}`, 'utf8');
+        configNewDevices = json5.parse(configNewDevices);
+    }
+    catch (e) {
+        console.error(`Could not parse new devices config file [${configNewDevicesFilename}]!`);
+        console.error(e);
+        process.exit(1);
+    }
+
+    try {
+        console.log(`Loading ${configDevicesFilename}...`);
+        configDevices = fs.readFileSync(`./${configDevicesFilename}`, 'utf8');
+        configDevices = json5.parse(configDevices);
+    }
+    catch (e) {
+        console.error(`Could not parse devices config file [${configDevicesFilename}]!`);
+        console.error(e);
+        process.exit(1);
+    }
+
+    try {
+        console.log(`Backing up devices config file [${configDevicesFilename}] to [${configDevicesBackupFilename}]...`);
+        fs.copyFileSync(`./${configDevicesFilename}`, `./${configDevicesBackupFilename}`);
+    }
+    catch (e) {
+        console.error(`Could not make backup of devices config file [${configDevicesFilename}] to [${configDevicesBackupFilename}].`);
+        console.error(e);
+        process.exit(1);
+    }
+
+    console.log('Indexing devices...');
+
+    // Create a dictionary for faster lookups with many devices
+    const configDevicesDictionary = {};
+    for (let configDevice of configDevices) {
+        configDevicesDictionary[configDevice.id] = configDevice;
+    }
+
+    console.log('Merging devices...');
+
+    // Add new devices and update existing devices
+    for (let configNewDevice of configNewDevices) {
+        let configDevice = configDevicesDictionary[configNewDevice.id];
+        if (configDevice == null) {
+            // Add new device
+            console.log(`Adding device: ${configNewDevice.name} (id: ${configNewDevice.id})...`);
+            configDevices.push(configNewDevice);
+            configDevicesDictionary[configNewDevice.id] = configNewDevice;
+            continue;
+        }
+
+        if (configDevice.allowMerge === false)
+            continue; // No merge updates allowed for this device
+        
+        // Update existing device
+        if (configDevice.name !== configNewDevice.name) {
+            console.log(`Updating device name: '${configDevice.name}' to '${configNewDevice.name}' (device id: ${configDevice.key})...`);
+            configDevice.name = configNewDevice.name;
+        }
+        if (configDevice.key !== configNewDevice.key) {
+            console.log(`Updating device key for: '${configDevice.name}' (device id: ${configDevice.key})...`);
+            configDevice.key = configNewDevice.key;
+        }
+    }
+
+    // Sort the devices by name
+    configDevices.sort((a, b) => a.name < b.name ? -1 : 1);
+    const configDevicesJson = json5.stringify(configDevices, { space: '    ', quote: '"' });
+
+    //console.log(configDevicesJson);
+
+    try {
+        console.log(`Saving devices config file [${configDevicesFilename}]...`);
+        fs.writeFileSync(`./${configDevicesFilename}`, configDevicesJson, { encoding: 'utf8' });
+    }
+    catch (e) {
+        console.error(`Could not write devices config file [${configDevicesFilename}]!`)
+        console.error(e)
+        process.exit(1)
+    }
+
+    console.log('Done!');
+}
+
+// Call the main code
+main();

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "tuya-mqtt",
-  "version": "3.0.0",
+  "version": "3.0.1",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {
@@ -1097,9 +1097,9 @@
       "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
     },
     "ini": {
-      "version": "1.3.5",
+      "version": "1.3.8",
-      "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.5.tgz",
+      "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
-      "integrity": "sha512-RZY5huIKCMRWDUqZlEi72f/lmXKMvuszcMBduliQ3nnWbx9X/ZBQO7DijMEYS9EhHBb2qacRUMtC7svLwe0lcw=="
+      "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew=="
     },
     "inquirer": {
       "version": "7.3.3",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "tuya-mqtt",
-  "version": "3.0.0",
+  "version": "3.0.1",
   "description": "Control Tuya devices locally via MQTT",
   "homepage": "https://github.com/TheAgentK/tuya-mqtt#readme",
   "main": "tuya-mqtt.js",

tuya-mqtt.js

@@ -64,8 +64,7 @@ async function republishDevices() {
         debug('Resending device config/state in 30 seconds')
         await utils.sleep(30)
         for (let device of tuyaDevices) {
-            device.publishMqtt(device.baseTopic+'status', 'online')
+            device.republish()
-            device.init()
         }
         await utils.sleep(2)
     }