FNG - getting started in webthings development

Hello, my name is Bill. I’m a career programmer (mostly Windows, but a bit of Pi and raspbian recently) and I’ve just set up a Pi with the latest build of webthings.io. Controlling zwave blinds and some lights quite well; want to do more. I’m interested in participating in development, particularly of addons. I built the image on a Pi, but I gather that’s not the ideal development platform. I have a windows box as my main platform and I vnc into the Pis.

I’ve not done anything in Node.js, but I have done some web development. What would be a good setup to get started? I’ve got a couple of Pi’s, one I use for “production” mode, running all my blinds and whatnot, and it took quite a while to set all those up. Is there a way to transfer that data to the “development” Pi? I don’t want to keep having to set up the system after I nuke the pi with a new image. I have installed other programs on the Pi that I also want to run (I’ve used the runprogram addin to launch them) that I don’t want to have to install every time I want to test. That’s enough questions for now. Thanks.

If you search the forum for “backup”, there have been several posts about transferring the configuation to a new system. Basically, all you need to do is identify the correct folder path and copy it in entirety to anoher host. If you have USB controllers then there are dependencies in the configuration, but you new that already…

On my rpi 2b+ I prefer the Docker WebThings image and map the GW configuration folder to a persistent folder on a USB stick. I can then just move the USB and my Z-wave controller to a new device, or to my development PC, and resume. I occasionally backup up the USB stick data too. See: Using HA-Bridge with Mozilla Gateway

1 Like

Transferring files between two linux systems is one of the simplest things to do when you know how.

Solution 1/ use sshfs to mount the remote filesystem, e.g. you are on your dev rpi, you mount your prod rpi’s filesystem remotely onto your dev one. Then use the normal tools for moving files: mv, cp, rsync.

Solution 2/ use rsync to remote copy.

Both rely on having sshd set-up on one of the pi’s (all it needs is enabling it as the software is installed as standard). But presumably you have that bit done as you are ssh’ing into the pi’s from your Windows box.

All that’s left is knowing which folder to replicate where. If you use the latest version, it should be .webthings, the parent of the addons folder (which contains config, data, log, media, ssl, uploads).

You can edit files in the addons folder directly, just remember to generate the SHA hashes before restarting the gateway.

That said i have never played with the gateway SD image, so I am not sure about the specifics of how to make my suggestion work for you.
I started using a deb install into a headless Raspberry OS set-up, so I could control my way what I wanted to run on my pi and how. I have switched to using the docker image and after a few teething (learning) problems, it works like a dream. (this is irrelevant but my main desktop machine is Linux based).

Thanks so much! Ok, I’m old school but I guess it’s time to learn what docker is all about. I’ll do my own reading. Thanks for the info on which folders need to be moved between images; I got everything except how to regenerate the SHA before restarting the gateway. Where does the SHA hash live, and what tool do I need to investigate to generate it? I’m newish to linux, but obviously I can get around a little since I’ve built the gateway and done some other development on the pi.

Installing docker on the pi:

curl -sSL https://get.docker.com | sh # Install as user, not admin
sudo usermod -aG docker pi
#logout/login pi user
groups # shows docker
docker run hello-world # shows expected result, not error messages
docker version # mine is 20.10.2
python3 -m pip install docker-compose # not sure I have used this yet, might come in handy later
reboot
docker-compose version # mine is 1.28.0

Installing the docker image for the gateway:

docker pull webthingsio/gateway
# configure gateway image, run it & auto-restart, copy files from existing installation
# Make sure to include config/db.sqlite3 and ssl/* so the connection to the webthings.io tunnel works

Running the gateway docker image

docker run \
      -d \
      -e TZ=Europe/London \
      -v /home/pi/<<your chosen path for local gateway files outside of the docker image>>:/home/node/.webthings \
      --network="host" \
      --log-opt max-size=1m \
      --log-opt max-file=10 \
      --name webthings-gateway \
      webthingsio/gateway:latest

To start fresh again if things go badly (and change some configuration):

docker stop webthings-gateway
docker container rm webthings-gateway

As to the SHA image, every addon has a SHA256SUMS file so the gateway can check they have not been tampered with. If you edit addon files in their gateway directory directly, you need to update this file. See further for details.

Example of the files in a typical addon:

> ll thing-url-adapter/
total 92
drwxr-xr-x 3 pi pi  4096 2021-01-18 19:07:18  .
drwxr-xr-x 9 pi pi  4096 2021-02-17 23:16:32  ..
-rw-r--r-- 1 pi pi   317 2020-12-23 01:29:26  index.js
-rw-r--r-- 1 pi pi 16725 2020-12-23 01:29:26  LICENSE
-rw-r--r-- 1 pi pi  1867 2020-12-23 01:29:26  manifest.json
drwxr-xr-x 6 pi pi  4096 2020-12-23 01:29:26  node_modules
-rw-r--r-- 1 pi pi   787 2020-12-23 01:29:26  package.json
-rw-r--r-- 1 pi pi  1730 2020-12-23 01:29:26  README.md
-rw-r--r-- 1 pi pi 18232 2020-12-23 01:29:26  SHA256SUMS
-rw-r--r-- 1 pi pi 24019 2020-12-23 01:29:26  thing-url-adapter.js

For my own addons, I created a file called sha.sh with contents as follows:

#!/bin/bash

# first make sure the owner is correct for every file in this directory and subdirectories
chown -R pi:pi .
# then find every relevant file and add the shasum to the SHA file
find . -path './pkg/__pycache__' -prune -o -type f -not -name '*.swp' -exec sha256sum {} + > SHA256SUMS
cat SHA256SUMS 

One last tip: whenever an instruction tells you to pip install XYZ, or pip3 install XYZ, I have found it more reliable to replace this instruction with python3 -m pip install XYZ.

Excellent information! I will keep the board posted as I move along and get up and running.

So my game plan is as follows. I have a pi 4 on which I followed the instructions for building the full gateway. This resulted in an image that I could presumably deploy to a fresh install for a pi. This might still be an option, but then I followed the above instructions instead. Docker was already installed as part of the build process, so I started with

docker pull webthingsio/gateway

Then I also did an

scp -r pi@gateway:.webthings/ .webthings/

to put the production information onto the local pi. I am now ready to run. I presume what you meant with your “your chosen path for local gateway files outside of the docker image” notation was to do this:

docker run \
-d \
-e TZ=America/Chicago \
-v /home/pi/.webthings:/home/node/.webthings \
–network=“host” \
–log-opt max-size=1m \
–log-opt max-file=10 \
–name webthings-gateway \
webthingsio/gateway:latest

Edit: The above appears to have worked. I got a string of hex digits at the command line after running the above command, which I copied into a runit.sh file and then did a

chmod 755 runit.sh

, so I could ultimately type:

. runit.sh

to start up the gateway. At that point, I then got on my windows computer and went to the url of devgateway.lan:8080 and was able to see all the things I had added to my production gateway! Success! So now the next step is to move the zwave dongle over and make sure it all still works and my zwave things come alive (they are present but off at the moment obviously). One other thing I’d like to know: How do I shut down the production webthings gateway so that I can restart it later if I so choose? I don’t want production and development running at the same time as it might cause confusion to the webthings.io server.

Edit 2:
To further document what I did to get things working with my ZWave adapter (a fifth gen Aeotec USB stick).
First I added the following line to runit.sh, just above the --network=“host” line:

--device /dev/ttyACM0:/dev/ttyACM0 \

My understanding is this lets the virtual docker container know about and use the usb dongle that exists in real life.
Secondly, I had to do this at the command line one time:

sudo chown 1000 /dev/ttyACM0

which in my understanding gives ownership of the USB dongle to the docker user, who is 1000, so it can use it. I stopped, deleted, and restarted the docker container as outlined in your message, and I was able to see all the zwave devices after I did a little clicking around (zwave can be fussy and needs a nudge every now and then, such as going into a thing to try to read its on-off properties, for example). This nudge woke one up, then the rest of the network followed. I’m now ready to begin work on addons. I will start with the experimental date/time addon that allows time parameters for sunset/sunrise, which is a widely requested feature.

Seems like you are in control.
Thanks for sharing the USB set-up - I have not played with that sort of thing yet.

Stopping and starting the container is as simple as follows:

docker start webthings-gateway
docker restart webthings-gateway

1 Like

Hi @EricEdberg @asoundmove just thought I’d share my progress. I made some fancy updates to the date-time-adapter and put in the appropriate pull requests to expose my fork as the latest release 1.0.2. If you want to preview it, it’s available at https://github.com/wfahle/date-time-adapter/releases/download/1.0.2/date-time-adapter-1.0.2.tgz

Edit: looks like it was approved! All you have to do is go to the adapter config and choose update. It takes a good while to build; I’ll be working on that next: https://github.com/tomasy/date-time-adapter/issues/23

I’ll have to apologise as I have no time these days: struggling with recovering data from a main hard drive corruption that in turn corrupted the backup! Until this is in order I won’t have time for anything else.

No worries, you jumped in and got me started! I’m excited to continue the journey.

Thanks for helping support the tool and updating the addon. Upgraded my development server to test.

I’ve place the log of my installation at the URL below for reference. There are some errors in it that I’m not sure are critical.

https://www.edbergnet.com/DateTimeAddonInstallLog.txt

The new “offset” option defaults to -60 minutes. If I read it right, there is always a secondary sunrise/sunset event created and the event default is always -60 minutes. Many folks may not want a secondary offset created and others may want the offset more that 60 minutes.

  1. Do not create a secondary offset unless it’s manually provisioned (assuming this is what currently happens)

  2. Change slider to a text value allowing user to input the offset they want in minutes

Q: How does the user determine if the real sunrise/sunset event fired or the custom sunrise/sunset event fired? When I attempted to create a new rule, there was the legacy sunrise/sunset events, but no sunrise_wOffset or sunset_wOffset choices. Hopefully 2 new sunrise/sunset events were created.
EDIT:
After looking at the addon configuration screen, I noticed there is a “DARK” property that might be able to be used to determine the unmodified sunrise/sunset (assume it means sunrise/sunset but may not know the difference).

Eric Edberg

  1. There are INFO “DEBUG” statements printed continually in the log file every minute. Too verbose…

    2021-03-10 07:49:01.300 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: even = False (True)
    2021-03-10 07:49:01.302 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: minute = 49 (48)
    2021-03-10 07:49:01.304 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: azimuth = 111.25612493744813 (111.06934658440495)
    2021-03-10 07:49:01.309 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: elevation = 16.678167651201406 (16.503844148109756)
    2021-03-10 07:49:13.324 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: next_event = 604.7833333333333 (605.8)
    2021-03-10 07:49:13.325 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: last_event = 96.73333333333333 (95.73333333333333)
    2021-03-10 07:50:01.394 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: even = True (False)
    2021-03-10 07:50:01.395 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: minute = 50 (49)
    2021-03-10 07:50:01.397 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: minutes5 = 50 (45)
    2021-03-10 07:50:01.397 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: azimuth = 111.44329944146315 (111.25612493744813)
    2021-03-10 07:50:01.399 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: elevation = 16.85227429578691 (16.678167651201406)
    2021-03-10 07:50:13.417 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: next_event = 603.7833333333333 (604.7833333333333)
    2021-03-10 07:50:13.418 INFO : date-time-adapter: date_property.py:53 DEBUG Device: DateTime Property: last_event = 97.73333333333333 (96.73333333333333)

  2. My influxdb addon core-dumped (but restarted) with asimuth property error. Not sure if it’s related…

    2021-03-10 07:36:12.145 ERROR : influxdb-bridge: /home/node/.mozilla-iot/addons/influxdb-bridge/node_modules/webthings-client/lib/device.js:170
    2021-03-10 07:36:12.145 ERROR : influxdb-bridge: throw Error("Unknown property " + key);
    2021-03-10 07:36:12.145 ERROR : influxdb-bridge: ^
    2021-03-10 07:36:12.145 ERROR : influxdb-bridge:
    2021-03-10 07:36:12.146 ERROR : influxdb-bridge: Error: Unknown property azimuth
    2021-03-10 07:36:12.146 ERROR : influxdb-bridge: at WebSocketConnection. (/home/node/.mozilla-iot/addons/influxdb-bridge/node_modules/webthings-client/lib/device.js:170:75)
    2021-03-10 07:36:12.146 ERROR : influxdb-bridge: at WebSocketConnection.emit (events.js:314:20)
    2021-03-10 07:36:12.146 ERROR : influxdb-bridge: at WebSocketConnection.processFrame (/home/node/.mozilla-iot/addons/influxdb-bridge/node_modules/websocket/lib/WebSocketConnection.js:554:26)
    2021-03-10 07:36:12.146 ERROR : influxdb-bridge: at /home/node/.mozilla-iot/addons/influxdb-bridge/node_modules/websocket/lib/WebSocketConnection.js:323:40
    2021-03-10 07:36:12.146 ERROR : influxdb-bridge: at processTicksAndRejections (internal/process/task_queues.js:79:11)
    2021-03-10 07:36:12.152 INFO : Plugin: influxdb-bridge died, code = 1 restarting after 0

Thanks for the report! It looks like most of that was a consequence of the old configuration already existing, but it is without sunset_offset_mins or sunrise_offset_mins. If the configurtion had been missing altogether, it would have defaulted to 0. So, when there was no sunset_offset_mins in the configuration, it defaulted to None. With mins of None or 0, it does not create a new event and yet it erroneously displays -60 as the value. I will fix this in the next rev, but in the mean time if you set the configuration to real values it should work and add new events.

I’m not sure what the issue was with the azimuth property, I’ll have to look into that. It’s curious that it didn’t have a problem with the elevation property, because they are mostly cut and paste of each other. I will make the info reports into debug reports to cut the verbosity. The next revision will install/build much more quickly.

Also a note: if you have set your horizion to a nonstandard value besides -0:34 (like -6) you will probably want to change it back to the default so that the sunrise and sunset events happen when expected. Then you can offset from those actual known values instead of trying to offset from the end of civil twilight or whatever. With the -0:34 setting, sunset and sunrise will happen when expected, and with proper values (-60 is allowed, but you have to set it to something else first to get it to register a real -60) the new events will exist, with a label like “Sunset offset -15 mins”. Note that if you later change the offset, the labels of the events will change, but the rules will display the old values while still working properly. I suppose I should report that to the core team, but it’s a minor issue. If you alter the rule to sunset, exit it, and go back in and set it to the offset sunset event, it will sort itself out display-wise.