IOT all devices remain disconnected

WebThings
#1

Hello;

I have been using Mozilla iot successfully for many months now. However, starting 2 days ago, all my zigbee devices remain disconnected – even after a reboot.

I confirmed that my zigbee adaptor is not the issue (functions properly when connect to my PC). I am running version 0.11.0 – on a raspberry pi 3b+.

Now, if have also tried to upgrade to version 0.12 – but is keeps showing ‘in process’ – with no hint it is doing anything.

Any help would be appreciated.

Here is a bit of the log following a reboot:

Opening database: /home/pi/.mozilla-iot/config/db.sqlite3

2020-04-11 13:36:39.931 INFO : Opening database: /home/pi/.mozilla-iot/log/logs.sqlite3

2020-04-11 13:36:41.202 INFO : wifi-setup: waitForWiFi: networks exist: [ ‘Connector_EXT’, ‘Connector’ ]

2020-04-11 13:36:41.291 INFO : wifi-setup: waitForWifi: connection found

2020-04-11 13:36:41.297 INFO : DNS config is: raspberrypi03

2020-04-11 13:36:41.300 INFO : Service Discovery: changing profile - local domain is now: raspberrypi03

2020-04-11 13:36:42.424 INFO : Service Discovery: state changed to: true

2020-04-11 13:36:42.456 INFO : HTTPS server listening on port 4443

2020-04-11 13:36:42.459 INFO : Redirector listening on port 8080

2020-04-11 13:36:44.666 ERROR : Error getting value for thingId: 2e22e7a62dfff91225f0f1999aeb056e property: active

2020-04-11 13:36:44.668 ERROR : getProperty: device: 2e22e7a62dfff91225f0f1999aeb056e not found.

2020-04-11 13:36:44.672 WARN : Rule get failed { code: 500,

message: ‘getProperty: device: 2e22e7a62dfff91225f0f1999aeb056e not found.’ }

2020-04-11 13:36:44.678 ERROR : Error getting value for thingId: 2e22e7a62dfff91225f0f1999aeb056e property: active

2020-04-11 13:36:44.688 ERROR : getProperty: device: 2e22e7a62dfff91225f0f1999aeb056e not found.

2020-04-11 13:36:44.691 WARN : Rule get failed { code: 500,

message: ‘getProperty: device: 2e22e7a62dfff91225f0f1999aeb056e not found.’ }

2020-04-11 13:36:44.695 ERROR : Error getting value for thingId: f4773995eeb886393de9b2d316354458 property: active

2020-04-11 13:36:44.697 ERROR : getProperty: device: f4773995eeb886393de9b2d316354458 not found.

2020-04-11 13:36:44.699 WARN : Rule get failed { code: 500,

message: ‘getProperty: device: f4773995eeb886393de9b2d316354458 not found.’ }

2020-04-11 13:36:44.702 ERROR : Error getting value for thingId: f4773995eeb886393de9b2d316354458 property: active

2020-04-11 13:36:44.704 ERROR : getProperty: device: f4773995eeb886393de9b2d316354458 not found.

2020-04-11 13:36:44.707 WARN : Rule get failed { code: 500,

message: ‘getProperty: device: f4773995eeb886393de9b2d316354458 not found.’ }

2020-04-11 13:36:44.976 INFO : Loading add-on: cron-adapter

2020-04-11 13:36:45.590 INFO : Loading add-on: scheduler-adapter

2020-04-11 13:36:45.998 INFO : Loading add-on: zigbee-adapter

2020-04-11 13:36:47.973 INFO : cron-adapter: Opening database: /home/pi/.mozilla-iot/config/db.sqlite3

2020-04-11 13:36:48.027 INFO : zigbee-adapter: Opening database: /home/pi/.mozilla-iot/config/db.sqlite3

2020-04-11 13:36:48.177 INFO : scheduler-adapter: Opening database: /home/pi/.mozilla-iot/config/db.sqlite3

2020-04-11 13:36:49.893 INFO : zigbee-adapter: Loading add-on zigbee-adapter from /home/pi/.mozilla-iot/addons/zigbee-adapter

2020-04-11 13:36:49.999 INFO : cron-adapter: Loading add-on cron-adapter from /home/pi/.mozilla-iot/addons/cron-adapter

2020-04-11 13:36:50.002 ERROR : zigbee-adapter: Error: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.28’ not found (required by /home/pi/.mozilla-iot/addons/zigbee-adapter/node_modules/@serialport/bindings/build/Release/bindings.node)

2020-04-11 13:36:50.005 ERROR : zigbee-adapter: at Object.Module._extensions…node (module.js:682:18)

2020-04-11 13:36:50.006 ERROR : zigbee-adapter: at Module.load (module.js:566:32)

2020-04-11 13:36:50.008 ERROR : zigbee-adapter: at tryModuleLoad (module.js:506:12)

2020-04-11 13:36:50.014 ERROR : zigbee-adapter: at Function.Module._load (module.js:498:3)

2020-04-11 13:36:50.016 ERROR : zigbee-adapter: at Module.require (module.js:597:17)

2020-04-11 13:36:50.018 ERROR : zigbee-adapter: at require (internal/module.js:11:18)

2020-04-11 13:36:50.020 ERROR : zigbee-adapter: at bindings (/home/pi/.mozilla-iot/addons/zigbee-adapter/node_modules/bindings/bindings.js:112:48)

2020-04-11 13:36:50.022 ERROR : zigbee-adapter: at Object. (/home/pi/.mozilla-iot/addons/zigbee-adapter/node_modules/@serialport/bindings/lib/linux.js:1:98)

2020-04-11 13:36:50.024 ERROR : zigbee-adapter: at Module._compile (module.js:653:30)

2020-04-11 13:36:50.026 ERROR : zigbee-adapter: at Object.Module._extensions…js (module.js:664:10)

2020-04-11 13:36:50.029 INFO : scheduler-adapter: Loading add-on scheduler-adapter from /home/pi/.mozilla-iot/addons/scheduler-adapter

Screeching to a halt
#2

All my Zigbee devices stopped working today too. I’m using the Digi XStick. I had previously managed to get them working again in 0.12 by following the steps in this post, including re-installing the Zigbee add-on.

But now the following error was surfaced as a toast in the UI:

Failed to start add-on Zigbee: /lib/arm-linux-gnueabihf/libc.so.6: version 'GLIBC_2.28' not found (required by /home/pi/.mozilla-iot/addons/zigbee-adapter/node_modules/@serialport/bindings/build/Release/bindings.node)

Neither a reboot nor re-installing the add-on seemed to help.

Currently sitting in the dark :new_moon_with_face:.

#3

Screeching to a halt again.

Today, at about noon PDT, all three of my instances of the things gateway quit working. Syslog on all three systems show the same error as was fixed earlier in the week:

Apr 11 15:19:53 bigyurt run-app.sh[556]: 2020-04-11 15:19:53.150 ERROR  : zigbee-adapter: Error: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.28' not found (required by /home/pi/.mozilla-iot/addons/zigbee-adapter/node_modules/@serialport/bindings/build/Release/bindings.node)

I’ve also tried removing and reinstalling the zigbee addon to no avail.

When this happened earlier this week, I turned off automatic updates. So I’m surprised that this problem has suddenly happened again and to all instances simultaneously.

My spouse has requested that I remove the system entirely and, reluctantly, I’m inclined to agree. Relying on this system for home heating/lighting control is not wise at this time.

#4

I tried following the same instructions that fixed the problem earlier this week.

It doesn’t help.

#5

This is the same problem as with the gateway on 0.12. It seems like things are going to be problematic from here on out if you’re still using a Stretch based image. I’d recommend backing up your ~/.mozilla-iot directory, flashing a fresh 0.12 image, and restoring the directory.

#6

By the way, this is because I released a new version of the Zigbee adapter yesterday. Add-ons are built with the same toolchain as the gateway images, which uses the newer glibc version.

#7

When you say “back up the ~/.mozilla-iot directory”, do you mean the whole thing recursively? I ask because there is a directory called addons in .mozilla-iot that contains code. Copying that directory from an older installation into a newer installation means that older code will be overwriting newer code. That doesn’t seem right.

#8

I think it’s safe to copy the whole directory @2braids. The add-ons get automatically upgraded anyway.

If anyone is wondering how to do that, I did the following:

  1. Enable SSH via the gateway’s web interface, in Settings -> Developer
  2. Log into the gateway over SSH from a desktop computer
    $ ssh pi@192.168.1.xxx

(with xxx set to the IP address of your gateway)

  1. Install the zip tool and zip up the directory (you could use tar instead if you prefer):
    $ sudo apt-get install zip unzip
    $ zip -r mozilla-iot-backup.zip .mozilla-iot

  2. From the desktop computer, secure copy the backup file from the gateway:
    $ scp pi@192.168.1.xxx:/home/pi/mozilla-iot-backup.zip ./

(where xxx is the IP address of the gateway.)

You would basically then do the reverse, with unzip to restore it once you’ve re-flashed.

However, I haven’t done that yet because…

#9

I wanted to try to see if there is an alternative upgrade path by upgrading the underlying OS with:

$ sudo apt update
$ sudo apt dist-upgrade
$ sudo reboot

Unfortunately I still get the same glibc error as before from the Zigbee add-on.

@mstegeman I’m wondering whether we could come up with a manual upgrade solution which could in future be automated?

Short of switching to a base OS which supports automatic upgrades, can you think of a path to get the upgrade working manually on Raspbian, in a way that could potentially be automated in future?

#10

dist-upgrade will not update you from one major release to the next without some extra steps. There is a process for doing that, but it can be pretty dangerous. In my experience, I tend to end up with quite a few broken things. It’s typically recommended to reinstall.

#11

Oops, you’re right. I missed the crucial step.

FWIW, I just upgraded Raspbian to buster and now my Zigbee devices work again.

I wouldn’t necessarily recommend this because it may make problems you have in future less reproduceable by others than if you just re-flash with a newer clean image. And if you do try it, definitely make sure you back up your .mozilla-iot directory first as described above in case something goes wrong.

Here’s what I did.

$ sudo nano /etc/apt/sources.list

Replace instances of “stretch” with “buster”

$ sudo apt update
$ sudo apt upgrade
$ sudo apt dist-upgrade
$ sudo reboot
#12

I have installed a fresh 0.12 image which looks to be running fine. I have also uninstalled and reinstalled the Zigbee adapter - but the same problem remains - or no connection to zigbee devices.
In the log message of GLIBC_2.28 not found. Can this library be added manually?

#13

So, you downloaded and flashed the full 0.12 image? You shouldn’t be seeing the GLIBC message anymore after doing that. Can you SSH into your Pi (or use a keyboard/monitor) to run lsb_release -a? It should look like this:

pi@gateway:~ $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

I’d also like to see the output of apt-cache policy libc6, which should look similar to this:

pi@gateway:~ $ apt-cache policy libc6
libc6:
  Installed: 2.28-10+rpi1
  Candidate: 2.28-10+rpi1
  Version table:
 *** 2.28-10+rpi1 500
        500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
        100 /var/lib/dpkg/status
#14

Hello, It looks like looks like the issue may be that i am still running Jessie on this Pi that iot is running on. I have installed iot by building from source - and not the pre-built image.
So based on the discussion on this topic, i guess my solution would be to back-up my current mozilla-iot directory, re-image my Pi to Buster, and then copy back my mozilla-iot directory?

#15

Yes, that would be the best solution. Make sure you’re backing up ~/.mozilla-iot, not ~/mozilla-iot (notice the dot).

#16

I’m having the same issue where all of my Zigbee devices are disconnected.

I’m using the ConBee II adapter.

I tried rebuilding the node modules as per the instructions here: Screeching to a halt, and I reinstalled the Zigbee addon, but it’s still not working.

Anyone have any other ideas/suggestions for me to try?

pi@gateway:~/.mozilla-iot/addons $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

pi@gateway:~/.mozilla-iot/addons $ apt-cache policy libc6
libc6:
  Installed: 2.28-10+rpi1
  Candidate: 2.28-10+rpi1
  Version table:
 *** 2.28-10+rpi1 500
        500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
        100 /var/lib/dpkg/status
#17

Can you post your latest log file from ~/.mozilla-iot/log?

#18

Thank for replying, @mstegeman! I got it working, I saw that in the other thread another option was to backup and reinstall (Screeching to a halt), so that’s what I tried and that fixed it! :smiley: