Towards Decentralized Taxonomy


(David Teller) #1

During the work week, we identified a number of limitations/issues with Taxonomy. After lengthy brainstorming with @mcav, I have been working on a new version that should hopefully lift most of these limitations and issues. This new version is not quite ready to land, but the prototype is pretty advanced.

Here is a short list of issues and fixes.

ChannelKind is too centralized

Currently, we rely upon a mega-enum ChannelKind that lists all the possible kinds of channels in the world. Every time we add a new kind of channel, we have the choice between updating Taxonomy’s ChannelKind or using the ChannelKind::Extension, with also the temptation of using a mismatched ChannelKind just because it has the right type. None of these options is satsifying.

With Decentralized Taxonomy, ChannelKind disappears. Instead of a mega-enum, we now have a much simpler type Feature, which can be extended by any crate. So, instead of having ChannelKind::LightOn, ChannelKind::HeaterOn, ChannelKind::MediaOn, etc. we will rather define

let light_on = Feature {
  implements: vec!["light/is-on"],
  type_: types::on_off,
  // ...
};

We will, of course, keep providing a library of standard Features, but this library will be a help, not a limitation.

(more on this below).

This will make it much easier to experiment with new kinds of adapters/devices and to write third-party adapters. This will also help us a bit towards out-of-process and non-Rust adapters.

This will require trivial modifications to existing adapters. Using a standardized feature basically means replacing the current calls to add_getter/add_setter with

adapter_manager.add_feature(&id, &library::light_on)

No boilerplate needed, unless the adapter developer decides to introduce a new Feature, and even then, it’s pretty trivial.

Value and Type are too centralized

Similarly, we rely upon a mega-enum Value and a mega-enum Type that list all the possible values in the world. This has basically the same drawbacks as ChannelKind.

With Decentralized Taxonomy, we break the centralized enums. The new definition ofValue and Type are based on Rust’s built-in dynamic typing. Basically, this means that you can use as Value just about anything that you know how to (de)serialize from/to JSON + Binary.

As a secondary benefit, this means that the day we start working on out-of-process and/or non-Rust adapters, it will be easier to send them a payload that they can parse in their process/language.

Existing adapters will need a few trivial changes. Basically, instead of

match value with {
  Value::OnOff(foo) => /// Proceed
  _ => // Type Error
}

we now write

match value.cast::<OnOff>() {
  Some(foo) => // Proceed
  None => // Type Error
}

No boilerplate needed for adapters that use built-in types. Adding new types will require a little boilerplate, essentially the definition of (de)serializers.

We don’t support CRUD

Experience by @mcav and @dhylands working on the Thinkerbell Adapter and the Camera Adapter show that we are not very good with CRUD. So far, they have needed to work around this limitation by introducing weird channelkinds.

The problems identified are:

  • no good way to express the DELETE part of CRUD;
  • no good way to list all the rules/images/entries – which is a shame, given that we already have many ways of listing stuff;
  • no good way to find out which UPDATE or DELETE matches which READ.

With Decentralized Taxonomy, we rework the definition of channels to solve these issues:

  • instead of separate Getter and Setter, we register a Feature with a single ID, which may (or may not) support FETCH/SEND/DELETE/WATCH operations – this simplifies considerably detection of features and finding out the correspondence between UPDATE, DELETE, etc.;
  • oh, yes, there is new operation DELETE, by the way;
  • with these changes, we can much more easily use one Feature implementation per rule/image/entry, which lets us in turn use the existing selectors and operations to access these, instead of having to roll out our own.

Existing adapters will not need changes besides what is already listed in section “ChannelKind is too centralized”. No new boilerplate needed.

Native API, JSON API

The Decentralized Taxonomy now offers two high-level APIs. The Native API is statically-typed and designed for Rust clients (e.g. Thinkerbell) while the JSON API offers a dynamically-typed front for the Native API, designed for non-Rust clients (e.g. REST API, possible future clients written in other languages).

The main consequence is that we can perform better testing on some parts of the JSON API without having to rely upon Selenium. This should mean better documentation for the REST API.

Too much boilerplate

A few minor changes there.

  • The Decentralized Taxonomy will make it a little easier to implement an Adapter which does not support FETCH, SEND, DELETE or WATCH.
  • The Decentralized Taxonomy should remove the need to implement custom lists operations (see the section on CRUD).

For the moment, that’s it. I have heard requests for me to find a way to remove the boilerplate for WATCH operations, but I haven’t found out how to just yet.

What about improving the REST API?

That’s another piece of work. I’m planning to work on this after landing the Decentralized Taxonomy. If you wish to take part in the discussions on the topic, see:

ETA?

No ETA yet, as I’m concentrating on the demos.

There is a working prototype, which misses:

  • plenty of documentation;
  • DELETE operation;
  • the JSON implementation of WATCH operation;
  • a few tests that haven’t been ported to the new API yet;
  • a few new features that need tests;
  • porting the adapters and the foxbox itself.

I don’t think that this list will take me more than one week, perhaps with the exception of the JSON implementation of WATCH operation, which is surprisingly more complicated than the native implementation.

If you wish to take a look, development takes place here.


(Fabrice) #2
  • “Feature” is a bit confusing I think. Maybe “Endpoint” would be actually better with this new api?

  • It’s unclear to me also which operation returns a value. Or are they all?

  • How do we “unwatch” ?


(David Teller) #3

Well, I’m using Feature as a replacement for ChannelKind. I should perhaps list the name ChannelKind as one of the problems we encountered: experience in the team shows that pretty much nobody understands at first that ChannelKind and the field kind of selectors are about feature discovery. Feature and the field implements are much clearer on this account.

I have considered allowing all of them to do so, but I haven’t found a nice way to go in this direction and I’m not sure it’s useful. For the moment, FETCH returns a single value, WATCH returns successive values, DELETE and SEND return nothing (except error codes).

Exactly as we do it now. No change here.