Synchronous Messaging at Mozilla - Trial Servers & Feedback

Mozillians
#35

We used Mattermost at Mozilla Foundation for a little over a year before the company migration to Slack. We had a lot of non-technical contributors for which IRC wasn’t an ideal candidate. It worked, but wasn’t a favorite compared to Slack.

Mobile support left a lot to be desired at the time, the apps were slow and buggy. It lacked some features and nice-to-haves that you might expect from a more mature platform. Not sure if that is still the case.

I will say, from experience it’s better to have everyone on one platform if you can - although understandably this doesn’t always happen. With the creation of the Slack there was a duration of a few months where contributor engagement was low because employees would understandably choose to just use the one platform. Which led to delayed responses increasingly as time went on.

Contributors ended up being folded into the MozFest Slack, and tied back into the official Mozilla slack through the #contribute-code channel. This seemed to be a good solution, it was never fully tested because shortly after was the transition away from learning products (phase out of Thimble, etc)

1 Like
#36

Good to know. @ryanwse do you happen to have the list of requirements that were considered when analyzing these tools in the past? It would helpful to frame some conversation here.

Thanks!

1 Like
#37

I love all three options for an open source code and I like it in the next order:

1-Rocket
I think it is more personalized and have a installer to linux and app movil, i like it.

2-Mattermost
This remember a slack, i very like that has installer in gnu/linux but this is not too much necessary, i think. I don’t like that it isn’t very customizable.

3-Riot
In this I don’t like it too much but it’s not bad, mmmm i like this but it is no my favorite option.

1 Like
#38

Around 2016/2017, the Mozilla Philippines Community tried to have a self-hosted Mattermost instance. We abandoned the project because:

  • Most of the local Mozillians are in Slack;
  • Mattermost needs an above average server specs

In my opinion, Mattermost is the better option for us to promote FOSS and at the same time, adopt it as an organization.

#39

After 10 days, here my impressions:

I’ve been daily testing the three options, but I have to admit that I stopped opening RocketChat after a few days, from my point of view its set of of features and UI seem too simplistic for the need we have.

Mattermost seems to have a nicer UI/UX and parity with known slack workflows, but I have to admit when you look a bit deeper, its back-end is not as flexible as I would like it to be. Being have to enforce our CPG and easily handle moderation is a must for me, and it’s seems mattermost is not specially strong on this one (no report, no moderation, no clear way to control a public space where random people will join).

Riot/Matrix initially looked like a less-advanced UI/UX as mattermost, but I have to say that the core matrix team have been collecting all our feedback and even implementing fixes in realtime. This has been a surprise to be and demonstrates a potential for further engagement in case we need paid-modifications to suit our needs. Simple theming and changing some defaults seems to be totally doable, and these were my only cons when talking about the web client.

When talking about backend, Matrix seems to be the most flexible and mature one, not only in terms of moderation tools and policy (which are really strong) but other things like potential federation and integrations.

I suspect a lot of people will provide feedback only based on the main clients UI, but I think that the minor personal preferences we might have can easily changed to fit out needs (specially in the case of riot).

That’s why I would recommend we go with matrix/riot and just understand the minor UX changes needed, since I’m confident the backend will serve us better in the long term than the other options.

Cheers.

13 Likes
#40

I wanted to wait until I post my feedback, but I really couldn’t have done it any better than Nuke just did. I fully, whole-heartedly agree with everything he said.

5 Likes
#41

Hi,

Going to add more information about Mattermost, specially related to security.

Mattermost is really careless about deps in their project components having vulnerabilities, quick example:

The desktop client is not exception, I have done a few checks time ago and found deps with vuln not being taken care, and is being something that keeps happening from time to time.

Be sure to check the vuln record and how they proceed to deal with this type of issues, for example:

Hope this brings a different point to consider.

6 Likes
#42

To avoid burying the lede, I’ll come right out and say Riot/Matrix is my first choice and it wasn’t close.

Disclaimer: I’ll admit I didn’t give Rocketchat and Slack the same fair shake as the other two. The former due to poor first impressions and lack of activity. The latter due to previous familiarity and bias (against).

Interface

In terms of the interface, I’d say Riot and Mattermost are a wash. At first I liked Mattermost’s interface better, but after spending a bit of time tweaking my settings I managed to make Riot look much more appealing and less cluttered. I learned that a lot of these settings can have a server default, so we can fix this for people if we want. But overall they both seem to tick most of the major boxes in terms of usability, and getting into the minor details here isn’t very interesting to me.

Moderation

The main reason to switch away from IRC is to make it possible to fight abuse. And here Riot/Matrix seems head and shoulders above the others. One feature I particularly liked is the ability to mute users. This means if anyone is experiencing abuse either in channel or in direct messages, they can take matters into their own hands without needing to wait for an admin/moderator to step in. This feature didn’t seem to exist in Mattermost. Matrix also has a handy moderation guide. While I haven’t tested out everything myself, it looks like Matrix supports many ways to deal with spam and abuse. In contrast I searched the Mattermost docs for “ban”, “abuse”, “kick”, “moderation”, “moderator” but none of those came up with any relevant hits.

Support

All that being said, the biggest draw to me is how dedicated Matthew and the Matrix team are to making this work. From standing up the moderation guide, to fixing bugs in near realtime to simply hanging out and answering questions. It’s clear they are dedicated and willing to support Mozilla (at least for the time being). It has been a very pleasant surprise and I think both orgs will benefit from the relationship (Mozilla gets support, Riot/Matrix gets a large user study). The interactions I’ve witnessed so far have made me confident that any problems Mozilla might have with Riot/Matrix will be taken seriously and addressed relatively quickly.

Extras

I have several other reasons for preferring Riot/Matrix too, though they aren’t criteria in the doc so I’ll just list them quickly.

  • Read-only guest mode
  • E2E Encryption (*not quite fully implemented in Riot yet)
  • Open protocol / not a silo
  • Federation. I know this was a non-goal, but being able to jump into the channels of other open source projects that we interact with (and letting them jump into ours) is useful. If we can enable this without making the moderation story harder, it would be nice to do.
  • IRC / Slack bridges (I think Mattermost supports these too)
  • Other bridges (not so relevant to Mozilla, but nice for individuals who might want them)
  • Several clients to choose from
  • Markdown (Mattermost also has this)

But even without all these extras, from my experimentation Riot/Matrix seems like the best choice for Mozilla.

10 Likes
#43

I agree with everything @ahal said. Riot is also for me my first choice and Mattermost my second.

Here’s one thing I’d add that hasn’t been mentioned but I think is important for me.

App client

An advantage with Riot is that they have a native app available for Android and iOS. The other chat platforms only offer a wrapper of their web version. I’ve tested the Mattermost one and it doesn’t offer a great user experience. Personally, the new platform should have a mobile app that is seen as a first-class citizen by their developers

Also, on desktop there’s a few native clients for Matrix that integrate well with the OS they’re targeting (e.g Fractal with Gnome on Linux). It also has a plugin for Weechat if you like terminal apps.

There’s also Matterhorn for Mattermost but the developers have no intention to integrate SAML/SSO connection and that’s a dealbreaker for me.

6 Likes
#44

Agree with everything said by @nukeador and @ahal about Matrix.

Matrix aligns with Mozilla Mission compared to other options we have, it is a open protocol that enables interoperability and federation, which is a must needed thing to improve the internet health in the current times.

8 Likes
#45

Hi, everyone: we’re now less than 24 hours away from the shutdown of our chat system trial instances. Tomorrow morning - October 9th, morning Eastern time, and with little or no further notice - those machines will be shut down and permanently deleted . As much as I would love to leave these things spinning until it was time to announce the successful candidate, we’ve made a specific commitment about how user data would be handled on these services and we are going to follow through on that commitment.

We’re grateful for everyone’s participation and the feedback we’ve received so far, and we’ve learned a lot. If you have any information from those instances that you would like to preserve, you need to act on that immediately.

Again, thank you for all of your time, effort and feedback. I’m looking forward to the next steps in the process.

6 Likes
#46

And there it is. As announced, we’ve shut down our trial servers.

I’d like to thank everyone who’s participated in this process and given us your feedback. I’m grateful for the time and effort everyone’s put in to testing these services, and we’ve learned a lot.

I hope that we’ll be to distill it all down, make a decision and announce our next steps shortly.

Again, thank you.

4 Likes
#47

I missed the trial (my bad). FWIW there’s some commentary in Reddit:

#48

Slack

I rarely need/needed to use instances of Slack, but the experiences made me uneasy in ways that were difficult to quantify/describe. There was, I think, an underlying sense of “Yet Another System”; plus an overarching sense of things occurring in relative isolation.

Matrix

IHMO this will be a fine choice for Mozilla.

To counter the many plus points, my only significant criticism is of the notifications UX in the https://riot.im/app/ web client.

(For me, the long-standing shortfalls require use of a last-century technology – e-mail – to plug the gaps in an otherwise very pleasantly forward-looking solution. YMMV; I’m in https://riot.im/app/#/room/#riot-web:matrix.org for feedback.)

HTH

Re: Publicly visible streams · Issue #4817 · zulip/zulip and linked issues, I imagine that the current lack of true public visibility was an additional factor in exclusion of Zulip from the final contest.

Also: re: https://chat.zulip.org/#narrow/stream/7-test-here/topic/push.20notifications I couldn’t get push notifications. @rht join me there, if you like; I may be missing something.

#49

The blog post says that “assuming we can gather that support” the new service will be set up on December 1st. Did that happen?

#50

Being Canadian I didn’t account for American Thanksgiving in my calculations. So I won’t have an announcement for you today, unfortunately. I expect (though cannot promise) to have more to say very shortly.

#51

I definitely look forward to us finishing up the transition process. The current schism we have between IRC and Slack has made a mess of communication, with far too many contributor queries being missed because everyone was looking at the other service.

1 Like
#52

Well, you’ll still need to look at two services, since that whatever new thing will not replace the internal Slack. But the interface might be a bit nicer. :wink:

#53

That’s
true – however, currently we have a number of channels set up on the internal slack that shouldn’t be, because people are using it instead of
public channels that had already existed. Hopefully we will strongly encourage those to be closed down in favor of the new public system. Otherwise, our community engagement will continue to suffer as it has the past few months as more people have spend less time looking at IRC and more time looking at Slack.

Indeed, as I’ve watched things the last few months, I’ve seen a number of IRC channels kind of collapse
because the staff that used to be active there have been more focused on Slack instead. This makes me nervous about any ongoing setup in which
we have split chat areas. The more things people have to look at, the more likely they are to either ignore some of them entirely or to at least poorly distribute their time across them.

Since having to start watching Slack in addition to IRC and various GitHub repositories and Bugzilla and the documentation tracking system we have scattered across Trello, Google Sheets, and GitHub, and so on, a lot of stuff has just gotten casually glanced at if at all for months now. There’s just too much stuff in too many places now. We can’t keep adding more things to look at if we expect people to keep up with all of it.

If IRC isn’t good enough for our needs anymore, we need to choose a replacement
that meets our needs well enough that we can minimize or eliminate our Slack usage. The number of user or contributor questions I see going unanswered these days even though there are people around is much higher
than it used to be; I’ve seen it repeatedly today, and it makes me sad.
This didn’t used to happen nearly this much before we had so much reliance on a second chat solution.

We need to be aware of these risks when rolling out a new system.

5 Likes
#54

Since there still seems to be an IRC community: Matrix and Slack can also be bridged to IRC. Don’t know about Slack, but for Matrix that works fine. Matrix users will have a modern UI, while IRC users will receive a link where they can view pictures someone posted on Matrix side.