Measuring Impact For Mozilla Events

Being in this week’s ReMo vidyo call with @rosana , the topic of measurement came up. Namely, what does it mean to have impact in a Mozilla event and how to measure it?

You can watch the recording here: https://air.mozilla.org/reps-weekly-20160407/
Etherpad: https://public.etherpad-mozilla.org/p/remo-call-07-04-16

Right now there is a simple metric in place on how to consider ‘impact’: conversations started. I was wondering if the idea of measurement can be expanded somehow to also include the length and depth of conversations and the kinds of people engaged. If we engage a smaller number of people, but with longer, more meaningful conversations, could that increase impact in some cases? Would it matter to go after influencers (leaders in other open projects, journalists, political and business leaders) in particular? That’s not to say influencers should be the focus of these events, but that their inclusion can have a multiplier effect to the participation as a whole.

The second aspect is the social media impact. People with lots of social media followers are able to have a greater impact by virtue of their involvement. Would being an effective rep or ‘leader’ entail a ‘soft obligation’ to join and be active on social media networks?

@Ioana also noted that people in different communities use different networks: Twitter has less of an impact (relatively speaking) in Indonesia as opposed to Romania. Therefore, relying on shares/likes/tweets is not a definitive ‘metric’, but may be something to watch out for.

As a counter-point, there is an advantage in simplicity too. Metrics alone may not determine fuzzy ideas like ‘impact’. If you’re of this opinion, I’d like to hear from you too.

I think that is a question for the Meta WG but I saw in the past that the metric don’t work very well.
I agree that some social have different impact, In Italy depends from the knowledge of the smartphone. The majority of people use only FB and it is not easy to track or in the events not share anything on social of an event.

I think that we have do draft a document with all the suggestion for the metrics like I have done for the Reps portal improvement.

Maybe this is a conversation for Reps but time ago the Council improved their way to check on events measuring the impact. Right now there are a few mandatory questions when asking for budget and I would like to remark few of them:

  1. Which functional area goals is this supporting and how?
  2. What’s success for this event?
  3. What are the event goals?
  4. How are you going to measure it?

These questions allow to understand:

  1. How are we going to advance Mozilla as organization? Why are we doing this event?
  2. What should happen by the end of the event to know we were successful?
  3. What are the things we want to accomplish to be successful?
  4. Which specific items we are defining that can be measured that support these event goals?

So my recommendation would be to stop thinking on metrics first and do the mental exercise of planning from the top (mozilla goals) to the bottom (metrics).

What about:
-number of installs of Firefox to Android. Number of active users a month
later? To me thats a really important metric that I don’t even know if we
can measure.
-number of blog posts, instagram posts, or Facebook posts across given
time using a campaigns suggested hashtag?
-number of Thimble accounts created and active 1 mon, 6 mon after event or
campaign?

I know @adamlofting is working on impact and social network analysis
ratings for other projects. We maybe able to poach some of the metrics MLN
is using for engagement.

+1 @nukeador on deciding on what you want to measure (goals) before
deciding on how you want to measure (metrics).