Request for feedback: Locale Health Dashboard for managers

Pontoon
#1

Hi everyone,

We’ve been working to create role-specific dashboards for Pontoon users that help them to meet the expectations and responsibilities unique to their roles. These dashboards give users powerful information to help them find ways to have the greatest impact in their roles.

We started with the manager role and have created community health dashboards to help managers identify issues within the community related to peer reviews, completion, and activity level between the three user action groups. Matjaz has created a prototype that we’d like you have a look at and offer feedback on.

The spec for this new feature is here: dashboard specs.

The prototype is here (it uses fake data, and it’s identical for all locales): dashboard prototype.

Some questions you might keep in mind while playing with the dashboards:

  1. Will this data help me be a better manager for my community?
  2. What problem areas could you expect to find with this data?
  3. What data is missing to help you understand the health of your community?
  4. How often would you check in on these dashboards?
  5. What pieces of data are most interesting to you?
  6. Do the tooltips explain each visualization well enough for you?
  7. Are the graphs easy to understand without a tutorial?
  8. What tweaks to the graphs would you recommend?

Note that we’re only incorporating a certain set of data in these dashboards. We’re also using your feedback to understand what additional data Pontoon should register and use in similar dashboards in the future. If you have recommendations, we’d love to hear them.

This feedback period will only be open for one week (ends 13 November). Please take a few minutes to experiment with the prototype and give us your feedback in this discussion thread (not in github, please).

Thanks!

L10n-drivers

1 Like
#2

Thank you Francessco for sharing this with us.
I have these suggestions:
1.It would be better to include most active translators (Who did most translation) in a panel or some place to show them.
2. Showing the projects or branches that are more importand.
3. The Unreviewed suggestion lifespan graph i don’t understand well, If there is more tutorial on it will be better.

Best Rgeards.

1 Like
#3

Great idea, Francesco. Here are a couple of suggestions:

  1. Word count. The panel now offers counts based on strings; maybe an option could be implemented to show word count, which might provide a more accurate picture.
  2. Download rejected/modified as translation memory. I know this might be beyond the scope, but if rejected and/or modified translations could be exported to a translation memory, along with the final version of specific strings, that could give a lot of information relevant for research purposes related for instance to automatic post-editing.

Best!

1 Like
#4

It would be better to include most active translators

This information is already available in the CONTRIBUTORS tab, but it might make sense to include it here (maybe limited, e.g. the top 5 contributors). Not sure how easy/hard it would be to do from a technical point of view.

Showing the projects or branches that are more importand.

This dashboard shows information about data over a time period, while this would be “current” information, and it’s already available in the PROJECTS tab. Not sure if a different summary (e.g. project completion by priority) would fit.

The Unreviewed suggestion lifespan graph i don’t understand well, If there is more tutorial on it will be better.

Did you already see the explanation displayed on the “i” icon? Does that help?

How much time it takes on average to review a suggestion within the selected time frame.

#5

Word count

The challenge is that Pontoon currently shows information about “number of strings” anywhere, so I believe that should be a larger change (and I believe we already have bugs on file). For now, we should probably aim for consistency.

Download rejected/modified as translation memory

Definitely out of scope for this conversation :wink: Note that you can open new discussions in this section if you want to talk about features.

#6

Thank you for this initiative. I really appreciate it and it may finally push me to aim to improve the review process for our team, as I promised but didn’t.

Active users

What stopped me first in the mockup was the color coding of active users. For some reason the active users widget got me confused what it actually means. Whether it’s how many people from the whole team are managers/reviewers/contributors out of the whole team, not sure why.

Even after I realized what’s it all about, I find the color coding a little bit unclear, especially when colors like green and red are used. I see they are connected to the colors used for string status, but maybe I would prefer to have all the gauges the same colour (e.g. green like the team completion percentage above is now), or some other color coding, that indicates the status (how good the engagement is).

Unreviewed suggestions lifespan

Abbreviating months with a single letters looks odd to me. We never do such think in our locale at least. Having three Js there is definitely not ideal, but I like how the letters form the name JASON.

As for the values, what is actually plotted there, from what strings? If for November there is a value of 60 days, does it mean the strings that were reviewed in that months were waiting for that review for 60 days in average?

Just to clarify the mockup, the months shown on the X axis are the last 12 months, right? So now in November, it would actually show Dec '19 as the first month and Nov '20 as the last, is that right? Then the tooltip (and X axis, if there is enough space) should show not just the month, but the year too.

Translation activity

I am a little puzzled why there are values like 49.9 or 71.5 Human/Machinery translations. Shouldn’t that be a number of strings and so an integer?

I would prefer if the tooltips for the line chart (completion) and stack chart (number of strings) were separate. Their Y axis and units are different too.

Review activity

Basically the same as for translation activity. Plus what is the unit for Unreviewed? Is that absolute number of percentage (of what)? If that’s absolute number, I wouldn’t mix it with the number of reviews. Instead I would put it into a separate chart in the size of Unreviewed suggestions lifespan. Not sure whether bar or line chart would be better then. Maybe bar to keep consistence that line charts are used for percentage and bars for absolute values.

1 Like
#7

Oooh, now I see there are some data like “New source strings” and “New” for reviews hidden by default. I enabled them now.

I like those are shown next to the stack, not as part of it. But for the “Review activity” chart, I would appreciate if the label says “what new” without having to look into the information tooltip, i.e. instead of “New” show “New suggestions”.

Thinking more about it, I would propose the show the total unreviewed strings count in this char after all, but not as line, but as a third bar and with the same Y axis as all the other bars.

1 Like
#8

Good point. Maybe 3 colors depending on the percentage (green, yellow, red).

I believe so, but probably better for @mathjazz to confirm, since I normally looked at that data at a point in time.

Yes, that should be only an issue with the underlying fake data. The only percentage is completion.

#9

Ciao @flod!

Thanks for showing us this and taking our feedback.

  1. I believe this would be useful.
  2. This could highlight if community members are disengaging.
  3. Can we get averages of time between logins for the community members or some other metric similar to that to measure engagement, maybe opt-in as now that I read it it seems to nosy.
  4. Biweekly or monthly.
  5. Active users
  6. Explained in the comments below.
  7. I believe they are easy to understand.
  8. Explained in the comments below.

image
I agree with @mstanke and yourself it would be better for these dials to change color based on % i.e. red if less than 30%, yellow if less than 70%, green if above or something like that.

image
Could this be Jan, Feb, Mar, etc. maybe going downwards to stay consistent with the other two graphs?

image
Should this tool-tip have a % for the “Human translations” too like this other one?
image
The tool-tip when hovered could easily fit the complete name of the month.

image
image
image
Could these get a tool-tip when hovered perhaps with just the definition of what I’m hovering on? It took me a bit to figure to hover on the [i] to get the definitions.

Should “Rejected” just be another shade of blue instead of red.

Hope this is helpful to y’all.

Grazie,
Alex

1 Like
#10

Thanks everyone for the feedback. We’ll go through the actionable items and update the specs accordingly.

#11

Thanks everyone for a very valuable feedback! :handshake:

I’ve deployed an updated dashboard which incorporates your suggestions:
https://pontoon-chd-v3.herokuapp.com/en-GB/insights/

For comparison, you can still access the previous version here:
https://pontoon-chd-v2.herokuapp.com/en-GB/insights/

A quick summary of changes:

  1. Active users: Use same color for all pie charts (and render data as fractions to hopefully add more clarity)
  2. Unreviewed suggestion lifespan: Use 3-letter month name in X-Axis
  3. Translation activity: Show relative share of “Human translations” in the tooltip
  4. Review activity: Rename “New” to “New suggestions”
  5. All tooltips: Use full month name and year
  6. Use integers for more realistic data

Things I haven’t changed:

  1. Separate tooltips for line chart and bar chart. I haven’t found an easy way to have 1 tooltip for the line and 1 for the entire stack; it can also result in a bad UX for selecting the bar/line when the values are very low or when bars and line overlap.
  2. Plot Unreviewed as bar instead of line. The amount of Unreviewed suggestions is a result of items in bars (some sort of summary), so it’s put into a spotlight by being plotted as a line.
  3. Use a specific shade of blue for “Rejected” instead of red. The reason to use red is because we also use it in the History panel for the reject button and the rejected status.
  4. Add tooltips on legend items. I believe having explanations in two places is a bit overwhelming. I’d prefer to keep the “Info” buttons, because they are explicitly visible and can be used on touchscreen devices.

Ideas stored for future iterations:

  1. Include a short list of most active translators.
  2. Add project priority filter to e.g. only include projects with 4+ stars in the charts.
  3. Show averages of time between contributor logins or similar metric to measure engagement.
#12

Nicely done.

I understand. May I just ask then to change the order of items in the tooltip and put Completion/Unreviewed as the first row in the tooltip, or somehow visually separate is from the rest, so it’s clear that’s the tooltip row for the line chart?

Especially now in the mockup, when the line chart is actually the highest, it’s confusing that in the tooltip it’s the last row. And all the colors in each graph are too similar for me to quickly see what is what (of course until a few days/weeks of using it my eye memory would get used to it, but…).

1 Like
#13

That’s a good point.

Done.