DefaultAPISidebar, APIRef and GroupData

wbamberg · May 12, 2019, 9:14pm

DefaultAPISidebar, APIRef and GroupData

In the last couple of weeks I’ve been digging into the DefaultAPISidebar, APIRef and GroupData macros. Having learned some things about them, and developed some opinions on Things We Should Do About It, I thought it would be worth writing it up here, partly to check my understanding, partly to share my understanding, and partly to propose some possible improvements.

This particular post will just describe how AFAICT these things work and what they do. I’ll follow up with replies describing pitfalls and problems, and what we might do about them.

Which sidebar should you use when?

DefaultAPISidebar and APIRef both build sidebars for pages under https://developer.mozilla.org/en-US/docs/Web/API. So which should you use when?

Concepts: APIs and Interfaces

As far as I can see there’s a conceptual distinction in MDN between “APIs” and “Interfaces”. (This is terribly confusing to me because I think of an API as an interface - that’s what the “I” is for.)

An interface is a single concrete type of object that a developer might interact with, like Request.
An API is a more abstract collection of features, that may include multiple interfaces and also individual methods belonging to other interfaces. Generally it seems that APIs align with specifications.

For example. There’s an API we call the Fetch API, that contains all the stuff under the abstract concept of Fetch. This includes specific objects (interfaces) like Request and Response, but also includes the fetch() method of the WindowOrWorkerGlobalScope interface. It has its own spec, that defines all these bits.

Corresponding MDN page types

API pages

To document this abstract thing called “Fetch”, we have an overview page whose name is usually (but not always) of the form “APIName_API”. This page gives an overall description of the API. It may also have subpages, which are guides to aspects of the API.

For example: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API, which has child pages like https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Cross-global_fetch_usage.

Interface pages

To document interfaces we have pages whose name is the name of the interface. This page lists the properties, methods, events that are defined for that interface, and these things all get subpages under the interface’s page.

For example, https://developer.mozilla.org/en-US/docs/Web/API/Request describes the Request interface, and has child pages like https://developer.mozilla.org/en-US/docs/Web/API/Request/method.

Corresponding sidebars

Corresponding to this distinction:

API pages (aka landing pages, aka overview pages), and their children should use DefaultAPISidebar.
Interface pages (aka API reference pages), and their children should use APIRef.

What do they do?

I won’t go into all the details of what the macros do. For APIRef there are some more details in this comment, and for DefaultAPISidebar and GroupData there are some more details in this comment.

Briefly, though.

GroupData

GroupData.json is a big blob of JSON. It contains one object for each “group”, where a “group” is basically an API. So there’s an entry called “Fetch API”, one called “WebVR API”, and so on.

Groups may contain:

the name of an overview page
names and links to guides about the API
interfaces, methods, properties, and events that comprise the API

Note that the methods, properties, and events lists do not include items that are under the interfaces: they are for items defined under other interfaces. For example, the entry for “Fetch API” lists WindowOrWorkerGlobalScope.fetch() but not Request.json().

APIRef

APIRef expects to be found on an interface page or the child page of an interface page, like https://developer.mozilla.org/en-US/docs/Web/API/Request or https://developer.mozilla.org/en-US/docs/Web/API/Body/json.

Its content is driven mostly by the pages it finds under that interface page, and their tags. It collects the children of the main interface and sorts them by tag into “constructors”, “methods”, “properties”, “events”, and these links end up in corresponding sections of the sidebar.

It also refers to InterfaceData.json to list other interfaces that this interface implements or inherits from.

It also accepts an argument which is the name of a group, and if this is given, it also uses GroupData.json to fetch interfaces, methods, and properties that belong the given group. These things then get stuffed all together into a “Related pages” bit of the sidebar.

DefaultAPISidebar

DefaultAPISidebar’s content is mostly driven by GroupData.json. It is given a group name as a parameter. It fetches the named group from GroupData, and from the group adds a link to the overview page, if given, guides, interfaces, properties, methods, events.

Also, if an overview page was listed in the interface, it fetches subpages of that overview, which it adds to the list of guides.

ListGroups

One last macro is ListGroups. This is used in just one page, the root Web/API page. It fetches GroupData and adds a link to the overview page for each group (i.e. for each API). The link is derived from the overview name, if that exists, or the group’s name. So there’s an implicit assumption here that overview URLs map to overview names or group names: if they don’t, these links will be broken.

wbamberg · May 10, 2019, 10:30pm

People use the wrong macro

Perhaps because the distinction between “APIs” and “Interfaces” isn’t always very clear, or because these things both live in the same namespace, or because the macros are rather confusingly named, people choose the wrong macro for the page.

Usually they choose APIRef when they should choose DefaultAPISidebar (this is probably just because APIRef is much much more common so people are more likely to see it as an example to copy).

But if you use APIRef on an API page (as opposed to an Interface page) then the sidebar will be garbage:

it will look for subpages with tags like “Constructor”, “Event”, “Method” and so on, but won’t find any
it won’t attempt to list any guide pages, or interfaces the API contains, and so on.

Doing a survey of APIs listed in Web/API, and excluding obsolete APIs and APIs that don’t have overview pages at all, we find that:

49 overview pages correctly use DefaultAPISidebar
13 pages, about 1/4, are incorrectly using APIRef and are showing garbage sidebars as a result

For example, https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API which should be showing this data.

Of course fixing these 13 pages is quick, but this tells us a couple of things:

writers are confused. Perhaps renaming DefaultAPISidebar might help? As would moving the API pages so they don’t share the same namespace as interfaces. But that seems like a much bigger change.
readers don’t appear to notice sidebars that don’t make much sense. Perhaps we have trained them not to.

Edit: I just updated all these pages to use the right macro, so the WebVR sidebar doesn’t look like that any more. Unfortunately, now it has the right sidebar it is exhibiting the duplicated-guides problem, which I was going to talk about next. It’s a bit like https://www.youtube.com/watch?v=aI0euMFAWF8.

sheppy · May 10, 2019, 9:33pm

I think that’s what we absolutely should do. We should rename DefaultAPISidebar to StandardOverviewSidebar, or similar. Personally, I would love to be able to rename APIRef to something more descriptive, like APIRefSidebar, but that’s pretty unrealistic.

exe-boss · May 10, 2019, 10:45pm

I personally recommend APIOverviewSidebar, since StandardOverviewSidebar could be confused with other namespaces.

wbamberg · May 12, 2019, 9:37pm

I personally recommend APIOverviewSidebar , since StandardOverviewSidebar could be confused with other namespaces.

Yeah, I think so too. I’m a bit sceptical that renaming will help much though, tbh. I did find this: https://wiki.mozilla.org/MDN/Archives/Meetings/Community/2015/09-23#News which is helpful especially in explaining “Default”:

This work well for all simple API, that have a couple of guides and some reference

For large APIs, we will still need to build more complex sidebars, like for WebRTC: https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API

This is another of my gripes, which I will eventually get to, that several APIs have their own handcrafted sidebar macros, and I think we should aim to converge on one* sidebar implementation.

* (or at least < 30 )

Edited to add: I’m very tempted to suggest that instead of talking about “APIs” and “Groups”, we talk about “Specifications” throughout, since that seems like a more accurate definition of these things. It’s telling that the section listing all these things in Web/API is called “Specifications”: https://developer.mozilla.org/en-US/docs/Web/API#Specifications.

Among other things, that could mean renaming DefaultAPISidebar -> SpecificationSidebar.

exe-boss · May 11, 2019, 12:48am

I’ve got a few plans for this, including adding sidebar generation and handling shared code to the mdn namespace:

(needs to be rebased, depends on mdn/kumascript#1107)

wbamberg · May 12, 2019, 3:40pm

Guides get listed twice

DefaultAPISidebar lists guides in the sidebar. It takes guides from subpages of the overview page and from the guides listed in the group. If a single guide appears in both places, it gets listed twice in the sidebar.

See e.g.

This seems like an easy fix: have DefaultAPISidebar check the URLs and remove duplicates.

Unfortunately it isn’t, because of page moves and redirects. Suppose a guide page is initially created outside the API overview page, and is listed in GroupData under that URL. Then we move it under the API overview page, it will appear twice and a simple check of the URLs will not show that. Perhaps page JSON can tell us about redirects? - but still this doesn’t look like a good direction.

My first preference here was to fix this by removing the “guides” property of GroupData and only looking for guides as subpages of the API overview page. That seems simpler, and a rule that guides should live under the API overview page seems good generally for documentation coherence.

However there are a couple of problems: the obvious one is that sometimes there might be a good reason for guides to live outside the API subtree. Perhaps we should give writers that freedom. The less obvious one (which is more convincing to me) is that writers probably want to order the guides in the sidebar, like:

getting started with the foo API

advanced uses of the foo API

some really niche stuff about the foo API

The WebVR API sidebar is quite a good example of that. The first half of the sidebar lists guides as they are returned by subpages, and it doesn’t make sense:

Using VR controllers with WebVR

Using the WebVR API

WebVR concepts

The second half has the same guides in the order they come in GroupData, and does make sense:

Using the WebVR API

WebVR concepts

Using VR controllers with WebVR

So I think we should change the macro so it doesn’t look in API overview subpages. Correspondingly, we’ll need to add “implicit” guide pages to GroupData.

exe-boss · May 12, 2019, 11:06pm

One option would be to have guides in GropData override subpage guides, which would be placed after the last subpage guide in GroupData or the beginning, if GroupData only contains non-overview subpage guides.

Option two would be to only do overview subpage guides if the GroupData guides key is missing or doesn’t contain subpages of the overview page.

Also, $json should support returning redirect metadata, which would help hide Firefox OS APIs from the API sidebar.

wbamberg · May 13, 2019, 3:33am

One option would be to have guides in GropData override subpage guides, which would be placed after the last subpage guide in GroupData or the beginning, if GroupData only contains non-overview subpage guides.

Sorry, I don’t understand this.

Option two would be to only do overview subpage guides if the GroupData guides key is missing or doesn’t contain subpages of the overview page.

We could do this. But as I said above, I think it’s inappropriate for guides to be taken from subpages, since guides typically want to be ordered. In this respect guides are different from something like a list of methods.

Also, $json should support returning redirect metadata, which would help hide Firefox OS APIs from the API sidebar.

Yes, as I said above, whether or not it’s possible, I think this is the wrong way to go. I would prefer us to be trying to simplify this situation, not to make it more complex.

chrisdavidmills · May 13, 2019, 9:12am

Thanks for doing all this work Will, to make sense of this mess. I must admit to being confused about the difference between these two macros for years. The wrong usage of them is probably largely my fault.

I’d be happy to have a name change to make them less confusing. APIRef is used in too many places to change the name, so renaming DefaultAPISidebar to SpecificationSidebar sounds good. Or maybe APILandingPageSidebar, to make it clearer that this is just to be used on the API landing pages?

One problem is that just about anything can be referred to as an API, from an entire spec of related functionality, to a single method. So maybe yours is better?

I’d be very happy to change DefaultAPISidebar so that it only includes the guides listed in groupdata, and we can pick and choose which ones we list, and the order in which they appear.

The only question I have left is — APIRef doesn’t list any of the guides that might exist that relate to that interface, or that method, etc. To find more out you have to go back to the API landing page. Are we happy with that? I guess that’s OK, and we are looking to simplify. To list specific guides per interface or whatever would make things too complex.

sheppy · May 13, 2019, 12:00pm

So… I think we need to have it both ways – homehow. We need to primarily list the ordered guides from the data, but we should find a way to also be sure every guide page in the hierarchy is also listed somewhere. Whether that’s in the sidebar, or perhaps in the content of the overview page doesn’t matter much to me. But some method to ensure that every guide shows up even when the writer forgets to add it to the database (which will happen in a world of so many contributors) is necessary, IMO.

I concur with the name SpecificationSidebar in principle although I’d suggest SpecSidebar instead because it’s shorter.

wbamberg · May 13, 2019, 6:12pm

I do think that APIRef should list guides, yes. I’ve so far avoided discussing the particular design of these sidebars, because I think it’ll be a morass, but I do think adding guide pages is an obviously useful thing to do.

In general I think the API sidebar experience is too fragmentary, but this issue intersects with the Great Mixin Debate, and in general how we should present the organization of APIs to users.

For example, suppose I’m interested in knowing about Fetch.

I start at https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API, where I get the GroupData sidebar.
I see Request, think it’s useful and visit https://developer.mozilla.org/en-US/docs/Web/API/Request. I now get the APIRef sidebar for Request. Looking at that page, I decide I want the json() method, so I click that.
I’m now at https://developer.mozilla.org/en-US/docs/Web/API/Body/json, and get the APIRef sidebar for Body.

That’s three closely related pages in the same quite small API, with three different sidebars - not just a bit different, but completely different. For me this makes navigation very difficult.

But yes, as a small step, I agree that including guides in APIRef seems like a good plan (also, now we have tests, we can do this with some degree of confidence).

Edited to add, yet again, sorry.

As well as guides, we could also add a link at the top of APIRef sidebars to the API overview page, which would help tie things together a bit.

So at the top of the sidebar on https://developer.mozilla.org/en-US/docs/Web/API/Body/json, we’d have a link to https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API.

wbamberg · May 13, 2019, 5:54pm

Well… I’m torn on this TBH. I do appreciate that it’s more convenient for guides to be listed automatically. But:

I think it’s important to list guides in a curated order. If we effectively make the group.guides property optional for subpages, then noone will use it.
It’s not like people are writing hundreds of guide pages every year. Here are all the subpages (level 1) of all the overview pages listed in GroupData (excluding the archived groups I want to remove): https://developer.mozilla.org/en-US/docs/User:wbamberg/all-guide-pages. There are 110 of them, and this is the product of several years’ worth of writing for all of Web/API.
This is similar to the way guides are added for static sidebars like AddonSidebar, LearnSidebar, JsSidebar, and it seems to work OK in these cases. It’s just part of the workflow: you write a guide, you update the sidebar.

exe-boss · May 14, 2019, 1:32am

{{APIRef}} supports this as long as you pass the correct group name argument.

wbamberg · May 14, 2019, 2:57am

Oh yes, sorry, I forgot about that.

sheppy · May 14, 2019, 4:02pm

Yes, definitely.

While I agree this is important, I think it’s also important that all guides get listed at all, and as things stand, with the sidebar having to be updated on GitHub, it’s far too easy to create a subpage that never gets added to the menus or sidebars. This is why I’ve always been in favor of some degree of automation, and why I think it’s important that we automatically add to the end of the page list any articles not included in the curated article list.

I would be more in favor of requiring every page to be manually added to the sidebar if editing content and metadata and the like all had a more integrated experience.

exe-boss · May 14, 2019, 10:46pm

That is already supported, as I explained in my reply:

wbamberg · May 14, 2019, 11:59pm

Web/API broken links

The Specifications section of Web/API uses the ListGroups macro to list all groups in GroupData, as links to the group’s API overview page. Many of these links are broken, which contributes to this page looking sad and unmaintained. Sometimes real users complain about this.

I’ve been through the broken links, excluding those that would be removed by my PR to clean up GroupData. The table below lists them all, together with the reason for the broken link.

API                             Reason for bad link
------------------------------------------------------------
CSSOM View                   -> no API overview page
Console API                  -> no API overview page
DOM Events                   -> no API overview page
Device Orientation Events    -> no API overview page
File API                     -> no API overview page
Geometry Interfaces          -> no API overview page
Image Capture API            -> needs overview property
Media Source Extensions      -> needs overview property
Resize Observer API          -> no API overview page
SVG                          -> no API overview page
Screen Orientation API       -> no API overview page
URL API                      -> no API overview page
Web Components               -> not under web/api
Web MIDI API                 -> no API overview page
WebVTT                       -> needs overview property
XDomain                      -> remove?

Nonexistent overview pages

So in most of these cases the problem is the obvious and not cheap-to-fix one: the overview page just doesn’t exist. We should file bugs to write overview pages for all these APIs.

Another thing we could do is this. At the moment, ListGroups will use the group’s overview property to build a link, if the overview property exists. But if the overview property does not exist, ListGroups will use the group’s name to build a link (https://github.com/mdn/kumascript/blob/master/macros/ListGroups.ejs#L45-L52). This means that an author can specify the group explicitly using overview or implicitly using the group name. I think we should stop doing this, and only use overview, and update GroupData to set overview for all groups that actually have overview pages.

This means ListGroups would stop showing broken links for nonexistent overview pages.

Also note that this ListGroups behavior is inconsistent with APIRef and DefaultAPISidebar, which only use the overview property to get an overview URL. So making this change would make all three macros consistent, and remove a bit of obscure magic from the ListGroups macro.

There are also a couple of nonexistent groups which are a bit different.

“CSSOM View”: this doesn’t have an overview page, but the overview page for CSSOM lists all the “CSSOM View” interfaces, treating them as a single thing. Perhaps we should fold “CSSOM View” into “CSSOM”.
“DOM Events”: this doesn’t have an overview page, but the overview page for DOM lists all the “DOM Events” interfaces, treating them as a single thing. In this case there’s even overlap in the interfaces the groups contain: Event and EventTarget appear in both “DOM” and “DOM Events”.

Other excuses

Apart from these, there are some easy fixes:

“Image Capture API”, “Media Source Extensions”, and “WebVTT” all have overview pages but the implicit name-derived links are incorrect, so this can be fixed just by giving them explicit overview properties.
“Web Components” already has an overview page but it’s not under Web/API, it’s at https://developer.mozilla.org/en-US/docs/Web/Web_Components. We should move it, right?
We’re surely never going to write an overview page for “XDomain”, so I suggest deleting this group (not deleting the interface page itself, just the group entry).

chrisdavidmills · May 15, 2019, 7:59am

@wbamberg there isn’t really anything I disagree with here. I think creating landing pages for these API is important, and something I would like to do early on as part of the maintenance project. Perhaps we could file an Epic and substories for “Fix API landing pages”, and do some estimates on how long it might take, then look at wherre we could fit it in, in Q3? I’m not saying you should do that necessarily; I’d be happy to help.

For web components, I ended up putting the landing page in a different place because I wasn’t so sure about it being an API in the same way as the others — it’s kind of a couple of related APIs, plus a couple of HTML elements, and some other bits.

We could argue that the Custom Elements API and the Shadow DOM API should have their own landing pages…but they make more sense being bundled together as web components, IMO.

But I don’t feel particularly strongly about this. I would be OK with moving it under Web/API.

wbamberg · May 15, 2019, 5:13pm

Thanks Chris, I have filed https://github.com/mdn/sprints/issues/1537 for adding the new pages. I didn’t create all the sub-issues yet though :).

Ah, that is interesting and a thing I hadn’t thought of. Then I think we should probably delete this group (since it’s not a “Web API” and GroupData is quite explicitly for Web APIs).