# Proposal: Have MDN API provide data for all specs linked to in Specifications tables

**URL:** <https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768>\
**Category:** MDN\
**Created:** [September 14, 2018, 4:38am UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768 "2018-09-14T04:38:24Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sideshowbarker](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/sideshowbarker/32/23860_2.png) [@sideshowbarker](https://discourse.mozilla.org/u/sideshowbarker)\
**Post date:** [September 14, 2018, 4:38am UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/1 "2018-09-14T04:38:24Z")

</div>

I’m interested in discussing with others here the utility (and feasibility) of having the MDN API provide data for all specs that are linked to in Specifications tables in MDN articles.

I filed an enhancement request/issue here:

> **[1491194 - MDN API should provide data for all specs linked to in Specifications...](https://bugzilla.mozilla.org/show_bug.cgi?id=1491194)**
>
> UNCONFIRMED (nobody) in developer.mozilla.org - API. Last updated 2018-09-13.

Quoting that issue’s description:

A concrete use case for this data is, I want to add annotations in the HTML spec for all defined terms in the spec which have a corresponding MDN article that contains a Specifications table with a link to that term’s URL (with fragment).

Currently, lacking a way to use the MDN API to get that, I need to crawl the MDN site and parse/scrape articles in order to the data and put it into a form I can use in the build process that generates the published HTML spec.

If the MDN API provided the data, I wouldn’t need to do that crawling/scraping.

Here’s a proposed JSON format for the structure of the exposed data:

```json
{
  "https://html.spec.whatwg.org/multipage/#2dcontext": [
    {
      "path": "API/CanvasRenderingContext2D",
      "title": "CanvasRenderingContext2D",
      "summary": "The CanvasRenderingContext2D interface is used for…"
    },
    {
      "path": "API/Canvas_API",
      "title": "Canvas API",
      "summary": "Added in HTML5, the HTML <canvas> element can be…"
    }
  ],
  "https://html.spec.whatwg.org/multipage/#abstractworker": [
    {
      "path": "API/AbstractWorker",
      "title": "AbstractWorker",
      "summary": "The AbstractWorker interface…"
    }
  ],

```

So the structure is:

- the JSON object has spec URLs as keys
- the value for each key is an array containing an item for each MDN article that has a Specifications link to that spec URL
- each item in the array is an object with “`path`”, “`title`” and “`summary`” keys
- the “`path`” value is a URL segment for that article’s path within the [https://developer.mozilla.org/en-US/docs/Web/](https://developer.mozilla.org/en-US/docs/Web/) tree
- the “`title`” value is the article’s display title
- the “`summary`” value is the article’s `class=seoSummary` content — or, if the article has no `class=seoSummary`, the contents of the article’s first non-empty paragraph

Of course the data would not just have [https://html.spec.whatwg.org/](https://html.spec.whatwg.org/) spec URLs, but would also have URLs for referenced terms in CSS, etc., specs too.

---

<div class="post-metadata">

**Author:** ![fscholz](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/fscholz/32/25986_2.png) [@fscholz](https://discourse.mozilla.org/u/fscholz)\
**Post date:** [September 14, 2018, 8:59am UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/2 "2018-09-14T08:59:33Z")

</div>

Thanks for the suggestion, Mike.

Over at mdn/browser-compat-data, it has also been requested to add spec links to features. It would allow mapping MDN docs, compat data, and specifications. The current idea is to add "`spec_urls"` next to the already existing “`mdn_url`”.  
Issues:

> <https://github.com/mdn/browser-compat-data/issues/1336>
>
> I noticed that some info about standards track is already in the schema. The tab…le of spec docs (and their statuses) stands out to me as another staple on MDN articles that would be very useful in a machine readable format.
> 
> Would it make sense to include these data in \`mdn/browser-compat-data\`? This would obviate the need to scrape MDN articles to deduce linkages between APIs/features and spec docs.

> <https://github.com/mdn/browser-compat-data/issues/1531>
>
> Rather than maintaining the "standards-track" status for each feature tracked in… BCD, it would probably be best if a given feature could be attached to a given spec (or set of specs) - that way, the standards-track status of the page could be determined from the \[separately maintained status of specs\](https://github.com/mdn/kumascript/blob/master/macros/SpecData.json).
> 
> If so, we should be able to do a one-time scraping of spec infos from existing MDN pages to automatically bind features to the specs that define them to update BCD en masse.

(Detail: it might replace “`standard_track`” in the bcd schema, as providing a link to a standard answers the question whether it is a standard feature or not).

I think this approach would help to link to MDN docs from specs and also display detailed compat data in specs, which is something that Marcos Cáceres was interested in for the Payment spec.

---

<div class="post-metadata">

**Author:** ![chrisdavidmills](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/chrisdavidmills/32/21688_2.png) [@chrisdavidmills](https://discourse.mozilla.org/u/chrisdavidmills)\
**Post date:** [September 14, 2018, 9:58am UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/3 "2018-09-14T09:58:15Z")

</div>

I like this idea too. This would answer the need to have a place to give feedback on specs in MDN pages.

Could we update the Spec2 macro to perhaps show a link to the spec as well as the standardization process badge? This’d be a lot quicker than having to manually update every page.

---

<div class="post-metadata">

**Author:** ![sheppy](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/sheppy/32/20903_2.png) [@sheppy](https://discourse.mozilla.org/u/sheppy)\
**Post date:** [September 14, 2018, 5:49pm UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/4 "2018-09-14T17:49:33Z")

</div>

I like this idea in general, other than details of how it would fit in with other proposals for data we’re discussing. I do have thoughts on this JSON structure though; the form you’ve suggested seems very specifically only useful to your use case. 🙂

First, some observations and questions, then some proposed changes.

Questions:

1. This proposal looks like all specs and items within specs are in one huge JSON file (otherwise, you wouldn’t need the keys to be full URLs). That seems impractical and unnecessary. Instead, why not have each spec in its own file?
2. Could the specs be identified using the same strings we use to identify them on MDN? We could provide a way to map these bidirectionally, from our identifying string to URL and back.

Proposed changes:

- Have each specification in its own JSON file.
- The JSON is a series of objects whose keys are the names of the individual objects in the API (interfacename, dictionaryname, elementname, css-attribute-name, etc)
- Each object may contain nested objects if the item has members of its own, such as properties, attributes, values for CSS attributes, and so forth
- Each object has the following members as well:  
– `"slug"` - the same as “path” in the original, but using the same terminology we use elsewhere in our data  
– `"title"` - same as the original; the name of the item. _This is really only needed if the keys are not using correct capitalization though_.  
– `"summary"` - same as before; the SEO summary of the item or appropriate substitute as needed

---

<div class="post-metadata">

**Author:** ![sheppy](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/sheppy/32/20903_2.png) [@sheppy](https://discourse.mozilla.org/u/sheppy)\
**Post date:** [September 14, 2018, 5:50pm UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/5 "2018-09-14T17:50:45Z")

</div>

One problem with replacing `"standard_track"` with the spec URL link instead of having both is that you lose the ability to cover the case where “this was once part of the specification but was removed in later specification updates”, which we currently can easily handle.

---

<div class="post-metadata">

**Author:** ![sideshowbarker](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/sideshowbarker/32/23860_2.png) [@sideshowbarker](https://discourse.mozilla.org/u/sideshowbarker)\
**Post date:** [September 15, 2018, 12:34pm UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/6 "2018-09-15T12:34:07Z")

</div>

> [@fscholz](#):
>
> Over at mdn/browser-compat-data, it has also been requested to add spec links to features. It would allow mapping MDN docs, compat data, and specifications. The current idea is to add " `spec_urls"` next to the already existing “ `mdn_url` ”.

Yes — having the `spec_urls` data in BCD itself would address the main need here.

But for my use case at least, in order to avoid the need to for me to scrape MDN myself, what would also need to be included is the MDN article titles and summaries.

However, since we’d not want to maintain the article titles and summaries in BCD (I think), one way the could be provided together with the BCD data is by regularly running some kind of build that pulls the data from MDN itself and integrates it with the BCD data and then incorporates into a published distribution:

> <https://github.com/mdn/browser-compat-data/issues/2836>
>
> It would be useful to have an enhanced form of BCD that included not only the st…atic data maintained in this repo but also the titles and (seo)summaries from the corresponding MDN articles.
> 
> Providing such a distribution could require some build step that pulled/scraped the corresponding MDN article titles and summaries from MDN itself.
> 
> Also, until the \`spec\_urls\` data (as discussed in #1336 and #1531) is added/moved to BCD, the build would need for now to also pull the spec-URL data from MDN itself.
> 
> https://github.com/epistemex/mdncomp-data already provides such a distribution, with the MDN summaries and spec URLs — and may \[eventually also include the article titles\](https://gitlab.com/epistemex/mdncomp/issues/2) — but I hope that the MDN staff team can consider providing the same thing upstream in some way.

As I noted in that issue, [https://github.com/epistemex/mdncomp-data](https://github.com/epistemex/mdncomp-data) already provides such a distribution, with the MDN summaries and spec URLs — and may [eventually also include the article titles](https://gitlab.com/epistemex/mdncomp/issues/2) — but I hope the MDN staff team can consider providing the same thing upstream in some way.

---

<div class="post-metadata">

**Author:** ![sideshowbarker](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/sideshowbarker/32/23860_2.png) [@sideshowbarker](https://discourse.mozilla.org/u/sideshowbarker)\
**Post date:** [September 15, 2018, 12:38pm UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/7 "2018-09-15T12:38:03Z")

</div>

> [@sheppy](#):
>
> This proposal looks like all specs and items within specs are in one huge JSON file (otherwise, you wouldn’t need the keys to be full URLs). That seems impractical and unnecessary. Instead, why not have each spec in its own file?

I’d be happy to have it in any form at all. For my use case, it wouldn’t need to be distributed as one big file — or even as a file at all. What I’d really prefer is an API that exposed it.

---

<div class="post-metadata">

**Author:** ![exe-boss](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/exe-boss/32/43293_2.png) [@exe-boss](https://discourse.mozilla.org/u/exe-boss)\
**Post date:** [September 15, 2018, 12:46pm UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/8 "2018-09-15T12:46:34Z")

</div>

Another solution might be to create an npm package containing specification data (it might also make it possible to move `SpecData.json` out of the KumaScript repository.

---

<div class="post-metadata">

**Author:** ![sheppy](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/sheppy/32/20903_2.png) [@sheppy](https://discourse.mozilla.org/u/sheppy)\
**Post date:** [September 16, 2018, 3:41pm UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/9 "2018-09-16T15:41:45Z")

</div>

> [@exe-boss](#):
>
> Another solution might be to create an npm package containing specification data (it might also make it possible to move `SpecData.json` out of the KumaScript repository.

Yeah, that would be a good approach.

> [@sideshowbarker](#):
>
> I’d be happy to have it in any form at all. For my use case, it wouldn’t need to be distributed as one big file — or even as a file at all. What I’d really prefer is an API that exposed it.

I think that would be awesome, except for the issues of hosting and the costs of maintaining the service. I don’t think we have the bandwidth on a personnel level to handle it.

---

<div class="post-metadata">

**Author:** ![sideshowbarker](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/sideshowbarker/32/23860_2.png) [@sideshowbarker](https://discourse.mozilla.org/u/sideshowbarker)\
**Post date:** [September 17, 2018, 12:29am UTC](https://discourse.mozilla.org/t/proposal-have-mdn-api-provide-data-for-all-specs-linked-to-in-specifications-tables/31768/10 "2018-09-17T00:29:29Z")

</div>

> [@sideshowbarker](#):
>
> Yes — having the `spec_urls` data in BCD itself would address the main need here.

So I’m actually now realizing that having the `spec_urls` data in BCD wouldn’t be sufficient for my use cases — and I think for other use cases as well.

The reason is that BCD only contains data for a subset of the material having MDN articles with Specifications tables.

Take for example the following article:

> **[Event handling (overview) - Event reference | MDN](https://developer.mozilla.org/en-US/docs/Web/Events/Event_handlers)**
>
> Events are signals fired inside the browser window that notify of changes in the browser or operating system environment. Programmers can create event handler code that will run when an event fires, allowing web pages to respond appropriately to...

That MDN article has a link to this HTML spec section:

[https://html.spec.whatwg.org/multipage/webappapis.html#event-handler-attributes](https://html.spec.whatwg.org/multipage/webappapis.html#event-handler-attributes)

But since the MDN article itself isn’t an article for a single interface/object or method or property or whatever, there’s never going to be a item in BCD with an `mdn_url` for that article.

Among the ~750 IDs in the HTML spec that are linked to from MDN articles, it looks like there are about 50 of them that don’t (won’t) have corresponding data in BCD.

So I think even if/when the `spec_url` data is added to BCD, we still have use cases that would benefit from having the kind of which-articles-link-to-which-specification-URLs data proposed here.

Along with this being useful for the HTML spec, it would also enable the [Bikeshed](https://github.com/tabatkins/bikeshed#bikeshed-a-spec-preprocessor) and [Respec](https://github.com/w3c/respec) tools to add MDN annotations to specs.

Given that Bikeshed and Respec are the tools used for generating the published versions for most all the current specs for the web platform, that’d mean we’d have most all specs with annotations linking back to MDN articles.