@t1m1 and I have been working on transitioning the gateway source to TypeScript, which brings it inline with the recent changes to the webthing-node and gateway-addon-node libraries.
This is going to take some time, so before we go too far down that road, @benfrancis and I wanted to see if anyone had any opposition to this. I’ve noted some benefits and drawbacks here.
Please feel free to weigh in here or on GitHub. We’d also welcome contributions! 
Thanks @mrstegeman,
We discussed migrating the WebThings Gateway codebase to TypeScript a few years ago and at the time we decided against it for various reasons, but the balance of tradeoffs may have changed over time and as the code base has grown, so I think it’s worth revisiting now.
Here are some obvious pros and cons of a move to TypeScript I can think of…
Pros
Cons
Maybe some people with actual experience with TypeScript can comment on or add to these pros and cons?
Reliability
Making the gateway more reliable really appeals to me as one of my main goals for the gateway this year is to make it more production ready (e.g. by considering replacing Raspberry Pi OS with a more production quality base OS with better security and automatic software updates).
Arguably other ways to improve the reliability of the code are to increase the test coverage of unit tests and integration tests, but there will always be some bugs that slip through the net which could have been caught by static typing.
I do have a number of reservations though…
Barriers to Contribution
The first reason is simply that I haven’t learnt how to use TypeScript myself yet, which on its own is not a good reason for an architectural decision, but there are some real practical considerations to think about. Currently there are only three people who can review core gateway code: @mrstegeman, Tim Hellhake and me. If I can’t review that code then if one us writes code there’s only one person who can review it, which could slow down development.
The obvious solution to that is for me to learn TypeScript, which I’d like to do, but I’m conscious I may also need to learn Rust (to hack on the registration server) and get deeper into Android development (to work on a mobile app) this year, and it’s very unlikely I’m going to have time to do all of that extra learning whilst also starting a business.
The other (more scalable) solution to that problem is to get more contributors experienced with working on the core gateway code so that we can assign more peers, but that process is going to take time. Moving TypeScript also arguably makes that harder simply because there are fewer TypeScript developers than JavaScript developers.
Increased Build Times
The second reservation I have is about the increase in build times and its effect on developer productivity. Since webpack and TypeScript were introduced to the codebase our install and build times have increased by an order of magnitude.
This was particularly noticeable to me when developing the transition UI for 1.0, when a one line CSS change would require an extremely long wait for a re-build to test. We can work around that for front end development to a certain extent by using browser developer tools to write code before copying it across to an editor, but that doesn’t help back end development and it’s still a very painful development cycle.
When the project first started we had live reloads of static resources on the front end and plain interpreted Node.js on the back end so the development cycle was extremely fast. The introduction of webpack as a build step eventually put an end to the live reload, which I wasn’t very happy about at the time, but I couldn’t argue with the end user performance benefits. TypeScript seems to have made the situation even worse.
Is there a way we can provide a better developer experience without compromising on the user experience?
Obsolescence
My third reservation is that I would like to build WebThings Gateway to last, ideally for decades. I have a hypothesis that TypeScript is not going to have a particularly long lifespan and will eventually be made obsolete by a future version of ECMAScript (JavaScript). This is because TypeScript exists to address what some people perceive as shortcomings of JavaScript, but over time such features tend to get integrated into the ECMAScript standard. So will TypeScript still exist in ten years time? How would we feel about converting the codebase into TypeScript, only to convert it back to JavaScript at a future point in time once it has grown even bigger?
I would argue that keeping the code base as vanilla as possible and reducing the number of dependencies helps to make it more future proof. There’s clearly a balance to be reached.
What other pros and cons are there for the move to TypeScript that I’m not aware of? These issues are far from being unique to the WebThings project, so what experiences do people have from other projects which could be relevant here?
I can think of a few options going forward:
My instinct is option 1, but mainly because I am suspicious of TypeScript and I don’t like change Rationally, I know there are significant benefits of TypeScript which means we should give it serious consideration. Also, no amount of automated testing is ever going to catch all the bugs that static typing could, and vice-versa.
The wildcard option 4 brings all the benefits of static typing, and much more, but would be a very significant effort we probably don’t have the resources to pull off at this point. (It’s probably more realistic and helpful to convert some adapter add-ons to Rust first, rather than the core gateway application.)
I feel uncomfortable about option 2 for all the above reasons, but option 3 feels like it could be a reasonable compromise, if we can improve the developer experience. We could keep the front end as vanilla JavaScript to reduce build times and make it more future proof, but reap the benefits of TypeScript on the back end where it can have the most impact and where I think the future proofing argument is harder to make (given Node.js already diverges from the ECMAScript standard).
What does everyone think?
P.S. This might be a good topic to discuss at the first monthly public WebThings meeting, which we’re hoping to organise soon!
Response to some of @mrstegeman’s points on GitHub (to try to keep my comments in one place)…
it’s not changing the architecture at all, and in most cases the transpiled code is identical to the original
It’s true that the code which is executed on the gateway is still JavaScript, but it does change the development workflow and the set of skills which developers need in order to contribute.
I don’t remember any conversations about moving things to TypeScript, only Rust, so maybe that was before my time.
Ah OK, it was very early on in the project.
Code is more understandable.
This is subjective. It’s not more understandable by someone who has never used TypeScript! What do you like about it, specifically?
Static typing helps find lots of issues (I’ve already found and fixed several). This leads to fewer bugs.
Agreed, this is the big benefit.
The webthing-node and gateway-addon-node libraries have already been moved over to TypeScript.
That’s not necessarily a good reason to move the gateway code to TypeScript, I was hesistant about moving those components too. A JavaScript developer may come along wanting to contribute to webthing-node, only to discover that webthing-node is actually webthing-typescript. But at least those components are quite self-contained and don’t include any front-end code which runs in a browser.
part of the gateway source is already TypeScript. Moving everything will lead to a less confusing code base.
I agree, that’s confusing. But there are two ways to fix that problem and the other way is a lot quicker!
Moving things over has already brought in major contributions to the other libraries mentioned above, so I’m hopeful it will encourage new contributors to the gateway as well.
That’s interesting, because my assumption would be that it would have the opposite effect. There are obviously going to be the TypeScript fans who are attracted by it, but statistically I would have thought it may risk putting off more developers than it attracts.
TypeScript is new to some people. However, I’ve been able to pick it up without ever reading any TypeScript docs, so it’s a pretty small learning curve.
That’s good to know, although a small learning curve is not no learning curve and currently I’m personally very time poor but would like to keep contributing to the core gateway application.
These are all good arguments. I’m interested to hear whether anyone else has an opinion?
I don’t think that’s an entirely accurate assessment. By design, ECMAScript won’t ever fully replace typescript. Will the spec include some way for type hinting or similar? Possibly, however I doubt the spec will every do anything past run-time guarantees (see also PHP, for example). As such the “compile time” checks will stay with typescript, or eslint or whatever.
Then there is of course the case of syntax. Would it change with it being standardized? Probably, however TypeScript has had a history of introducing features that didn’t have official syntax yet and then gracefully migrating to the speced version.
Arguably you don’t have to learn it for every contribution, but to someone that’s never seen TS seeing the code for the first time will be confusing. In the meantime, you could (at least in theory) still contribute without knowing typescript, even if in some cases it could mean additional maintainer effort to accept the contribution.
To be honest, that’s something that’s always scared me about the gateway frontend. It felt too big to just simply understand as a plain project. Sure, typescript doesn’t mean modules, but it does at least mean that my editor will be much smarter and can help me better with understanding the code. And then also splitting the code into modules (as webpack allows) makes it much easier to just jump in and fix something, because you know it is fairly isolated. Of course finding the correct place for a fix is still a challenge, but I don’t think that really changes given the technology used. So I’d say the increased build times make the code more accessible.
My user-perspective comment – better performance and more reliable (less bugs) for me = a good thing.
Also, if ECMAscript might eventually replace both, and the code base would need yet another transition, is there any difficulty difference between transitioning from JS to ECMA or TS to ECMA?
And lastly, I’d say that listening to the most active developers is a good plan to keep the community engaged.
First of all: I’m not a native speaker.
If anything sounds offensive, I’m truly sorry for it.
This is in no way my intention.
There are fewer TypeScript developers than JavaScript developers, which may make growing the number of contributors harder
I think the weight of this argument is strongly dependent on which languages you compare.
If you chose something like JavaScript and Erlang, this would be a major issue.
But I’m not aware of two languages that are as close as TypeScript and JavaScript.
TypeScript is basically JavaScript + Types.
If you ignore the types, basically every JavaScript developer can read the code.
Even the types you use in JavaScript are the same.
TypeScript adds an additional compile step to our build process, which could harm developer productivity
On paper, this might be true, but in practice, there are far more time-consuming things than waiting on the compiler.
The compilation step is often only a fraction of the effort you have to put into a new feature.
Arguably other ways to improve the reliability of the code are to increase the test coverage of unit tests and integration tests, but there will always be some bugs that slip through the net which could have been caught by static typing.
I fully agree.
Barriers to Contribution
Migrating the code to TypeScript eliminates a lot of ambiguities and makes assumptions explicit.
This helps to understand big unknown code bases.
I would argue that this even lowers the barrier to contribution.
While looking at JavaScript might be more convenient, I don’t think that it can outweigh the insight it brings to an unknown codebase.
When the project first started we had live reloads of static resources on the front end and plain interpreted Node.js on the back end so the development cycle was extremely fast.
I agree that live reloads make a lot of sense in the frontend and can largely increase your productivity.
But the backend isn’t stateless like a CSS file or an HTML file.
If you change something, you need to reload the application.
Loading an adapter alone takes some time.
Compiling the TypeScript sources isn’t a big deal here.
How would we feel about converting the codebase into TypeScript, only to convert it back to JavaScript at a future point in time once it has grown even bigger?
Actually, pretty good.
The TypeScript migration alone increases the code quality a lot.
It forces you to think about things you normally wouldn’t think about because it just works in JavaScript.
Even if you go back to JavaScript, the effort isn’t wasted.
- Revert the whole code base back to JavaScript but increase test coverage to improve reliability
I think you would need to put a lot of effort into writing tests to achieve the same confidence level.
Which, in my opinion, reduces productivity and makes it harder for people to develop new features.
- Convert the whole code base to TypeScript but find a way to improve build times and developer productivity
I’m in favor of this proposal.
- Convert the back end to TypeScript , but leave the front end as plain JavaScript
My impression is that the webpack process is far heavier than the TypeScript compilation.
I don’t have numbers, but my feeling is that there is little to gain by removing TypeScript while still using webpack.
- Convert the whole back end to Rust instead
I also like this idea.
It would also be pretty cool to have a gateway that is just a single binary with a very low memory footprint.
Also, no amount of automated testing is ever going to catch all the bugs that static typing could, and vice-versa.
Exactly 
Hi all,
I don’t have experience of writing in either JS or TS and so far haven’t had chance to learn them to contribute to the gateway (although I would like to in future, this is unlikely to change in the near-term).
Despite that I’ll share my thoughts 
Messy types always annoy me, I get fed up trying to understand code that does “clever” things just because it works (weak typing in C, for example), and I also get fed up trying to debug issues that only get revealed at runtime as for me they’re harder to find. I know TS vs JS is static vs dynamic typing, not weak vs strong, so my second gripe is more important in the context of this decision, but my general preference is for clear, unambiguous and explicit types and revealing of bugs, because it helps reduce bugs.
I know there are more JS devs than TS ones according to the various statistics, but does that make them more likely to contribute to WebThings? Given that lots of people in IoT are actually used to working with microcontrollers rather than web technologies I think they might actually prefer TS, because the workflow is more similar to microcontroller development (compiled rather than interpreted code).
Also, since all TS devs probably also call themselves JS devs that will skew the stats from any surveys.
Would moving to TS preclude using WASM or any other front-end performance enhancing dru- excuse me features
Is some of the reticence to use TS because it’s made by MS by any chance? After all there is some history between Mozilla and MS…
If TS is dropped by MS in future then aren’t they likely to provide tools to support migration to TS’ successor? That might be JS or it might be something else.
Rust is a very nice prospect and I would love the performance improvements which it should bring but wouldn’t that require that all current contributors (Ben said there are three) have to go away and learn Rust? Wheras for option 1 one dev has to learn TS, which apparently isn’t very different from TS anyway? @benfrancis do you find the new TS code for the gateway understandable? For completeness, option 2 requires nobody to learn a new language as far as I can tell.
From a user perspective less bugs is nice, but the main reason it’s nice is because it allows you clever contributor/maintainer devs to spend more time on tasty features and just generally enjoying yourselves more 
Cheers 
Thank you very much for all the insightful feedback, opinions so far seem pretty unanimously in favour of TypeScript!
It sounds like there could be many benefits of migrating the code base to TypeScript. Improvements in reliability, but also making the code easier to understand for newcomers (once people grok the small differences in syntax between TS and JS). It’s very interesting to know people think it may even lower the barrier to contribution as that’s the opposite of what I expected.
I’m reassured that people think switching back to plain JavaScript in the future if necessary wouldn’t be too painful. If I understand correctly, by default the TypeScript compiler just strips the type definitions and leaves the rest of the JavaScript (including formatting and comments etc.) intact (until webpack mangles it all that is), so it pretty much automates that process. I also understand that the authors of TypeScript are involved in the ECMAScript standardisation process and usually integrate new ECMAScript features back into TypeScript, so that helps reduce the divergence between TypeScript and JavaScript.
@t1m1 Your written English is better than most native speakers, so don’t worry about that!
@madbilly The connection with Microsoft isn’t a factor from my point of view, GitHub is owned by Microsoft after all.
I’ll leave the topic open to further responses, but so far I see no reason for @mrstegeman and @t1m1 to not continue with their work. I would still strongly prefer to start with migrating the back end to TypeScript first if possible, but it sounds like there’s support for eventually migrating the front end to TypeScript too.
In which case, does anyone have any suggestions for how to reduce build times and/or improve the developer workflow? I recall that at some point either npm run watch or npm run watch:build allowed hot reloading of static resources, but it doesn’t seem to work any more.
My concern about this move is the volume of learning required to become a productive contributor. I’ve close to 40 years as a developer and have put the effort in to contribute in the addons. Including patching TypeScript.
I’d like to make some enhancements in the core too, but already face high barriers there. What I’d like to see is a set of simple procedures help contributors (who are not necessarily deep Javascript and TypeScript professional developers) get started without needing to learn all the options of all the tools, with pointers to any deeper learning resources.
My understanding is that this proposal would not stop contributions of new and changed JavaScript addons. That would be a real problem for me.
I didn’t say there are three contributors, I said there are three people who can review the code (module owners and peers). There are 99 contributors to the gateway codebase, with varying levels of activity.
I was just reading through some of the new TypeScript code, and if I’m completely honest, no I don’t. It will require me to spend some time learning about TypeScript.
Realistically that’s going to prevent me from contributing to and doing code reviews of any of the migrated code for the forseeable future. I will endeavour to learn TypeScript when I can, but I currently have a lot on my plate.
That’s correct, JavaScript add-ons can still be written in JavaScript. This is just about the core gateway application.
Correct, this won’t affect add-ons at all. They will continue to work as they always have.
My mistake, please excuse me, I should have checked back to what you wrote.
I had a quick look at some simple examples comparing JS and TS and it looks like the only difference is that TS requires variables are declared before use and given a type.
I’m sure it’s more complex in practice:
I’m not saying it’s easy, because as someone who can’t programme in JS anyway it’s not really possible for me to judge the differences.
Yes I know you’ve got a lot on your plate, running a business is not really conducive to having time to focus on programming, never mind learning a new language. Best of luck
Cheers
I stumbled across an article today while looking for something else, and thought it might provide an interesting contribution this TypeScript discussion.
The author describes a way of bringing type safety to JavaScript using JSDoc comments which are then checked by the TypeScript compiler, rather than writing the code itself in TypeScript and compiling the TypeScript to JavaScript.
I thought was an interesting approach, and may even finally provide a reason for why I’ve been (badly) writing JSDoc style comments for my JavaScript code all these years without actually generating any documentation from it!
I’m curious what people think.
My 2 cents:
IN THE BALANCE: 1/ how about a survey of the 99 contributors - weighing their answer based on the amount they contributed so far?
FOR TS: 2/ I used to develop for defense projects a long time ago, all this was done in Ada, a very strongly typed language. Pros: catches a lot of errors, great for reliability (intent is clearer), readability, exception management; Cons: heavy, time consuming, complex.
AGAINST TS: 3/ I am not sure I have this correctly, but if TypeScript essentially belongs to MS (or whatever corporate entity), then I would be very against it, until it becomes an accepted standard which does not depend on the whims of only a single corporate owner.
Weak argument AGAINST TS: 4/ Yet another language to learn (for me, rust would also fit in there). That will only be worth it if there is truly a growing trend or ground movement towards substantial market share. Personally, why not a python back-end? But that’s not typed either.
IN THE BALANCE: 5/ what is the impact on all the add-ons?
Interesting idea, but there’s no easy way to contact them all and hopefully those who care will comment here, as most of the current active contributors already have. (Also I’d rather not give the impression that decision making on this project happens by democracy, as in practice it’s more of a meritocracy, for want of a better word).
TypeScript doesn’t “belong” to Microsoft as such, it’s an open source project licensed under the Apache 2.0 license which is open to pull requests just like WebThings. Most of its top ten contributors do work for Microsoft though, so I imagine they have a large amount of control over its direction. I don’t personally have a problem with something owned by Microsoft otherwise we wouldn’t be using GitHub. I do think that JavaScript being based on the open ECMA-262 standard makes it potentially more future proof, though Node.js already diverges from that standard on the back end.
Yes, this is a problem for me too. There does seem to be a lot of momentum around TypeScript at the moment, but it could still turn out to be a fad which is forgotten in ten years time.
I don’t think we’d get much benefit from switching to Python over Node.js. I think if we were going to switch language completely for the back end it would probably be to Rust. Every time we’ve explored that possibility so far we’ve concluded Rust would be good for adapter add-ons (systems programming type features) but is still a bit immature on the web side of things.
None. Add-ons can still be written in JavaScript.
While we have been continue to discuss this, @mrstegeman and @t1m1 have almost finished migrating the back end of the gateway to TypeScript.
I think the thinking now is to pause for a while and see how things go, before deciding to migrate the front end as well.
I’m interested to see what impact the change has on finding contributors, bug finding and the developer experience. I’ve done some measurement of build times and the good news is that the migration of the back end to TypeScript doesn’t seem to have had much of an impact there. It seems like the biggest factor is webpack, as @t1m1 suggested.
On balance I think it seems like a good decision to switch, but I still have some reservations. I also read an interesting article today from 2019 which tries to offer a cost-benefit analysis for TypeScript and suggests it certainly isn’t a foregone conclusion that the benefits outweigh the costs.
We will see.