In case you didn’t see it already (link), I made something similar, but in python. The project uses rasa for nlu parsing and automatically builds domain language models out of the installed skills for improved stt performance.
Maybe you can find some inspirations for your project there, you can find it here: