You can’t really use self-signed certificates in conjunction with a .mozilla-iot.org subdomain, because the certificates need to be generated by LetsEncrypt on behalf of Mozilla in ordered to be trusted by browsers.
You either need to opt into the .mozilla-iot.org subdomain during first time setup and allow LetsEncrypt to automatically generate the certificates for you, or choose your own domain (or local host) and generate your own certificates.
We haven’t created a UI for configuring a custom domain in settings yet (that’s bug https://github.com/mozilla-iot/gateway/issues/82) so you’d need to do that yourself on the command line.
If you’re running the gateway on a Raspberry Pi you might be better off using the pre-built image (instructions on the blog) which also does other useful things like configure iptables so the gateway process doesn’t have to run as root to run on port 80/443.
If you want to configure a custom domain yourself, you just need to point that domain at your IP address, but if you Raspberry Pi is behind a firewall you’ll need to open up ports on your router and forward them to the Raspberry Pi. This is something else the pre-built image solves automatically, by using a tunnelling service.