I have a scraping application that uses Firefox to read finance variables in certain web pages every minute. It’s implemented as an addon with content script and background script. The content script refreshes the page once a minute, waits for the DOM to finish building, and then reads the values in the table. It sends these values to a webserver.
I have made it work, but at a cost. My first problem was that the DOM isn’t built in tabs that are out of focus. I mean, when I click on another tab and the content script refreshes the page that is now inactive the new DOM isn’t built and there is nothing for the script to read. I solved this by running the app in a separate Firefox window. Blurred windows don’t behave like that.
But there is another problem: When the screen goes black because I haven’t pressed a key for some minutes Firefox again stops working. It’s evidently made for interactive use and not very automation friendly. I solved this by configuring the power settings never to let the screen go black.
Now it works, but these ad hoc solutions are not satisfactory. What’s the proper way to do this? Can I put FF in non-interactive mode somehow? On about:config there are a zillion properties that can be set and I assume something can be done there, but I need somebody in the know to guide me. Is it enough to order all tabs to run in separate processes? What about the screen blackout problem?