XPath Behaves Differently in Inspect Console & Extension

Add-ons Development
#1

Hi. I am writing my first extension. Basically, I want to screen scrape a webpage and read data to generate a URI. I’ve tried a lot of things and then I broke it down to basics. There are three commands that work fine in the console. Really, it’s just an “evaluate” with an XPath that gets an element, an iterateNext to get the element, and a textContent to read the value. Here are the actual commands:

let td = document.evaluate("//*[@id='ride-1286927']/td[@class='pickup-location']", document, null, XPathResult.ANY_TYPE, null)

let td1 = td.iterateNext()

td1.textContent

However, in my JavaScript for the extension it fails :frowning: It’s the iterateNext that fails by returning null.

Any ideas? I’m getting frustrated.

#2

Maybe the page isn’t fully loaded yet? Injected content script may run “too soon” for the elements to be loaded yet.

#3

Hi, Juraj. I don’t think so but I’m not an expert. Here’s how I tested:

  1. Opened browser to website being scrapped.
  2. Test #1 - Extension
    a. Installed temporary extension.
    b. Tested - Got null on call to iterateNext().
  3. Test #2 - Console
    a. Entered three commands on console.
    b. Expected text is displayed.
  4. Tested extension again - same results.
  5. Refreshed webpage.
  6. Tested extension again - same results.

AFAIK, the page should have been fully loaded at point #1 because there was no activity and all the results displayed - including the table row that I’m using for testing.

I don’t know if these were valid attempts but I was trying to ensure that the page was loaded to eliminate that possibility.

I understand the basics of “injected content” for the scripts but I don’t know enough about the details to know if that is the cause.

Is there any way to confirm if this is the issue? If it is, is there a way to fix it?

Thanks.

Benny

#4

Hi, Juraj. I have been getting some help in the Add-ons Chat group. It turns out that I was trying to access webpage content in the background script, which is not allowed. I refactored my code and created a content script too. It’s almost complete. I’ll continue in Chat.

Thanks.

#5

Okok, good luck :slight_smile:
Also, I can recommend reading the docs: