How to close popup browser window

but its not working

That is strange. I just successfully did that in both Chrome and Firefox. Are you calling .close() on the top-level frame (and not inside an embedded iframe)?

An alternative approach would be:

async function closeSelf() {
    const { id: windowId, } = (await browser.windows.getCurrent());
    return browser.windows.remove(windowId);
}