Alerts and reports notification fails due to selenium exception

[EinavDanielDX]

After scheduling a report for either chart of dashboard, using smtp, The report fails to send the message with the following error in worker pod:

[2022-03-21 14:29:00,298: INFO/ForkPoolWorker-15] Init selenium driver
/usr/local/lib/python3.8/site-packages/celery/platforms.py:800: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

  warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
Scheduling alert my_test eta: 2022-03-21 14:30:00
[2022-03-21 14:30:00,102: INFO/ForkPoolWorker-16] Scheduling alert my_test eta: 2022-03-21 14:30:00
Report state: Report Schedule is still working, refusing to re-compute.
[2022-03-21 14:30:00,293: INFO/ForkPoolWorker-16] Report state: Report Schedule is still working, refusing to re-compute.
Selenium timed out requesting url https://my-domain/superset/slice/75/?standalone=true&standalone=3
Traceback (most recent call last):
  File "/app/superset/utils/webdriver.py", line 122, in get_screenshot
    element = WebDriverWait(driver, self._screenshot_locate_wait).until(
  File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

[2022-03-21 14:30:45,988: WARNING/ForkPoolWorker-15] Selenium timed out requesting url https://my-domain/superset/slice/75/?standalone=true&standalone=3
Traceback (most recent call last):
  File "/app/superset/utils/webdriver.py", line 122, in get_screenshot
    element = WebDriverWait(driver, self._screenshot_locate_wait).until(
  File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

Report state: Failed taking a screenshot local variable 'element' referenced before assignment
[2022-03-21 14:30:46,136: INFO/ForkPoolWorker-15] Report state: Failed taking a screenshot local variable 'element' referenced before assignment
Scheduling alert my_test eta: 2022-03-21 14:31:00

Also, in under reports -> logs in UI screen i see this error
Failed taking a screenshot local variable 'element' referenced before assignment

How to reproduce the bug

1 - setup alerts, smtp and google chrome driver for selenium
2 - schedule a report
3 - follow logs

Expected results

An email with the relevant dashboard/chart should be sent

Actual results

Selenium fails to take a screenshot fails, I'm getting an email with a failure message as described above

Screenshots

Environment

• browser type and version: chrome 99.0.4844.51
• superset version: 1.4.1
• python version: Python 3.8.12
• deployment method: helm chart
• any feature flags active:

my configuration

      FEATURE_FLAGS = {
          "DYNAMIC_PLUGINS": True,
          "ENABLE_TEMPLATE_PROCESSING": True,
          "VERSIONED_EXPORT": True,
          "ALERT_REPORTS": True
      }

      from celery.schedules import crontab

      REDIS_HOST = env('REDIS_HOST')
      REDIS_PORT = env('REDIS_PORT')

      class CeleryConfig:
          BROKER_URL = 'redis://%s:%s/0' % (REDIS_HOST, REDIS_PORT)
          CELERY_IMPORTS = ('superset.sql_lab', "superset.tasks", "superset.tasks.thumbnails", )
          CELERY_RESULT_BACKEND = 'redis://%s:%s/0' % (REDIS_HOST, REDIS_PORT)
          CELERYD_PREFETCH_MULTIPLIER = 10
          CELERY_ACKS_LATE = True
          CELERY_ANNOTATIONS = {
              'sql_lab.get_sql_results': {
                  'rate_limit': '100/s',
              },
              'email_reports.send': {
                  'rate_limit': '1/s',
                  'time_limit': 600,
                  'soft_time_limit': 600,
                  'ignore_result': True,
              },
          }
          CELERYBEAT_SCHEDULE = {
              'reports.scheduler': {
                  'task': 'reports.scheduler',
                  'schedule': crontab(minute='*', hour='*'),
              },
              'reports.prune_log': {
                  'task': 'reports.prune_log',
                  'schedule': crontab(minute=0, hour=0),
              },
          }
      CELERY_CONFIG = CeleryConfig

      SCREENSHOT_LOCATE_WAIT = 100
      SCREENSHOT_LOAD_WAIT = 600

      # # Slack configuration
      # SLACK_API_TOKEN = "xoxb-"

      # Email configuration
      THUMBNAIL_SELENIUM_USER = '[email protected]'
      ENABLE_SCHEDULED_EMAIL_REPORTS = True
      SMTP_HOST = env('SMTP_HOST')
      SMTP_USER = env('SMTP_USER')
      SMTP_PASSWORD = env('SMTP_PASSWORD')
      SMTP_PORT = env('SMTP_PORT') 
      SMTP_SSL = False
      SMTP_STARTTLS = True
      SMTP_MAIL_FROM = env('SMTP_MAIL_FROM')

      # WebDriver configuration
      # If you use Firefox, you can stick with default values
      # If you use Chrome, then add the following WEBDRIVER_TYPE and WEBDRIVER_OPTION_ARGS
      WEBDRIVER_TYPE = "chrome"
      WEBDRIVER_OPTION_ARGS = [
          "--force-device-scale-factor=2.0",
          "--high-dpi-support=2.0",
          "--headless",
          "--disable-gpu",
          "--disable-dev-shm-usage",
          "--no-sandbox",
          "--disable-setuid-sandbox",
          "--disable-extensions",
      ]

      # This is for internal use, you can keep http
      WEBDRIVER_BASEURL=env('DASHBOARDS_BASE_URL')
      # This is the link sent to the recipient, change to your domain eg. https://superset.mydomain.com
      WEBDRIVER_BASEURL_USER_FRIENDLY=env('DASHBOARDS_BASE_URL')

[srinify]

Hey there, moving to Discussion so hopefully someone in the community will help with debugging 🙏 If there's a clear bug you can help the community reproduce, then definitely open a Github Issue!

[fourierlion]

Maybe try WEBDRIVER_BASEURL="http://superset:8088/". Also want to double check that you are using your own domain. The reason I am asking is that your error message shows https://my-domain

Selenium timed out requesting url https://my-domain/superset/slice/75/?standalone=true&standalone=3

[EinavDanielDX]

I've tried with localhost, and also the superset k8s service. I got this same error as with the FQDN.
Also, I made sure https://my-domain/superset is resolved from the running server.

[sujiplr]

Seems this error is coming because of the screenshot timeout issue. You need to try making changes to the timeout values and check it.

  SCREENSHOT_LOCATE_WAIT = 100
  SCREENSHOT_LOAD_WAIT = 1200

[EinavDanielDX]

I've tried setting

 SCREENSHOT_LOCATE_WAIT = 1000
 SCREENSHOT_LOAD_WAIT = 6000

still get the same result.. just have to wait few more minutes for failure

Error: Failed taking a screenshot local variable 'element' referenced before assignment

[sujiplr]

I also faced the similar issue and it was due to the dashboard load issue.

[JunhaK95]

How did you solve the issue?

[MPalix86]

@sujiplr How did you solve the issue?

[sujiplr]

As I mentioned in above thread it was due to dashboard load issue. So need to adjust the wait times.

[JunhaK95]

So I noticed that this issue is only prevalent for me when I use the new filter (the version that is built into the dashboard as opposed to being a chart type). Maybe the issue is related?

[sfirke]

I tried testing this hypothesis by scheduling a report on a dashboard with native filters and a report on a single chart. I only get this error in the logs for the dashboard - not conclusive, but seems to support this idea.

[muralikrishnankh]

Hi, I have tried different scenarios for this. I am able to reproduce the error whenever I am using the native filters(new filters on the left panel) along with passing Jinja templating. In all other cases reports are successfully sent for me.

[bonamim]

Hi,

We were facing the errors below:

[ERROR/ForkPoolWorker-3] A downstream exception occurred while generating a report: xxx-xxx-xxxx-xxxxx
Traceback (most recent call last):
  File "/app/superset/utils/webdriver.py", line 119, in get_screenshot
    element = WebDriverWait(driver, self._screenshot_locate_wait).until(
  File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)

Failed taking a screenshot local variable 'element' referenced before assignment

Failed generating csv local variable 'response' referenced before assignment

and we noticed a connection error too:

    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/local/lib/python3.8/http/client.py", line 922, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.8/socket.py", line 808, in create_connection
    raise err
  File "/usr/local/lib/python3.8/socket.py", line 796, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Solution:

After analyzing the errors and listing when the problem started to happen, we identified that the Superset admin user had been deactivated.
So when activating that user again, everything worked again 🚀

[muralikrishnankh]

Okay. It is already true for me. I think the problem mentioned by @EinavDanielDX is different.

[bonamim]

@muralikrishnankh,
Is the username still admin or has it been renamed? If it was renamed the error can still happen.

[sujiplr]

Yes @bonamim , I also faced this issue initial time, when I made the user as Admin during the deployment. Then I changed the user as "Admin" through UI as well.

[muralikrishnankh]

@bonamim Username was renamed. But in the config file, we have overridden it with THUMBNAIL_SELENIUM_USER = <new_username>.

[bonamim]

We also tried to change the THUMBNAIL_SELENIUM_USER variable, but without success, so we changed it back to Admin.

[seanstock91]

I had the same issue, turned out to be one of the filters having the "Select first filter value by default" option selected. Changed this to a specified default value in the "DEFAULT VALUE" dropdown (I required a default for jinja templating) and all sprung to life.

[sfirke]

Was that filter out of scope? This PR fixed a bug related to select-first-value-by-default on out of scope filters, I wonder if you'll experience the need to do this on version 2.1.0 (coming soon) which has that PR.

[QuyNguyenVu]

I have the same issue: "Selenium timed out requesting url http://superset:8088/..."
How did anyone solve the issue?

[chathawee]

@QuyNguyenVu Could you solve this?

[QuyNguyenVu]

@chathawee No, I could not solve this.

[carlosjepard]

I was able to solve this (I don't use csp btw). I was using superset latest-dev tag but i changed it to tag 2.1.0, and it worked fine. I think that is also worth mentioning that the documenation regarding the reporting the dockerfile may be outdated (if you try to use tag 2.1.0), since i did have to install wget in order to follow the steps to install de headless browser (i installed firefox) and changed geckodriver version to 0.33.0. Hope this helps!

[SamraHanifCareem]

@carlosjepard can you guide me what changes exactly you've made to make it work and in which file?

[MPalix86]

@carlosjepard I agree with @SamraHanifCareem can you write down here a short guide ?

[onyname]

Another tip for those using Openshift/Kubernetes deployments. In the YAML file for Route in HTTP specify TLS settings:.
port:
targetPort: http
tls:
termination: edge
wildcardPolicy: None

If this is not done WEBDRIVER URL will always be rejected and Selenium won't be able to do its part