aws-quickstart / quickstart-apache-superset Goto Github PK

Hi, thank you for providing this great quickstart template for Apache superset.

I was wondering, is there a recommended way to install additional drivers for Apache Superset resources provided by the quickstart ?

At this moment, I was thinking of creating a new Dockerfile using the ECR image as parent and running pip install from there.

Could you recommend a better way to do this ?

I am trying to make use of the Superset/AWS Athena connection and am running into some permissions issues.

I’ve successfully opened up a connection using a SQLAlchemy URI:

awsathena+rest://:@athena.us-east-1.amazonaws.com:443/default?s3_staging_dir=s3%3A%2F%2Fmy-athena-bucket%2Fathena-output%2F 

And engine parameters:

{"connect_args":{"role_arn":"arn:aws:iam:::role/AwsDataCatalogdefaultAthenaAccessRole"}}

This connection works when using SQLLab.

However, when I attempt to create a chart, I am met with the following access denied error:

superset.charts.commands.exceptions.ChartDataQueryFailedError: Error: An error occurred (AccessDeniedException) when calling the StartQueryExecution operation: You are not authorized to perform: athena:StartQueryExecution on the resource. After your AWS administrator or you have updated your permissions, please try again.

I am aware that if you do not include user credentials within the Athena connection string, then superset falls back on the permission of its ECS task role. In this case, the error is caused by the ECS task not having the right access permissions.

But I assumed that the chart request would be made using the role arn that's defined in connect_args. Clearly, that's not happening. Any idea why this might be?

Hi, I am new to CloudFormation and I have deployed Apache Superset v.2.0.0 on CloudFormation stack using Quick Start doc. I want to install Snowflake driver into Superset, but I couldn't find a way like superset_config.py (to enable embedding feature) to inject Snowflake into it and I didn't figure out which stack the instance belongs to otherwise I can connect to it and execute the installation, can anyone help out please?

I want to change some configs like adding the Mapbox key and SMTP details. How can I do it, given that I am using https://aws.amazon.com/quickstart/architecture/apache-superset/ ?

Hi there,
First of all Thanks for proving the cloudformation template. It worked for me on an existing VPC and I was able to play with it easily.
Now I want to explore some more features and hence seek guidance. Feature that I want to use are:

• Upgrade superset version to the latest minor version.
• Want to query Snowflake database which we are using for reporting purpose.
• Want to try the alerts (https://superset.apache.org/docs/installation/alerts-reports/), and so want to setup additional configuration for SMTP, modify configurations like superset_config.py, and install required packages like Chrome headless browser

Would it be possible for you to share some tips/samples for this?

What is the best way to install snowflake driver into the image, I actually tried to compose another superset image by doing:
FROM public.ecr.aws/p9r6s5p7/superset:v1.4.1
RUN pip install snowflake-sqlalchemy==1.2.4
and use the new image still doesn't work.

Just reporting this issue, looks like something is missing in the docker image.

Hello,
when I'm trying to deploy Apache Superset into a new VPC on AWS I'm facing this error:

My IAM user has AmazonVPCFullAccess policy.
Please advice.

Using template for new VPC and running on issue:

CREATE_FAILED | The following resource(s) failed to create: [SupersetService, SupersetnodeService, SupersetworkerbeatService, SupersetinitService, SupersetworkerService].
-- | -- | -- | --
upersetService | CREATE_FAILED | Resource timed out waiting for completion (RequestToken: 73791ad8-b294-7f58-cf24-b149f574eb86)
SupersetinitService | CREATE_FAILED | Resource timed out waiting for completion (RequestToken: 0af95f33-1556-f0c1-cde3-544c61209225)
SupersetnodeService | CREATE_FAILED | Resource timed out waiting for completion (RequestToken: 1e7663ac-a658-60b3-60a9-7ac920097089)
SupersetworkerbeatService | CREATE_FAILED | Resource timed out waiting for completion (RequestToken: 3fb94cf1-c7e9-0a65-6f01-59e0ee79770e)
SupersetworkerService | CREATE_FAILED | Resource timed out waiting for completion (RequestToken: 099634fa-39d1-5656-ac41-d65c74451155)

Happens for multiple locations (N.Virginia, Frankfurt, Stockholm).

We are looking to deploy the Superset quickstart and notice the Superset container used seems to be owned by "untitled-registry" in the ECR public gallery.
https://gallery.ecr.aws/p9r6s5p7/superset

Is this provided by AWS or a third party - and what level of validation has been performed to ensure the software is legitimate and not infected?

Many thanks for your help.

Opened a case with AWS support this morning as the template for Deploy Apache Superset into a new VPC is not working and returning an error CREATE_FAILED immediately.

Status reason:
S3 error: Access Denied For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

AWS support identified a potential cause is the template within the template is not accessible:
https://superset-sandbox-quickstart.s3.us-east-1.amazonaws.com/quickstart-apache-superset/submodules/quickstart-aws-vpc/templates/aws-vpc.template.yaml

Link to error screencap: https://imgur.com/a/2CJmftN

All the docs claim the architecture uses an ALB, but the CFN templates clearly define an NLB. Is this an error, or an intentional change which didn't make it to the docs?

We are looking to deploy AWS Quickstart and would like to roll out the latest version of Superset if possible.

I have a number of questions about upgrades:

• I notice on a separate issue that you upgrade the version every quarter - please confirm.

• If there is a security vulnerability identified, do you update the version sooner?

• Does the upgrade take effect automatically or is there action we need to take to activate it?

Many thanks!

Valid passwords according to the description get rejected by the pattern.

I'm facing difficulty accessing the database of the superset container. I want to connect to the default PostgreSQL database and update some user data programmatically. If anyone knows how to do this, then please share the procedure. @troy-ameigh @DelfinGala

Thanks to this quick start guide - https://aws-quickstart.github.io/quickstart-apache-superset/, we are able to launch Superset as ECS cluster. The guide uses a docker image - public.ecr.aws/p9r6s5p7/superset:v1.4.1.

I was able to find this docker hosted in AWS Public images gallery - https://gallery.ecr.aws/p9r6s5p7/superset, however this is more than 9 months old. I wish to extend this and base it on the latest version of superset master - release 2.0.0. However I cannot find the Dockerfile associated with this, can you please share the Dockerfile so I can extend this?

I am happy to raise a PR with latest version of superset if you share instructions

Hi, not sure if this is the correct place to post this but I have superset up and running using this quick start guide. I need to edit the config file to increase the timeout for my queries. How do I go about doing this?

I m trying to build superset using template for my POC, i m getting below , this is because i have different DNS options set for my existing VPC since it creates its own private DNS zone which doesn't have forward in our DNS, if can mount via that will be solution but got an error for botocore

ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: Failed to resolve "--" - check that your file system ID is correct, and ensure that the VPC has an EFS mount target for this file system ID. See https://docs.aws.amazon.com/console/efs/mount-dns-name for more detail. Attempting to lookup mount target ip address using botocore. Failed to import necessary dependency botocore, please install botocore first. : unsuccessful EFS utils command execution; code: 1

Please would someone be able to confirm if there are any additional steps required to connect superset, deployed into a new VPC, to Athena. I've tried to use the Athena+rest connection string, with a key:secret extracted from an IAMs administration user - the same user used to create the S3 bucket and AWS GLUE/Lake Formation.

I am currently hitting the following popup error when trying to test the database connection "Failed to Fetch Results" - wondering if I'm missing some steps to make AWS Athena available to the VPC.

Any help appreciated,
Cheers

When trying to set Athena connection according to https://superset.apache.org/docs/databases/athena/

Showing error messages "ERROR: No JVM shared library file (libjvm.so) found. Try setting up the JAVA_HOME environment variable properly."

I just tried running the quickstart and deploying it into my existing VPC. Everything in the cloud formation script completes successfully and I'm able to load the login page. However, the username and password I configured earlier don't work.

I also noticed there is no output inside cloud formation and I ended up getting the URL from the load balancer page directly.

What else can I do to figure this out? Thanks

The process to create the stack fail with this message:

Step: Superset

Embedded stack arn:aws:cloudformation:us-east-1:XXX:stack/apache-superset-v4-Superset-TK4HO6ZIKJ87/4e42a4f0-6309-11ee-b21f-0e27148c5c1b was not successfully created: The following resource(s) failed to create: [SupersetnodeService, SupersetworkerbeatService, SupersetworkerService].

I use a new VPC, no more logs are registered

We based our stack off of the quickstart stack and got everything running well! However, we wanted to enable Google OAuth and that required updating the superset_config.py file. So, we decided to roll with a Docker image consisting of the following:

## Main Superset Docker image
# Root image available here: https://gallery.ecr.aws/p9r6s5p7/superset

FROM public.ecr.aws/p9r6s5p7/superset:v2.0.0

USER root

RUN pip install Authlib

COPY superset_config.py /app/pythonpath

USER superset

With superset_config.py:

from os import environ
from flask_appbuilder.security.manager import AUTH_OAUTH

# Set the authentication type to OAuth
AUTH_TYPE = AUTH_OAUTH

OAUTH_PROVIDERS = [
    {
        'name':'google',
        'whitelist': ['@example.com'],
        'token_key':'access_token',
        'icon':'fa-google',
        'remote_app': {
            'client_id': environ["GOOGLE_OAUTH_CLIENT_ID"],
            'client_secret': environ["GOOGLE_OAUTH_CLIENT_SECRET"],
            'client_kwargs':{
                'scope': 'email profile'
            },
            'access_token_method':'POST',
            'access_token_params':{
                'client_id': environ["GOOGLE_OAUTH_CLIENT_ID"]
            },
            'access_token_headers':{
                'Authorization': 'Basic Base64EncodedClientIdAndSecret'
            },
            'api_base_url':'https://www.googleapis.com/oauth2/v2/',
            'access_token_url':'https://accounts.google.com/o/oauth2/token',
            'authorize_url':'https://accounts.google.com/o/oauth2/auth'
        }
    }
]

AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = "Admin"

By basing it off of the image used in this quickstart, we hoped that we would retain the AWS specific settings that made it so easy to setup and run. However, this was not the case and we're having to manually reapply all the AWS configuration that enabled things like communicating with Postgres and using Redis.

So, would it be possible for either one of two things to happen:

1. Sharing the Dockerfile for the image used in this stack (public.ecr.aws/p9r6s5p7/superset:v2.0.0) so we can duplicate the missing configuration (preferable)
2. Adding from superset_config import * that the official config.py file uses at the end of the superset_config.py file in the AWS image to overwrite configurations (see below)

"""The main config file for Superset
All configuration in this file can be overridden by providing a superset_config
in your PYTHONPATH as there is a ``from superset_config import *``
at the end of this file.
"""

Either one of these solutions would make developing an AWS specific deployment of Superset easier. Related issue

When I refer the helm charts in superset code base or the docker compose we see below service

• redis - caching / celery process
• db - persistent state management for superset
• superset-init - Runs migrations, sets up example tables, charts, dashboards etc
• worker
• worker-beat

I don't see any separate service for node.

I made changes in superset front end code to change font, primary, secondary colors. Rebuilt the docker image from source for Superset and deployed the template with updated docker image for superset. superset-node is using a separate docker public.ecr.aws/p9r6s5p7/superset-node:v1.4.1 which I have not replaced. I see the updated css, font family when I open superset frontend, but no logs or activity on superset-node

Hi there,

Is there a way to upgrade superset after deploying via CloudFormation? I notice that the CF template to deploy to an existing VPC notes that the platform version should be 1.4.0, but the new VPC template file has no reference to a version being used. We want to upgrade to the latest version as there is no BigQuery database connection types included. Any clarification on this would be greatly appreciated!

Thanks
Naz

Which specific superset components actually need CloudMap to communicate with each other internally?